PERSPECTIVES: Social Security Administration Payments to State Vocational Rehabilitation Agencies for Disability Program Beneficiaries Who Work: Evidence from Linked Administrative Data

Introduction

The federal-state vocational rehabilitation (VR) program is administered by the Department of Education's Rehabilitation Services Administration (RSA) to offer counseling, medical and psychological services, job training, and other individualized assistance to people with disabilities. VR is increasingly viewed as a widely available early-intervention program to keep workers with disabilities from becoming dependent on federal disability benefits (Stapleton and Martin 2012; Schimmel Hyde, Honeycutt, and Stapleton 2014; Dean and others 2014). Disability program beneficiaries may face more obstacles to finding work than other potential VR clients because they have impairments that met the stringent evidentiary standards for program eligibility; nevertheless, many of them seek VR services to pursue employment. In fact, disability program beneficiaries represent more than one-quarter of VR applicants. Among recipients of VR services, employment outcomes are poorer for disability program beneficiaries than for nonbeneficiaries, in part because work earnings might affect benefits (Stapleton and Erickson 2004). Beneficiaries who receive VR services are more likely to be employed than are those who do not (Dean and others 2014), and beneficiaries' earnings often increase after receiving VR services, although not frequently to the level that would allow them to forgo disability benefits (Government Accountability Office 2007).

Recognizing the important role of the VR program, the Social Security Administration (SSA) offers compensation to state VR agencies (SVRAs) that assist beneficiaries who maintain substantial work and thereby exit the disability rolls (SSA 2012). SVRAs can be compensated by SSA in one of two ways. Under the first, SSA reimburses SVRAs for the cost of services provided when beneficiaries sustain employment. Under the second, the Ticket to Work (TTW) program authorizes SSA to offer specified payment amounts to SVRAs if beneficiaries achieve certain employment-related benchmarks. We describe the payment schemes in more detail later. During our study period (2002–2012), SSA paid approximately $100 million annually in SVRA cost reimbursements; its TTW payments, much lower at the outset, increased steadily in that span, reaching nearly $28 million in 2012 (SSA 2016).¹ Compared with RSA and state contributions to the VR program, SSA's payments to SVRAs are modest. The majority of VR funding comes from the Department of Education through formula grants to states amounting to more than $3 billion annually; in addition, states are required to match federal funding at a rate of 21.3 percent. Each state has one or two SVRAs; many operate with tight financial resources and have to prioritize services based on need when demand is particularly high (Honeycutt and Stapleton 2013; Schimmel Hyde, Honeycutt, and Stapleton 2014). For SVRAs that are particularly effective in assisting disability program beneficiaries to maintain substantial work, SSA payments might free up resources that can be used to serve additional clients or to offer more extensive services.

In this article, we examine SSA payments to SVRAs and assess how they relate to the long-term employment-related outcomes of beneficiaries seeking VR services. To conduct our analysis, we use SSA data on beneficiary status, cash benefits, personal characteristics, and agency payments to SVRAs; we then link that information to RSA case-closure records known as RSA-911 files. The data from this combination of administrative sources allow us to determine the outcomes for virtually all beneficiaries who sought VR services during calendar year 2002, and to track their outcomes for more than a decade after VR application. By grouping beneficiaries based on their demographic characteristics, impairment types, and VR service receipt patterns, we are able to assess the relative value of SSA's investment, which might in turn inform better targeting of VR services. Additionally, the linkage to SSA data allows us to compare VR payment amounts with the amounts of disability benefits forgone for work (BFW), a particularly salient measure with which to assess SSA's investment in VR services.

From SSA's perspective, paying for VR services is a sound investment if the payment amounts provide a positive return on investment for the agency. Two factors determine the return on VR payments: (1) the value of cash benefits forgone when a beneficiary leaves the program rolls because he or she sustains work at a substantial level, and (2) the proportion of those savings that are attributable to VR services. Because it is possible that the benefits would have been forgone without the services, VR may be responsible for only some, or even none, of the change. Yet beneficiaries seek these services believing them to have some value, so it is reasonable to expect that VR is responsible for at least some of the benefit reductions we see. If 100 percent of the cash benefit reductions of VR service recipients were directly attributable to the services, the dollar value of those reductions would merely have to exceed the VR payment amounts to provide a positive net return for SSA; thus, if VR services are responsible for only part of the benefit reductions, a positive return on SSA's investment requires a greater differential between BFW and VR payments. In other words, the more the BFW exceed the VR payments, the more confidence we can have that SSA realizes a positive net return. This cost-benefit calculation need not be positive for each beneficiary who receives VR services, yet it should be in the aggregate. It is also important to note that calculating the net return for SSA excludes any consideration of the positive effects of VR services for the client beneficiaries themselves, in terms of both the monetary and nonmonetary rewards of meaningful workforce engagement.

In this analysis, we rely on advances in data rather than improved analytic methods. From SSA's perspective, previous analyses of the utility of VR services to disability program beneficiaries had two major drawbacks. First, nearly all such studies used the earnings of the VR client to measure success (for example, Dean, Dolan, and Schmidt 1999). Although SSA is interested in the earnings of VR clients, the question of more direct policy importance is whether VR services lead to a decreased reliance on cash benefits, and more specifically whether the reduction in benefits linked to VR services is greater than what SSA pays for those services. Past studies have relied on earnings data by necessity, because benefit-savings data were unavailable. Recent versions of SSA's Disability Analysis File (DAF), however, include an algorithm that provides monthly estimates of BFW that can be summed to calculate values accrued over time.² By linking VR data from RSA with the DAF data, we are now able to assess a cost-benefit relationship that is of direct interest to SSA and policymakers.

The second drawback of past studies—that VR effects cannot be assessed using experimental evaluation methods because VR is a nationally available and voluntary program—remains an obstacle. To date, researchers have used various nonexperimental methods to examine the effectiveness of VR services using comparison groups of individuals who are deemed to be similar to participants. These techniques generally find positive returns on VR investment for client earnings (for example, Dean and others 2001). However, we can infer that the comparison groups in those studies, nonparticipants, differ from those who did receive services based on their decision to seek VR, and as such, earlier studies are limited by inherent differences between the treatment and comparison groups. Although we do not purport to solve this methodological dilemma in this article, we note that advancements in data and measurement can enable us to better understand the relationship between services and outcomes for beneficiaries with disabilities. Although it is not definitive, such information can provide useful evidence of VR's effectiveness. In particular, by assessing the relationship between SSA payments to SVRAs and the decrease in benefits paid by SSA to beneficiary VR clients in the months and years after receipt of services, we can determine the extent to which the evidence supports a hypothesis that VR services reduce benefit outlays. Benefit reductions that exceed SSA costs certainly do not prove VR effectiveness, but they are a necessary condition of any such effectiveness; and relatively greater reductions in the postservice benefits of VR clients at least suggest possible positive effects.

We will show that among all disability program beneficiaries who applied for VR services in 2002, relatively few generated payments from SSA to SVRAs; only 3.0 percent did so within 4 years of VR case closure, and 3.6 percent did so by the end of 2012. The average VR payment generated by these individuals was around $13,500. Among all applicants, the average value of BFW after VR application was more than 10 times the average value of SSA's total VR payments.³ This outcome suggests that if just 10 percent of the BFW were attributable to VR, SSA's overall investment in these services produced net savings.

Across beneficiary subgroups, we find substantial differences in the likelihood of generating a VR payment, the payment amount, and BFW. However, for every subgroup of beneficiaries we consider, we find that the ratio of total BFW to total VR payments was greater than 1, generally ranging from 4 to 10.⁴

The analysis in this article highlights outcomes for beneficiaries who sought VR services in 2002 and our findings are similar to those for beneficiaries who sought services in 2003 through 2007 (Schimmel Hyde and O'Leary 2017). The relatively low share of beneficiaries generating VR payments highlights opportunities to better identify potential beneficiary VR applicants who might be eligible for services that would generate a payment. Likewise, differences in the ratio of BFW to payments across beneficiary subgroups may offer avenues for considering additional investments, as we will explore.

SSA Payments to SVRAs

SSA's cost reimbursement system for VR services has been in place for decades. SVRAs are reimbursed for qualifying service costs once a client beneficiary attains earnings at or above an annually adjusted threshold designated as substantial gainful activity (SGA)⁵ in 9 out of 12 consecutive months. Once the SVRA can properly document that the beneficiary has met these conditions, it can request reimbursement (SSA 2012).⁶

SVRAs can also be compensated by SSA for providing services to beneficiaries through the TTW program. These payments are not tied to the cost of services provided to a beneficiary. Instead, they are paid in predetermined amounts that accrue in months when beneficiaries achieve specified earnings outcomes, and are the same whether the services are delivered by an SVRA under TTW or by another provider type known as an employment network (EN). SVRAs and ENs must choose from one of two TTW payment schemes (“milestone-outcome” and “outcome-only”), which they must then use for all beneficiaries they assign under TTW. Unlike ENs, SVRAs can choose whether to seek cost reimbursement or TTW payments on a beneficiary-by-beneficiary basis (SSA 2012). Most SVRAs serve most or all SSA disability program beneficiaries under the cost-reimbursement system, and those using TTW largely do so under the milestone-outcome payment scheme (Schimmel Hyde and Stapleton 2015).

Data Sources

To conduct our analysis, we combined administrative data from SSA (the DAF) and the Department of Education (the RSA-911 files). The DAF aggregates information from multiple sources to create a research data set with one record for each beneficiary who is aged from 18 through SSA's full retirement age and who received a Disability Insurance (DI) or Supplemental Security Income (SSI) benefit in at least one month from 1996 onward. We used the DAF to identify participation in DI and SSI at a monthly level, along with beneficiary work activity and benefit payment amounts. The DAF also contains information on payments from SSA to SVRAs for both payment systems at the beneficiary level, using cost-reimbursement records housed in the VR Reimbursement Management System (VRRMS)⁷ and TTW records (both milestone and outcome payments) housed in the EN Payment System.

We used RSA data to identify annual cohorts of VR applicants who were also disability program beneficiaries. The RSA-911 files contain information on all VR cases that close in a fiscal year, regardless of the reason for closure. We stacked data from the closure files to allow us to observe all VR closures from fiscal years 2002 through 2012. To track client experiences from the time they first sought services, we reoriented the closure records to sort by application date. From these files, we identified individuals who first sought VR during calendar year 2002. Approximately 92 percent of VR cases close within 4 years of application and 98 percent close within 7 years (Schimmel Hyde, Honeycutt, and Stapleton 2014). By including case closure records through fiscal year 2012, we expect that our RSA-911 analysis file contains data on virtually all beneficiaries who applied for VR in 2002.

We also linked the 2002 VR applicant file based on the RSA-911 data to the administrative information contained in the DAF. A key advantage of using these linked data is that we can use administrative records to verify beneficiary status during the VR spell; using RSA-911 data alone would provide information on beneficiary status at application only as collected by the agency and would be subject to errors relative to the beneficiary's actual status. From the universe of first-time applicants who sought services in 2002, we selected those who had at least 1 month during their VR service spell (between application and case closure) during which the SSA records identified the person as a beneficiary of the DI program, the SSI program, or both. By considering beneficiary status at any point during the VR spell, we aimed to identify all VR applicants who might have been eligible to generate a payment from SSA to the SVRA. To focus on first-time applicants for whom we could reasonably assume that the receipt of VR services would be correlated with the observed outcomes, we excluded from our analysis those beneficiaries who had also applied in the 4 calendar years preceding 2002.

In light of SSA rules regarding payment to SVRAs, we intentionally cast a broad net in identifying beneficiaries to include in our analysis. We included those in current-payment status, those suspended or terminated for work, and those suspended or terminated for any other reason in at least 1 month between VR application and case closure, provided the beneficiary was aged 18 or older, up to full retirement age, in the month that he or she met this definition. Sixteen percent of our study sample did not meet our definition of beneficiary in their VR application month, but began to receive DI or SSI benefits prior to case closure (comparable to findings in Stapleton and Martin 2012; and in Schimmel Hyde, Honeycutt, and Stapleton 2014). Approximately 2 percent of those identified as beneficiaries during their VR spell had had their benefits terminated prior to their application month, but of these, 26 percent returned to current-payment status at some point during their VR spell.

Outcomes of Interest

We used information in the DAF to measure SSA payments to SVRAs and the cash benefits that beneficiaries gave up when they found work after applying for VR services. Here again, we capitalized on the linkage of the administrative records; by using the DAF in conjunction with the RSA-911 files, we were able to follow applicant outcomes for many years, even after the VR case closed and the SVRA no longer tracked the client.

VR Payments

VR payments is a measure that aggregates all payments by SSA to SVRAs regardless of payment system (traditional cost reimbursement or TTW). We included all payments that were triggered by a client's work activity between the VR application month and December 2012 (provided the payment had been processed by mid-2013, when the data were extracted for the DAF). We identified a single VR payment value for each beneficiary. Cost-reimbursement payments accounted for more than 90 percent of the total amount SSA paid to SVRAs for vocational services, consistent with other evidence on VR payments under the two competing payment systems (Schimmel and others 2013).

BFW

BFW is a measure available in the 2012 DAF of the inflation-adjusted monthly dollar amount that a beneficiary would have received if his or her benefits had not been reduced, suspended, or terminated because of work. We did not count benefits that are reduced, suspended, or terminated for reasons other than work in this measure, even if work was a contributing factor. The DAF includes separate BFW measures for DI and SSI benefits because of differences in each program's work incentives and eligibility rules, but we combined the forgone DI and SSI benefit amounts to create a single measure for each beneficiary.

Under DI, BFW do not begin to accumulate until after a beneficiary has exhausted a 9-month trial work period plus a 3-month grace period. Thereafter, the beneficiary enters suspended-benefit status and receives no cash benefit for any month in which earnings exceed the SGA level. The SGA level is a value that SSA adjusts for inflation each year and which, for nonblind beneficiaries, ranged from $780 per month in 2002 to $1,010 per month in 2012. SSA reduces the amount due to SSI recipients based on their earnings levels. Generally, SSA excludes the first $80 in monthly income in this calculation and reduces the amount due to the SSI recipient by $1 for each $2 in earnings above $80. If earnings are high enough to reduce the amount due to $0 through this calculation, SSA considers the SSI recipient's eligibility to be suspended. Because SSI recipients in suspended status generally remain eligible for Medicaid as long as they meet the medical criteria for SSA disability programs, their SSI eligibility is rarely terminated because of work activity. The BFW calculation effectively estimates the difference between the cash benefits DI and SSI beneficiaries would have received had they not worked and the benefit amount they actually received, then sums these values across the months and years the beneficiaries are in reduced-, suspended-, or terminated-benefit status because of work.

We included all BFW from the date of VR application through December 2012, the last date available in the 2012 DAF. This reflects an assumption that VR services might affect BFW in any month after VR application, but there is no way of knowing whether receipt of VR services actually affected BFW in any or all of the months counted. In fact, as we discuss later, many applicants do not ultimately receive VR services, although some of those applicants accrue BFW.

We consider BFW through December of 2012 to provide the most complete picture of the benefits not paid because of work after seeking VR, even though that extends the study period to many years after the month of application. Thus, some would argue that we should not attribute BFW to VR services in these later years even though SVRAs may still qualify for payments from SSA. To provide a more conservative estimate, we also present alternative findings for a narrower specification in which we only count BFW accrued by the end of the 4^th calendar year following VR case closure. As we will show, that time period accounts for most of the accrued VR payments.

We report all BFW and VR payment values in 2012 dollars using SSA's annual cost-of-living adjustment (SSA n.d. a). Payment values are adjusted based on the month in which they were made, which can be months or even years after the triggering work activity, depending on how quickly a claim is filed by the SVRA and processed by SSA.

Comparing VR Payments to BFW

Although we expect to find a positive correlation between VR payments and BFW following VR participation because both outcomes are predicated on beneficiary work activity, there are multiple reasons not to expect a deterministic relationship. For example, DI beneficiaries are allowed a 9-month trial work period during which they can earn any amount without losing benefits. If a VR applicant's work activity occurs entirely during that period, SSA could pay the SVRA under the cost reimbursement system without the beneficiary accruing any BFW. Similarly, SSA could pay SVRAs under the TTW program if DI beneficiaries have low levels of work earnings that generate milestone payments but do not result in cash benefits being suspended for work. Further, SVRAs must request payments from SSA; SVRAs that lose track of clients may not receive payments even when those clients meet the work requirements for VR payment and generate a string of BFW months.

The ratio of BFW to VR payments conveniently summarizes the relative value of SSA's investments in VR and allows comparisons across subgroups; the higher the ratio, the greater the indicated return on the investment. Although the ratio offers a useful metric to assess the relative strength of SSA's investment for each applicant subgroup, we must interpret the ratio with care in some circumstances. Specifically, if the VR payment (the denominator) is particularly small, the ratio can be quite high even when the BFW value is small as well. This is most pronounced in subgroups for which relatively few beneficiaries generate any VR reimbursement, as may occur if a substantial proportion of beneficiaries who apply for VR drop out before any services are provided. In such cases, when we include large proportions of $0 VR payments in the denominator, any BFW in the numerator will exaggerate the ratio relative to the ratios for other client subgroups. Likewise, a lower ratio does not necessarily indicate against offering services to particular subgroups, a point we revisit in our conclusions.

Beneficiary VR Applicant Characteristics

Using the beneficiary selection criteria outlined above and limiting the data to applications to SVRAs in the 50 states and the District of Columbia, we identified 266,039 first-time beneficiary VR applicants in calendar year 2002 (Table 1). For brevity, we refer to this cohort simply as “applicants” hereafter, noting that “application” in this context is for VR services rather than for DI or SSI benefits. Although disability program beneficiaries represent a meaningful share of individuals who seek VR in a given year, VR applicants are a small share of the overall beneficiary population. In 2002, there were about 11.3 million beneficiaries in current, suspended, or terminated payment status (Schimmel and others 2013); of those, about 2.4 percent sought VR.⁸

The attributes of DI and SSI beneficiaries who are VR applicants may affect the likelihood that they achieve the earnings levels required to trigger VR payments. Work-related characteristics affect eligibility for SSI and DI benefits differently, so we would expect results for these beneficiary types to differ as well. Mamun and others (2011) estimated the distribution of disability program beneficiaries by program for 2007 as about 60 percent for DI only, 29 percent for SSI only, and 11 percent for concurrent DI and SSI. Relative to those estimates, our sample of VR applicants has a higher proportion that receives SSI only (40.1 percent) and a lower proportion that receives DI benefits only (32.7 percent). The share of applicants that received concurrent DI and SSI benefits during the study period (27.2 percent) also exceeds that group's share of 2007 beneficiaries overall, although this finding mainly reflects our inclusive definition of concurrent receipt for this analysis. We categorize a beneficiary as being in both programs if he or she had at least 1 month in DI and at least 1 month in SSI over the duration of the VR spell, even if the month(s) did not coincide.

DI eligibility rules generally require a claimant to have a work and earnings history that qualifies for a cash benefit. As such, qualifying DI beneficiaries are also more likely than SSI recipients to have other savings or pensions they can rely on to supplement their cash benefit. Because other resources are more likely available, and because beneficiaries exited the labor force before seeking benefits, DI-only beneficiaries may have weaker incentive to supplement their cash benefit through work; further, those who want to work are more likely to have employment skills that make VR services less necessary. SSI-only recipients, by comparison, tend to receive lower amounts from SSA and have few supplemental income resources, sparse work histories, and fewer employment skills. Together, these circumstances make work an attractive means of supplementing or replacing SSI, and often make VR critical to attaining a successful work outcome. Stapleton and Martin (2012) also suggested that SSI recipients may apply for VR so they can work enough to become eligible for DI and, eventually, Medicare. The circumstances of individuals who receive concurrent SSI and DI benefits lie between these SSI-only and DI-only extremes: They have substantial work histories, but low benefits and resources. Such beneficiaries are more likely to have work skills from their time in the labor force, but those skills may not lead to earnings opportunities that make employment an attractive option. VR services could supplement those skills to make work more remunerative for these beneficiaries. In some cases, DI-eligible SSI recipients may apply for VR during the 5-month DI waiting period and convert to DI soon after their VR application, meeting our definition for concurrent benefits although the two benefits never actually overlap.

Beneficiaries in our applicant cohort are concentrated in younger age groups with lower education levels (Table 1), groups in which beneficiaries typically are less likely to earn at high levels. Reflecting the relative distribution of applicants across SSI and DI, it is not surprising that applicants tend to be younger. One-quarter of 2002 applicants were aged 25 or younger, in contrast with 2.0 percent who were aged 60 or older. Younger applicants may be more likely to work, but may also do so at lower wages. This may be especially likely given that the majority of applicants in 2002 had low education levels: 27.0 percent had not finished high school, while another 45.9 percent had no more than a high school diploma or its equivalent.

Among impairment categories, psychiatric conditions accounted for the largest share of beneficiary VR applicants (30.2 percent), followed by intellectual disabilities (14.2 percent). Applicants with a musculoskeletal condition represented 6.9 percent of all applicants, lower than that category's share among all beneficiaries.⁹ A small minority of applicants were already working when seeking VR services (11.4 percent), and about one-quarter of those reported working full-time at application.

Share of Applicants Who Received VR Services

In most instances, VR service delivery formally begins when an applicant signs an individualized plan for employment (IPE), which is recorded in the RSA-911 file and indicates that the applicant and VR counselor have agreed to a set of services to be provided. Not every applicant receives services before his or her case is closed. There are a variety of reasons why an IPE may not be signed. For instance, applicants may choose not to follow through once seeing their service plan; or, they cannot be located once the SVRA has space for them. Many SVRAs have limited resources. They may not have the capacity to offer an IPE to all clients, and must prioritize services based on applicant need.

About 60 percent of applicants in our analysis sample received an IPE, which is consistent with statistics for both nonbeneficiaries and beneficiaries in other years (Schimmel and others 2013; Schimmel Hyde and O'Leary 2017). Signed IPEs were notably more likely among the youngest applicants (78.2 percent of those aged 17 or younger at VR application), those who were not yet beneficiaries when they applied for services (72.7 percent), those who had sensory or communication impairments (75.4 percent), and those who were already working when they applied for VR services (74.1 percent).

Cumulative VR Payments and BFW

By year-end 2012, beneficiaries who applied for VR during 2002 had accounted for $1.48 billion in BFW and generated $128 million in VR payments (Chart 1). From the outset, BFW increased much more rapidly than VR payments, reflecting the lag effect of the cost-reimbursement model, which requires a client to attain 9 months of earnings at or above the SGA level before the SVRA can submit a claim. This lag effect may also reflect cases in which clients have either short-term success that generates BFW but not a VR payment, or longer-term success for which the SVRA never requested payment. Further, SSA processes some claims quickly; these may reflect milestone payments under TTW, which can occur even if earnings do not reach SGA level. Because TTW was not fully rolled out in the first years of the study period, early payments may have been less common than they were in later years.

Few clients who applied in 2002 generated VR payments (Table 2). Among the 266,039 members of the 2002 applicant cohort, 9,510 (3.6 percent) generated any payment by the end of 2012. Among that group, the average VR payment amount was $13,517 (roughly consistent with statistics published in SSA n.d. b). Median values (not shown) were lower than the mean values, in part reflecting much higher payments to SVRAs serving blind individuals, who constitute a small share of total VR clients.

BFW were many times higher than VR payments for 2002 applicants. For the entire cohort, BFW accumulated through year-end 2012 were 11.5 times the cumulative amount of VR payments. By the end of 2012, virtually all VR payments that would ever be generated had been paid, yet BFW can continue to accrue until beneficiaries return to the disability rolls, reach retirement, die, or have their benefits terminated for another reason. Between the end of the 4^th calendar year after case closure and year-end 2012, VR clients generated an additional 10 percent of VR payments, but they accounted for an additional 40 percent of BFW. When limiting follow-up to the end of the 4^th calendar year after VR case closure, the ratio of BFW to payments was 7.8.

Differences in BFW and Payments Based on the Receipt of VR Services

Although 40.2 percent of 2002 VR applicants did not sign an IPE, that group accrued about 21 percent of the total BFW through the 4^th calendar year after case closure and about 28 percent of the total BFW through 2012. The group's disproportionately low share of BFW may indicate that a lack of VR services left them underprepared for a successful job search relative to those who received services. Conversely, failure to sign an IPE could also signal that this group was less likely to work. However, the group accrued more than one-quarter of the total BFW by year-end 2012, suggesting that many of the applicants who did not receive VR services were nevertheless interested in and able to work at a significant level.

Applicants who signed an IPE accounted for 60 percent of the applicants overall but they accounted for 86 percent of the clients who generated VR payments by year-end 2012. This finding is predictable because the SVRA must document the provision of services to claim reimbursement. However, it also implies that 14 percent of the beneficiaries who generated payments did not receive an IPE prior to closure of their 2002 application. Because receipt of services is required for a payment to be generated, these applicants therefore must have later reapplied for VR, received services, and worked at a level sufficient to generate a payment. For those who did not sign an IPE, the ratio of BFW to VR payments as of year-end 2012 is substantially higher (30.6) than the ratio for those who did sign an IPE (9.3). This last finding is driven, at least in part, by the relatively small number of non-IPE cases with a VR payment, which tends to exaggerate the BFW-to-VR payment ratio.

VR Payments and BFW by Applicant Subgroup

The likelihood that beneficiaries will work at a level that generates VR payments and accrues BFW varies depending on their individual characteristics. Table 3 presents statistics on VR payments, BFW, and the ratio of BFW to payments through the 4^th calendar year after case closure for each applicant subgroup. Although only 3.0 percent of applicants overall generated a VR payment, some subgroups were much more likely to do so; for example, 7.0 percent of those categorized by SSA as having a sensory or communication impairment and 6.5 percent of those whose VR case was not closed in less than 4 years. Payments were relatively less likely for VR cases that were closed quickly (a maximum of 1.8 percent of applicants with closure in less than a year) and for older applicants (no more than 1.7 percent of those who applied at age 50 or older).

Relative to the other subgroups within each category, the ratio of BFW to VR payment amounts was higher for DI-only and SSI-only applicants, men, African Americans, older applicants, applicants with longer periods as beneficiaries or with intellectual impairments, applicants who were already working when they sought VR, and those whose cases were closed relatively quickly. The reasons for these differences may vary across groups. Older applicants, for example, may have a higher ratio than younger applicants because they generally have higher monthly benefit amounts to forgo; but they are also less likely to generate a VR payment than younger applicants are. This type of difference also explains the relatively high ratio for applicants with intellectual impairments—many of those individuals likely receive only SSI and forgo a relatively small amount over many months as they sustain work, but do not accrue work earnings sufficient to generate VR payments.

To better understand the differences in the ratios across subgroups, Table 3 also shows the percentage distributions of all applicants, of applicants who generated a VR payment, of VR payment amounts, and of BFW, by subgroup for each characteristic category. For example, DI-only beneficiaries represented 32.7 percent of all applicants, 42.1 percent of applicants generating a VR payment, 40.5 percent of VR payment dollars, and 41.9 percent of BFW. When a subgroup's share of applicants with a given outcome is greater than that subgroup's share of all applicants, the members of that subgroup had a disproportionately strong outcome. Therefore, DI-only beneficiaries generated disproportionate shares of payments and BFW relative to their share of the applicant pool. Conversely, when a subgroup's share of all applicants is greater than its share of applicants with a given outcome, members of that subgroup had a weaker result for that outcome.

Although the ratio of BFW to VR payments was the same among SSI-only and DI-only applicants (8.1), SSI-only applicants accounted for disproportionately low shares of VR payments and BFW, in contrast with DI-only beneficiaries. Although BFW can accrue for each month in which an SSI-only recipient works, the SSI monthly amount received tends to be low and work often does not result in full loss of cash benefits in the month. Moreover, SSI recipients generally lack significant work experience, reducing the likelihood of working at the level required to generate a VR payment.

The ratio of BFW to VR payments is significantly lower among applicants younger than 18 (3.4) than that of applicants aged 50–59 (12.0). The difference is driven largely by the lower share of applicants generating a VR payment for the older group (8.6 percent, versus that group's 15.2 percent share of applicants overall) in contrast with the higher share for the younger group (6.4 percent, versus their 4.0 percent share of applicants overall). For the younger group, the share of VR payment dollars (9.3 percent) is even higher than the share of applicants generating payments; this might reflect expenditures for more extensive vocational, training, or postsecondary education programs for this group. Because these beneficiaries could accrue BFW for many years, successful returns to work are particularly important for this group and we would likely find that their BFW dwarfs the payments they generate to SVRAs if we followed them for longer periods.

Beneficiaries who had been on the disability rolls for longer periods when seeking VR had a higher ratio of BFW to VR payments than did those who were recent awardees (9.2 for those who had been beneficiaries for more than 10 years and 6.8 for those who had been receiving benefits for 2 years or less). A more recent onset of beneficiary status may indicate relatively less stable health and disability status, leading to less stable employment as well, which could underlie the reduced BFW we see for this group. Individuals who received SSI as children but sought VR after completing school as young adults would have been on the rolls for many years and could begin accruing BFW soon after applying for VR. Some applicants may have already begun accruing BFW when seeking VR, looking for additional assistance to maintain employment. These possibilities also explain why, in this category, the subgroup with the lowest ratio of BFW to VR payments is the individuals who were not receiving benefits at the time of VR application; they had fewer potential months of observation as current-pay beneficiaries and therefore had fewer opportunities to accrue BFW during the study period.

Among SSA impairment categories, applicants with sensory and communication disorders had the lowest ratio of BFW to VR payments while those with intellectual impairments had the highest ratio.¹⁰ Applicants with sensory impairments accrued a disproportionately large share of VR payment dollars, perhaps reflecting the cost of funding workplace accommodations such as screen readers for those with visual impairments; Schimmel Hyde and O'Leary (2017) showed that the average payment amounts for SVRAs that serve blind clients are much higher than those for SVRAs serving nonblind clients. Applicants with intellectual impairments had a low share of VR payment dollars but a relatively higher share of BFW. Individuals with intellectual impairments often receive SSI and may work at modest wage levels for sustained periods, which could result in BFW for each month worked at a level that does not generate a VR payment under the cost-reimbursement system.

Applicants whose cases remained open for longer periods had lower ratios of BFW to VR payments than did those whose cases closed relatively quickly. Very short spells are likely to be concentrated among those who applied but did not receive an IPE for reasons related to the limited availability of services or lack of client interest in continuing with services. It could also be that those with short spells found jobs on their own before VR services became available. If so, those jobs were not particularly successful, given the subgroup's relatively low average BFW values relative to those for other groups. Strikingly, the subgroup with a VR spell lasting at least 48 months represents about 10 percent of all applicants but more than 40 percent of VR payment dollars. Longer spells could indicate that the SVRA had a long-term service relationship with the client resulting in a successful work outcome (because the client was receiving education or training or because the case was left open until the beneficiary had earnings high enough to generate a payment) or because the accrued cost of providing long-term services eventually led the SVRA to seek payment.

Interestingly, we did not find large variations in outcomes based on the time spent waiting for services (not shown; those results are presented in Schimmel Hyde and O'Leary 2017). Honeycutt and Stapleton (2013) found that the more an SVRA had to resort to placing DI-beneficiary VR applicants on wait lists in a given month, the poorer the clients' employment outcomes 48 months later. Our findings may at first seem inconsistent with theirs, in part reflecting that their waiting-time measure was defined from the perspective of the agency and not the applicant. Additionally, the discrepancy may highlight the importance of length of follow-up for beneficiaries who applied for services. Individuals who face a long delay in receiving services may be slower to attain positive employment outcomes, such that they are not realized until after the end of a 48-month observation period. Moreover, although the length of the wait for services did not seem to affect the likelihood that those services would lead to BFW and VR payments, waiting for services would certainly have delayed when BFW and VR payments occurred, which has meaningful implications for beneficiaries, SVRAs, and SSA.

Subgroup Differences in BFW and VR Payments Conditioned on the Receipt of VR Services

As described above, the likelihood of signing an IPE and receiving services varies across subgroups. Because receiving services is requisite for generating a payment (at least for the current VR spell), Table 4 shows VR payments, BFW, and ratio of BFW to payments through the 4^th year after case closure for applicants who signed an IPE, by subgroup. For every subgroup of applicants with an IPE, the ratio of BFW to VR payments is considerably lower than the overall ratio for applicants without an IPE. Beneficiaries who signed an IPE generated 94 percent of total VR payments but accounted for only 79 percent of the BFW, as we can calculate from Table 2. Because the BFW-to-VR payment ratios of IPE signers reflect the comparatively high VR payments in the denominator, their ratios are lower than those of applicants without an IPE. Table 4 shows little variation in the ratios across the subgroups of applicants with an IPE, with most ranging between 6 and 7. Overall, though, the cross-subgroup patterns of the ratios shown in Table 3 for all applicants are not substantively different from those seen in Table 4 for applicants who signed an IPE.

Discussion and Conclusions

We find that relatively few of the beneficiaries who sought VR services generated a payment from SSA to an SVRA. This may reflect two factors. First, beneficiaries face substantial barriers to entering the labor force, and even if they succeed, they may not sustain work at a level that triggers VR payment. This may be especially relevant under the cost-reimbursement system that SVRAs use for most beneficiary applicants, which requires earnings at or above SGA level to be sustained for 9 out of 12 consecutive months. This means that beneficiaries who work at SGA level but only for a few months, or those who work for many months but at a lower level, would not generate a payment. Under the TTW milestone-outcome payment system, payments may be generated for lower levels of work earnings, but relatively few SVRA applicants have been assigned to that system, even in recent years (Schimmel Hyde and Stapleton 2015).

The relative rarity of applicants who generate a VR payment may also reflect instances of SVRAs failing or opting not to request payment, even for beneficiaries who work at the level that would warrant one. Schimmel Hyde and O'Leary (2017) identify differences across SVRAs in the share of applicants who generate a payment, even in cases with comparable BFW amounts. SVRAs differ in how well they follow clients who have found work, how well they document costs for which they request reimbursement, and how difficult they find the process for requesting reimbursement. SVRAs that are relatively less financially constrained by the funding they receive from RSA may see less value in seeking reimbursement from SSA. The Government Accountability Office (2009) reported that SVRA staff members often thought the costs of tracking beneficiaries outweighed the benefit of reimbursement. Similarly, ENs participating in the TTW program noted difficulty in tracking applicants; once clients are stabilized in work, they often lose interest in providing current earnings information. When that occurs, ENs can find it challenging and time-consuming to track client earnings, ask SSA for payment, and follow up accordingly (Altshuler and others 2011). Even agencies that can effectively track clients over time may perceive or observe that the costs and paperwork involved in doing so outweigh the value of the reimbursement.

After our analysis period ended, SSA instituted changes that may have increased the likelihood that SVRAs receive payments for which they are eligible. Specifically, starting in 2013, the agency offered to alert SVRAs when their beneficiary clients attained earnings above SGA level, if so requested. Since then, the number and value of payments made by SSA to SVRAs have been markedly higher; in fiscal year 2015, SSA paid more than 12,000 claims—more than double the number paid in fiscal year 2012 (SSA n.d. b). We do not know the extent to which this increase was because of the new SGA-alert program versus other influences such as an improved economy. Additionally, even with the number of payments doubling, the share of applicants who generate one for their SVRA is still low. Revisiting this analysis using data for 2013 and later might indicate whether additional improvements are possible.

Even if the share of applicants who generated a VR payment were to increase significantly, the return on SSA's investment may still be good if payments are judged relative to BFW. We find that BFW for 2002 applicants amounted to about 10 times the value of VR payments; even under our more conservative accounting using a shorter follow-up period, the ratio is about 8. As such, payments would have to rise substantially to equal the BFW amount that accrues after application for VR services. Most VR payments are made through the cost-reimbursement system, so SVRAs already recoup all or most of their client rehabilitation costs when SSA pays them. Total VR payment amounts could feasibly increase only if SVRAs:

1. Increase the share of SSA program beneficiaries served,
2. Provide more expensive/intensive services per beneficiary than they do now,
3. Extend employment services for clients approaching the threshold that generates a VR payment (for example, by providing additional support to clients who work at or above the SGA level for 8 months and are served under the cost-reimbursement system), or
4. Improve the monitoring of employed clients so that the SVRA can claim and receive payments for a higher proportion of clients with long-term success.

Options 1 and 2 seem reasonable in that they would likely increase BFW amounts as well as VR payment costs and could thereby represent improved outcomes for beneficiaries, SVRAs, and SSA. Still, even with the possibility of additional VR payments, SVRAs are likely to be constrained by other factors that will hinder or prevent their expansion of services. Since 2008, SSA has encouraged option 3 through an initiative called Partnership Plus, which enables a client whose SVRA services conclude to continue to receive services from an EN. Additionally, as mentioned above, SVRAs can request an SSA alert when a client beneficiary has earnings above SGA level, and this initiative may bear additional fruit in the near future. Increased VR payments from SSA, even if they equaled BFW, would still be small relative to the total funding provided to SVRAs by RSA and state matching funds, and could free SVRA resources to provide additional services. In this way, additional payments made possible through option 4 could make more services available through options 1 and 2, and could improve service outcomes under option 3. This is particularly important in light of our finding that a significant number of beneficiaries who are not served on their initial VR spell reapply, generate BFW, and go on to generate a VR payment after reapplying. Letting these beneficiaries fall through the cracks—even if temporarily—seems like a lost opportunity in terms of their work activity, well-being, and potential BFW.

Although our work identified the subgroups that are most likely to receive services, generate a payment, and accrue BFW, we did not delve into cross-group differences. A multivariate consideration of characteristics might identify the factors most likely to predict employment success and thus could allow SSA to identify target groups, inform them of employment opportunities, and encourage them to consider and pursue their employment options.

More study is also needed to identify whether VR services cause beneficiaries to find work—and if so, the types of services that might be most beneficial. Our analysis was designed to document an association between VR payments and BFW and was not meant to capture the extent to which VR services resulted in either outcome. In fact, our work suggests that the relationship is not strictly causal, given the finding that many applicants who never signed an IPE had BFW. For that group, VR services seemed not to be necessary to lead to employment. However, our results also showed that some beneficiaries in that group reapplied for VR services—indicating that they saw value in assistance—and we do not know if outcomes may have been better, had services been received, among those who did find work.

Designing a study that identifies a causal mechanism between certain VR services and employment outcomes will continue to be a challenge because those services are available to all SSA disability program beneficiaries and application is voluntary. The difficulty of identifying causation underlies past studies that consider the effect of VR services on client outcomes and, as such, no nonexperimental study of VR has provided definitive answers. The inherent weakness of nonexperimental analysis of employment programs has been quantified by reanalyzing experimental findings using nonexperimental methods (see, for example, Bloom and others 2002; and Glazerman, Levy, and Meyers 2003). Beyond this general problem, there is evidence that nonexperimental methods may be less accurate for persons with disabilities because there tends to be a poor correlation between available observable characteristics and the likelihood of successful employment for those populations. Peikes, Moreno, and Orzol (2008) compared experimental and nonexperimental outcomes specifically for employment supports for SSA program beneficiaries with disabilities. They found the two methods gave results that differed in both magnitude and direction, likely because unobservable characteristics, such as motivation, locus of control, and health status were more important than the extensive observable characteristics that were available for the treatment and comparison groups. O'Neill and others (2015) similarly found that matching on observable characteristics might not fully account for differences between SSA disability program beneficiaries in treatment and comparison groups. More recent examinations have used improved nonexperimental methods to examine the marginal effect of VR (Dean and others 2014), but whether these methods provide true estimates of VR effects remains unresolved.

Although the causal relationship is important, data-driven approaches can also shed light on the likely effectiveness of the VR program. In the absence of reliable nonexperimental methods, it is useful to know that BFW after VR application not only exceed the VR payment amounts but, in general, are 8 to 10 times their size. This is not conclusive proof, but it is at least consistent with the notion that VR services are effective. It will remain important to develop and weigh the other available evidence to determine whether it too indicates a positive return on VR investment. To borrow from legal jargon, in the absence of evidence beyond a reasonable doubt, a preponderance of the evidence provides the best available guidance to policymakers on these important services.