Ticket to Work Evaluation (January 2006)
Early Outcomes for Ticket Recipients
The principal objective of the Ticket to Work program is to move SSI and DI recipients into employment and off the beneficiary rolls. This chapter presents findings from our initial analyses of outcomes for Ticket recipients that directly pertain to this objective: benefit receipt, benefit amount, and employment. We report on these outcomes both before and after Tickets were mailed. However, even though we restricted the analysis to the Phase 1 states where TTW was rolled out first, the follow-up period was relatively brief—we have data on beneficiaries for at most 15 months after Tickets were mailed. This timeframe may be too brief to observe meaningful changes in outcomes for participants. For example, prior studies indicate that the average SVRA client receives services for 25 months (Gilmore 2004). Findings are presented for all Ticket recipients and separately for Ticket participants (beneficiaries who assigned their Tickets to ENs or SVRAs) and nonparticipants (beneficiaries who received Tickets but did not assign them).
Some of the most interesting findings presented in this chapter are from comparisons of outcomes between TTW participants and nonparticipants. Outcomes were often different for these two groups even before Tickets were mailed. Some of these differences persisted virtually unchanged in the postmailing period, while others changed substantially. Whether the latter occurred because of TTW is an open question that will be addressed in future reports under this evaluation.
The previous two chapters reported that Ticket participants are distinguished by a number of characteristics—they are younger, have more education, and are less likely to be married than Social Security disability beneficiaries in general. Our analysis of outcomes revealed that participants are distinctive in other dimensions as well. For most of the year before TTW was rolled out in Phase 1 states, the beneficiaries who ultimately received and assigned their Tickets were more likely than nonparticipants to have zero benefits and to be substantially employed. Following the rollout, there were only small changes in benefit receipt and employment among all Ticket recipients; however, employment and program exits did appear to increase slightly among beneficiaries who assigned their Tickets relative to those who did not, at least among SSI beneficiaries. These small differences may have been the result of several factors, including the documented differences between the two groups in their characteristics, levels of impairment, and attitudes toward work. They also may have been partly a result of the TTW program itself. At this time, we are reporting just the differences without making firm judgments about what caused or influenced them. Our next report will attempt to identify the causal factors and, in particular, to assess TTW’s influence. Nevertheless, the observed differences in outcomes between participants and nonparticipants after Tickets were mailed are at least consistent with the principal objective of TTW, and it is hard to imagine that the program could have its intended effects without generating these sorts of differences.
A. TARGET POPULATION AND OUTCOME MEASURES
On January 12, 2002, SSA selected from its beneficiary rolls all individuals in Phase 1 states who satisfied the Ticket eligibility criteria.1 SSA mailed Tickets to these beneficiaries on a staggered schedule beginning in February 2002 and ending in October of that year.2 Beneficiaries who became Ticket eligible after the initial selection date were mailed a Ticket soon after achieving eligibility but no earlier than if they had been eligible on that date. We conducted separate program-specific analyses of the 1,511,299 DI beneficiaries and the 940,876 SSI beneficiaries who were selected by SSA on January 12, 2002 3 and were still alive and under age 65 at the end of the analysis period, 15 months after Tickets were mailed.4 These beneficiaries constituted the target population for our analysis. Included among them were 11,565 DI recipients and 7,757 SSI recipients who had assigned their Tickets.
This chapter describes time trends in three key outcomes for DI and SSI beneficiaries: (1) the receipt of zero benefits, (2) the receipt of zero benefits combined with substantial employment, and (3) the combined SSI and DI disability benefit amount. The time covered ranges from 12 months before the Tickets were mailed to either 6 months or 15 months after they were mailed.5 We selected these outcomes because they are useful indicators of whether TTW has been successful in moving beneficiaries into employment and off the rolls, and they will be the focus of the evaluation in future reports.
B. OUTCOMES BY PROGRAM
This section describes outcomes before and after Tickets were mailed for all Ticket recipients who satisfied the criteria to be included in the analysis, as noted in the previous section. Findings are presented separately for DI and SSI beneficiaries.6 Section C presents outcomes for the same Ticket recipients, but the findings are further disaggregated by whether or not they were TTW participants by virtue of having assigned their Tickets to ENs or SVRAs.
1. Zero Benefits
We classified individuals as being in “zero benefit status” (or receiving “zero benefits”) if they were not eligible for disability benefits under either the DI or the SSI program. If TTW is successful in terms of its principal objective, then we would expect it to increase the incidence of zero benefits among Ticket recipients. Before Tickets were mailed, the incidence of zero benefits among those in the target population who ultimately received them was falling. This was an artifact of the Ticket eligibility criteria. Individuals who were mailed Tickets had varying benefit starting dates in the months leading up to the mailing date, with all of them being on the rolls as of the January 12, 2002, selection date. Among those who were selected, a few were no longer receiving positive benefits when their Tickets were subsequently mailed. Figure IV.1 shows that 1.2 percent of DI recipients and 2.2 percent of SSI recipients were in zero benefit status in the Ticket mail month. Under TTW program rules, these individuals were prohibited from assigning their Tickets until they returned to positive benefit status. Twelve months earlier, 3.1 percent of DI beneficiaries and 3.8 percent of SSI beneficiaries were in zero benefit status.
Outcomes after Tickets were mailed are central to the evaluation of TTW. If TTW is effective in moving beneficiaries off the program rolls, then we would expect the percentage of Ticket recipients in zero benefit status to increase following the mail month. Figure IV.1 shows that the percentage of both DI and SSI beneficiaries in zero benefit status did, in fact, increase after the mail month, but the increases were small. During the 15 months after Tickets were mailed, the proportion of SSI recipients with zero benefits increased by 1.5 percentage points to 3.7 percent. Meanwhile, the proportion of DI recipients in zero benefit status increased by only three-tenths of 1 percentage point to 1.5 percent. These increases may reflect the continuation of weak propensities for program exit that existed in the absence of TTW. On the other hand, it is also possible that TTW contributed to them.
As mentioned, Section C presents outcomes separately for Ticket participants and nonparticipants. Those results are more suggestive about TTW’s contribution to the increase in the incidence of zero benefits after Tickets were mailed, at least among SSI recipients. However, they too do not provide convincing evidence that TTW had its intended effect.
2. Zero Benefits and Substantial Employment
Substantial employment, defined as working and earning in excess of the substantial gainful activity (SGA) level, in combination with receiving zero benefits is a more telling outcome measure for TTW than zero benefits alone. It captures both the program participation and employment dimensions of the principal objective of TTW. We created this measure from monthly indicators of zero benefit status and from annual earnings in SSA summary earnings records.7 Because the earnings data apply to calendar years, while the benefit data apply to months, we had to construct the combined measure by using the simplifying assumption that earnings were received only during months when there were no disability benefits. We calculated average earnings during zero benefit months by dividing earnings for a calendar year by the number of zero benefit months in that year. If the resultant value equaled or exceeded SSA’s monthly substantial gainful activity level for that year, then we designated the zero benefit months in that year as months of substantial employment. All other months—those in which benefits were positive and those in which benefits were zero but average earnings were below SGA—were designated as months without zero benefits and substantial employment.8
Almost no Phase 1 Ticket recipients received zero benefits and were substantially employed (Figure IV.2). This is not surprising given that SSI, DI, and TTW rules rely on a definition of disability that is based on the inability to perform substantial employment. Only four-tenths of one percent of both SSI and DI beneficiaries met our criteria for having zero benefits and substantial employment when Tickets were mailed. The shallow U-shaped time pattern that we observed for zero benefit status alone is barely perceptible when substantial employment is also considered because substantial employment for Ticket recipients was extremely rare throughout the analysis period. Specifically, before DI beneficiaries were selected to receive a Ticket, and 12 months before the Tickets were mailed, just 1.4 percent of them were substantially employed and receiving zero benefits. And for SSI beneficiaries, we know from Figure IV.1 that 3.8 percent were in zero benefit status at that time; however, only about one in seven of those individuals were substantially employed by our definition. The resultant very low rate of combining zero benefits with substantial employment, just one-half of one percent, reflects the especially weak employment histories of SSI recipients.9
Rates of combining substantial employment with zero benefits edged up almost imperceptibly for both SSI and DI beneficiaries during the six months after Tickets were mailed, by about one-tenth of one percent to about one-half of one percent. This suggests that if TTW did affect the rates at which Ticket recipients combined substantial employment with zero benefits, the impacts must have been very small. Of course, six months is not much time to allow impacts to materialize. Subsequent reports on this evaluation will present findings based on longer follow-up periods.
3. Disability Benefit Amount
If TTW is successful in moving DI and SSI recipients off the beneficiary rolls and into substantial employment, then we would expect it to result in lower disability benefits. Furthermore, under TTW’s outcome-only payment option, SSA pays ENs and SVRAs only when they have helped beneficiaries work to the extent that their benefits are reduced to zero. Hence, we examined the disability benefit amounts received by Ticket recipients before, during, and after the mail month. Among the Phase 1 Ticket recipients included in this analysis, 19 percent of DI beneficiaries and 31 percent of SSI beneficiaries participated concurrently in both programs during the mail month. In light of this fact, the benefit measure reported in this chapter is the combined benefits from the two programs, adjusted for inflation.10 In this and all other analyses discussed in this chapter, program classification (DI/SSI) was determined as of the mail month and remained fixed for all preceding and succeeding months. So, for example, an individual who was an SSI beneficiary in the mail month but subsequently qualified for DI benefits would continue to be classified in the SSI program group for analytic purposes.
Average disability benefits for Phase 1 Ticket recipients increased gradually during the year preceding the Ticket mail month, reaching $801 for DI participants and $546 for SSI participants at Ticket mailing (Figure IV.3). These increases were driven by entry onto the active beneficiary rolls by individuals who had been receiving zero benefits 12 months before Tickets were mailed.11
During the 15 months after Tickets were mailed, the average benefit received by SSI participants fell by $14, or 2.5 percent, to $532. This change mirrors the small increase in the incidence of zero benefits during the same period (see Figure IV.1). Just as those earlier results are consistent with TTW having increased the incidence of zero benefits among SSI participants, so the results presented here are consistent with TTW having reduced their average benefit. But, as noted in our discussion of the zero benefit results, consistency is not strong evidence that TTW actually caused the changes in the outcome measures.
For DI participants, the average benefit increased by $16, or 2.0 percent, to $817 during the 15 months after the Ticket mail month. This change was not a result of inflation because we adjusted all benefit amounts to January 2004 dollars. It was also not a result of the exit of low-benefit individuals from the DI program because we retained all Ticket recipients in the postmail month analysis regardless of their program status. The incidence of zero benefits was relatively stable for DI participants following the mail month (see Figure IV.1) and thus exerted virtually no downward pressure on benefits, in contrast to the situation described in the preceding paragraph for SSI participants. The small, gradual increase in the average benefit received by DI participants probably reflects the functional relationship of DI benefits (but not SSI benefits) to recipients’ past earnings. We know from Figure IV.2 that some DI recipients had substantial earnings shortly before the Ticket mail month. After those earnings were posted to their accounts, those recipients may have qualified for higher DI benefits.
C. OUTCOMES BY PROGRAM AND TTW PARTICIPATION STATUS
This section compares outcomes for Ticket recipients in Phase 1 states who participated in TTW by assigning their Tickets to ENs or SVRAs with outcomes for recipients who did not participate. These outcome measures were introduced and discussed in the preceding two sections—zero benefit status, zero benefits in combination with substantial employment, and the disability benefit amount. Readers are cautioned not to interpret participant/nonparticipant differences in these measures after Tickets were mailed as estimates of the impacts of TTW. Some, perhaps even all, of the differences might have existed even if the Tickets had never been mailed. In particular, participants may be more capable of working and more inclined than nonparticipants to seek substantial employment, creating more favorable outcomes for participants both before and after Tickets were mailed. For example, TTW participants who responded to our survey of beneficiaries were much more likely than nonparticipants to report that they expected to earn their way off the disability rolls (refer to Figure III.15 in the previous chapter). Further, recipients who would have entered employment and left the rolls even if they had never been mailed a Ticket are likely to find it advantageous to assign their Ticket because in doing so, they can obtain useful assistance or even cash (if, for example, they assign it to an EN with policies like those of AAA TakeCharge).12
Subsequent evaluation reports will attempt to sort out the various factors shaping beneficiary outcomes and to estimate the impacts due to TTW. We will consider not only the actual outcomes of Ticket recipients but also their likely outcomes in the absence of TTW. The true impacts of TTW will be assessed by comparing those two sets of outcomes.
1. Analysis Approach
Our approach to the analysis of each of the three outcome measures was to compare levels and trends of Ticket participants and nonparticipants. We began by looking at the two groups before Tickets were mailed to see if they differed in pre-TTW outcome levels. We then looked for differences after mailing. Of particular note are postmailing differences between Ticket participants and nonparticipants in outcome levels that do not have the same sign as the premailing differences or that are substantially different in magnitude. If individuals were transitioning on and off the benefit rolls for similar reasons (into and out of employment, for example) before and after Tickets were mailed, then these differences may be suggestive of TTW’s impact. Postmailing differences between the two groups in outcome trends are also notable. A distinctly different rate of change in an outcome measure between participants and nonparticipants after Tickets were mailed could signal an effect of the new program.
Figure IV.1 shows that Ticket recipients who were SSI beneficiaries had a less consistent pattern of benefit receipt during the year before Tickets were mailed than did recipients who were DI beneficiaries. Entry of new beneficiaries onto the program rolls is likely an important reason for inconsistent benefit receipt during the pre-mailing period for both SSI and DI. In addition, transitions between employment and dependency are likely an important reason for SSI, because SSI program rules are conducive to such transitions. Such transitions also occur for DI among those in their extended period of eligibility, but are likely less frequent. After Tickets were mailed, we expect that most transitions off assistance by members of the target population in either program were to employment.
Because transitions between employment and dependency are likely an important source of inconsistency in benefit receipt both before and after Tickets were mailed for SSI recipients, pre-post comparisons of differences in levels of outcomes between TTW participants and nonparticipants are an attractive way to assess the possible effects of TTW on those beneficiaries. In contrast, because we think that inconsistency in benefit receipt for DI is primarily due to new enrollments before Tickets were mailed and to exits from the rolls for work after they were mailed, postmailing differences between participants and nonparticipants in outcome trends may be more useful indicators of the possible effects of TTW on DI beneficiaries than pre-post comparisons of differences in levels between participants and nonparticipants.
2. Zero Benefits
DI and SSI beneficiaries who participated in TTW were more likely than nonparticipants to receive zero benefits early in the year before Tickets were mailed (Figure IV.4). This is one more way in which participants differed from nonparticipants prior to mailing, in addition to the differences discussed in Chapter III. However, the participant/nonparticipant difference in this outcome diminished for both program groups as the mail month approached. It essentially disappeared by the mail month for SSI recipients. And for DI recipients, the sign of the difference switched from positive to negative, meaning that TTW participants were less likely than nonparticipants to receive zero benefits during the mail month.
After Tickets were mailed, the proportion of DI and SSI beneficiaries who were receiving zero benefits increased for both TTW participants and nonparticipants, but the increase was greater for participants. Consequently, about midway through the follow-up period, a positive participant-nonparticipant gap re-emerged for beneficiaries of both programs. The gap continued to grow through month 15, when it was six-tenths of one percentage point for DI recipients and 1.3 percentage points for SSI recipients. Among DI recipients, the participant-nonparticipant difference in the rate of receipt of zero benefits was essentially the same 15 months after Tickets were mailed as it was 12 months before they were mailed. In contrast, among SSI recipients, the participant-nonparticipant gap at the end of the follow-up period was twice as large as the gap of six-tenths of one percentage point a year before Tickets were mailed.
The fact that the participant-nonparticipant gap in the receipt of zero benefits by SSI beneficiaries was larger 15 months after Tickets were mailed than it was 12 months before they were mailed is consistent with the hypothesis that TTW may have helped some of the Ticket participants to make the transition to zero benefit status. The relatively rapid increase in the incidence of zero benefits among TTW participants in the postmailing period further supports this hypothesis for SSI beneficiaries. A comparison of the participant/nonparticipant gap in the receipt of zero benefits by DI beneficiaries is a less reliable indicator of the possible effects of TTW. As explained in the previous section, this is because the nature of instability in benefit receipt was different after Tickets were mailed (transitions between benefit receipt and employment) than before (new enrollments). For DI beneficiaries, the faster rate of growth in zero benefit status in the postmailing period by TTW participants relative to nonparticipants is a better indicator of the possible influence of TTW, and it is consistent with a positive effect.
3. Zero Benefits and Substantial Employment
The difference between Ticket participants and nonparticipants in the percentage who were substantially employed and receiving zero benefits was larger before and after Tickets were mailed than during the mail month itself (Figure IV.5). For both DI and SSI beneficiaries, the participant-nonparticipant difference in this outcome measure was sixtenths of one percentage point one year before Tickets were mailed. By the sixth month after Tickets were mailed, the participant-nonparticipant gap was three-tenths of a percentage point smaller than its former size for DI beneficiaries, whereas for SSI beneficiaries, it was nearly one percentage point larger.
Our interpretation of these findings for this combined outcome measure is generally similar to that for zero benefit status alone. Among SSI beneficiaries, relatively rapid postmailing growth in the percentage of TTW participants who were substantially employed and receiving zero benefits, and the resultant widening of the participant-nonparticipant gap in this outcome measure, are what we would expect if TTW had the effects intended by policymakers. In contrast, relative to non-participants, DI beneficiaries who participated in TTW experienced only slight growth in the percentage who were substantially employed and receiving zero benefits. This finding is only weakly consistent with a possible positive effect of TTW.
4. Disability Benefit Amount
The average combined DI and SSI benefit amount received during the months leading up to and including the Ticket mail month was different for Ticket participants and nonparticipants (Figure IV.6). The sign and magnitude of the difference varied dramatically between beneficiaries of the two programs. After Tickets were mailed, benefits for TTW participants fell relative to those for nonparticipants in both programs.
Among DI recipients, the average disability benefit during the months leading up to and including the Ticket mail month was markedly smaller (by about $62) for those who ultimately assigned their Tickets than for those who did not. The opposite was true for SSI recipients—TTW participants received larger benefits than nonparticipants before Tickets were mailed—but the magnitude of the difference (about $15) was smaller than for DI recipients.
After the mail month, the average disability benefit received by TTW participants fell relative to that received by nonparticipants for beneficiaries in both programs. For DI beneficiaries, the ratio of the average benefit received by Ticket participants to that received by nonparticipants was 92 percent in the mail month. Fifteen months later, it was slightly smaller—91 percent. For SSI beneficiaries over the same period, the sign of the participant/nonparticipant difference in average benefits changed from positive to negative as the benefit ratio fell from 103 percent to 98 percent. These results, particularly those for SSI beneficiaries, are consistent with, but not firm evidence of, TTW having helped to lower disability benefits.
D. SUMMARY OF FINDINGS
This chapter has presented preliminary findings regarding three key outcomes for disability beneficiaries in Phase 1 states who were eligible for TTW on January 12, 2002. The findings are based on data for TTW participants and nonparticipants in the 12 months before and up to 15 months after Tickets were mailed to these individuals. For SSI beneficiaries, differences in the participant-nonparticipant gaps in the outcome measures before and after Tickets were mailed are what we would expect if the new program had helped to increase employment and program exit to some extent. They suggest that TTW may have slightly increased the proportion of SSI beneficiaries who receive zero benefits and who combine zero benefits with substantial employment, and slightly decreased the average benefit amount. For DI beneficiaries, relative rates of growth in the outcome measures during the postmailing period for participants compared with nonparticipants provide weaker evidence that is consistent with the TTW’s intended effects.
These findings are tenuous for several reasons. Most importantly, we know from the survey results presented in Chapter III that many TTW participants are motivated to become employed and, as a group, would probably be more successful in the labor market than nonparticipants even if Tickets had never been mailed. The findings presented in this chapter are based on simple analytic techniques that do not fully control for such motivational differences. Consequently, they may suggest impacts of TTW that may fail to be substantiated by future more rigorous analyses under this evaluation that will do a better job of controlling for the influence of motivation and other factors external to TTW. In addition, the findings presented here are tenuous for two reasons: (1) they are based on data that reflect a short postmailing follow-up period, during which time relatively few beneficiaries assigned their Tickets, and (2) we could observe changes in their employment and benefits for only 15 or fewer months. This problem will be less severe in future analyses because they will be based on data for longer follow-up periods.
1 To be eligible for TTW, a disabled individual must be at least 18 years old and less than 65 years old and receiving a positive DI or SSI benefit. The disability may be either permanent (improvement is not expected) or nonpermanent (improvement is either expected or cannot be accurately predicted). Two small groups, accounting for about six percent of those satisfying the former criteria, are ineligible: (1) SSI beneficiaries who had been entitled to benefits under the childhood regulations but who have very recently turned 18 and have not undergone the process to determine whether they are disabled under the adult eligibility criteria and (2) SSI or DI beneficiaries for whom medical improvement was expected at the time of benefit award but who have not passed at least one medical continuing disability review. Return to Text.
2 Readers who would like more information on the rollout of TTW can consult Thornton et al. 2004, Chapter III. Return to Text.
3 We do not present findings for the Phase 1 beneficiaries who became eligible for TTW after January 12, 2002 (“newly eligibles”), for three reasons. First, many of them were mailed Tickets after the initial group of Ticket eligibles, so fewer months of postmailing data on outcomes are available for them. Second, newly eligibles differ dramatically from existing eligibles in the pre-mailing values of the outcome measures, so we were reluctant to analyze both groups together. Third, there are only about 10 percent as many newly eligibles as initial eligibles; consequently, the numbers of newly eligible Ticket participants are small (1,131 DI and 844 SSI). Thus, summary measures of outcomes would be much less reliable when computed on newly eligible participants than when computed on existing-eligible participants. Return to Text.
4 The requirements that beneficiaries be alive and under age 65 at the end of the analysis period ensure that the trends in outcomes presented in this chapter do not reflect the influence of the attainment of age 65— when many beneficiaries stop receiving disability benefits and begin receiving retirement benefits—or of death. Return to Text.
5 The postmailing follow-up period is shorter for employment outcomes based on data from SSA summary earnings records, which are posted only after a considerable time lag. Return to Text.
6 Concurrent beneficiaries, who were receiving DI and SSI benefits, were included in the DI analysis and the SSI analysis. Return to Text.
7 To preserve the confidentiality of beneficiary data in SSA’s summary earnings records, contractor staff did not directly access those records. Rather, the SSA project officer for this evaluation ran analysis jobs for us on SSA’s mainframe computer. Return to Text.
8 Future reports on the TTW evaluation will present analyses of a more direct measure of substantial employment combined with zero benefits—one that does not require the simplifying assumption underlying the current measure. The new measure will be based on SSA’s official determination that a (former) disability beneficiary left the rolls due to work. Return to Text.
9 There are several reasons why members of the target population who were not substantially employed (by our definition) at some time during the analysis period may not have been receiving disability benefits. First, they may have been ineligible for benefits at that time for reasons unrelated to their own employment, such as the absence of disability or, for SSI, disqualifying levels of income or assets from other family members. Second, the measure of earnings that we used to determine substantial employment has known coverage gaps. Most notably, many state and local government employees, as well as some long-term Federal government employees, are not covered by Social Security, so their earnings are not included in SSA’s summary earnings records, which are the source of our earnings measure. Future reports under this evaluation will present employment-related findings based on data from SSA’s detailed earnings records, which have fewer coverage gaps than the summary earnings records because they are not restricted to earnings in Social Security covered employment. Additionally, some of our future analyses of zero benefit status and substantial employment will be based on SSA records data on the reason for exit from the benefit rolls. Return to Text.
10 To ensure comparability of benefits received in different years, all benefit amounts were adjusted for inflation to January 2004 dollars using the CPI-W. Return to Text.
11 Recall from Figure IV.1 that the percentage of Ticket recipients in zero benefit status fell during the 12 months before Tickets were mailed by 1.9 percentage points for DI participants and by 1.6 percentage points for SSI participants. These individuals were included in our calculation of the average benefit amount in all months, so they tended to depress the average in early months when they were receiving zero benefits. Return to Text.
12 See Thornton et al. (2004), page 89, for a description of AAA TakeCharge. Return to Text.