XIII. TTW Outcomes and Impacts

If the TTW program is to achieve the objectives that policymakers originally envisioned, it must increase the enrollment of eligible beneficiaries in employment services, and/or change service delivery in a manner that increases the likelihood of program exit (Exhibit XIII.1). Such changes should subsequently translate into higher earnings and lower DI and SSI benefit amounts. Initial impacts should occur first on enrollment in services and the nature of service delivery, as beneficiaries assign their Ticket and/or become more aware of employment service options in their area. Any impact on earnings and, especially, benefits are expected to take longer to emerge; earnings increases are not likely to occur for some time after Ticket assignment, and DI benefits will not be reduced until earnings have exceeded the SGA level for as long as 12 months.

Exhibit XIII.1. Anticipated Impacts of TTW on Service Enrollment, Earnings, and Benefit Amounts of Ticket-Eligible Beneficiaries

[D]

This chapter documents TTW’s impacts on:

• Enrollment in employment services provided by SVRAs and ENs
• Annual earnings
• DI and SSI benefit payments

We examined several approaches to estimating impacts based on non-experimental models originally suggested in Stapleton and Livermore (2002). We have concluded that the strongest approach is to estimate a multivariate model that compares outcomes of beneficiaries in states where the Ticket had already been phased in with the outcomes of beneficiaries in states where it had not. The model was estimated using administrative data on 4.7 million beneficiaries with disabilities. The data were obtained from SSA and other federal agencies and include information on service enrollment, program benefits, and SSA-covered earnings.

According to our analysis, TTW appeared to have a small impact on promoting service enrollment during the first year of TTW rollout. We do not find any evidence that the TTW had negative impacts on SVRA or EN enrollment and our upper-bound estimates indicate that TTW increased service enrollment by up to 0.4 percentage points, which represents an increase of 4,675 beneficiaries receiving services in Phase 1 states. Under the assumption that impacts would be the same in Phase 2 and 3 states, we project an increase in service enrollment of 16,743 beneficiaries across the entire caseload in the first rollout year.¹ Using a more restrictive set of assumptions for service enrollment, we obtain a lower-bound estimate of the Ticket’s impact of 0.1 percentage points, which represents an increase of 1,169 beneficiaries in Phase 1 states and a projected Year 1 increase of 4,186 beneficiaries across the entire caseload.

We found no compelling evidence that TTW affected beneficiary earnings or benefits during its first two years. If TTW had any success in increasing beneficiary earnings or reducing benefit receipt, those effects were masked by the underlying variation in beneficiary outcomes across states and over time. Our results show that there were persistent differences in the trends in earnings and benefit amounts for beneficiaries in Phase 1 states relative to those in Phase 2 and 3 states before the Ticket rollout. This finding indicates that the environment that influenced earnings and benefit amounts in Phase 1 states differed from the environment in other states. The fact that this underlying difference between states cannot be explained accurately by the available data means that the differences in earnings and benefit receipt between beneficiaries in Phase 1 states and those in Phases 2 and 3 states cannot be used to generate precise estimates of the effect of TTW on those outcomes. However, we did not find the same difference across states in service enrollment trends, which is an important factor in our ability to estimate service enrollment impacts. We speculate that this is because service enrollment is less sensitive than earnings and benefits to state policy and economic changes.

Our impact findings for all outcomes are consistent with the expectation that changes in service enrollment would occur before changes in either earnings or benefit receipt. Additionally, the relatively small service enrollment impact (0.1 to 0.4 percentage points) is consistent with the low Ticket participation rate, which was less than one percent during the first year of the rollout in Phase I. Given the anticipated timing of impacts and the relatively small service enrollment impacts, it is not surprising that we found no compelling evidence of subsequent impacts on earnings and benefit amounts at this early stage.

In the next report, we will extend the estimates of TTW on service enrollment to the second rollout year and address the limitations in the existing model by examining the extent to which these impacts are concentrated in a small set of states. Our findings of state differences before rollout indicates Ticket outcomes could be related to state characteristics.

The remainder of this chapter describes the methodological approaches to estimating impacts originally suggested by Stapleton and Livermore (2002) (Section A), summarizes the data sources and econometric model used for estimating impacts in this report (Sections B and C), presents detailed impact estimates and trends in key outcomes (Sections D and E), and concludes with a discussion of directions for examining impacts in future reports (Section F). Appendix D discusses all findings and provides detailed estimates for each of the models presented in this Chapter.

A. Overview of Approach to Estimating Impacts

Stapleton and Livermore (2002) summarized a general approach to estimating impacts in their design report for the Ticket evaluation that we used as a framework for this paper. The proposed approaches exploited variation over time in the rollout (“pre-post”) and across states (“contemporaneous comparisons”) in the three phases of the TTW program’s rollout. The major challenge in estimating impacts in a situation like this, where beneficiaries were not randomly assigned to program and control status, was choosing a credible comparison group. Their approach to estimating impacts requires the use of SSA and Rehabilitation Services Administration (RSA) administrative data. These data were the only viable options for estimating impacts, given the absence of pre-TTW survey data and the prohibitive costs of collecting enough survey data to identify meaningful contemporaneous differences in outcomes across states.

Of the approaches proposed by Stapleton and Livermore (2002), we determined that the strongest approach was to estimate impacts using a longitudinal fixed effects model (see Appendix D for a description of all models considered in Stapleton and Livermore). This approach measures impacts as the differences in the values of the outcome measures for the treatment group (beneficiaries who were eligible for TTW and were living in states where TTW had already been rolled out) and the contemporaneous values for the comparison group (beneficiaries who were eligible for TTW but were living in states where the program had not yet been rolled out), after controlling for characteristics in the pre-rollout year. Our strategy allows each source of identification—cross-state, pre-post, and within-period cross-person—to play a role, where the relative influence of each is allowed to be determined by the data. Given the data and the nature of TTW’s implementation, this model maximizes opportunities to reduce bias from individual confounding factors, such as beneficiary motivation and severity of impairment, as well as annual factors that might affect outcomes in all states.

B. Data Description

The selected approach to estimating impacts used SSA and RSA administrative data to assess impacts. We included administrative data from multiple SSA and RSA administrative data sources to develop a multi-year longitudinal file for the purposes of generating impact estimates. We selected an initial sample of all Ticket eligibles from these files, which we stratified by nine age and program groups. We then used this sample to generate impacts of TTW on annual service enrollment, earnings, and benefit amounts.

1. Administrative Data Includes Multiple SSA and RSA Administrative Data Files

The SSA and RSA administrative data sources include the Ticket Research File (TRF), which contains SSA program administrative data on the full population of working-age SSI and DI beneficiaries; SSA’s Summary Earnings Records (SER), which contains annual earnings data for all workers who pay Social Security taxes; and the RSA-911 case service report, which contains data on closed SVRA cases.² The TRF file used in this chapter contains longitudinal data on approximately 17 million beneficiaries age 18 through 64 with disabilities who participated in the SSI or DI programs at any time from 1994 through October 2004. The SER provides person-level historical data on Social Security taxable earnings for each year from 1937 to the present, which was the end of calendar year 2003 for this report. The RSA-911 file is updated annually by RSA to include each SVRA case that closed, as reported by state agencies, during the most recent federal fiscal year.

One important aspect of the file construction is that service enrollment measures from RSA data are available through 2002 (the first rollout year) and earnings and benefit amounts are available through 2003 (the second rollout year). The amount of information on service enrollment is limited because the RSA data pertain to case closures not enrollment. Because it takes two or more years to close the cases for many beneficiaries who use SVRA services, the available RSA data, which covered closures through 2004, can only be used to measure enrollment through 2002. In contrast, the lag in obtaining SSA earnings and benefit amount outcomes is shorter, which allows us to estimate impacts for these outcomes through 2003.

2. Sample Includes Most TTW Eligibles Age 18 to 57 and Is Stratified by Age and Program Subgroups

The analysis sample includes a 2001 cohort of beneficiaries with disabilities age 18 through 57 who would have been eligible for TTW when the program was rolled out in 2002. If a beneficiary was determined to be eligible in at least one month during a calendar year, that beneficiary was considered eligible for that year in the longitudinal file. We included an upper age restriction to ensure that all beneficiaries were under the age of 60 at the end of the two-year period for which we had data (that is, through 2004). Our findings in Chapter III indicate participation declines substantially with age, so the predicted TTW impacts on service enrollment, earnings, and benefit amounts should also decline with age. For those over age 57 in 2001, we assume that any impacts of TTW are far too small to be detected. In future reports, we will test this assumption by estimating impacts for this older population. We have no reason to expect that older beneficiaries who participate would have negative impacts, which could counter any positive impacts for younger beneficiaries or pull overall impacts into negative territory.

We excluded beneficiaries who were ineligible for the TTW, new beneficiaries, and those who moved across a phase state. The only beneficiaries in this age group who were ineligible for TTW were those designated as Medical Improvement Expected (MIE) who had been on the rolls for less than three years and had not yet had a continuing disability review; and former child SSI recipients awaiting adult redetermination. We excluded individuals who were new beneficiaries at the beginning of the TTW rollout by requiring that all beneficiaries in our sample have 12 full months of benefits in 2001. We excluded this group because it is difficult to measure base-year earnings and benefit amounts for them.³ Finally, we excluded beneficiaries who moved from a state in one rollout phase to a state in another phase (e.g., from a phase 1 state to a phase 2 state) during the window of our sample because we used the phase residence as a proxy for having access to the TTW program.

Our choice to estimate impacts using a sample of all TTW eligibles is important for two reasons. First, it is not possible to determine which members of the comparison group would have participated in TTW had they received a Ticket during the same period. Second, TTW might have effects that extend beyond effects on those who assigned their Ticket. As shown in Chapter III, a small share of eligible beneficiaries had participated in TTW by the end of the analysis period, December 2003 (1.0 percent in Phase 1 states and 0.5 percent in Phase 2 states). However, these participation rates might understate program impacts for two reasons. First, TTW might have affected beneficiaries with disabilities regardless of whether they assigned a Ticket. For example, the process of rolling out TTW and training SSA staff might have led to general change in attitudes among SSA staff, providers, advocacy organizations, and others to more aggressively promote return-to-work activities (for example, encourage use of work incentives, refer beneficiaries for related work services) to all beneficiaries, including those who did not assign a Ticket.

As suggested by Stapleton and Livermore, to account for differences in anticipated impacts in outcomes across subgroups, we stratified the sample by nine age-program groups based on age and program titles; the age categories are 18-39, 40-49, and 50-57, and the program title groups, which are mutually exclusive, are DI-only, the SSI-only, and concurrent (DI and SSI) beneficiaries.⁴ As noted, impacts should be larger among younger beneficiaries because they have higher employment rates relative to older beneficiaries as well as higher Ticket assignment rates. Impacts could vary by program title because work incentives and participation rates differ across the SSI and DI programs (Titles XVI and II), though other differences, including age, education, work experience, and income, make it difficult to predict whether impacts should be larger for one program group or another.

3. Outcome Measures Include Annual Measures of Service Enrollment, Earnings, and Benefit Amounts

We assessed the TTW’s impact on annual measures of SVRA-only service enrollment, two measures of total (SVRA and EN) service enrollment, benefit amounts, and earnings (Exhibit XIII.2).⁵ The SVRA-only measure was of interest to assess whether the Ticket had any impact in either inducing or crowding out SVRA enrollment by beneficiaries. This impact could be negative because some beneficiaries who, under TTW, only receive services from ENs after the rollout would have enrolled for services at an SVRA in the absence of TTW. It could be positive if TTW stimulated enrollment at SVRAs. The estimate of the impact on SVRA enrollment might also be downward biased if the TTW rollout increased the number of Phase 1 SVRA enrollees who were not included in the RSA data available for the analysis because their cases were still open.

The first total service enrollment measure (upper bound) captured SVRA and EN participation as measured in the RSA-911 and/or TRF data files. This measure included beneficiaries who had assigned their Ticket or had an open SVRA case sometime during the course of that calendar year. It addressed a limitation of the SVRA-only measure by capturing impacts on the private rehabilitation market through the inclusion of EN service enrollment information. In years before the TTW rollout in a phase group, a beneficiary was counted as enrolled for services in a calendar year only if the beneficiary had an open case at an SVRA in at least one month as measured in the RSA-911 data. In the first rollout year for Phase 1 (calendar 2002), a beneficiary was considered to be enrolled for services if, in at least one month, the beneficiary had an open SVRA case and/or has a Ticket assigned to an EN or SVRA as measured in the RSA-911 and/or TRF data files.

We refer to impact estimates using this first total service enrollment measure as an “upper bound” because we were concerned that it included an upward bias related to a change in the methods used to account for SVRA and, to a lesser extent, non-SVRA participants after the Ticket rollout. In 2002, Phase 1 beneficiaries enrolled for services under a Ticket assignment to an SVRA would be counted as enrolled in the TRF even if their SVRA case had not closed, whereas before the rollout, only closed cases are counted. Thus, this total service enrollment impact estimates might capture increases in measured enrollment that reflects only changes in measurement that coincided with the TTW rollout. It might also miss some beneficiaries who used non-SVRA rehabilitation service providers before the rollout in each phase. However, we believe the bias associated with non-SVRA participation is minimal based on a finding from our process analysis that suggests that the vast majority of ENs had not served beneficiaries prior to the TTW rollout, except possibly under contract to provide services to SVRA clients (Thornton et al. 2004).

To address this potential upward bias, we created a second total service enrollment variable (lower bound) that measured SVRA participation using the SVRA-only measure and added in the proportion of Phase 1 beneficiaries who had assigned a Ticket to an EN during at least one month in 2002.⁶ We use this measure to generate a “lower bound” impact estimate because it assumed that, if anything, the SVRA-only estimates had a downward bias, and the non-SVRA providers rarely gave services to beneficiaries except under contract to SVRAs. Our qualitative findings from the first Ticket evaluation report suggest that this assumption is reasonable (Thornton et al. 2004).

The benefit amount was measured from the TRF and modified for the purposes of estimating impacts. We generated the benefit amount as the sum of the federal SSI amount paid and the DI benefit amount due in a year from the TRF and adjusted these values to reflect January 2004 real dollars.⁷ We then modified the adjusted benefit amount measure so that its values in 2002 and 2003 were fixed at 2001 levels unless the beneficiary was employed at some time during the analysis period. The modification was necessary because benefit amounts can vary for several administrative reasons (for example, overpayments or changes in state supplement payment rules for SSI) that are unrelated to TTW but could influence the impact estimates (see Appendix D for more details).

Finally, the earnings were based completely on information from the SER and included the amount of earnings from Social Security-covered employment received during a year. As with the benefit amount measure, we adjusted earnings to reflect January 2004 real dollars.

C. Econometric Model for Estimating Impacts

Our approach to estimating impacts follows a 2001 cohort of beneficiaries to track changes in outcomes over time and across the different phases of rollout schedule during the program’s initial two years, 2002 and 2003. During this period, some states had implemented TTW (Phase 1 states in 2002 and 2003, and Phase 2 states in 2002), and some had not (Phase 2 states in 2002 and Phase 3 states in 2002 and 2003) (Exhibit XIII.3). The rollout was gradual within each phase group, so during the first rollout year for each phase the 2001 cohort’s beneficiaries residing in the phase’s states were only eligible for part of the year. The estimated coefficients from our model represent an impact per TTW eligible.

Impact estimates within this approach are measured as the differences in the values of the outcome measures for the treatment group (beneficiaries who were eligible for TTW and were living in states where TTW had already been rolled out) and the contemporaneous values for the comparison group (beneficiaries who were eligible for TTW but were living in states where the program had not yet been rolled out), after controlling for characteristics (including earnings and benefits) in the pre-rollout year.

To isolate TTW impacts from other possible influences on eligible beneficiaries, we used the following fixed effects longitudinal model to net out the stable differences in individual or contextual characteristics between the treatment and comparison groups:

where:

Y icsy = outcome for individual i in county c in state s during year y(use of employment and training services; benefit receipt and amount; and employment and earnings)

a i = individual (observed and unobserved) fixed effects for individual i

b s= state (observed and unobserved) fixed effects for state s

c y = time fixed effects for year y

X cy = unemployment rates in county c in year y

T1 sy = mailing year TTW treatment indicator in state s in year y

T2 sy = year after mailing TTW treatment indicator in state s in year y (earnings and benefit amount equations only)

ε icsy = unobserved disturbance term for individual i in county c in state s in year y

The key coefficients of interest in the model are λ 1 and λ 2, which represent impacts in the year of the Ticket mailing and in the year after the Ticket mailing, respectively.⁸ The service enrollment equation includes an impact only in the year of the Ticket mailing (i.e., λ 1) because as noted above, RSA administrative data on SVRA enrollment in calendar year 2003 were incomplete when the analysis was conducted. The earnings and benefit amount equations include data for the full rollout that can be used to estimate impacts in the year of the Ticket mailing and in the year after the Ticket mailing (that is, λ 1 and λ 2).

We present impact estimates for each of our outcomes and use projections to translate these estimates to effects on the total number of beneficiaries affected by the TTW. Our impact estimates provide information on the change in each outcome since the TTW was rolled out and our projections illustrate the total number of beneficiaries potentially affected by the policy.

Sample Size. The sample size for each of the nine age-program groups was very large, ranging from a minimum of 193,000 (concurrent beneficiaries age 50 to 57) to 1.1 million (DI-only beneficiaries age 50 to 57). Across all of the groups, the total sample size was 4.7 million beneficiaries. Specific sample sizes for each estimation model are presented in Appendix D.

Credibility of Estimates. We assessed the credibility of the estimates by checking their consistency with our expectations about impacts for the nine age-program groups, and with our descriptive analyses in earlier chapters on overall TTW participation rates. The aggregated estimates provide a general summary of findings relative to the full caseload, and the age-program estimates provide detailed information on subgroups of policy interest. We expected the estimated impacts to be small relative to the overall caseload, relatively larger for younger beneficiaries, and close to zero for older beneficiaries. Moreover, because of the direct and relatively immediate relationship between TTW and service enrollment, we expected to find larger impacts on service enrollment during the first year relative to the impacts on earnings and benefits.

Robustness of Findings. We tested the robustness of our findings by comparing our impact estimates with those produced by applying the same empirical model for several pre-TTW cohorts. We estimated models for two pre-TTW cohorts (1998 and 1999 cohorts) for which we have data on all outcomes.⁹ In each case, the model was estimated over a three-year period that starts with the cohort year and ends before the Ticket rollout. Presumably, the impact estimate for these earlier cohorts should be zero because the Ticket was not available. Non-zero estimated impacts on outcomes for any of these early cohorts would suggest that impact estimates from the rollout period reflect differences in outcome trends across Phase 1, 2, and/or 3 states that existed in the pre-TTW period.

D. Impacts on Service Enrollment

The impacts on service enrollment apply to the beneficiaries enrolled in services during the first year of TTW rollout in Phase 1 states who were age 18-57 in 2002 and had been on SSA disability benefits for at least one year. We present estimates for the SVRA-only service enrollment measure and the two upper and lower bound total service enrollment measures.

1. Estimates by Age and Program Group Indicate Impacts Close to Zero of TTW on SVRA-only Service Enrollment

Our impact findings for the SVRA-only service enrollment measure indicate that the TTW did not have major impacts on the number of people being served by SVRAs. Our estimates are close to zero for all age-program groups (see Appendix D).¹⁰

2. Estimates by Age and Program Group Indicate Positive Upper-Bound Impacts of TTW on Total Service Enrollment

We present detailed estimates for our upper bound estimates of total service enrollment in Exhibit XIII.4 that shows statistically significant program impacts. The top chart in the exhibit summarizes estimates of the impacts of TTW, and the bottom chart summarizes the mean values of service enrollment for the treatment and comparison groups. The treatment-comparison difference in mean values in the bottom chart is the impact estimate shown in the top chart.¹¹ These two ways of presenting the estimates highlights both their absolute and relative size.

The impacts of TTW on total service enrollment are positive in all age-program groups and are generally larger among younger beneficiaries. As shown in the top chart, the impact estimates for beneficiaries age 18 through 39 imply an absolute increase of 0.5 percentage points (SSI and concurrent beneficiaries) to 0.6 percentage points (DI-only beneficiaries) in enrollment in SVRA and/or EN services during the initial rollout year, 2002. In contrast, the estimated impacts for the two older groups of beneficiaries are smaller, ranging from 0.1 percentage points (age 50 to 57 concurrent beneficiaries) to 0.4 percentage points (age 40 to 49 SSI-only recipients and age 40 to 49 concurrent beneficiaries). The larger impacts in younger beneficiaries are consistent with higher TTW participation rates for this population. In general, there are not large differences in impacts on service enrollment across program categories within each age group.

Exhibit XIII.4. Upper-Bound Impact Estimates on Total Service Enrollment for Ticket-Eligible Beneficiaries Age 18 to 57, by Age and Program Group

[D]

Source: Tabulations of econometric estimates based on linked TRF and RSA-911 longitudinal data files. Note: The absolute impact estimates are regression coefficients from separate econometric analyses for each age-program group. Full sets of coefficient estimates and sample sizes for this exhibit appear in Appendix Table D.3. The relative impacts represent the regression adjusted means of service enrollment for the treatment and comparison cases in the year of the Ticket mailing (2002). Sample sizes for the age-program groups range from 193,000 to 1.1 million. *Impact estimate is statistically significant at the .01 level.

The magnitude of the impacts ranges from 0.1 to 0.6 percentage points, indicating a small increase in overall total service enrollment in each of the age-program groups. The largest point estimate is for DI-only beneficiaries age 18 to 39 and the smallest is for concurrent beneficiaries age 50 to 57. The largest impact relative to the 2002 service enrollment value was a 10 percent change for concurrent beneficiaries age 40 to 49 (from 4.9 to 5.4 percent).

The aggregate upper-bound estimates of TTW’s impact on service enrollment, which we calculated by using a weighted average of the age-program group estimates from above, indicate that the impacts for each program group are roughly similar but that larger differences exist across age groups (Exhibit XIII.5). The aggregate impact estimate for the overall population is an increase of 0.4 percentage points. The magnitude of the impact for young beneficiaries (ages 18 to 39) is more than two times larger than that for the oldest group (ages 50 to 57, 0.5 vs. 0.2 percentage points, respectively). The estimated impacts on total service enrollment are fairly uniform across the program groups (an increase of 0.3 to 0.4 percentage points).

Exhibit XIII.6 summarizes our estimates of the total service enrollment impacts of TTW on individual Ticket-eligible beneficiaries and shows the implications of those estimates for the Phase 1-only states and projections for the full caseload. The 0.4 percentage point increase in service enrollment represents a 9.5 percent increase in overall service enrollment (from 4.2 to 4.6 percent). This impact translates to an increase in service enrollment of 4,675 beneficiaries in Phase 1 states during the first rollout year. Based on this estimate, the projected impact translates to an increase into an upper-bound impact on service enrollment of 16,743 across all three phases in their respective rollout years.

The impacts findings across the nine age-program groups and the projections of the overall effects across the entire caseload are consistent with the theoretical expectations. The results for the age-program groups are consistent with expectations, as the larger impacts are generally concentrated among younger beneficiaries in all program groups, and older beneficiaries had much smaller impacts. The magnitude of the impacts (less than 1 percentage point) is consistent with the TTW participation rates with the 1.1 participation rate by eligible beneficiaries through March 2004 in Phase 1 states.

Our confidence in the total service enrollment estimates is further bolstered by applying our model to earlier cohorts, where we show that our impact findings are distinctly different from observed differences in pre-TTW cohorts. As shown in Exhibit XIII.7, the results from our econometric models indicate that Phase 1 states had similar service enrollment trends relative to other states prior to the program rollout for the 1998 and 1999 cohorts (enrollment was less than 0.1 percent below the rate in other states). This finding is to be expected given that TTW was not yet available. Additionally, as shown in Appendix D, the differences within the age-program subgroups were also small or statistically insignificant. However, after TTW rollout, we show the Phase 1 difference in service enrollment, which is the impact estimate above, is substantially different from these earlier cohorts. Hence, these sensitivity tests indicate that trends in service enrollment only changed across states appreciably after rollout, thereby affirming our impact estimates above.

3. Lower-Bound Estimates Using an Alternative Total Service Enrollment Measure Indicate Smaller Impacts

Based on the findings of a zero impact on SVRA-only services, we generate a lower-bound estimate of the TTW’s impact on total service enrollment under the assumption that the only increases in enrollment were through non-SVRA ENs. Our process analysis findings in the second report indicate that just under 0.1 percent of Phase 1 TTW-eligible beneficiaries (approximately 10 percent of TTW participants in Phase 1 states) enrolled in a non-SVRA EN. Furthermore, those process analysis findings suggest that few, if any, non-SVRA ENs served beneficiaries prior to TTW except as subcontractors to SVRAs. Hence, a reasonable lower-bound estimate for the service enrollment impacts based only on non-SVRA ENs is 0.1 percent.

Exhibit XIII.7. Sensitivity Tests of Total Service Enrollment Impact Models Applied to Pre-Ticket Cohorts

[D]

Source: Tabulations of econometric estimates based on linked TRF and RSA-911 longitudinal data files. Note: The difference in service enrollment in Phase 1 states relative to other states represents a weighted average from all of the age-program group estimates. Results for enrollment in services pertain to the year when Tickets were mailed. Full sets of coefficient estimates and sample sizes for the age-program groups exhibit appear in Appendix Table D.3.

Exhibit XIII.8 summarizes our lower-bound estimates of the impacts of TTW based on the assumption that the only impacts on service enrollment are through non-SVRA ENs. The 0.1 percentage point increase in service enrollment represents a 2.4 percent increase in overall service enrollment (from 4.2 to 4.3 percent). This impact translates to an increase in service enrollment of 1,169 beneficiaries in Phase 1 states during the first rollout year. Based on this estimate, we project an impact that translates to an increase in service enrollment of 4,186 across all three phases in their respective rollout years.

4. Summary of Findings Indicates a Range of Small Positive Impacts on Total Service Enrollment

We conclude that the TTW did not have a negative impact on SVRA service enrollment, and that our estimates for total service enrollment are between 0.1 and 0.4 percentage points. While we cannot precisely estimate impacts, our findings from the models above underscore that TTW probably increased overall beneficiary enrollment in employment support services by a relatively small amount in relation to the overall caseload. We will further assess the size of these impacts in future reports as more data becomes available for later years.

E. Impacts on Earnings and Benefit Amounts Are Too Small to Differentiate from Historical Variation

To estimate TTW’ impacts on annual earnings and benefit amounts during each of the first two years of the rollout, we used the same model that was used to estimate impacts on service enrollment. We expected impacts on earnings and benefits to be minimal during the first rollout year, when participants are presumably receiving services, but thought that they might be large enough to detect in the second year.

The early impact results for beneficiary earnings and benefit receipt, however, are inconclusive. When we applied our model to the 1998 and 1999 cohorts, we found that earnings were higher ($33 in the 1998 cohort and $29 in the 1999 cohort) and benefit amounts were lower (-$20 in 1998 cohort and -$15 in 1999 cohort) in Phase 1 states relative to other states (Exhibit XIII.9). While we found that Phase 1 state beneficiaries had higher earnings levels ($23) and lower benefit amounts (-$20) in the year after Ticket mailing, we are skeptical that these differences represent true impacts because they are not different from the historical pattern in these outcomes for prior cohorts. Instead, the estimates based on earlier cohorts indicate the presence of a persistently positive trend in earnings levels and a negative trend in benefit amounts in Phase 1 states relative to Phase 2 and 3 states before the rollout.. As a result, it is not possible to tell if TTW had an effect on these outcomes or if TTW was merely rolled out first in states that had systematically different trends in beneficiary earnings and benefit receipt.

The differential trends in earnings and benefit amounts in the pre-TTW period across states were likely related to unmeasured state differences the economic and policy conditions. In general, Phase 1 states appeared more conducive to implementing the TTW program as beneficiaries in these states were more likely to be receptive to return-to-work activities based on their relatively higher earnings trajectories that resulted in lower benefit amounts. Indeed, SSA selected the Phase 1 states for this specific feature. The findings also indicate that the differences in state environments had a larger effect on earnings and benefit amounts than they did on service enrollment. One possible explanation of the differential impact on outcomes is that the role of the economy and policy environment has a stronger effect on the relative trends in earnings and benefit amounts than it does on the relative trends in service enrollment, which is plausible given the more direct effects associated with changes in economic conditions and earnings.¹²

F. Analyses of Impacts in Future Reports Will Focus on the State Level

For the fourth report, we plan to update our impact estimates of service enrollment as new data about service enrollment (not just case closures) become available for 2003 (for the second year of rollout in Phase 1 states and the first year in Phase 2 states). We anticipate that these impacts will be larger than those in the first rollout year, given descriptive data from Chapter III that show substantially higher participation in 2003 than in 2002.

Based on our findings of differences across states, particularly for earnings and benefit amounts, we will focus our further efforts in estimating impacts on outcomes at the state level. We conducted a preliminary analysis indicating that TTW impacts not only varied across states, but also did so in a manner that is roughly consistent with state differences in Ticket participation, as identified in Chapter III.¹³ In future reports, we will further explore state differences in impacts on service enrollment and assess whether these differences are related to other state differences, especially differences in EN participation and/or SVRA outreach efforts.

Exhibit XIII.9. Sensitivity Tests of Earnings and Benefit Impact Models Applied to Pre-Ticket Cohorts

[D]

Source: Tabulations of econometric estimates based on linked TRF and SER longitudinal data files for earnings and TRF files for benefit amounts. The difference in earnings and benefit amounts in Phase 1 states relative to other states represents a weighted average from all of the age-program group estimates. Results for earnings and benefit amounts pertain to the year after the Tickets were mailed (coefficient on ). The coefficients on , the year of Ticket mailing, are generally close to zero for both earnings and benefit amounts across cohorts. Full sets of coefficient estimates and sample sizes for the age-program groups appear in Appendix Exhibits D.5 and D.6.

The opportunities for generating additional longer-term impact estimates or estimates on additional other outcomes are limited. The methodology used here could be extended to impacts in later years. However, because TTW was rolled out nationally in subsequent time periods, an extension of the methods would mean making untestable assumptions about variation in the size of impacts across the three phases (for example, that impacts in each year after rollout are constant across the three phases). It will also continue to be difficult to distinguish between true impacts and historical differences in trends across the three phases. Finally, as described in more detail in Appendix D, the potential for using alternative methods for estimating impacts originally outlined in Stapleton and Livermore (2002), including historical cohort and propensity score matching methods, is likely limited.

To obtain additional information on TTW-related outcomes, we plan to descriptively track service enrollment, earnings, and benefits at the national and state level as well as other outcomes that are likely to be sensitive to TTW, such as the number of beneficiaries who leave the rolls due to work and participation in SSA work incentive programs, including the SSI Section 1619 program, the SSDI trial work period, and the DI extended period eligibility. These trends will provide descriptive information that policymakers can use to assess the extent to which these outcomes are moving in the direction that TTW, as well as many other initiatives, is designed to promote. Of particular interest will be the question of whether there has been an increase in the number of people who leave the rolls because of work that corresponds to TTW’s objectives of doubling that number. For example, if TTW meets its objectives, we might expect to find that the number of people who leave the rolls because of work in the future years increases from 0.5 to 1.0 percent. The decision to track outcomes instead of estimating impacts acknowledges the fact that although we cannot distinguish between the impact of TTW and the confounding effects of other factors, the evaluation findings can still inform policymakers and others with a stake in the system about the extent to which these outcomes are moving in the desired direction.

We also plan to examine the influence of state policies that are complementary to TTW, such as the Medicaid Buy-in program, on outcomes in states that have few or no complementary programs. If such complementary programs are effective in promoting employment, outcome trends in states with such programs should be more favorable than outcome trends in states without such programs. An analysis of state-level variation in outcomes might also help us to distinguish between the effects of the economy on outcomes and the effects of policy and program changes.

¹ It is possible that impacts may be larger in future years of the rollout because of differences in state policy environments and/or further refinements in the marketing of TTW to Phase 2 and 3 states. At this point, we do not have evidence that these impacts would be substantially larger or smaller than those in Phase 1 states. Hence, the experiences in Phase 1 states should provide a reasonable approximation for the potential experiences in Phase 2 and 3 states that we can use to calculate aggregate impacts across the entire caseload. (back)

² In accordance with the Internal Revenue Service/SSA data agreement, MPR researchers did not access earnings data with personal identifiers. (back)

³ For example, it is likely that many new beneficiaries, especially DI beneficiaries, will have at least some reported annual earnings according to the SER, although we cannot determine what portion of these earnings came before or after benefit receipt. Because of this issue, new beneficiaries could have received substantial base-year earnings before enrolling in the program, which could introduce measurement error in our earnings impacts of TTW in later years. Additionally, we anticipate the impacts on new beneficiaries will differ from existing beneficiaries. For these reasons, we plan to estimate impacts on these populations separately in future analysis. (back)

⁴ We excluded those over age 57 because beneficiaries nearing the retirement age have relatively fewer prospects for using TTW to return to work. (back)

⁵ We also examined three supplemental outcome measures—annual employment status, annual benefit receipt, and an indicator from SSA administrative records of beneficiaries who left SSI and DI programs specifically because of work (“left cash benefits due to work”)—that are not reported below but are available in Appendix D. These measures are more restrictive than the core measures of benefit and earnings outcomes shown in Exhibit XIII.2. We did not find any significant impacts on any of these outcomes during the two years of the TTW rollout. (back)

⁶ Unlike the upper bound measure, the lower bound measure did not include open SVRA participants measured in the TRF file in any month of 2002. (back)

⁷ The amount paid represents the benefit actually received by the beneficiary in a particular month and the amount due is the amount that SSA is scheduled to pay the beneficiary. The benefit amount paid and amount due can differ if there are changes in the beneficiary’s status. For example, if SSA retroactively has adjusted a beneficiary’s record for an overpayment due to excess earnings, the amount due will be less than the amount paid. In later months, collection of overpayments will reduce amounts paid relative to amounts due. We would have preferred to use the amount paid variables for both SSI and DI, because the amount paid accurately captures SSA’s benefit cost experience. At the time of our analysis, however, the DI benefit amount paid was not available. The implication for the measurement of this outcome is likely limited given that generally there are only relatively small differences between the amount paid and amount due variables in DI. (See Appendix D for more details.) (back)

⁸ The impact estimate in the year of Ticket mailing, represented by λ 1, includes the difference-in-differences from 2001 to 2002 in Phase 1 states relative to Phase 2 and 3 and, for the earnings and benefit equations, the difference-in-differences from 2002 to 2003 in Phase 2 states relative to Phase 3 states. The impact estimated for the earnings and benefit equations in the year after Ticket mailing, represented by λ 2, is the difference-in-differences from 2001 to 2003 in Phase 1 states relative to Phase 3 states. Because TTW was fully implemented in all states after 2003, there is no comparison group in the year after Ticket mailing for Phase 2 states. (back)

⁹ In Appendix D, we also present additional 1996 and 1997 cohort models for benefit and earnings outcomes to further test the sensitivity of our findings to different economic conditions. We do not have corresponding data on service enrollment outcomes for the 1996 and 1997 cohorts. (back)

¹⁰ Our estimates indicated negative impacts of –0.3 percent points for age 18 to 39 concurrent beneficiaries and a positive impact on SVRA enrollment of 0.1 percentage points for age 40 to 49 SSI-only beneficiaries. (back)

¹¹These charts present regression-adjusted means for the year before and after Tickets were mailed. The difference in regression-adjusted means for treatment and comparison beneficiaries is the impact estimate. (back)

¹² It is important to note that the differences in impacts represent relative trend differences across states, not aggregate state differences. It is likely that economic conditions affect all of our outcomes. While our econometric model makes adjustments for any stable initial differences that exist across states, our ability to control for any within-state changes in policy or economic conditions (beyond controls for the unemployment rate) is limited. We argue that it is these within-state differences that have a stronger influence on earnings and benefits relative to service enrollment. (back)

¹³ We derived these estimates by interacting the Ticket treatment indicators ( T1 sy and T2 sy) with state indicators from Phase 1 states. (back)