Disability Research Home

What's New

Major Activities

Demonstrations

Projects & Studies

Research & Data

Occupational Information System Project

Work Incentive Policy

Public Use Files

Completed Projects

More Information

About Us

Red Book

BPQY Handbook

Organization Chart

Related Links

Questions & Comments

CHAPTER 8
Next Steps for the Evaluation

The evaluation, like the TTW program itself, is in its early stages. This first report presents limited findings based on interviews with SSA, the Program Manager, and a small group of experienced ENs. The report also presents descriptive statistics based on preliminary TTW data. While these data sources provide useful operational information, they leave many key evaluation issues to be addressed by the data collection and analyses that will be completed during the next four and a half years. This chapter presents the data collection plan and the timing and content of future evaluation reports.

A. DATA COLLECTION SCHEDULE

Data collection for the evaluation is already under way and will continue over the next few years. Below we describe the plans for the evaluation’s three major data collection efforts.

1. Administrative Records

SSA administrative records will contribute to the analysis of TTW outcomes and impacts and help address virtually every priority research question for the evaluation (Table I.1).

The complex task of extracting beneficiary information from SSA’s extensive databases has already begun. We will create a longitudinal database from SSA administrative records that ultimately will include up to ten years of data for more than 15 million beneficiaries. This database will enable the evaluation to compare outcomes for beneficiaries before and after TTW implementation and in states with different employment and service environments. These comparisons are essential to measuring the extent to which TTW changes the employment and program participation outcomes for beneficiaries. The large size of this file means that analyses can be done for many beneficiary subgroups, including those included in the Adequacy of Incentives study and groups defined by their primary disabling condition, prior benefit receipt patterns, work history, or demographic characteristics.

The first version of this database will be analyzed in spring 2004. It will be updated in 2005 and 2007 and incorporated into the reports issued in those years.

Other administrative data will come from the Program Manager’s TTW system and the Rehabilitation Services Administration, which collects data about vocational rehabilitation service delivery. Those data will be essential to describing participation in TTW, particularly for beneficiaries who are served by SVRAs under the traditional payment system before and after the roll-out of the other TTW provisions. The Program Manager data is already being collected and should be available for all of the subsequent evaluation reports. SSA is currently negotiating with the Rehabilitation Services Administration to link individual vocational rehabilitation records with SSA records.

2. Surveys

A series of surveys will contribute to the outcomes and impacts analysis, the adequacy of incentives analysis, the process analysis, and the participation analysis. The surveys will provide detailed information about beneficiaries’ understanding of the TTW program, their attitudes and expectations about work and their actual work activity (including their wages, hours of work, occupations, and fringe benefits).

There will be two sets of surveys. First, the National Beneficiary Survey will interview a representative cross section of all active beneficiaries. This survey will be repeated each year starting February 2004 and continuing through 2007. It will track beneficiaries’ knowledge and views of TTW and other SSA programs as well as their levels of employment and service use. The evaluation will use this survey to analyze TTW participation and to understand the overall context within which TTW is fielded. Because it includes a representative sample of all beneficiaries, SSA will also use this survey to address broader issues, including beneficiary understanding and use of currently available work incentives, and to inform the design of future employment initiatives and beneficiary outreach and education strategies.

Second, the TTW Participant Survey will interview a representative sample of beneficiaries who have assigned Tickets to ENs. We will use data from this survey to assess beneficiaries’ experiences with ENs, particularly their service receipt, job placements, and satisfaction levels. This survey includes a new cross-section of beneficiaries each year as well as longitudinal follow-up interviews with a sample of the beneficiaries interviewed in the previous year. Altogether, more than 27,000 interviews are planned with beneficiaries and participants during the evaluation.

The beneficiary questionnaire is currently being finalized after a pretest conducted in December 2003. The sampling plan has already been developed and approved, and the sample for the first National Beneficiary Survey will be drawn from a list of all beneficiaries who received benefits in June 2003 (the survey will also include some beneficiaries whose benefits were temporarily suspended). The initial round of the Longitudinal TTW Participant Survey will be drawn from all beneficiaries who had a Ticket in use between January 1, 2003 and September 1, 2003. That is, it will include beneficiaries who had their Ticket assigned prior to September 2003 and had not terminated their Ticket assignment prior to January 2003. The sample excludes beneficiaries who participated only prior to 2003 because of concerns that they would not be able to remember details of their TTW experience when interviewed in 2004. It will take about six months to compete the interviewing for these surveys, so that data should be available for analysis late in 2004. The subsequent survey rounds will be rolled-out so that new survey data will be available each year through 2007.

3. Qualitative Data

The evaluation will track TTW operations using interviews, focus groups, document reviews, and case studies. We plan to contact representatives of every group involved, including SSA (central, regional and field offices), the Program Manager, SVRAs, ENs, former ENs that withdrew from the TTW program, non-participating providers who considered becoming ENs, and advocacy groups for beneficiaries (the beneficiaries will be interviewed directly in the surveys). This data collection effort will contribute to the outcomes and impacts analysis, the adequacy of incentives analysis, and the process analysis. During the early years of the evaluation, the qualitative data will also provide formative feedback to SSA as it considers ways to improve the program and its operations.

Qualitative data will be collected every year through early 2007. Table VIII.1 indicates the tentative schedule for qualitative data collection, although the plan may change in order to address emerging policy or operational issues.

Table VIII.1: Number of Planned Interviews with TTW Groups by Year and Group

B. ANALYSIS AND REPORT SCHEDULE

The evaluation will produce four more major reports on TTW, one in each of the next four years. As more data become available over time, the successive reports will provide additional analyses relevant to some of the research questions addressed in this report, and also analyze for the first time some issues that we were unable to address in the present report. Below we highlight some of the unique features of forthcoming reports.

Spring 2004. The next report will be the first to present results from analyses of SSA administrative data. These data will be used to describe the characteristics of TTW participants and nonparticipants, particularly their demographics, DI/SSI status, primary disabling condition, and work history. It will also present available information on beneficiary employment, earnings, and program exits because of work, but will not include an impact analysis. We will conduct an initial test of the methods developed to estimate impacts, as an assessment of the proposed methodology, but it will be too early in the TTW roll-out to provide policy-relevant estimates of effects. This report also will present additional findings from the process study, including a discussion of factors that affect organizations’ decisions to participate or not participate as ENs.

Spring 2005. This report will be the first to include results from the beneficiary survey, including additional information on participant characteristics and new information on beneficiaries’ knowledge of and participation in the Ticket program. This report also will be the first to address program effects on beneficiaries, presenting early impact findings on outcomes such as employment, earnings, participation in benefits programs, amount of benefits received, and total income.

Spring 2006. This report will draw on all data sources to present updated findings on all major elements of the evaluation—the process study, the participation study, and the outcomes and impacts study. This will also be the first report to analyze the longitudinal survey information on participants, which will track the duration of their employment and service use.

Spring and Fall 2007. These reports will revisit and update all the key research issues, summarize the evaluation’s most important findings and, if warranted, present final recommendations and matters for consideration by TTW stakeholders and policy makers, particularly SSA and the Congress.

Each report will include a special analysis of the adequacy of TTW incentives and the extent to which all groups of beneficiaries participate and benefit from the program. These analyses will pay particular attention to relatives levels of participation among different groups of beneficiaries (particularly those defined by the Ticket Act as being of special concern).

There will also be a series of specialized reports on the data collection activities and methods used in the evaluation. Non-technical summaries will also be prepared for the major reports produced in 2004, 2005, and 2007.