5
A Work Disability Monitoring System

The previous chapter discussed the National Study of Health and Activity (NSHA) developed by the Social Security Administration (SSA) to estimate the size and characteristics of the population eligible for disability benefits. When completed it should yield a rich set of data that should be valuable for policy development and planning. Effective management of SSA’s disability programs requires sufficient information to understand and predict changes in the size, characteristics, and distribution of the pool of persons eligible for disability benefits (applicants and beneficiaries), as well as to understand the factors that affect application volume and answer many policy questions. A single cross-sectional survey such as the NSHA will not provide adequate data in the future for either of these goals. Medical models of disability historically have been insufficient in explaining unexpected growth in the size of the applicant pool (Haber, 1971; Yelin et al., 1980; Stapleton and Dietrich, 1995; Stapleton et al., 1995; Bound and Waidmann, 2000). Factors extrinsic to the benefits programs—for example, cyclical changes in the economy, as well as social and cultural issues—have resulted, in the past, in changes in applications and awards rates and unexpected increases in program expenditures.

One means to understand the magnitude and characteristics of the potential eligible population as well as the intrinsic and extrinsic factors that influence the application for benefits is to develop a monitoring system related to work disabilities. The idea is not new, as is evident in the series of surveys sponsored by SSA throughout the late 1960s and the 1970s (for details, see Mathiowetz, 2001, in Part II). Of interest in a disabil-



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 88
The Dynamics of Disability: Measuring and Monitoring Disability for Social Security Programs 5 A Work Disability Monitoring System The previous chapter discussed the National Study of Health and Activity (NSHA) developed by the Social Security Administration (SSA) to estimate the size and characteristics of the population eligible for disability benefits. When completed it should yield a rich set of data that should be valuable for policy development and planning. Effective management of SSA’s disability programs requires sufficient information to understand and predict changes in the size, characteristics, and distribution of the pool of persons eligible for disability benefits (applicants and beneficiaries), as well as to understand the factors that affect application volume and answer many policy questions. A single cross-sectional survey such as the NSHA will not provide adequate data in the future for either of these goals. Medical models of disability historically have been insufficient in explaining unexpected growth in the size of the applicant pool (Haber, 1971; Yelin et al., 1980; Stapleton and Dietrich, 1995; Stapleton et al., 1995; Bound and Waidmann, 2000). Factors extrinsic to the benefits programs—for example, cyclical changes in the economy, as well as social and cultural issues—have resulted, in the past, in changes in applications and awards rates and unexpected increases in program expenditures. One means to understand the magnitude and characteristics of the potential eligible population as well as the intrinsic and extrinsic factors that influence the application for benefits is to develop a monitoring system related to work disabilities. The idea is not new, as is evident in the series of surveys sponsored by SSA throughout the late 1960s and the 1970s (for details, see Mathiowetz, 2001, in Part II). Of interest in a disabil-

OCR for page 88
The Dynamics of Disability: Measuring and Monitoring Disability for Social Security Programs ity monitoring system is not simply measurement of the prevalence and of the socioeconomic conditions linked to disability, but also understanding both the individual and the environmental factors that lead to changes in the application process. This chapter explores ways in which SSA could build from its experience with the NSHA to develop an ongoing disability monitoring system for Social Security programs to provide timely information on the prevalence and distribution of disability in the working population. This chapter discusses the need for such a system, essential principles of such a system, possible design choices, and a suggested planning and implementation strategy. NEED FOR A WORK DISABILITY MONITORING SYSTEM A well-designed monitoring system should provide SSA with the data needed to respond to a variety of policy and planning issues, including, but not necessarily limited to, the following: Size, distribution, and characteristics of the working populations with disabilities. The growth in the population eligible for SSDI and SSI during the past three decades and the concomitant growth in applicants and awards have raised questions as to whether continued expansion of these programs can and should be sustained. Demographic trends. The working age population has grown dramatically and its composition has undergone fundamental change since the inception of the SSDI and SSI programs. This working age population eligible for disability benefits is projected to increase in the coming years as the baby boom generation ages and reaches the ages at which chronic diseases and disabilities are more likely to occur. This growth will impact significantly the Social Security disability programs in many ways. Labor market dynamics. Structural changes in the economy such as the relative shift over the years to service industries and occupations have a significant impact on the types of impairments that result in work disability. Labor force participation rates among women have increased substantially while those of men have declined. These structural changes need to be fully understood and predicted accurately. Changes in economic conditions. During periods of slowdown in the economy and high unemployment, marginal workers especially low-wage workers with disabilities are more likely to apply for disability benefits. SSA needs to closely monitor these changes in economic conditions and their impact on Social Security disability programs.

OCR for page 88
The Dynamics of Disability: Measuring and Monitoring Disability for Social Security Programs Needs of minority and special populations with disabilities. Disabilities are disproportionately represented among minorities, the elderly, and lower socioeconomic populations. The causes of these differentials are not clear. People with lower socioeconomic status probably experience more injuries, higher mortality rates, less access to health care, and generally poor health. On the other hand, some people have lower incomes because their disabling conditions restrict their ability to work. These phenomena and their relationship to application and receipt of Social Security benefits need to be further studied. Quality of life for disabled workers. Quality of life is an important theme for all workers, but it is especially important for those with disabilities. An improved quality of life through provision of assistive technology in and out of the workplace could represent the difference between working and applying for Social Security benefits. Quality-of-life measures for the population with disabilities need to be developed by SSA in collaboration with other relevant agencies. Functional status. Health conditions differ in the degree to which they cause functional limitations and disabilities that may result in work disability. One of the committee tasks was to examine SSA’s research into functional assessment instruments for its redesign efforts and to provide advice for adopting or developing instruments for the redesigned decision process and the National Study on Health and Activity. A workshop titled Measuring Functional Capacity and Work Requirements was held on June 4–5, 1998. Following the workshop, the committee issued its second interim report in 1988 titled The Social Security Administration’s Disability Decision Process: A Framework for Research. The committee recommended that further research on functional assessment measures be conducted. The role of functional assessment in the disability decision process still remains an important issue. Role of the states. As noted in Chapter 2, in times of poor economy, cuts made in state and locally funded general assistance and other welfare programs result in shifting the burden from state and local programs to federal programs. In addition, welfare agencies routinely refer persons to SSA’s disability programs. The relationship between SSA’s disability programs and state and locally funded programs needs to be further studied. Legislative, regulatory, and judicial impacts. Legislative and regulatory changes and court decisions have a major impact on SSA’s disability programs. These need to be monitored on an ongoing basis.

OCR for page 88
The Dynamics of Disability: Measuring and Monitoring Disability for Social Security Programs As stated in the previous chapter, SSA considers NSHA the centerpiece of its long-term disability research to understand the growth in the disability programs. The NSHA is designed to provide SSA with an estimate of the extent of the prevalence of disability, the factors that enable some people with disability to remain in the workforce, a basis for gauging the effect of changes in disability decision criteria, and much needed insight into the problems of measuring disability in surveys. The committee believes that the NSHA should be considered the first important part of a long-term commitment by the SSA to produce reliable national data on the demand for and quality of its disability benefits programs. However, NSHA will be of limited utility in directly measuring long-term temporal market changes and demographic changes. Although the population may age in somewhat predictable ways as the baby boom generation moves into the vulnerable ages, the size and geographic distribution of the racial and ethnic makeup of the population will change in less predictable ways depending on the swirling currents of economic opportunity and the associated flow of immigration across the nation’s borders. Uncertainty about the future direction of legal and policy changes affecting the population with disabilities, and about ways in which medicine and technology might be used to enable Americans with disabilities to function more effectively, will also limit the long-term usefulness of NSHA data. The committee believes that SSA has a continuing need for current and reliable data to project growth in its programs, and to understand the contributing factors. Similarly, it needs data to effectively adapt its disability benefits programs to the changing needs of people with disabilities. Disability is a dynamic phenomenon that needs to be monitored and evaluated continuously. The conduct of in-depth periodic disability surveys will provide an essential database for understanding this dynamic process. Because of the time lag for research and development, as well as the costs involved in launching a survey of the magnitude of the NSHA, it is not feasible to repeat such a study design every year, or even every other year. Such a comprehensive survey should be conducted with regular periodicity, at least every 7–10 years. In the intervening years, however, SSA requires ongoing estimation of the size of the population eligible and applying for benefits as well as other essential data including the socioeconomic and demographic characteristics and distribution of the eligible population. SSA’s information needs for policy decisions, therefore, suggest a need for a two tiered measurement program: (1) periodic rich and deep national data on the size, distribution, and characteristics of the working population with disabilities that permit analysis and simulation of alter-

OCR for page 88
The Dynamics of Disability: Measuring and Monitoring Disability for Social Security Programs native decision criteria, and (2) macro level ongoing national data to monitor the size and characteristics of the population eligible for benefits. Maintaining periodicity of data collection is a common problem in constructing an indicator series. A government agency needs very detailed data to help administer a program addressing a social phenomenon. These detailed data, yielding themselves to years of alternative analyses, form the basis of long-range policy guidance. Large studies providing these data are supplemented by ongoing data for monitoring a small set of key indicators. One example of such a system is the quinquennial economic censuses, as benchmarks of the size and complexity of the U.S. retail sector, supplemented by periodic monthly data on retail sales, plant and equipment investments, and other variables. Another approach would be to undertake followback surveys of panels of the large survey. Only with continuous data collection will analysts and policymakers have the information to understand and predict the impact of changes in the environment on an individual’s propensity to apply for benefits and other similar issues. SSA should make the investment in resources to expand its infrastructure to develop a permanent information-gathering system to monitor the disability-related needs of those it serves and the impact of disability benefits programs it is required to maintain. Recommendation 5-1: The committee recommends that the Social Security Administration develop an ongoing disability monitoring system from the experience with the National Study of Health and Activity. The committee further recommends that SSA establish a clear set of objectives for guidance in developing and implementing the substantive content of the system. Specific objectives might include all or some of the following: develop the capacity to estimate the current, and project the future, prevalence of work disability and the characteristics of the population with disabilities on an ongoing basis; assess how well its programs are serving persons with disabilities; monitor the number and proportion of working age adults with impairments severe enough to apply for benefits; monitor allowance rates at all levels of adjudication and investigate reasons for variation across regions and over time; monitor changes in nonmedical risk factors associated with the application for benefits, including changes in demographic characteristics, nature of employment, and nature of disability compensation programs outside of SSA; SSA should be able to observe the

OCR for page 88
The Dynamics of Disability: Measuring and Monitoring Disability for Social Security Programs impact of changes in the demand for labor, nature of work, and other risk factors on the propensity of individuals with impairments to apply for benefits; and be able to foresee change in demand for benefits by identifying the precursors to change, such as the nature of employment, personal and public attitudes about coping with disability, and alternatives to SSA disability benefits, so that it is better able to anticipate the need for its disability benefits. Underlying these objectives, the committee believes that SSA must be aware at all times of the potential need for, and effectiveness of, its disability benefits. It must know how many Americans may need their benefits, who has been applying for them and why, how satisfied those receiving them are with the administrative apparatus that has been installed to deliver benefits, and why those eligible but not receiving benefits have not applied. A disability monitoring system would begin with a comprehensive measurement such as the NSHA, from which a reduced set of indicators of the size and characteristics of the “pool” of applicants for disability benefits would be identified. Every n years a similarly large and in-depth survey would be mounted. In the intervening years the reduced set of indicators (or estimations based on existing data) would be the source of national estimates of the size and distribution of the potentially eligible persons and of other issues. With each passing year the relevance of the comprehensive survey’s data and analysis declines. The magnitude of policy and social changes in the intervening years affects the periodicity of the comprehensive survey. If policy or social changes are large, SSA may need to mount another comprehensive survey of the richness provided by NSHA within a three- to four-year period. If changes are small, the periodicity of the comprehensive study might be extended to, for example, every 7 to 10 years. This periodicity also allows sufficient time for thorough evaluation, planning, and testing innovations. The next iteration of the NSHA-type survey with rich measurements conceivably might use a new set of measures, developed as a result of scientific progress or changes in program direction since the last large disability survey. Recommendation 5-2: The committee recommends that the disability monitoring system consist of a periodic, comprehensive, and in-depth survey to measure work disability; and a small set of core measures in the intervening years derived from other surveys, reinterviews, and/or administrative data.

OCR for page 88
The Dynamics of Disability: Measuring and Monitoring Disability for Social Security Programs SSA should collaborate with other federal agencies on the design and implementation of the monitoring system. CHARACTERISTICS OF A DISABILITY MONITORING SYSTEM The committee defines a disability monitoring system as an ongoing systematic collection, analysis, and interpretation of data essential to the planning, implementation, and evaluation of the Social Security Disability Insurance (SSDI) and the Supplemental Security Income (SSI) programs, closely integrated with the timely dissemination of these data to those who need to know. Monitoring systems typically rely on a variety of data sources originally designed for other purposes such as, but not limited to, the national surveys of the National Center for Health Statistics (NCHS), the Bureau of the Census surveys, and administrative data. No single standard exists in the design of monitoring systems; rather, they should be designed to meet the specific purposes of the specific system. Developing a monitoring system is not dissimilar to the design of a complex survey consisting of multiple components. The components depend on the objectives of the system. The utility of a monitoring system is a function of the extent to which the data are used to make decisions, set policy, or implement changes and is evaluated in terms of the objectives of the system. Design of a Monitoring System1 Disability (for adults) is defined in the Social Security Act as inability to engage in substantial gainful activity because of a medically determinable physical or mental impairment lasting at least 12 months. Therefore, one of the challenges in thinking about the design of a disability monitoring system is to understand how shifts in the nature of work over the past 40 years and into the future affect the meaning of disability, and to make this operational in household surveys and administrative databases. As stated earlier, of interest in the disability monitoring system is not simply the measurement of prevalence and the socioeconomic and demographic conditions linked to disability, but also understanding changes in both the individual and the environmental factors that lead to changes in appli- 1   Much of the information in this section is drawn from the background paper “Survey Design Options for the Measurement of Work Disabilities,” commissioned from Nancy Mathiowetz for use by the committee. The committee appreciates her contribution. The full text of her paper can be found in Part II of this report.

OCR for page 88
The Dynamics of Disability: Measuring and Monitoring Disability for Social Security Programs cations. For instance, the system will need to estimate the prevalence of persons eligible for disability benefits as well as develop measures that predict application. Key to such a system will be sufficient data to understand macro- and micro-level factors that distinguish participating and nonparticipating eligible populations. The design of a disability monitoring system must consider the informational needs of the system and the impact of alternative design options on meeting analytic goals as well as the impact on various sources of survey error (e.g., whether the design should include the use of household surveys). Alternative design components include the following: Data source or sources: Among the various data sources that could be included, alone or in combination, in the design of a disability monitoring system are data obtained from household-based surveys, physical examination, and administrative records. Among the options with respect to household data are stand-alone surveys that permit rich and deep national data on the size of the disabled population (e.g., similar to the NSHA), survey modules administered as part of preexisting data collection efforts (e.g., a supplement to the Current Population Survey or the Survey of Income and Program Participation), or the incorporation of a limited number of questions on existing national surveys (e.g., the National Health Interview Survey or the Behavioral Risk Factor Surveillance System [BRFSS]). Each of these options has implications for the error properties of the resulting estimates, including coverage, sampling, nonresponse, and measurement error. The use of administrative record data potentially suffers from similar sources of error. Periodicity of measurement: Decision on periodicity requires answers to several questions, such as: If survey data are collected, how often should the data collection occur? What are the ramifications of more frequent or less frequent data collection on the utility of the data? How is periodicity affected if one decides to use repeated cross-sectional data collection or a longitudinal design? Mode of data collection: For survey data collection, a decision will need to be made as to the mode or modes of data collection—such as telephone or personal interviews, self-reports, or observation and examination. Little is known about the effect of mode of data collection on the measurement error properties of self-reports of disability and impairments. In addition, the choice of a single mode of data collection has potential implications for the coverage of the population and the potential for nonresponse bias. Self and proxy response status: Questions that need to be resolved include: Should only self-response be accepted for household

OCR for page 88
The Dynamics of Disability: Measuring and Monitoring Disability for Social Security Programs surveys related to disability? If so, what are the ramifications on nonresponse bias? If proxy responses are accepted, what impact does this design choice have on the measurement error properties of the reporting of disability? As is evident from the alternative design components discussed above, each choice impacts the error structure of the estimates of disability and the analytic capabilities that can be addressed with the resulting data. SSA will need to decide how much error both ways it will be willing to tolerate, taking into consideration costs, information needs, and other factors. Also evident is the lack of information with respect to the specific impacts of design choices on the reporting of impairments and disabilities. One could consider a number of various permutations of design choices outlined above in designing a work disability monitoring system. These options could be arrayed along lines of richness of the data, quality of the data, and costs. At one end of the spectrum is a monitoring system characterized as a continuous, longitudinal, multimode household-based data collection, which may be supplemented periodically with medical examinations (for those meeting a particular threshold based on the household data and a subset of those who are classified in the category adjacent to the threshold) and links to administrative records. Such a design would facilitate analysis of change over time in the size of the pool of eligible population and applicants, as well as understanding of the individual and environmental factors that influence application for benefits, and would simulate the impact of alternative decision processes, provided that the household survey, medical examination, and administrative records collected or contained the information necessary for such modeling. This comprehensive design would be the most costly. At the other end of the spectrum are data characterized by a small number of questions on disability included as part of repeated cross-sectional surveys. Such a design would allow analysts to monitor the size of the pool of eligible population, and possibly, if crosswalk analytic capabilities had been developed, the size of the pool of applicants. However, it does not facilitate understanding of how individual, environmental, and macro level changes impact the application process. This minimal design would be least costly. Design Options for Continuous Monitoring Continuous monitoring should be undertaken using one or more design options; each of which requires some statistical coordination. These may include

OCR for page 88
The Dynamics of Disability: Measuring and Monitoring Disability for Social Security Programs sponsoring annual surveys based on self-report data from a reduced set of disability measures; funding additional survey questions, suitable for estimation of the size of the population eligible for disability benefits, as part of, or supplement to, an ongoing household survey; longitudinal data collection; forming a partnership with other ongoing surveys; linking survey information with administrative databases; and ad hoc special studies. 1. Reporting from Reduced Sets of Measures in the Intervening Years The NSHA should yield a complex and large set of measures that are used to identify alternative estimates of the number, distribution, and characteristics of the working age population in the United States potentially eligible for benefits under the Social Security disability programs. It is likely that the set of NSHA variables used to compute the “best” estimate of the pool of the eligible population would be too large to be feasible in ongoing monitoring because of the time needed for, and the high cost of, mounting a survey with such measures frequently. How large a set is needed to attain stable estimates? Sensitivity and specificity criteria often favor different subsets of indicators. In any case, the practical problem for SSA is the issue of how large a data collection budget can be allocated to ongoing measurement of these indicators. One key principle of an ongoing monitoring system for disability is the cost efficiency of measuring a small number of attributes continuously. These could probably be self-report measures that require only a few minutes of interview time for the respondent. Thus, the ongoing measurement will be less expensive to support than the large, comprehensive periodic disability surveys. What indicators should be measured continuously, and what should be measured less frequently? The set of measures in the periodic surveys defines the population of items from which the smaller set of continuous measures would be identified. Statistical analysis of the “best” sets of variables can be conducted (using item response theory notions or more traditional predictive analysis) with the goal of identifying a smaller set of measures that might be used more routinely to estimate the size of the eligible pool. Conceptually the problem of identifying the best subset of indicators devolves to measuring what portion of true eligibles is identified as eligible by the reduced set of measures (sensitivity) and what portion of actual ineligibles is identified as ineligible by the reduced set (specificity). The success of the ongoing monitoring measures depends on the success of the large periodic surveys.

OCR for page 88
The Dynamics of Disability: Measuring and Monitoring Disability for Social Security Programs After a reduced set of measures from the large periodic survey of a size manageable by SSA is identified for ongoing monitoring, two additional methodological steps are required. First, simulation of sample design requirements must be conducted for minimal levels of sampling variance of ongoing estimates of the size of the pool. Such simulations should provide the effective sample sizes required for all subpopulations of policy relevance, in order to inform the policy and budget functions of SSA. Some of the issues for consideration in the sample design simulations will be the needed frequency of national estimates and the desired sensitivity of the monitoring efforts to changes over time in the size of the pool of the eligible population. For example: Is it necessary to know the size of the eligible pool at any one point in time within a 1 percent tolerance, a 10 percent tolerance? If there is a 5 percent change in the size of the pool across adjacent years, must SSA be able to detect this for program management purposes? Are separate estimates required of the size of the pool for different age groups, regions of the country, occupational subgroups, or gender? The second methodological step required after the reduced set of measures is identified, is a test of the measurement performance of the reduced set. It is common to find that 10 items extracted from a set of 100 perform differently by themselves in survey measurement than they did in the context of the 100 items. That is, the size of the “pool” estimated by NSHA using the reduced set of items might be somewhat different from that obtained when those items are introduced into another survey context. To determine the sensitivity and specificity of the reduced set of items, SSA will need to test them on a set of respondents whose eligibility is or can be known. Such studies are expensive and are often restricted to small samples, with high internal validity at the cost of low external validity. Once the reduced set of items proves its worth on its own, the set of items is ready for production use. To summarize, SSA will need to (1) identify, through analysis of NSHA data, the set of NSHA variables needed to provide the best prediction of eligibility for NSHA respondents; (2) estimate the size and design of samples that would achieve desired levels of precision for the estimates of the pool; (3) test the reduced set of measures in a design permitting estimation of sensitivity and specificity; and (4) determine the SSA budget that can be allocated for ongoing monitoring of the size of the eligible pool. Once these tasks have been completed, SSA can then examine alternative ways to mount an ongoing monitoring of the size, distribution, and characteristics of the population eligible for benefits. There are two common ways in which program agencies monitor the size of the potential pool of program participants: (1) the agency sponsors ongoing surveys to estimate the pool; and (2) the agency enters into a

OCR for page 88
The Dynamics of Disability: Measuring and Monitoring Disability for Social Security Programs Survey Sponsor/ Contractor Topic Sample Design National Health Interview Survey NCHS/Census Health care utilization, conditions, health behavior; adult- and child- specific questionnaires Repeated cross section of addresses National Health and Nutrition Examination Survey NCHS/Westat Health status, including medical examinations Annual cross-sectional samples National Household Survey of Drug Abuse SAMHSA/RTI Drug and alcohol use State-level cross-sectional samples Survey of Income and Program Participation Census Program participation and eligibility, income; topical modules by wave of interviewing Panel of households NOTE: AHRQ = Agency for Healthcare Research and Quality, BJS = Bureau of Justice Statistics, BLS = Bureau of Labor Statistics, CDC = Centers for Disease Control and Prevention, NCHS = National Center for Health Statistics, RTI = Research Triangle Institute, SAMHSA = Substance Abuse and Mental Health Services Administration. SOURCE: Adapted from paper commissioned from Nancy Mathiowetz, see Part II. nation Survey (NHANES) and the Medical Expenditure Panel Survey (MEPS) are relatively small as compared to some other surveys (n = 5,000 and n = 15,000 persons annually, respectively); however, each of their designs benefits from a complementary component. In the case of the NHANES, the design includes a medical examination. In the case of the MEPS, the design includes data from medical care providers and providers of health insurance. Similarly, the National Household Survey of Drug Abuse (NHSDA)—a large survey (n = 67,000) producing state estimates—does not presently include any measures of functional limitation or disability; however, the design includes both an interviewer-administered questionnaire and a self-administered set of questions that may be beneficial in the assessment of disability. The National Health Interview Survey (NHIS) is a large continuing survey of approximately 43,000 households including 106,000 persons. The NHIS sample is drawn from each state but is too small to provide state-level data with acceptable precision.

OCR for page 88
The Dynamics of Disability: Measuring and Monitoring Disability for Social Security Programs Sample Size Frequency Mode 40,000 households annually; 98,000 persons Weekly replicate samples; quarterly and annual estimates Face to face 5,000 persons Annual Face to face; physical examinations 67,000 persons Annual Face to face; self-administered 36,000 households Quarterly; annual estimates Face to face; telephone The relevant questions to be addressed by SSA in choosing a partner survey(s) include the following: How large a sample is interviewed each year? What standard errors are likely to be obtained for key disability prevalence statistics? Will the addition of disability measures in the interview be consistent with the measurement goals of the original survey? Are there possibilities of context effects that could damage the accuracy of prevalence estimates? Are there existing measures in the survey that might be used as explanatory variables for disability status indicators? Can the survey offer SSA other informational benefits beyond being a vehicle to produce disability prevalence statistics? Is the survey of high quality? What evidence is there about coverage, nonresponse, and measurement error properties of key statistics?

OCR for page 88
The Dynamics of Disability: Measuring and Monitoring Disability for Social Security Programs How frequently can estimates be updated? Will monthly prevalence estimates be generated, annual estimates, et cetera? Is the mode of administration of the survey compatible with the measures chosen from NSHA? What restrictions, if any, will SSA have on access to the micro-data from the surveys? Can SSA analysts and independent researchers use the data for other analyses of importance to SSA, or will they be given only statistics produced from the survey data? If state variations in disability applications, approvals, and denials are important factors to SSA, should existing surveys, such as the Current Population Survey (CPS), and NHSDA that produce state estimates be given preference as SSA partners? Will the mission of the sponsoring agency be aided by a partnership with SSA in measuring disability status? With the obligation of many federal household surveys to provide indicators of disability, can SSA expertise in work disability be viewed as a desirable complement to the skills of the survey sponsor’s staff skills? What are the cost trade-offs for SSA in choosing a partner survey? SSA must weigh the reduced costs of partnership with another federal agency to produce the information it needs against the increased costs of mounting large periodic and interim surveys for which it would have complete control. The ideal partner(s) survey would have a sufficiently large sample to provide SSA with prevalence estimates that were stable enough to protect policymakers from erroneous results. It would have very low coverage and nonresponse errors. It would be conducted frequently, giving SSA the ability to model seasonal effects in the size of the eligible pool and to estimate the impact of economic shocks. It would contain other measures that would be of utility to SSA in addressing other important management problems (e.g., Are all demographic subgroups changing the disability prevalence in the same way over time? What are the major health and demographic correlates of disability status?). The chief obstacle to the feasibility of this partnering option for ongoing monitoring is that most federal household surveys are already using long and complex instruments, filled with measures of great value to existing constituencies. Seeking to add measures to these instruments faces zero-sum conflicts with existing obligations of the sponsors. The single most important sign of optimism is that several of the surveys are facing mandates to begin measurement of disability status in order to learn how the disabled subpopulation differs from others on the key topics covered by the surveys.

OCR for page 88
The Dynamics of Disability: Measuring and Monitoring Disability for Social Security Programs 4. Longitudinal Data Collection A longitudinal design, either independently sponsored or in collaboration with one or more other federal agencies, offers analytic capabilities that are not possible with repeated cross-sectional designs. This is especially the case for those designs related to the decision to apply for benefits, including both individual factors that influence the decision and the impact of environmental and macro level changes (e.g., economic) on the decision to apply for benefits. Selected reinterviews from large intermittent national surveys could provide needed information to assess change in status in different age, occupation, and gender groups. Much more can be learned from studying changes in individuals and their environments than from one-time cross-sectional measurement research. Such a design has high response rates, more ease in locating, and often better response reliability. Particularly where expensive screening was required for the initial sample, it need not be repeated and further subselection at different rates is possible. For example, one might follow all those currently on disability rolls, half of those with disabilities but not covered, a small fraction of those not reporting disabilities but with some health problems, and a still smaller fraction of the remainder of the population. What this means is that a combination of periodic large national survey with screening, and efficiently designed follow-up mostly by telephone, could continue the research on the disability policy questions, and the effectiveness of the process for determining eligibility for disability benefits. The committee also suggests that SSA consider sample cohort rotation and integration with other federal surveys for the design of its disability monitoring system. Since samples with planned overlap over time perform more effectively in measuring change than independently drawn samples at each time point, some sort of cohort feature might be considered for the system. Several possibilities in decreasing order of statistical effectiveness are cohorts at the person, address, and cluster levels. A cohort in which the same persons are followed over time has the advantage of following those for whom disability is measured although the cost of follow-up can be extremely high to retain a high percentage over time. Drawing respondents from the same first-stage sample cluster (or primary sampling units) is the least costly of these options but also the least advantageous statistically since clusters often account for a relatively small part of the total variation in disability measures. A compromise to these two extremes is to return to the same sample of addresses for each round of a continuous sampling process. This approach is operationally effective since one returns to the same place each round (although address samples must be updated to accommodate new construction), but people

OCR for page 88
The Dynamics of Disability: Measuring and Monitoring Disability for Social Security Programs move, so only a portion of the sample is retained from one time to the next. Longitudinal designs require that additional decisions be made concerning the length of the panel (that is, the number of years individuals are followed) and the frequency of data collection. In addition to the questions outlined above describing a periodic rich data collection effort supplemented by monitoring of the population through an abbreviated set of measures, the development of an ongoing, continuous panel design would have to address: the size of the sample needed to achieve analytic capabilities for a single calendar year versus pooled estimation across contingent years; the life of a single panel, that is, the number of years individuals will be followed through time; the periodicity of the data collection; the acceptability of mixed modes for data collection and its effect on the measurement and nonresponse properties of the resulting estimates; the use of a panel design requiring consistent response from the same respondents versus a mix of self and proxy response over time; and the ramifications of the decision on the error properties of the estimators. Several panel designs among the federal data collection efforts are shown in Table 5-1. However, pure person and cohort samples also have the disadvantage of higher respondent burden since respondents will be asked to participate in several rounds of data gathering. To control the added burden of fully retained cohort samples, some type of rotating cohort sample might be used in the design of a disability monitoring system. For example, SSA might consider something comparable to the 4–8–4 rotation scheme that has been used by the CPS, in which a sample household is sampled for four consecutive months, not interviewed for eight months, then interviewed again for four consecutive months (Census and BLS, 2000). A very different design—a continuous overlapping panel design—is used in the Medical Expenditure Panel Survey. In that survey, members of a panel are interviewed five times over 24 months; a new panel begins at the start of each calendar year so that panels can be pooled to produce calendar-year estimates. 5. Linking with Administrative Files Survey data can be made richer by linking with appropriate administrative files maintained by SSA for both the SSDI and the SSI programs. Administrative data usually have no information on persons who have not applied for benefits and little information on socioeconomic variables. Household population surveys, on the other hand, provide information on persons who have not applied and on a wide range of socioeconomic variables but contain little or no information on the person’s interactions

OCR for page 88
The Dynamics of Disability: Measuring and Monitoring Disability for Social Security Programs with the administering agency (Hu et al., 1997). Some examples of the administrative files that can be used are The Master Beneficiary Record (MBR) is the main file administrating Social Security retirement and disability insurance payments. It contains the data to administer the SSDI benefit program. The MBR record is initiated once the initial decision is made to award benefits, and entitlement and payment data are stored in the file. The Supplemental Security Record (SSR) is the main file for administering the SSI program. It provides the data needed to generate federally administered SSI benefit checks. SSA establishes and updates the SSR through local field office and teleservice site transactions, usually establishing the record as soon as a person files for SSI. The file stores eligibility and payment information. The Master Earnings File (MEF) contains earnings records for calendar years since 1951 and contains approximately 400 million records. Since 1977, the MEF has been derived primarily from Internal Revenue Service (IRS) Form W-2 data. MEF data are used for computing SSDI benefits. In addition to the MEF, the SSR is a source of earnings data from SSI applications. The 831 Disability file is related to both SSDI and SSI. When a person applies for disability benefits from either program, a medical determination is required. Medical decisions are made by the Disability Determination Services, reported to SSA’s Office of Disability (OD), and recorded in the National Disability Determination Services System. The 831 Disability file is extracted regularly from this system for research purposes. The 831 Disability file may contain data from decisions made at several levels of adjudication that represent ever-higher levels of appeal. Most records in the 831 files pertain to only two levels: (1) the initial medical determination; and (2) the reconsideration decision (i.e., the first level of appeal for medical denials). Decisions made as a result of SSA’s Continuing Disability Review process are recorded on OD’s 832 and 833 Disability files. The unit of observation in the 831 Disability file is a disability decision; the main data elements capture the primary impairment code, the regulation basis code (used to measure the severity of the impairment), the date of decision, the level of decision, and the result of the decision. In addition to the master files, the Continuous Work History Sample (CWHS) file is a 1 percent continuous work history sample from 1937—when payroll tax was first levied—until the present. The intent of the sample is to measure the working trends and employment of the population in relation to the Old-Age, Survivors, and

OCR for page 88
The Dynamics of Disability: Measuring and Monitoring Disability for Social Security Programs Disability Insurance (OASDI) program. The data in the CWHS are drawn from other master files—the Numident, MBR, and MEF—and from the IRS self-employed file. The CWHS also contains derived and constructed data elements that do not appear on any master file. Given the stringent restrictions on direct MEF access, the CWHS has become a substitute for the MEF in many instances. As of June 2000, access to the CWHS is very limited, but efforts are under way to make the file more widely available. 6. Ad Hoc Special Studies In its plans for a monitoring system SSA should include ad hoc studies on specific emerging policy issues as well as explore other questions that do not need continuing data. One example is a follow-up study of applicants for disability benefits to see whether some years later they are working (disabled or not), or not working (even if they were denied benefits). The ratio of denied who remain not working to the accepted who could be, or are, working, is some indication of the accuracy of the decisions. The total number of errors both ways could be some indication of the efficiency of the system. For example, Bound (1989) using data drawn from the 1972 and 1978 surveys of the disabled done by SSA, found that fewer than 50 percent of rejected male applicants work. Another example is a study of employers as well as people with disabilities to develop information on employer tolerance in hiring workers with disabilities and on the willingness of employers to display the flexibility often required to deal with workers with disabilities. SSA should explore the experience of other agencies in conducting such surveys. The National Center for Health Statistics had conducted a survey of employers and the Medical Expenditures Panel Survey also surveys employers. DEVELOPING AND PLANNING A WORK DISABILITY MONITORING SYSTEM The workshops held by the committee and input from experts in the field led to a clear conclusion that key concepts in disability were subject to debate among scientists, policymakers, and disability interest groups. Comparisons of U.S. social legislation on disability and that of other nations, arguments about the role of the social and physical environment of a person in defining disability, and the impact of macroeconomic forces on self-identification as work disabled, all led the committee to concerns that the concepts and measurements of disability over time could (and perhaps should) undergo change.

OCR for page 88
The Dynamics of Disability: Measuring and Monitoring Disability for Social Security Programs The prospect that key constructs will force new measurement challenges over time is a common problem in social measurement. The chief model to address this problem in other fields is to form partnerships with the scientific field allied to the constructs being measured. An ongoing program of measurement development is needed, allied with the conceptual developments in the field. Such research, when conducted outside an agency but guided by the mission of the agency, can supply the agency with proven measurement approaches when the new concepts become integrated into statutes guiding program designs. For example, small-scale studies examining how environmental impacts on disability self-reports manifest themselves can be valuable to the development of structured survey questions. The Disability Research Institute is one possible locus for such research. Because notions of disability and models of influences on disability are constantly changing, any ongoing monitoring system to monitor the phenomena must adapt and change over time. This can be accomplished only with ongoing investment in new methods of measurement. Planning for a Monitoring System In order to develop a monitoring system in collaboration with other relevant agencies, the following elements are necessary: SSA should set aside a multiyear planning period to systematically design and establish the proposed disability monitoring system; the system should be designed with sufficient flexibility to accommodate the evolving medical, legal, social, and policy perspectives of disability; the system should use as much as possible the design and data from existing federal surveys that measure disability in the population (e.g., NHIS, NHANES, the American Community Survey), by further cultivating partnerships with the agencies that conduct these surveys; and SSA should ensure the availability of sufficient qualified research staff to design and oversee the proposed disability monitoring system. While data gathering and analysis of the NSHA are under way, the committee encourages SSA to begin planning a national disability monitoring system to serve as its main information source for program planning and assessment. The general goal of the monitoring system would be to continuously monitor the number of Americans who are eligible to receive SSA disability benefits (i.e., the size and characteristics of the popu-

OCR for page 88
The Dynamics of Disability: Measuring and Monitoring Disability for Social Security Programs lation eligible for disability benefits) as well as public access to, utilization of, and satisfaction with the nation’s disability benefits programs. Although resources may dictate many of the system’s design features, the committee urges SSA to make its system one in which data are collected on a continuing basis from a valid and statistically efficient national sample of households. SSA should consider using data from existing federal surveys in designing the monitoring system and should use these information sources to supplement data generated from the SSA system. Besides serving as a supplementary source of disability data, integration with the design features of these surveys might prove beneficial both to SSA’s monitoring system and to the other surveys. This level of design integration has already been successfully accomplished between the Medical Expenditure Panel Survey conducted by the Agency for Healthcare Research and Quality and the National Health Interview Survey conducted by the National Center for Health Statistics (Cohen, 1999). To design and establish a high-quality SSA disability monitoring system, the committee suggests that a planning period of approximately three years would be needed. Three important tasks must be completed during this planning phase. First, SSA must recruit a sufficient cadre of qualified staff to conduct the design work and directly oversee the initial field-testing of the system largely from within SSA. The implication is that SSA should develop this system largely from within and not rely heavily on external contractors to do the work. Among those professional expertise areas that SSA would need to recruit are experts in: functional ability, cognitive measurement, survey design, and analysis. The next planning phase would be for SSA to develop a detailed blueprint for the system. This step suggests that pilot testing an early version first in a few states or metropolitan areas before national implementation might best phase in the system. The goal here would be to produce a system design that is useful to SSA, scientifically sound, and able to withstand careful scrutiny, yet sufficiently flexible to adapt to changing information needs over time. Toward these ends, the committee suggests that SSA support a careful study of the cognitive and process effects of measuring disability in a survey context. At a minimum this research should answer the following questions: What are the effects of the mode of gathering the data (e.g., self-administered, telephone interview, face-to-face interview)? How is the portrayal of disability influenced by who provides the data (e.g., the disabled person, a proxy caregiver, a health care provider)? How do question wording, context, and format affect the picture of disability painted for a disabled person? Ultimately, the role of this research on measurement effects is to understand the effect of the survey design strategies used to measure disability

OCR for page 88
The Dynamics of Disability: Measuring and Monitoring Disability for Social Security Programs in NSHA and other major surveys and thus enable SSA to interpret disability data from all current sources. Suggested steps in a multiyear planning and implementation schedule starting in 2003 shown earlier illustrate a strategy that is divided into three, sometimes overlapping phases involving strengthening the infrastructure and developing interagency collaboration, initiation of research on design, and a testing phase. These steps are shown below in Box 5-1. The committee recognizes that despite its many benefits, developing and implementing the recommended disability monitoring system raises several important issues that would require careful examination and resolution during a three-year planning period before final decisions can be made on the details of the design. Many of these issues relate to conceptual definition, method, timing, collaboration with partner agencies, and resource requirements. The committee has provided SSA with a conceptual framework with alternative choices for SSA to decide on; it has also recommended that further research be undertaken on unresolved methodological and logistical issues to reach informed decisions on implementing the details of the design. In the committee’s opinion, SSA would benefit from technical guidance provided by an external group of technical experts. BOX 5-1 Suggested Implementation Schedule Phase I: 2003–2004 Explore agency collaboration Obtain necessary funding for extramural research Ensure a cadre of qualified research staff Select expert technical advisory committee Phase II: 2004–2006 Develop operational definition(s) of work disability Conduct studies relating to design features Search for and test performance of “best” set of measures Investigate statistical benefits of overlapping designs Develop model-based respondent imputation strategy Decide on design options Phase III: 2006 Design and pilot-test interim monitoring component Analyze results and make needed adjustments

OCR for page 88
The Dynamics of Disability: Measuring and Monitoring Disability for Social Security Programs Recommendation 5-3: The committee recommends that the Social Security Administration establish a continuing, external technical committee of experts for the planning and implementation of the recommended disability monitoring system. One model for such a committee is the working group established under the auspices of the Section on Survey Research Methods of the American Statistical Association, which currently advises the Centers for Disease Control and Prevention (CDC) on the BRFSS and the Census on the Survey of Income and Program Participation. This arrangement has been an effective vehicle for the CDC and Census; it could serve as a model for SSA to consider in carrying out the above recommendations. In conclusion, the committee emphasizes that developing and implementing a work disability monitoring system as recommended in this chapter will contribute toward a significantly improved and efficient system of measuring and monitoring work disability and effective fiscal management of the programs.