Page 32
4
Measurement Issues
Congress has called on the Bureau of Justice Statistics to measure crime against those with disabilities using the National Crime Victimization Survey (NCVS). Drawing from the paper by Richard McCleary and Douglas Wiebe, this chapter examines some of the methodological challenges involved in adapting extant surveys, such as the NCVS, to people with disabilities. Specifically, we address potential biases introduced by the sample design and methodological difficulties associated with interview methods.
For example, it is important to remember that criminal victimization of the types counted in the NCVS is relatively rare, that is, in any given reference period, the vast majority of respondents do not report any victimization. Very large household samples are therefore required for reliable estimates, and detailed subgroup analyses are problematic. For a variety of reasons, some important subgroups are not covered at all, and smaller research studies of crimes against these subgroups often have problems of statistical power because of small sample sizes in most cases. Furthermore, as with most crime surveys, the NCVS involves self-reports from victims, a method that presents a range of methodological challenges, for example, with regard to standard definitions of events, and the asking of sensitive questions. Although the methodological issues involved in measuring victimization are well known, new ones will emerge when data from the sub-population of persons with disabilities are integrated. In their paper, McCleary and Wiebe examine the current state of knowledge, outline a set
Page 33
of emerging methodological issues, and discuss potential solutions to the implied validity problems. This chapter summarizes these themes.
Although all people with disabilities are of interest in this context, the discussions here are limited to the methodological issues associated with the integration of people with developmental disabilities, because each sub-population of people with disabilities—cognitive, sensory, and physical— poses unique problems that alone would warrant a book-length essay.
SAMPLE DESIGN ISSUES
The Sample Frame
The first step in designing a sample is to define the population and the sample frame. In the case of people with developmental disabilities, the difficulty is to construct a list of sampling units—a frame. For example, the sampling frame for the NCVS is all U.S. households from which a nationally representative sample of 50,000 U.S. households comprising nearly 100,000 people is drawn. From this frame large segments of the general population, such as men, women, the elderly, and some minority groups such as blacks and Hispanics, can be identified, and the frequency characteristics and consequences of their victimization experiences can be explored through interviews. In the case of people with developmental disabilities, however, there is no exhaustive list that could serve as a sample frame. Household lists (of addresses, phone numbers, etc.) are problematic because a large proportion of this population does not reside in household units but in institutions, intermediate care facilities, and the like, so household lists would miss them. While this difficulty might be overcome, locating the population that resides in households also would require that the household member who answers the telephone or door identify a resident with a developmental disability as such. Many individuals with developmental disabilities will not admit having a disability, and some individuals without developmental disabilities will say they have a disability, causing incorrect households to be identified (see section below on screening bias).
The problems of using existing nonhousehold lists of persons with developmental disabilities are illustrated by the Client Development Evaluation Reports (CDERs) in California. In that state, an annual evaluation of each person with a developmental disability results in a CDER report. With some difficulty, these reports can be linked over time for tracking or monitoring individuals, or they can be arrayed in a cross-section for mea-
Page 34
suring the prevalence of developmental disabilities. (Although CDERs have a wealth of clinical information and other data, they contain no information of victimization experience.) As a sample frame, CDERs are limited both because they do not include every person in the state with a developmental disability (enrollment in the state system is voluntary), and because funding exigencies may bias data in other unknown ways. When a group of researchers from UCI Mental Retardation Research Center investigated the adequacy of CDERs for program evaluation, they found that only 3,067 of the 8,502 patients listed in a centralized medical reimbursement database could be located in CDERs. Thus, this reporting system is not a reliable sample frame.
Screening Bias
In the case of developmental disabilities, screening is likely to yield a biased overestimate of the disability-specific victimization rate because of the rarity of a particular disability. As Hemenway (1997:1434) noted: “No matter how specific the test, if the population is at low risk for having the disease, results that are positive will most likely be false positive. With a huge number of actual negatives, virtually any screen or screening question will pick up a sizeable absolute number of false positives.”
Thus, McCleary and Wiebe concluded, if items were added to the NCVS to screen for developmental disabilities, a large majority of the self-identified population of persons with developmental disabilities would be false positives. Because these respondents would presumably have lower risks of victimization, the screening items would bias the relative risk of victimization toward zero.
INTERVIEW METHODS
Although the methodological problems associated with sampling can be nearly insurmountable for a survey, in many respects, they are minor compared with the methodological problems associated with interviewing. Specifically, within this population group, three serious problems, according to McCleary and Wiebe, are incidence of caretaker-perpetrated victimizations, the widespread use of proxy respondents, and a wide variety of language and communication disabilities among the victimized population that may require individualized interviewing techniques.
Page 35
Caregiver-Perpetrated Victimization
Conventional approaches to estimating the victimization risk of persons with developmental disabilities require the active cooperation of care-takers. A major problem of assessing victimization is that the victim may know the perpetrator and may be unwilling to report the event for fear of his or her reprisal. For example, O'Brien (1985) found in rape incidents that this fear may depend on the proximity of the relationship between the perpetrator and victim. The likelihood of a rape being reported is less if the perpetrator and victim are acquainted (51 percent) than if the perpetrator is a stranger (66 percent), and less still if the perpetrator and victim are related (44 percent).
A general unwillingness of people with disabilities to disclose victimization to anyone may be especially problematic because many of the crimes against them may be committed by known offenders on whom they depend for assistance. For instance, when Sobsey (1994) examined offender-victim relationships with data from the University of Alberta Abuse and Disability Project, he found that among 215 cases of abuse among adults with a developmental disability, 52 percent were victimized by someone who was associated through contact with disability services. Other research has also found that a large proportion of victimizations are committed by service providers and family members and by peers with disabilities. Turk and Brown (1992) found that 58 percent of offenses against adults with disabilities occurred in the home of either the victim (48 percent) or the perpetrator (10 percent). Because many surveys, including the NCVS, are conducted from the respondent's home, where perpetrators may be within earshot, victimization may go unreported.
Proxy Respondents
Proxy interviews are often used as a tool to help obtain interviews from individuals who would otherwise be excluded from a survey or from types of individuals who would be underrepresented in a sample. The methodological and ethical problems posed by interviewing people with developmental disabilities—using standard interview methods, at least—seem to argue for proxy interviews.
In the case of people with developmental disabilities, however, the quality of proxy data may be affected in a number of ways. For example, the accuracy of data gathered may vary depending on the relationship with the
Page 36
subject. A proxy who is a live-in relative of a respondent will presumably be better able to provide accurate information than a more distant acquaintance. Still, the ability to respond accurately does not guarantee that genuine responses will be provided. For instance, the accuracy of data is jeopardized when assessing the victimization of a person whose proxy interview is conducted with the person who is the perpetrator.
The NCVS allows respondents to be interviewed by proxy (usually another member of the household) if the respondent is a minor child, away for the entire interview period, or physically or mentally incapacitated; incapacitation can range from hearing impairment to severe mental retardation.
In 1997, approximately 4 percent of all NCVS personal victimization interviews were conducted by proxy. Of these, 30 percent were justified by physical or mental incapacity. Just 37 victimization incidents were reported by proxies, suggesting that mentally or physically incapacitated people faced significantly lower risks of victimization than the general population (see Table 4-1 ). If specific crimes are examined, however, mentally or physically incapacitated persons had a significantly higher risk of sexual assault—they were nearly twice as likely to report (by proxy) a sexual assault. The higher risk of sexual assault is consistent with the findings of Wilson and Brewer (see Table 4-2 ) and with the prevailing theory of victimization. The lower risks of assault and robbery may be an artifact of the proxy interview method or differences in the sample. In addition, people with disabilities
Crime |
Number of Victims |
Relative Risk |
Any |
37 |
0.13 |
Sexual assault |
4 |
1.95 |
Burglary |
2 |
0.76 |
Assault |
4 |
0.14 |
Robbery |
11 |
0.21 |
Auto theft |
0 |
NA |
Theft |
18 |
0.10 |
Other theft |
2 |
0.17 |
All other crimes |
10 |
0.13 |
Page 37
Intellectually Disabled |
|||
Crime |
Yes |
No |
Relative Risk |
Assault |
11.4 |
4.0 |
2.8 |
Sexual assault |
3.2 |
0.3 |
10.7 |
Robbery |
5.1 |
0.4 |
12.8 |
Total personal |
19.7 |
4.7 |
4.2 |
Auto theft |
0.6 |
0.7 |
0.9 |
Theft |
7.6 |
6.4 |
1.2 |
Burglary |
11.4 |
6.4 |
1.8 |
Household theft |
4.4 |
3.7 |
1.2 |
Total property |
24.0 |
17.2 |
1.4 |
may be at lower risk for such crimes as theft and robbery if their mobility is restricted and they spend limited time in public places.
Proxy interview data are also widely used in clinical practice and research on elderly patients with senile dementia. In these types of interviews, the elderly patient is too cognitively impaired to respond to standard neuropsychological test items, so the caregiver—usually an adult child or spouse—gives proxy responses. A 1995 study conducted by researchers from the UCI Alzheimer's Disease Center (McCleary et al., 1996b) brings the validity of proxy interviews into question.
In a subset of this study (McCleary et al., 1996a), 81 percent of care-takers for 221 elderly demented patients were adult children of the patients—who were roughly similar to caretakers of persons with developmental disabilities. Three events—injury, aggression, and loss of a functional activity—were examined. Researchers found that these events were grossly underreported by the caretaker proxy. The actual annualized incidence rate of injury, for example, was more than five times higher than the rate reported by the caretakers.
Communication Barriers
The validity of survey responses depends on the subject's ability to understand the question and whether the subject comprehends what is a criminal event. But some people who have been victimized, such as those
Page 38
who are young or developmentally disabled, may not even recognize that the event was inappropriate or illegal. In addition, because there are no published studies like that of Goodman, Toby, Batterman-Faunce, Orcutt, Sherry, Shapiro, and Sachsenmaier (1998) on children's recall of traumatic events, little is known about the ability of people with a developmental disability to recall stressful events accurately. On this issue, McCleary and Wiebe could only describe unpublished research conducted for the Arc of the United States to suggest that the NCVS may not be adequate for assessing self-reported victimization among respondents with developmental disabilities.
SURVEYING THE INSTITUTIONAL SUBPOPULATION
A large proportion of people with developmental disabilities resides in nursing homes, halfway houses, hospitals, and jails and prisons. As stated above, because NCVS interviews are conducted by households, this population is automatically excluded from the sample data.
With the exception of the Bureau of Justice Statistics' annual Survey of Inmates in Local Jails, few surveys are designed to adequately assess institutional risk, especially in cases in which a large proportion of the subpopulation of interest may reside in institutions instead of households.
The 1996 survey, which was based on in-person interviews with a national representative sample of 573,000 inmates in U.S. jails, identified inmates with disabilities as people who were having difficulty seeing regular newsprint (even with glasses), difficulty hearing normal conversation (even with a hearing aid), a mental or emotional condition, a learning disability, speech impediment, or physical or emotional condition that limited work. The survey found that, in general, inmates with a learning disability, difficulty seeing, a physical or mental condition that limits work, or a mental or emotional condition, were at higher risk for violence and that those with a mental or emotional condition were at significantly higher risk (Harlow, 1998). When these data were reanalyzed and demographic, criminological, and contextual confounds were controlled for, however, the risks for inmates who reported a mental or emotional condition were found to be insignificant. These new findings illustrate three types of problems with the initial survey: (1) screening biases that led to significant numbers of false positives in the disability categories (because these nondisabled inmates face lower risks, their experiences bias the relative risk toward zero); (2) conventional survey interviewing methods are not appropriate here; institutional
Page 39
pressures on victims create an incentive to misreport; and (3) vagueness of the disability item limits the utility of the data.
Better Survey Design
Based on their review of the problem and the literature, McCleary and Wiebe conclude that no existing survey can be adapted to provide estimates of disability-specific victimization. They noted that valid estimation of disability-specific victimization requires instead a sample survey designed specifically for that purpose. They provide specific guidance on such a survey's sample design, indicating that it should tap both the institutional and noninstitutional subpopulations, tap the proportion of the population that is moving between the institutional and noninstitutional subpopulations, and be sufficiently large to ensure nominal power for contrasts with the population of people without disabilities. Furthermore, McCleary and Wiebe note that the survey's interview methods must use items designed specifically for the subpopulation with developmental disabilities, use proxy responses only as a last resort, and use interview prompts in a manner guaranteed to produce responses of known validity.
Bureau of Justice Statistics Action Plan
Despite the difficulties noted above, the Crime Victims with Disabilities Awareness Act of 1998 requires the Bureau of Justice Statistics to collect data under the NCVS. In response, BJS has conducted its own review of the literature cited in this report and has undertaken a number of activities to address the law's mandate. These include meeting with national and state officials working in the area of disability and becoming active participants in two working groups dealing with disability statistics: a governmentwide group, the Federal Interagency Subcommittee on Disability Statistics (ISDS), which meets on a monthly basis to discuss the issues related to identifying people with disabilities and coordinate research in this area; and a research group convened by the Bureau of Labor Statistics to help design questions related to disability on the Current Population Survey (CPS). BJS has also established relationships with people in the disability research and advocacy communities in order to ensure that its efforts would be accepted as true reflections of the victimization experiences of people with disabilities.
In August 1999, BJS convened a group of experts from inside and
Page 40
outside the federal government to begin the coordination process, obtain information about other disability-related surveys, and brainstorm on the issues and possible strategies. In October 1999, BJS officials participated in the workshop that is summarized in this report. Based on all of these activities and a review of the attributes of the NCVS, BJS staff began to construct a strategy, the components of which are:
1. Adding questions to the existing NCVS to determine whether a person has a disability.
2. Developing modifications to question wordings, proxy respondent rules, interview procedures, and interviewer training to improve the information-gathering process in the context of the NCVS interview.
3. Exploring enhancement of the NCVS sampling frame to oversample people with disabilities.
4. Exploring enhancement of the NCVS sampling frame to include an institutional component.
Given the difficulties, detailed in this chapter, of using the NCVS to measure the victimization of people with disabilities, this strategy is being implemented in phases. The first two items constitute the first phase and are the focus of current efforts. The last two items have been discussed but are presently tabled as future activities until BJS can determine whether the first phase can be completed successfully.1
1 The material on current and future BJS plans is briefly summarized here from a paper entitled "Developing the Capability to Measure Crime Victimization of People With Disabilities,” delivered at the 2000 National Conference of the American Society of Criminology in San Francisco, November 17, 2000, by Michael Rand, Victimization Statistics, Bureau of Justice Statistics.