–4–
Matching Design Features to Desired Goals

FROM OUR REVIEW OF THE GOALS of the National Crime Victimization Survey (NCVS; Chapter 2) and the challenges it faces (Chapter 3), we find seven fundamental goals to have particular salience and use them as the basis for evaluating various NCVS design options. As we elaborate below, we suggest this as a set of desirable goals; they are certainly not the only possible goals, and others may place different weights on particular goal statements. Four of these goals are historical in nature, in that they reiterate or reflect the various formal task statements recounted in Chapter 2. They are:

  • Production of a national measure of crime independent of official reports to the police;

  • Provision of information on the context, consequences, and etiology of victimization;

  • Ability to measure aspects of crime beyond the production of basic, overall rates; and

  • Utility for producing information on hard-to-measure crimes that are difficult or impossible to detect in police reports.

The remaining three goals for the NCVS that we consider to be particularly relevant come from current data uses and needs:



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 81
–4– Matching Design Features to Desired Goals F of the National Crime Victimiza- ROM OUR REVIEW OF THE GOALS tion Survey (NCVS; Chapter 2) and the challenges it faces (Chap- ter 3), we find seven fundamental goals to have particular salience and use them as the basis for evaluating various NCVS design options. As we elaborate below, we suggest this as a set of desirable goals; they are cer- tainly not the only possible goals, and others may place different weights on particular goal statements. Four of these goals are historical in nature, in that they reiterate or reflect the various formal task statements recounted in Chapter 2. They are: • Production of a national measure of crime independent of official re- ports to the police; • Provision of information on the context, consequences, and etiology of victimization; • Ability to measure aspects of crime beyond the production of basic, overall rates; and • Utility for producing information on hard-to-measure crimes that are difficult or impossible to detect in police reports. The remaining three goals for the NCVS that we consider to be particu- larly relevant come from current data uses and needs: 81

OCR for page 81
82 SURVEYING VICTIMS • Capability to readily provide information on emerging crime problems; • Capability to provide small-domain and subnational data of direct in- terest to states or localities; and • Timeliness of resultant data (by which we mean that policy-relevant data can be collected and tabulated sufficiently quickly as to assess emerging trends and inform policy responses). This chapter summarizes the relationships between the goals of the NCVS, the design of the survey, and the implications of various designs in terms of cost, error, and utility. We first consider alterations to the current NCVS design that could be put in place quickly for the purpose of saving money, while more substantial changes in design are assessed and imple- mented (Section 4–A). We then consider long-term changes in the form of the NCVS (4–B), outlining a set of survey design packages, some represent- ing relatively minor changes to the current design and others overhauling the basic approach to measuring victimization. Section 4–C describes the trade-offs in cost, error, and utility associated with various design features and presents our general assessments. 4–A SHORT-TERM FIXES: COST REDUCTION STRATEGIES AND THEIR IMPLICATIONS Any thoroughgoing redesign of the NCVS will require research and de- velopment work. At the same time, the survey is in dire straits with respect to funding. The Bureau of Justice Statistics (BJS) has managed this problem by making short-term changes in the survey that will allow it to continue while the panel does its work and while some research and development work is done. These changes are designed to reduce cost with the most min- imal disruption to the survey. However, most have been implemented with no empirical tests of their likely impact, a very risky survey management strategy. In this section, we review a number of cost reduction strategies and assess their implications for the cost, error, and utility of the survey. Some of these strategies have been introduced into the survey and others have not. Table 4-1 describes some short-term changes that might be carried out in order to reduce NCVS costs (some of which are already planned for im- plementation in 2007 as part of the most recent cost-saving efforts by BJS). One of these changes—reduction in sample size—has been the alternative of most frequent resort over the history of the NCVS. The table does not include a change in the reference period of the NCVS (i.e., from 6 to 12 months) even though—as we discuss in Section 4–C.1—we favor it as an alternative to continuing to reduce the NCVS sample size.

OCR for page 81
MATCHING DESIGN FEATURES TO DESIRED GOALS 83 Two small-group expert meetings convened by Lauritsen (2006b:9) sug- gested reductions in the length of the NCVS questionnaire as a potential source of cost savings. Several of the alternatives we discuss in the next section include attempts to streamline the content of the survey; however, we do not include reductions in questionnaire length in the table of short- term alterations. As the summary of those meetings indicates, cost-saving implications from questionnaire length are not immediately obvious: Efforts to shorten interviewer screening time are likely to produce very small savings because the field costs of conducting the screening inter- view are likely to be the same. Eliminating or vastly reducing incident report details for less serious crimes may result in greater savings, espe- cially if such experiences account for some of the field costs associated with future efforts to improve retention and participation. In general, condensing the content of the incident report portion of the ques- tionnaire was seen as a better alternative because “it would reduce respon- dent burden and perhaps minimize errors such as survey fatigue and future participation.” However, we again note that large portions of the NCVS costs arise not in interviewing but in contacting and gaining the cooperation of sample households. We think that it is critical to emphasize that even small changes to the design of a survey can have significant impacts on resulting estimates and the errors associated with them. Design changes made (or forced) in the name of fiscal expediency, without grounding in testing and evaluation, are highly inadvisable. Recommendation 4.1: BJS should carefully study changes in the NCVS survey design before implementing them. We use “study” in wording this recommendation because the appropriate measures may vary based on the specific methodological changes being con- sidered. The comprehensive redesign of the NCVS included a sufficiently large bundle of changes that implementation was phased in over the course of several years; during that time, data were available using both the old and new survey instruments, for comparison and evaluation purposes. Not all possible changes would require such a lengthy and costly phase-in process, but some would; these include changes in the stratification of the sample of addresses or a shift in reference period from 6 to 12 months. A study of proposed changes may include smaller scale testing or the reanalysis and re- calculation using existing data; changes in this class might include decisions on how to count series victimizations in estimates (see Section 3–B.2). As noted in Table 4-1, one adjustment that BJS decided to implement in 2007 (for the production of 2006 NCVS estimates) was to include the first

OCR for page 81
84 Table 4-1 Short-Term Changes to the NCVS to Achieve Cost Reductions Changes to Current Design Cost Error Utility Using the bounding interview in NCVS This would reduce costs if the effective Some increase in standard errors due to Availability of unbounded interviews ad- estimates∗ 1/6 increase in sample was offset by a adjustment. vantageous for methodological research. sample cut. The methodological work on the adjustment would involve costs. Keeping units in sample for an eighth Effectively increases sample size by 1/6, Some increase of time in sample bias. No None. interview which would reduce costs if it was offset adjustments currently made for known by a cut in sample. time in sample bias. Including series incidents in the estimates Reductions in standard errors due to Some reduction in standard errors due to High-end users have always had access to higher victimization rates would permit the increase in the rate of victimization, series incidents. However, including them sample size reductions or other cost especially violent victimization and the in the general estimates could benefit other savings. Some methodological work inclusion of some suspect events that NCVS consumers, as the products would necessary to select maximum number of cannot be accurately allocated to reporting more accurately estimate the magnitude incidents to allow in a series. years or crime category. of such incidents as domestic assaults and assaults at school. Reduce or eliminate allocation to central- Unclear; use of centralized CATI was Pilot work on CATI showed large effect on The mode of interview would be some- ized CATI facility∗ intended as major cost saver, but BJS and reporting, but early implementation sug- what more uniform across the sample. Census Bureau experience suggests no gested that centralized CATI use yielded There might be some reduction in inter- gain over current mix of in-person and higher reported incidence rates (Hubble, view refusals due to caller ID screening. informal CATI interviews. 1999). This seems likely to be due to the automation of the instrument, generally; estimates may not vary much between interviewer-initiated calls using a CAPI instrument and calls from a centralized CATI center. SURVEYING VICTIMS

OCR for page 81
Sample reductions Cost savings in proportion to the size of Increases in standard errors of estimates Decreasing precision of the estimates for the reduction. commensurate to size of the reduction. violent crime; further reductions would seriously limit usefulness of estimates. Biannual estimates Current BJS reports already combine two Aggregating two years of data would Threatens the relevance of survey as a years of data to assess short-term change; increase the precision of the estimates source of annual information on crime. this would formalize this approach by somewhat. switching to a 2-year release cycle. This would reduce costs if the increases in precision were offset with sample cuts or other cost reductions. Subsampling prevalent incidents for re- This could reduce time in household but ceiving incident form since much of the cost is in contacting households this will not save much. Use police statistics to improve stratifica- To the extent that stratification reduced If stratification was used without sample This revision stops short of a wide-scale MATCHING DESIGN FEATURES TO DESIRED GOALS tion in the sample standard errors, commensurate sample cuts, then there would be gains in preci- revision of survey content but is not cuts could reduce costs. sion with changes in sample stratification. really a short-term effort. In addition to planning, it would take several years to phase in and implement. * Planned for implementation in 2007. NOTE: CATI, computer-assisted telephone interviewing. CAPI, computer-assisted personal interviewing. 85

OCR for page 81
86 SURVEYING VICTIMS interview at a sample address in the estimates. The NCVS has always in- cluded some mix of bounded and unbounded interviews in its estimates due to movers—households that depart an address during that address’s 3.5-year stay in the survey sample and are replaced with new households; the first in- terview with the “new” household at an “old” address cannot be bounded by the previous interview. Catalano (2007:131–132) observes that the level of unbounded interviews included in the NCVS estimates “has fluctuated between 10.7 and 14.7 percentage points during the course of the survey”; Lauritsen (2005:Note 3) comments that the fraction “is roughly 6% and does not appear to have changed much over the past decade.” Bounding had been the focus of some studies prior to the 1992 NCVS redesign (see, e.g., Biderman and Cantor, 1984; Murphy, 1984; Woltman et al., 1984), but there was a paucity of research on combining bounded and unbounded data in postredesign NCVS estimates. Addington (2005) was able to assess the relative effects of bounding and residential mobility on reported victimiza- tions in what may be the only published, peer-reviewed article in the postre- design era. However, due to its reliance on public-use files, that analysis was limited to data from an administration of the School Crime Supplement for which incoming respondent (first interview) data were included. The deci- sion to include unbounded, first interviews in NCVS estimates was made as our panel was being established and assembled, and so we do not think it proper to second-guess it; we understand the fiscal constraints under which the decision was made. However, it serves as an example of a seemingly short-term fix with major ramifications, and it would have benefited from further study prior to implementation.1 4–B ALTERNATIVES TO THE CURRENT DESIGN IN THE LONG TERM Tables 4-2 and 4-3 present a set of design packages that can be compared with the current NCVS design. The columns of Table 4-2 describe a set of specific design features on which the packages vary. Each design feature poses some trade-off decision for BJS, offering benefits of some type, but also potential costs or risks of not fulfilling the BJS mission with regard to the measurement of crime. We discuss first the columns of Table 4-2, the specific design features, by pointing out the benefits and costs of each 1 On December 12, 2007, the first 2006 estimates from the NCVS were released. The inclusion of the first-time-in-sample interviews, combined with the effects of a new sample based on 2000 census information, led to “variation in the amount and rate of crime [that] was too extreme to be attributed to actual year-to-year changes”; BJS concluded that “there was a break in series between 2006 and previous years that prevented annual comparison of criminal victimization at the national level” (Rand and Catalano, 2007:1).

OCR for page 81
MATCHING DESIGN FEATURES TO DESIRED GOALS 87 feature; assessments of advantages and disadvantages of each design package follow in Table 4-3. We emphasize—and elaborate in greater detail in Section 4–B.2—that the set of design packages we discuss in this chapter is not, and is not in- tended to be, exhaustive of all design possibilities for the NCVS. Instead, they are meant to suggest a range of choices and broad visions for the sur- vey. It should also be noted that we do not have the capacity to provide an estimated price for all of these various options; while some can clearly be assumed to be less costly than the current NCVS design, others may not. Our focus is on the mapping of possible design choices to desired goals, and the weight one puts on the end cost of the design (relative to other choices) is important in selecting one; we think that it is our charge to sug- gest possibilities but not to impose any particular weighting of goals and design characteristics. 4–B.1 Characteristics of Possible Design Packages Table 4-2 compares each of these alternative design packages across nine characteristics, five related to the general survey design and four specific to the type of instrumentation used to implement the design. Single Survey Versus System of Surveys The wealth of information needed to help policy analysts and researchers alike is ever changing. One survey—even one that is omnibus in nature— may not be able to handle changing needs. A “system of surveys” is a set of independent but coordinated data collections; in the context of victim- ization, the individual surveys could vary by the type of crime studied, the setting of the crime, or the population victimized. A system of surveys may be better suited to focus on measuring a particular type of crime or a par- ticular social context; a system of surveys may also be amenable to a “quick survey” capability, being able to provide meaningful data in a compressed time period (a few weeks, rather than several months or years) in response to critical social events, such as a college or school shooting. A system- of-surveys approach could increase the richness of covariates and so may be more useful for the study of correlates of victimization. However, this greater flexibility must be reconciled with an increased burden of coordina- tion and heightened dependence on the continued existence (and funding) of, and access to, multiple other surveys. By comparison, the single-survey approach allows for focusing of effort and consistency of measurement.

OCR for page 81
88 SURVEYING VICTIMS General Structure of Sample By “general structure of sample,” we mean the combination of two con- cepts. One is the unit of analysis used in the survey: choices include house- holds, persons, addresses, or phone numbers. The second is the way the sample of those units is selected: for instance, an element survey makes draws directly from a frame or listing of eligible units, compared with clus- tered or multistage approaches. Choices for the sample design affect the efficiency of the data collection or the degree to which respondents can be found at units chosen for the sample. Due to extensive clustering, a multi- stage sample is less efficient than an element sample, as is the case in a pure random-digit dialing (RDD) survey. Oversampling (state, local area, or small population) improves the precision for those areas or populations in which sample is added but not the overall national picture. Under this heading, we also include potential alternatives to the NCVS current design that would serve to boost basic efficiency. One such approach would be adding crime-related questions as supplements to existing surveys, such as the American Community Survey (ACS), National Health Interview Survey (NHIS), or Current Population Survey (CPS), all of which are con- ducted by Census Bureau. Cross-Section or Panel Panel surveys are generally useful to assess the change over time and per- mit longitudinal analysis of change in behavior. The NCVS is a 7-wave panel of addresses, not individuals; this permits analysis of longitudinal behavior of nonmoving households.2 A rotating panel design also permits bound- ing interviews with nonmoving households, using information collected in a prior interview to make sure that events are not duplicated. Rotating panels are less costly than the repeated cross-sectional surveys and produce lower cost per interview. However, the advantage of rotating panels on response rates is likely to be a function of how frequently sample households are vis- ited; households may drop off in their cooperation with a survey the longer they remain in the sample. By comparison, repeated cross-sectional surveys may have a lower response rate but can be more flexible in terms of content and design modifications. 2 It is important to note that the NCVS panel of addresses is different from the panel of persons used in some other surveys, such as the Survey of Income and Program Participation and the National Longitudinal Survey of Youth. Although a panel-of-persons structure would be ideal for capturing within-household experiences of victimization (and reactions to them), a panel of addresses is a substantially less costly compromise approach.

OCR for page 81
MATCHING DESIGN FEATURES TO DESIRED GOALS 89 Reference Period The NCVS currently asks respondents to report victimizations occurring in the six months prior to the interview (“a 6 month reference period”). The current practice came about as part of the 1992 NCVS redesign and was intended as a simplification for respondents. Prior to the redesign, the survey questionnaire emphasized boundaries on the reference period, ask- ing respondents “to report for the 6-month period ending at the end of the previous month and beginning 6 months prior” (e.g., January–July for an interview conducted in August). The new, and current, questionnaire im- plicitly asks respondents to report victimizations up to and including the day of the interview by referencing only the past 6 months (Cantor and Lynch, 2000:112). By comparison, most other ongoing victimization surveys (e.g., the British Crime Survey and the International Crime Victimization Survey; see Appendix E) use a 12-month reference period. Between 1978 and 1980, the Census Bureau conducted a reference pe- riod research experiment using parts of the National Crime Survey sample; see Bushery (1981a,b); additional early work on reference period under the initial design is described by Singh (1982) and Woltman and Bushery (1984); see also the summary by Cantor and Lynch (2000). Mode Telephone interviewing reduces the proportion of time that interviewers are engaged in noninterviewing activities (i.e., travel). Thus, the mode tends to be cheaper per interview than face-to-face interviewing. However, the telephone model appears to be more sensitive to longer interviews, exhibit- ing higher break-off rates and partial interviews. Furthermore, the telephone prohibits use of visual aids in measurement, such as the flashcards currently used in the NCVS to clarify responses to some questions (e.g., race, His- panic origin, employment, education) and provide respondents with infor- mation about survey privacy (U.S. Census Bureau, 2003:A7-6–A7-9). For some types of populations (e.g., those in multiunit structures and walled subdivisions), the telephone might yield higher contact rates, but for oth- ers there appears to be no mode difference. The face-to-face mode, while costly, generally is viewed as yielding the highest cooperation rates. The self-administered mode (e.g, paper questionnaires, web administration) is gaining increasing attraction. Self-administration often has very low per- unit costs. Self-administration is found to yield more honest reporting on sensitive topics (e.g., domestic violence, rape).

OCR for page 81
90 SURVEYING VICTIMS Screener and Incident Form The current NCVS separates the functions of counting and listing vic- timization events from more detailed classification of those crime events through its separate screening questionnaire and incident report. This sep- aration of screening from classification is an effective way of promoting re- call and filtering out ineligible events; moreover, the additional information collected in the more detailed incident form permits a fuller description of crime events and their context. It is possible for surveys to take a different, more omnibus approach, integrating screening and classification in one pass. Surveys that use responses to screening questions to classify events may be less burdensome, but they may yield data of lower quality. (Further discus- sion of the nature of the screener portion of the questionnaire follows under “Event History.”) Core and Supplement Some surveys have a set of questions that are consistently asked of all re- spondents, sometimes labeled the “core.” The full survey questionnaire con- tains core questions and a rotating set of supplement questions. Scheduled supplements allow topical reports from the survey, enriching the breadth of reports. These supplements might change over time, to reflect the changing nature of crime. Subsampling Frequent Events In the current NCVS design, respondents are asked to complete an in- cident report for every victimization incident counted during the screening portion of the questionnaire. A possible approach to reduce overall inter- viewing time (and hence lower costs somewhat) is to subsample the most frequently occurring victimization types (e.g., theft, simple assault) and to collect the detailed incident report data for only some of these incidents. The subsampling rate could depend on the frequency with which a particu- lar crime is reported as well as complex structures involving multiple crimes. For example, a matrix sampling framework could be used to subsample inci- dent reports to preserve certain associations or joint occurrences. Probability sampling of such events permits estimation of standard errors of estimates. Subsampling could be implemented with slight modification of the current computer-assisted instrument used by the NCVS. Event History We include “event history” as a category as one relatively new technique in survey methodology that may be useful to consider in promoting accu-

OCR for page 81
MATCHING DESIGN FEATURES TO DESIRED GOALS 91 racy in the screening portion of the NCVS interview, in addition to contin- ued emphasis on completeness of recall. As described in Section 3–B.1, the problem of respondent recall—that survey respondents often have difficulty recalling the number or exact timing of events and that telescoping may occur as a result—is a long-standing one in survey research. Accordingly, a major objective of the 1992 NCVS redesign was to improve the screen- ing portion in order to promote completeness of reporting. As Cantor and Lynch (2005:296) observe, the structure and wording of the screener ques- tions made the new NCVS the first federal statistical survey to be designed based on psychological models of the survey response process, or what have come to be known as cognitive aspects of survey measurement (see, e.g., National Research Council, 1984; Sudman et al., 1996; Sirken et al., 1999; Tourangeau and McNeeley, 2003). Specifically, the new NCVS screener em- phasized the cognitive steps of comprehension and retrieval, adding a large number of “short cues” and structuring the interview into multiple frames of reference (Cantor and Lynch, 2005:297–298). Martin et al. (1986) re- view screening procedures considered in the redesign. (In addition to the revisions to the screening questions, the NCVS retained its high-level ap- proach to dealing with recall problems by withholding the first interview at an address and using it only to “bound” responses given in the next in- terview, until that practice was changed for the production of 2006 NCVS estimates.) In recent years, survey research has suggested methods for structur- ing and designing questionnaires that can improve the recall and dating of events. These event history methods typically involve the use of calendar- type structures in questionnaires, along with special cues, that permit re- spondents to make best use of thematic groupings and landmarks in the way memories are coded in the human mind. See, for example, Belli (1998) for an overview of event history methodology and Belli et al. (2001) for a direct comparison between event history and standard “question list” methods. The implementation of event history methods in the NCVS would require major restructuring in the existing screener portion of the survey. However, they could become particularly important if respondents were asked to recall victimization incidents over a longer time frame, as would be the case with a switch from a 6-month to a 12-month reference period. Although they tend to improve recall of events, event history methods generally require more interviewing time. They are also generally easier implement in face-to-face interviews—a setting in which a flashcard or a questionnaire section struc- tured in the form of a calendar grid—than in telephone interviewing, which is the mode most commonly used in NCVS interviews. However, event his- tory methods have been developed for use in telephone surveys; Belli et al. (2007) compare the implementation of an event history approach for tele- phone interviews in the Panel Survey of Income Dynamics with standard

OCR for page 81
106 SURVEYING VICTIMS The second basic conclusion is that there is nothing inherently wrong or lacking in the current NCVS design. As noted elsewhere, the NCVS has benefited from years of experience and methodological probing, particularly the intensive redesign effort that culminated in 1992. The NCVS design is a model that has been adopted by international victimization surveys as well as subnational surveys in the United States, and it is a good and useful ex- emplar. The principal fault of the current NCVS design is not a design flaw or methodological deficiency, or even that the design inherently costs too much to sustain, but rather—simply—that it costs more than has been ten- able under current federal budgetary priorities. We argue in Chapter 3 that collection of victimization data is substantially undervalued in the United States. There are design packages suggested in our tables that would ultimately be cheaper than the current NCVS, but they can involve considerable com- promises in the quality and detail of knowledge that the present NCVS has been able to provide about crime. For instance, the partnership model (7) that would dissolve the current NCVS and allocate topics to other survey vehicles would be likely to be less expensive in the long run than the cur- rent NCVS (depending on the choice of topics for smaller, focused surveys). However, this model would involve “piggybacking” some questions (for gen- eration of basic victimization rates) onto some omnibus survey vehicle like the American Community Survey or Current Population Survey. This piggy- backing raises a number of challenges: • By design, the NCVS screening interview is long, in order to more completely elicit victimization incidents from respondents. For consis- tency with other ACS/CPS questions—and to keep respondent burden under control—any victimization questions added to another survey would almost certainly have to be radically simplified, with a corre- sponding lack of accuracy. • NCVS screener questions are asked of all persons in the household over age 12, in turn, whereas other surveys like CPS have only one respondent per household. That single respondent may then provide information about other household members or about characteristics of the household as a whole. Separate interviews with multiple house- hold members would be a major departure from usual procedure for the ACS and the CPS, while reliance on proxy information from a sin- gle household respondent will lead to underreporting of victimization incidents. Likewise, the crime poll (10) option is arguably the least expensive of the options in the table, but the sacrifices in information would be profound. Although important information about attitudes toward crime and public

OCR for page 81
MATCHING DESIGN FEATURES TO DESIRED GOALS 107 safety would flow readily from this type of model, there would be no capac- ity to obtain details about the characteristics of specific incidents. For continuity of measurement, the best-case situation would be the pro- vision of budgetary resources sufficient to keep the current NCVS in opera- tion at a stable size. For long-term viability of the NCVS, the ideal would be an increase in budgetary resources—and sample size—so that at least some subnational (e.g., large metropolitan statistical areas) can be made available on some regular basis. Barring those very optimistic outcomes, we also rec- ognize that—in terms of its sample size—the current NCVS is at a critical point: its sample size has slipped sufficiently that terminating the survey would effectively be preferable to sustaining additional across-the-board cuts in sample size. 4–C.1 Length of Reference Period In light of these arguments, we suggest switching to a 12-month refer- ence period (thus achieving savings by reducing the number of contacts with sample households) in the event that resources to continue the NCVS using the current 6-month reference period are not available. Recommendation 4.2: Changing from a 6-month reference pe- riod to a 12-month reference period has the potential for im- proving the precision per-unit cost in the NCVS framework, but the extent of loss of measurement quality is not clear from exist- ing research based on the post-1992-redesign NCVS instrument. BJS should sponsor additional research—involving both exper- imentation as well as analysis of the timing of events in extant data—to inform this trade-off. If this recommendation is followed, a decision will have to be made about what number of waves (interviews conducted at a particular address) would best balance cost, measurement error, and nonresponse error. Relative to other countries’ victimization surveys, the NCVS 6-month reference period is something of a luxury, with a 12-month reference period being the norm. It bears repeating, though, that the switch would decrease the cost of the survey but would not be without consequences. The choice of a best-length reference period for the NCVS (e.g., 3, 6, or 12 months) has been a source of discussion and research since the survey’s inception; tests varying reference periods were recommended by National Research Council (1976b:68), and Cantor and Lynch (2000:110–112) sum- marize reference period experiments conducted following that recommen- dation. The effects of length of reference period have also been treated in general survey methodological research. The results of these studies, NCVS-

OCR for page 81
108 SURVEYING VICTIMS specific and otherwise, are generally quite consistent. Cognitively, an opera- tive phenomenon is telescoping: distant events tend to be moved forward in time relative to when they occurred, while recent events tend to be shifted to an earlier date. Traumatic distant events (like victimizations) may be more easily recalled and subject to more severe forward telescoping (Bradburn et al., 1994). Accordingly—until adjustments in 2007 led to first-time-in- sample interviews being included in estimation—the first interview with an NCVS household has historically been withheld and used as a bound—a check that events are not projected forward into the 6-month reporting time window. A switch to a 12-month window means that forward telescoping might be less problematic but that complementary problems would become prominent: backward telescoping victimization incidents so that they are outside of the 12-month recall period or, for some less traumatic incident types, simply forgetting that they occurred within the past year. The extent of underreporting may differ by crime; studies like those summarized by Cantor and Lynch (2000) suggest about a 30 percent general reduction in reporting by doubling the length of the reference period to 12 months. Consequently, it is important that a move toward a 12-month reference period be paired with research on developing event history techniques and other methods for improving accurate recall over a lengthy time window. This could involve major redesign so that events are placed on calendars or that the list of recalled victimization incidents is anchored to important dates and events (such as birthdays or anniversaries). At the same time that the screener-type questions are revised to promote more accurate recall, attention would also have to be paid to protocols for the detailed incident form. Even allowing for the possibility that some less recent (and less traumatic) incidents may be forgotten, the net number of incidents experienced by a sample interviewee can be expected to be larger for a 12-month window than a 6-month window. Hence, respondent bur- den would correspondingly increase if each and every incident was subject to the detailed incident form questioning; break-offs in the interview and general nonresponse could be expected to increase as a result. Accordingly, protocols for sampling which events elicited in an interview are subject to the full incident form questionnaire would have to be tested and evaluated. The change in reference period will also have an impact on the vari- ances of some statistics that involve multiple incidences because of increased intraperson correlation. Due to these logistical and technical complications, our recommendation concerning a change in reference period is deliberately nuanced. It is not a change that should be rushed into, in the name of fiscal savings, but rather one that needs grounding in pilot work and testing. Changing to a 12-month reference period while maintaining continuous interviewing in the field (as the Census Bureau currently does) throughout

OCR for page 81
MATCHING DESIGN FEATURES TO DESIRED GOALS 109 the year would also warrant reconsideration of the production cycle for an- nual NCVS estimates. Table C-2 in Appendix C illustrates the current time coverage of events using the NCVS 6-month reference period, illustrating the distinction between collection year and data year estimates. Seeking “to publish more timely estimates from the survey,” BJS switched to collection- year estimates in 1996 (the actual data release cycles have since been syn- chronized so that both NCVS and Uniform Crime Reports estimates of crime are made public each September). The change to collection year estimates rather than data year estimates was discussed in Bureau of Justice Statistics (2000:163), acknowledging that these “estimates for any given year will in- clude some incidents that actually took place in the previous calendar year and will exclude some incidents that would have been reported in interviews conducted in the following calendar year.” In support of that decision, col- lection year and data year estimates for 1995 were compared, disaggregating by type of crime, and no statistically significant differences were detected (Bureau of Justice Statistics, 2000:Appendix Table 1). Extending Table C-2 to reflect a 12-month reference period, a straight collection year estimate for year t would more thoroughly overlap events that actually occurred in calendar t − 1. Labeling these as “year t estimates” would thus be inher- ently misleading, conceptually; presenting them as estimates “for the past 12 months of collection” (and avoiding explicit mention) could be confusing to the media, the public, and other users of the data. However, reverting to a data year estimate for calendar year t would require waiting until interviews were completed in December t + 1 (and then allowing time for processing), which would have serious implications for the timeliness of the data. The BCS-type model (3) described in our tables would combine a switch to a 12-month reference period with another major change: converting from a rotating panel, multiple-interview-per-household design to a cross- sectional single-interview-per-household structure. (The model also assumes that some of the cost savings would be redirected into the fielding of regular, systematic supplements, providing additional flexibility in measurement.) To be clear, we do not recommend a change to a cross-sectional design. How- ever, it is worth noting that a 1-year reference period and cross-sectional design could be combined with a third major change to interesting effect. Specifically, continuous interviewing throughout the year could be replaced with focused, intensive data collection in the first few months of year t + 1. Doing so, respondents would have a very natural common sense reference period to work with (calendar year t ), and the resulting estimates would likewise be easy to interpret as “year t ” estimates. Logistically, the drawback to this scenario is the strain on field operations that would result. Prior to its switch to annual operations, the BCS was collected on this basis and the workload required the use of multiple private survey groups to carry out the interviewing workload. Such a large once-a-year effort would be a major

OCR for page 81
110 SURVEYING VICTIMS change from current NCVS operations by the Census Bureau and would be unique among the Census Bureau’s demographic survey programs. As we discuss further in the next chapter, exactly how other survey research orga- nizations might be able to carry out the work as the NCVS data collection agent is an open question. It is also likely that the cost of moving away from continuous data collection would be prohibitive since the advantage of hav- ing a survey infrastructure in place would be lost (and such infrastructure would have to be stopped and rebuilt annually). In addition, this change in cycle would raise the problem of seasonality: the volume and type of crime varies by the month of the year. Hence, seasonality and respondent bias toward reporting more recent events could combine to distort the overall picture of crime. 4–C.2 Role of Supplements Table 4-3 gives generally high marks to models that emphasize the role of topic supplements. Core-supplement models would streamline the body of the NCVS questionnaire to a minimal “core” and structure the remainder of the survey around regular topic supplements. Likewise, although we argue against the BCS-type model in the previous section due to a reluctance to switch away from a rotating panel design, we do find much to admire in the BCS concentration on supplements as a regular and structural part of the broader survey. Recommendation 4.3: BJS should make supplements a regular feature of the NCVS. Procedures should be developed for solic- iting ideas for supplements from outside BJS and for evaluating these supplements for inclusion in the survey. The role of supplements in the NCVS—as is part of the current design of the British Crime Survey (see Appendix E)—is to enhance the flexibility of the survey content. It is meant to prevent the overall content of the survey from becoming stale; in some cases, a topical supplement could serve as a “methods panel” for testing new questions that might, in time, be added to the core NCVS content. We think that the implementation of such NCVS topic supplements as the Police-Public Contact Survey and the School Crime Survey have broadened the scope of the victimization survey and that further branching out into topic supplements will serve to firm up the constituencies for the NCVS and other BJS products. Although a focus on a systematic set of topic supplements would, on its own, fall short of correcting a long- standing shortfall of the NCVS—the lack of geographic detail in estimates— it could be the vehicle for much richer substantive detail.

OCR for page 81
MATCHING DESIGN FEATURES TO DESIRED GOALS 111 Recommendation 4.4: BJS should maintain the core set of screening questions in the NCVS but should consider streamlin- ing the incident form (either by eliminating items or by changing their periodicity). To be clear, we do not suggest trimming the screening questions on the NCVS questionnaire. As noted in Section 4–A, reducing the screener portion of the interview could slightly decrease interview length and yield very small cost savings, but the quality of resulting data would suffer. That said, we have not comprehensively reviewed the incident form to suggest items to cut, either. The point of this recommendation is that if reductions in interview length are deemed necessary (particularly in order to facilitate a more regular set of topical supplements), then some means of condensing incident form content should be considered. It follows from this recommendation that we think that BJS and the De- partment of Justice should dedicate staff to find ways to effectively market the sample for periodic supplements while insisting that supplement costs are fully covered by the sponsors. Conceptually, an advantage of a core-supplement design (or other design that includes a strong role for supplements) is that it allows the NCVS to play to its strengths as a survey. 4–C.3 Supporting Subnational Estimates Many federal government national surveys measure key social phenomena—transportation and travel patterns or health risks, for example—that have importance to the country as a whole as well as to local areas and small subpopulations. Crime is certainly a phenomenon that has policy relevance for local, state, and federal governments. Thus, it is not surprising that a long-term tension affecting support for the NCVS has been how useful it is for local interests versus national interests. NCVS’s role as a major national social indicator (and a point of com- parison with international victimization rates) notwithstanding, we think that the long-term viability of the NCVS will depend critically on its ability to provide small-domain, subnational information. “Small-domain” means subpopulations that may or may not be spatially proximate but offer impor- tant policy issues; these may include such groups as new immigrants, persons over 70 years of age living alone, or persons in areas with different growth rates (e.g., fast-growing counties, relative to stable or declining-population counties). “Subnational” refers to levels of geography smaller than the na- tion as a whole, such as regions, states, metropolitan statistical areas, and large cities and counties.

OCR for page 81
112 SURVEYING VICTIMS As Pepper and Petrie (2003:5) summarize: Public attention to crime and victimization often focuses on particular subgroups in which deviant behavior may be most troublesome. For ex- ample, hate crimes, crimes committed by youth, and crimes committed against vulnerable subpopulations including children, the elderly, and people with disabilities have all been the subject of recent investigations and legislation. The current NCVS is capable of generating estimates by these basic demo- graphic criteria. However, for very small subpopulations, sample sizes from each year of the NCVS may be too small to yield stable estimates. As crime mapping has become an increasingly useful tool in assessing trends and plan- ning police activities, and as geographic information system (GIS) software and Internet mapping tools have become more widespread, markets have developed for richer spatial data on social variables like crime and victim- ization. Recommendation 4.5: BJS should investigate the use of model- ing NCVS data to construct and disseminate subnational esti- mates of major crime and victimization rates. As described in Section 3–D.2, state and local agencies do still find value in having a national-level measure from the NCVS as a benchmark, in many cases because NCVS data are the only available source for some analyses. Hence, it is an overstatement to say that the NCVS would be relevant or important to state and local users only if subnational reporting was provided. It is also the case that—perhaps with some intuitive kinds of adjustments for differences between the national and local levels—some of the existing demographic splits available in the NCVS can be applied to good effect. Small states, for instance, may be able to make effective use of estimates derived only from the NCVS sample for rural areas. When the national or other larger area estimate is used as a proxy for a local estimate, there arises an additional component of error because the proxy is imperfect. A better alternative than using a proxy often can be found using meth- ods for small-area estimation (Pfeffermann, 2002; Rao, 2003). Using such methods, an estimate for each area of interest can be reported along with an estimate of its mean squared error (MSE). The MSE estimates reflect both sampling error and area-to-area variability but not biases from response er- rors or nonresponse errors. It is also possible to pool data over time (if short-run change analysis is not the focus) and thereby increase the sample size for local areas. Small-area estimation techniques have greatly advanced since the NCVS redesign efforts yielded the redesigned questionnaire in 1992. The NCVS

OCR for page 81
MATCHING DESIGN FEATURES TO DESIRED GOALS 113 would seem to be a good candidate for evaluating small-area models of vic- timization estimates both for spatial domains and for demographic domains. The attraction includes the possibility of borrowing estimation strength from the police-report data of the UCR and other demographic data (perhaps en- riched by the new American Community Survey data), as well as the rich covariates in NCVS, to model the victimization rates of small domains. Fur- thermore, the fact that the NCVS is a longitudinal study of addresses would allow the small-area models to take advantage of the covariances across time in repeated measurement areas. Since the redesign efforts, the United States has now become accustomed to small-area poverty estimates for program administration (National Research Council, 2000b,a, 1997, 1998, 1999); in terms of geography, these estimates extend to the county and school district levels and have been expanded to include estimates of health insurance pro- grams. The Bureau of Labor Statistics Local Area Unemployment Statistics program combines data from the monthly Current Population Survey and state unemployment insurance records, along with other sources, to gener- ate estimates for states and selected cities and counties (National Research Council, 2007:256). Folsom et al. (1999) describe methodology for produc- ing small-area estimates of drug use, and Raghunathan et al. (2007) discuss the utilization of two survey sources to estimate cancer risk factors at the county level. In addition to small-domain modeling using NCVS data, it may also be useful to explore ways to strengthen victimization surveys conducted by states and localities. The surveillance model (9) we describe in our table has drawbacks in its full form, as it would shift the burden for collecting victimization data on states (or groups of states) and make the “national” survey a concatenation of the state measures. Although we lean against its implementation in full, we think that the basic idea underlying the model is a sound one. Currently, BJS operates a program under which it develops victimization survey software and provides it to interested local agencies; however, those agencies must supply all the resources (funds and manpower) to conduct a survey. An approach to strengthen this program would be to make use of BJS’s organizational position within the U.S. Department of Justice. The bu- reau is housed in the Office of Justice Programs, the core mission of which is to provide assistance to state and local law enforcement agencies; it does so through the technical research of the National Institute of Justice and the grant programs of the Bureau of Justice Assistance (BJA), among others. We suggest that OJP consider ways of dedicating funds—like BJA grants, but separate from BJS appropriations—for helping states and localities bol- ster their crime information infrastructures through the establishment and regular conduct of state or regional victimization surveys.

OCR for page 81
114 SURVEYING VICTIMS Recommendation 4.6: BJS should develop, promote, and coor- dinate subnational victimization surveys through formula grants funded from state-local assistance resources. Such surveys would most likely involve cooperative arrangements with research organizations or local universities and make use of the existing BJS statistical analysis center infrastructure. 4–C.4 Efficiency in Sample Design The sample design of the NCVS uses data for stratification of primary and secondary units that are available from the decennial census. Stratifica- tion and the size of the sample are the two features that are tools to reduce the standard errors of NCVS estimates. When variables are available on the frame to preidentify groups that will vary on their victimization experience, the standard errors of the estimates might be reduced. One possible approach for using external information would be to stratify areas based on other crime-related data; UCR data might be considered for this purpose. Although the sample for the NCVS was originally designed so that each household had the same probability of being selected into the sample, such a design is not optimal for the uses of the NCVS data. Since many of the uses are at the local level, reallocating the sample to strengthen the estimates for the smaller areas will be advantageous even though the sampling variance for the largest areas (and national level) will increase, provided that those increases are not too severe. Optimization for multipurpose samples is dis- cussed in Cochran (1977:119–123) and Kish (1976), for example. Review- ing the optimality of the numbers of primary sampling units versus number of housing units within primary sampling unit may also be beneficial. Efficiency gains can also occur if information can be exploited to pre- dict which housing units tend to have higher victimization rates; in that case, one can then sample the blocks containing such housing units at higher rates. Properly done, such “oversampling” can improve precision, some- times markedly. Several approaches to predicting housing unit victimization rates may be considered: (i) Merge block-cluster covariates into the NCVS data file and use the covariates to develop a predictive model based on past NCVS data. Prior research using the area-identified NCVS has shown that tracts with the highest levels of socioeconomic disadvantage (e.g., percentage living below poverty) have respondents with the highest victimization rates (Lauritsen, 2001). (ii) Ask a question in the ACS about victimization and use the results to improve on the predictions in (i).

OCR for page 81
MATCHING DESIGN FEATURES TO DESIRED GOALS 115 (iii) Use the first wave results for housing units to modify retention proba- bilities for the second wave, so that housing units that report victimiza- tion in the first wave are more likely to be retained. Recommendation 4.7: BJS should investigate changing the sam- ple design to increase efficiency, thus allowing more precision for a given cost. Changes to investigate include: (i) changing the number or nature of the first-stage sampling units; (ii) changing the stratification of the primary sampling units; (iii) changing the stratification of housing units; (iv) selecting housing units with unequal probabilities, so that probabilities are higher where victimization rates are higher; and (v) alternative person-level sampling schemes (sampling or subsampling persons within housing units). 4–C.5 Other Improvements In early redesign efforts in the early 1990s, a consistent finding that CATI interviews yielded higher reports (Hubble, 1999) led to the belief that CATI interviewing could both save money and obtain higher quality data. Now that the dispersed interviewing of the NCVS uses computer-assisted per- sonal interviewing, the role of CATI in the overall cost-error structure of the NCVS is worth reconsidering. Both CATI and CAPI can use the same soft- ware for automatic routing and editing of responses, and the only distinction between the two approaches is connected with (a) different interviewers and (b) centralization of CATI that might affect impacts of supervision. There are time and administrative costs from shifting cases to and from CATI facilities to field interviewers in cases in which the mode initially as- signed is not desired by the respondent. These tend to reduce the response rates of the NCVS. It seems likely that the cost-quality attributes of the NCVS might be improved by shifting all telephone calling to interviewers’ homes. Recommendation 4.8: BJS should investigate the introduction of mixed mode data collection designs (including self-administered modes) into the NCVS. As with most federal household surveys, the person-level response rates (known as Type Z rates in the NCVS) are declining over time. Whether these declines have produced estimates with higher nonresponse bias is not clear from the data at hand. Increasing evidence from survey methodology

OCR for page 81
116 SURVEYING VICTIMS has shown that the relationship between nonresponse rates and nonresponse bias is more complicated than previously believed to be true (Curtin et al., 2000; Keeter et al., 2000; Groves, 2006). It is quite likely that response rates will continue to fall in the NCVS. If BJS attempts to keep response rates constant, it is likely that costs of the NCVS will increase. Thus, great importance should be given to determining whether low propensities to respond to the NCVS are related to different likelihoods of different types of victimizations. There are diverse methods of such nonresponse bias studies: studying the movement of victimization rates by increasing effort to measure the cases, follow-ups of samples of nonrespondents, examining changes in estimates from alternative postsur- vey adjustments, and so on. These methods and others have been noted and promoted in recent U.S. Office of Management and Budget (2006b,a) guidelines for federal surveys. Recommendation 4.9: The falling response rates of NCVS are likely to continue, with attendant increasing field costs to avoid their decline. BJS should sponsor nonresponse bias studies, fol- lowing current OMB guidelines, to guide trade-off decisions among costs, response rates, and nonresponse error.