Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 5
Ensuring the Quality, Credibility, and Relevance of U.S. Justice Statistics Summary THE BUREAU OF JUSTICE STATISTICS (BJS) of the U.S. Department of Justice (DOJ) requested that the Committee on National Statistics of the National Academies (in cooperation with the Committee on Law and Justice) convene this Panel to Review the Programs of the Bureau of Justice Statistics. The panel has a broad charge to: examine the full range of programs of the Bureau of Justice Statistics (BJS) in order to assess and make recommendations for BJS’ priorities for data collection. The review will examine the ways in which BJS statistics are used by Congress, executive agencies, the courts, state and local agencies, and researchers in order to determine the impact of BJS programs and the means to enhance that impact. The review will assess the organization of BJS and its relationships with other data gathering entities in the Department of Justice, as well as with state and local governments, to determine ways to improve the relevance, quality, and cost-effectiveness of justice statistics. The review will consider priority uses for additional funding that may be obtained through budget initiatives or reallocation of resources within the agency. A focus of the panel’s work will be to consider alternative options for conducting the National Crime Victimization Survey, which is the largest BJS program. The goal of the panel’s work will be to assist BJS to refine its priorities and goals, as embodied in its strategic plan, both in the short and longer terms. The panel’s recommendations will address ways to improve the impact and cost-effectiveness of the agency’s statistics on crime and the criminal justice system. This is the panel’s second and final report. At BJS’s request, the panel devoted the first phase of its work to considering broad options for conducting the National Crime Victimization Survey (NCVS), arguably the agency’s flagship data collection and certainly its most expensive. We described such options in detail in our interim report, National Research Council (2008b),
OCR for page 6
Ensuring the Quality, Credibility, and Relevance of U.S. Justice Statistics and the summary of that report is included in this volume as Appendix B. In this final report, we complement our work on the NCVS by considering the balance of BJS’s data collection series, its work in providing assistance to state and local authorities, and the functions and priorities of the agency as a whole. Established in its present form in 1979, BJS is one of the smallest of the principal agencies in the U.S. federal statistical system, yet it shoulders one of the most expansive and detailed legal mandates among the statistical agencies. Formally, BJS is tasked to provide information on crime and the operation of the justice system, across all levels of government. Like its administrative parent, the Office of Justice Programs (OJP), BJS was originally created as part of the former Law Enforcement Assistance Administration (LEAA); accordingly, BJS also inherits the LEAA’s strong mandate to assist state and local authorities—both financially and technically—in developing justice information systems. ASSESSMENT OF THE BJS DATA COLLECTION PORTFOLIO In our review of BJS programs, we have used as a guide a model of the series of events in the justice system that was first developed by the President’s Commission on Law Enforcement and Administration of Justice (1967). Graphically, this model resembles a funnel, in which large numbers of crimes and victimization incidents are processed—in decreasing numbers—through major parts of the system, including police work, prosecution and adjudication, and correctional supervision. Through our review of existing BJS data collections and the manner in which they map to the funnel model of the justice system, we conclude that BJS’s data collection portfolio is a solid body of work, generally well justified by public information needs or legal requirements. It represents a better-than-good-faith effort to marshal data relevant to an astoundingly large mandate, given that fiscal resources have typically been less than commensurate. Within its resources and the topics it has chosen to address, BJS has done well in the sense that nothing in its portfolio is obviously frivolous, wasteful, or inconsistent with its legal mandates. Our basic observations about BJS’s portfolio include: BJS’s key victimization series, the NCVS, is its most expensive, most flexible, and most scrutinized collection. It is also, arguably, the agency’s most underutilized collection, undercut by scarce resources, diminishing sample size, and—to a degree—a lack of innovation in analyzing and promoting the data. BJS’s corrections data series is a good example of a well-designed and integrated system of collections, wisely using different methods (per-
OCR for page 7
Ensuring the Quality, Credibility, and Relevance of U.S. Justice Statistics sonal interviewing and facility or administrative records) at certain time intervals for a range of facilities (prisons, jails, support agencies). The set of collections is designed such that censuses build the frame for subsequent, more detailed surveys. Going forward, the challenge for BJS’s efforts in the general area of corrections is to expand its coverage to include prisoner reentry issues, broadly, and to improve on its previous solid (but infrequent) studies of recidivism. BJS’s work in law enforcement is hindered by an overly restrictive focus on management and administrative issues, with little direct connection to data on crime, much less providing the basis for assessing the effectiveness of police programs. It is also in the area of law enforcement—with the proliferation of numerous special-agency censuses and little semblance of a fixed schedule or interconnectedness of series—where the need for refining the conceptual framework for multiple data collections is most evident. Our critique of BJS’s work in adjudication is more a reflection on the general difficulty of measurement in the justice system than a criticism of BJS. Information systems in state court systems—and, indeed, the structure and jurisdiction of those courts—vary strongly in their accessibility and sophistication. The dominant impression is of the agency (with its data collection partners) doing the best it can with what it has. That said, there are numerous areas where improvement is needed: bolstering the adjudication series’ basis in statistical sampling and patching important gaps in statistical coverage of the justice system “funnel” (particularly declinations to prosecute and out-of-court settlements) would dramatically upgrade the relevance and utility of BJS’s data series. Through comparison to the funnel model, we identify four major substantive gaps in BJS’s portfolio: White-collar crime, including various types of fraud, public corruption, and Internet crimes; Civil justice (as opposed to criminal justice) matters—ranging from prosecution of nonviolent crimes to property disputes to divorce and custody arrangements—which are currently covered by only a single BJS collection (the Civil Justice Survey of State Courts) that, by design, is not able to collect information on out-of-court settlements; Juvenile justice, authority for data collection of which is largely ceded to BJS’s fellow OJP bureau, the Office of Juvenile Justice and Delinquency Prevention (OJJDP); and Contextual factors such as the interaction between drugs and crime
OCR for page 8
Ensuring the Quality, Credibility, and Relevance of U.S. Justice Statistics that cut across various parts of the system and are thus difficult to measure completely and consistently. These are important topics and ones on which the principal statistical agency of DOJ should be able to speak authoritatively. However, each of these areas involves major methodological complexities; to fill any of these gaps in BJS’s portfolio would require increased and sustained support from Congress and the administration in terms of staff and fiscal resources. In the interim, we recommend (Recommendation 2.1) that BJS play a stronger “clearinghouse” role related to these and other substantive gaps, documenting and organizing those statistics that are available and pursuing research on what new statistics could be feasibly and usefully developed. Our review of BJS programs leads us to suggest four broad strategic goals for the agency over the coming years. A STRONG POSITION OF INDEPENDENCE Strategic Goal 1: To establish and maintain a strong position of independence as a statistical agency; to serve as an independent and objective source of statistical information on crime and the administration of justice. BJS generally espouses the expected principles and practices of a federal statistical agency, but it has sustained major challenges to its independence as a national statistical resource in recent years. These include: An attempt by Justice Department officials to alter the content and substantive emphases of a statistical press release announcing new estimates on police-public contact during traffic stops, an incident that led to the dismissal of a BJS director; The legal imposition, through the Prison Rape Elimination Act of 2003 (PREA), of reporting requirements on sexual violence in correctional facilities that oblige BJS to play a regulatory role, holding individual facilities up for sanction; and Legislative changes that have the effect of tethering BJS to the policy objectives of its immediate administrative parent, OJP, and DOJ generally. BJS’s uniquely precarious position—as a statistical agency nested within a program agency (OJP) that is dedicated to furthering policy objectives, nested in turn within the nation’s principal law enforcement department—means that its independent function as a source of independent and objective information is under constant threat. For this reason, we identify the building of a position of independence as a statistical agency as BJS’s chief strategic goal. The panel concludes that this goal requires strong corrective
OCR for page 9
Ensuring the Quality, Credibility, and Relevance of U.S. Justice Statistics actions: changing BJS’s administrative placement within DOJ and the term of service of the BJS director. Organizational Changes There exists no organizational arrangement that can completely shield a statistical agency from threats to its independence, guaranteeing complete freedom from political or structural interference or the appearance thereof. That said, the measures we recommend are strong and should reduce the likelihood that BJS and its officials are inappropriately treated. Recommendation 5.3: BJS should be administratively moved out of the Office of Justice Programs, reporting to the attorney general or deputy attorney general. Recommendation 5.4: Congress and the administration should make the BJS director a fixed-term presidential appointee with the advice and consent of the Senate. To insulate the BJS director from political interference, the term of service should be no less than4years. By these recommendations and the suggested strategic goal statement, we also suggest that preserving a strong position of independence should be BJS’s first criterion for undertaking new data collections or revising existing ones. Because of the experience with PREA reporting requirements, we recommend (Recommendation 5.1) that BJS should not provide or be required to provide individually identified data in support of functions that compromise BJS’s role—functions that involve collecting and analyzing statistical data for policy-furthering, tactical, regulatory, and operational purposes. We further reinforce guidance issued by the U.S. Office of Management and Budget in the wake of the press release incident described above, recommending that any Justice Department review of BJS statistical products and related communications preserve the content, the release schedule, and the mode of dissemination planned by BJS (Recommendation 5.2). BJS should position itself as a statistical resource to DOJ, not an “arm” for the furtherance of any policy objectives. To this end, reciprocal outreach and cooperation are necessary: The BJS director needs to reach out to other agencies within DOJ, forming partnerships to propose information collection initiatives (Recommendation 5.5), and DOJ needs to build provisions for BJS collection of statistical information—the raw material for monitoring and evaluation—into its program initiatives aimed at crime reduction (Recommendation 5.6).
OCR for page 10
Ensuring the Quality, Credibility, and Relevance of U.S. Justice Statistics Ensuring the Quality of the NCVS Statistical agencies can safeguard their independence through the quality and visibility of their products. They are better suited to ward off threats to independence if their data are essential indicators of national well-being; this produces support by a vocal set of stakeholders who are committed to their objectivity and accuracy. In our assessment, the NCVS can and should be positioned as such a critical social indicator, with its unique ability to provide insight into the etiology of crime and the characteristics of crimes not reported to police. As such, the NCVS should have clear primacy in BJS’s resource allocations. It is sufficiently core to BJS’s legally mandated duties and its basic function as a statistical agency that it is difficult to imagine an effective BJS without a strong and continuing NCVS. Accordingly, we reiterate in this report two recommendations that we first offered in our interim report: Recommendation 6.1: BJS must ensure that the nation has quality annual estimates of levels and changes in criminal victimization. Recommendation 6.2: Congress and the administration should ensure that BJS has a budget that is adequate to field a survey that satisfies the goal in Recommendation 6.1. We further recommend (Recommendation 6.4) that any additional resources made available for the NCVS should be used not only to increase the reliability of annual estimates (i.e., rebuild the NCVS sample size) but also to supplement the survey in ways that increase understanding of criminal victimization and keep the content of the survey fresh. This includes seeking ongoing topic supplements such as the School Crime Supplement, with the regular support and cooperation of other federal agencies. It also includes fuller use of the NCVS to measure citizen involvement with other parts of the system (e.g., experiences with adjudication procedures and law enforcement). Given the utility of NCVS information on the needs of victims and the compensation and assistance goals of the Justice Department’s Office of Victims of Crime—we also suggest that Congress permit the use of funds obtained through the Victims of Crime Act to provide some funding for the collection and improvement of victimization data (Recommendation 6.3). Continuing and Strengthening State Justice Statistics Program and SAC Partnerships Through its State Justice Statistics program, BJS supports the operation of Statistical Analysis Centers (SACs) in states and several U.S. territories. The SAC network includes a mix of organizational arrangements—some
OCR for page 11
Ensuring the Quality, Credibility, and Relevance of U.S. Justice Statistics SACs function as independent state agencies, whereas others are affiliated with state justice or law enforcement departments or academic institutions. In the panel’s assessment, the BJS-state SAC network is a strong example of the federal-state cooperative systems that are important to statistical agency functioning. Support for the SAC network stands as a relatively low-cost activity on BJS’s part with great dividends in terms of outreach and feedback, as well as dissemination of data and products to state policy makers. Moreover, the SAC network is mutually beneficial—states gain from the technical assistance that might otherwise be unavailable or cost-prohibitive to obtain and from national-level benchmarking that is useful for framing policy developments in the states, and BJS benefits from the inventiveness of research performed by its SAC partners. As a vital part of BJS’s operations, we urge continued strengthening of the relationship between BJS and its SAC partners: Recommendation 4.1: Through its Statistical Analysis Center and State Justice Statistics programs, BJS should continue to develop its ties with the states, and more fully exploit the potential for using states as partners in data collections. We further suggest that the experience of BJS’s SAC partners be tapped as BJS pursues methodological improvements, including developments toward longitudinal and small-area measurement systems (Recommendation 4.2). BUILDING STATISTICAL SYSTEMS AND CONCEPTUAL FRAMEWORKS Strategic Goal 2: To build, maintain, and utilize statistical systems that describe the extent and characteristics of crime in our nation and the status and response of the justice system. More than a basic statement of topic, the language of this goal suggests attention to “statistical systems” in two basic respects: first, ensuring that BJS’s data collection portfolio has a greater sense of coherence and interconnectedness than it does at present and, second, making use of computerized record systems—including those supported by BJS’s grants for system development—to develop new sources of data. Conceptual Frameworks Based on our review of past and current BJS data collections, several of our recommendations concern the refinement of conceptual frameworks for what BJS does, with respect not only to the whole structure of BJS’s portfolio, but also to the design of BJS collections in major topic areas.
OCR for page 12
Ensuring the Quality, Credibility, and Relevance of U.S. Justice Statistics With respect to the whole portfolio, we recommend that BJS’s strategic plan articulate a blueprint of interrelated data collection and product activities that could then be used to evaluate new opportunities (Recommendation 3.7). The BJS-developed mapping of data series to the criminal justice system funnel model is an important first step in this regard; our concern is that the unique characteristics, shared areas of substantive overlap, and interrelationships of BJS’s current set of collections are not always immediately clear. We find this lack of an overall framework to be most striking in BJS’s law enforcement data collections, where the agency’s large number of special-agency censuses has an unmistakable scattershot appearance. In particular, we recommend that BJS develop an integrated conceptual plan for the periodicity of these law enforcement agency censuses and surveys, abandoning those with no meaningful prospect of being repeated (Recommendation 3.10). New one-time collections should have a clearly defined role within the broader portfolio. Within BJS’s major survey vehicles, we urge the implementation of a core-supplement framework, where the base content of the survey is reduced to a simplified common “core” of recurring content and coupled with recurring, structured topic supplements. We recommend such a structure for the NCVS in Recommendation 3.8 and for the Law Enforcement Management and Administrative Survey (LEMAS) in Recommendation 3.9. We think that such a structure is an important part of overcoming limitations in BJS’s current holdings in the area of law enforcement generally, which are currently principally limited to the organizational and administrative focus of LEMAS. We suggest the use of LEMAS topic supplements to expand the substantive scope of that survey and the analysis of LEMAS-type data in combination with crime data (thus coming closer to informing assessment of police policy effectiveness). We further recommend (Recommendation 3.11) that the NCVS and its supplements be used more effectively as a tool for studying law enforcement, both in terms of the types of crime that are reported (and not reported) to police and the action that results from the reporting of a crime (such as has been done by the current Police-Public Contact Survey supplement). As noted above, the panel’s general impression of BJS’s data collections in the area of adjudication is one of the agency doing the best it can with what it has—making good use of data collection partners but facing the basic problem of wide variation in the operational structure of state courts, in their levels of automation, and their use of computerized case management systems. Thus, it is more a statement on the condition of court information systems than on BJS’s efforts that we conclude that BJS’s adjudication series lack an effective basis in sampling. We recommend (Recommendation 3.12) that BJS work to implement more rigorous methods of probability sampling in its adjudication series as court records become more accessible
OCR for page 13
Ensuring the Quality, Credibility, and Relevance of U.S. Justice Statistics through computerized case management systems; its recent steps to redesign the State Court Processing Statistics collection exemplify developments in this regard. We further recommend that BJS develop a research program to build representative samples of courts and to assess strategies for collection of case records for even such a small, but representative, sample (Recommendation 3.13). One major omission in BJS’s current work in court statistics is information on declinations to prosecute and the use of alternative dispute resolution techniques; collecting these data in full detail would be a major enterprise, but we suggest that one low-cost way to get at least some information on this missing piece would be to add some basic questions on aggregate case processing to the National Prosecutors Survey, which currently has a purely administrative focus. Making Use of Record Systems Interpreting “statistical systems” more technically, this strategic goal statement also includes two recommendations regarding the use of computerized record systems. First, it is part of BJS’s explicit legal mandate to provide both technical and financial assistance to state and local authorities for purposes of developing information resources. Much of this work has been done under BJS’s National Criminal History Improvement Program (NCHIP), under which BJS provides grants to local authorities to help their development of criminal history record databases. These local databases are used, in turn, to populate national-level record databases maintained by the Federal Bureau of Investigation (FBI) that are used, among other things, to conduct background checks for firearm purchases. From the purely statistical standpoint, BJS’s NCHIP work is ripe for criticism because BJS’s role is strictly a money-transfer operation; although it pays to develop the local databases, BJS has no access to the resulting data for research purposes. Other than generating rough summary statistics of firearm-purchase background checks, BJS has not been able to utilize the criminal history record data that its grant monies help to develop at the state and local levels. This failure has occurred despite a formal set-aside of funds in all of those grants for BJS evaluation purposes. Very recently, however, the FBI has begun to permit BJS the same access to criminal history records that is available to law enforcement agencies. Access to compiled history record data for research purposes opens very exciting avenues for study, which BJS intends to first explore in studies of prisoner reentry to the community and recidivism. We encourage BJS to pursue this work and expand it to include broader studies of criminal behavior: Recommendation 4.3: BJS should actively utilize the NCHIP program to improve criminal history records necessary for longitudinal studies of crime.
OCR for page 14
Ensuring the Quality, Credibility, and Relevance of U.S. Justice Statistics Second, we suggest that BJS study the feasibility of a records-based data collection that would be capable of providing a timely indicator of major crime trends. In the panel’s assessment, the panel is well served by having multiple indicators of crime and justice in the United States; the FBI’s Uniform Crime Reporting (UCR) program has the weight of being an official measure of crimes reported to police while BJS’s NCVS has unique strengths in describing the etiology and contexts of crime and violence (including those incidents not reported to police). However, both series suffer in public perception from lengthy lag times between data collection and release of results. It has been argued that the UCR program would be better administered if it were transferred from the FBI to BJS; we do not recommend such a transfer at this time, arguing that the move would pick unnecessary turf fights and would incur a much more intensive—and costly—redesign than the simple organizational switch suggests at first glance. However, we suggest that BJS study the feasibility of an entirely different approach that is better in some respects, in order to determine whether crime incident data collected by local police departments on their own (and, in some cases, published on their websites) can be compiled on a probability sample basis. This new collection is not intended to be duplicative of the UCR, because it puts the burden of structuring the sample and compiling the data on BJS rather than requiring local departments to compile and record the counts in the UCR format; rather, it is simply intended to gain leverage from the availability of existing local data. Recommendation 4.4: To improve the timeliness of crime statistics, BJS should explore the development of a crime reporting system based on a probability sample of police administrative records. The goals of such a system would be national representativeness, high response, high data quality, timeliness and flexibility in terms of crime classification and analysis, and national statistics for the monitoring of crime trends. IMPROVING COVERAGE OF THE JUSTICE SYSTEM Strategic Goal 3: To provide comprehensive statistical coverage of all parts of the criminal justice system, including the longitudinal flow of persons and cases through the system and their return to the community. Through this goal statement, we return to our analysis of the funnel model for the criminal justice system sequence of events and our conclusions from that analysis. Accepting the funnel model as a premise, this goal might be restated as saying that data collections should rise and fall in importance relative to their proximity to the funnel: The further a specific
OCR for page 15
Ensuring the Quality, Credibility, and Relevance of U.S. Justice Statistics phenomenon is from the activities described by the funnel model, the less central it is to the mission of BJS. By its nature, the funnel model has strong implications for priority setting. First, it suggests that BJS should focus on activities that affect the most people and affect them most extensively. By definition, this means the incidence of crime and victimization, which the funnel graphically describes as the most prevalent phenomenon in the criminal justice system. Following this logic, and all other things being equal, describing crime, victimization, and the immediate consequences should be the principal focus of BJS. Second, however, the nature of a funnel also directs attention to the late stages of the process. As offenders make their way through the funnel, their numbers decrease but the consequences of justice system decisions increase, in terms of impacts on lives and public budgets. At the far end of the criminal justice system, in the correctional area, the effects of criminal justice decisions are so extensive that obtaining adequate data on these populations has importance well beyond their numbers. This principle for establishing the importance of any given activity need not correspond to the allocation of resources on that activity because some data can be collected more cheaply than others. Nonetheless, it should guide decisions as to where to put the intellectual energy of the agency if not in a one-to-one correspondence with its fiscal resources. A logical decision process may be to separately value activities by their proximity to the funnel, and then assess their relative cost; the final portfolio is a function of both importance and cost. The word “longitudinal” in this goal statement is particularly critical. As the mapping of BJS data series to the funnel makes clear, BJS generally has good coverage of all the steps in the criminal justice process but, almost exclusively, this coverage is cross-sectional in nature. BJS’s steps in true longitudinal measurement of persons and cases through multiple steps of the criminal justice system have been rare; in large part, this is due to the structural problem that there is no common identifier attached to a person (or case) that follows the person (or case) through all major steps (police, court processing, entry into corrections, and reentry into the community) to facilitate easy longitudinal measurement. We see a longitudinal approach as essential to study of the performance of the justice system as a whole, and we recommend a variety of strategies for improving longitudinal designs. Some of these are low-cost techniques that simply build on longitudinal structures within existing data resources such as building methods for record linkage into existing and emerging correctional data collections; one example is linking data from recidivism studies to the Census of Adult Correctional Facilities to understand how correctional facilities and programming affect recidivism (Recommendation 3.3). We also recommend that BJS work to emphasize transitions and flows in prison and jail populations, whereas BJS’s current publications tend to emphasize current “stocks” or fixed-time
OCR for page 16
Ensuring the Quality, Credibility, and Relevance of U.S. Justice Statistics counts of inmates. We suggest that BJS’s regular correctional data products include both stock (Recommendation 3.1) and flow (Recommendation 3.2) information capable of disaggregation and cross-tabulation by state, offense category, and demographic group. We recognize that other suggestions are more costly, but would ultimately be beneficial in enhancing understanding of the justice system: Conduct research on the “common identifier” problem discussed above, determining the feasibility of measuring individuals’ experiences in the justice system on a prospective, longitudinal basis, beginning as early as practicable in the process (arrest) and ending with their eventual exit (ranging from early dismissal of charge through completion of sentence) (Recommendation 3.4); Develop an approach to measure the victimization of experiences of individuals (as contrasted with the NCVS focus on households) on a longitudinal basis, beginning from a focal victimization and following the victim forward in time, measuring subsequent victimizations (Recommendation 3.5); and Develop a panel survey of people under correctional supervision to understand how individuals move between institutional and community settings, and to understand the social contexts of correctional supervision (Recommendation 3.6). To extend this last point, BJS and the Justice Department should look for opportunities to leverage studies called for by the Second Chance Act of 2007 (enacted in April 2008) to build an active program on recidivism and reentry. Specifically, we recommend that BJS mount a feasibility study of the flow of individuals between correctional supervision and community settings. Repeated interviews of samples of about-to-be-released prisoners that track their successes and failures in reintegrating with the community would enhance understanding of this critical policy issue (Recommendation 3.14). We also suggest that BJS explore ways to reactivate its studies of probationers and parolees. Under this general heading of justice system coverage, we further recommend that BJS continue to develop and study the measurement of emerging or hard-to-reach groups and develop more appropriate approaches to sampling and measurement of these populations (Recommendation 5.14). In addition, BJS (in conjunction with OJJDP) should develop juvenile victimization, crime, and justice statistical series suitable for describing the patterns of offending and victimization of youth, as well as studies of the longitudinal progression of youth through the juvenile and criminal justice systems (Recommendation 2.2).
OCR for page 17
Ensuring the Quality, Credibility, and Relevance of U.S. Justice Statistics FACILITATING ACCESS AND IMPROVING DISSEMINATION AND OUTREACH Strategic Goal 4: To ensure access to statistics and data on crime and justice by the American public, the U.S. Congress, the U.S. Department of Justice and other executive agencies, and state and local government agencies. Outreach and dissemination are areas that already play considerable roles in BJS operations and in which BJS has made laudable strides. We have already noted and endorsed BJS’s formation of a strong communications and contact network with the states through its SACs. BJS also deserves credit for the extensive backfile of reports and summary tabulations that are available from the agency’s website and the dissemination of its reports through the OJP-administered National Criminal Justice Reference Service. High-end users are well served by BJS’s microdata holdings that are available for download through the National Archive of Criminal Justice Data. Through this goal statement, we encourage still further developments along these lines as BJS works to increase its public profile and its relevance in policy debates, while maintaining the high quality standards it has set for itself. In the panel’s assessment, there is value in BJS pursuing both formal and informal mechanisms for obtaining user input and feedback and for shaping possible new data collections. The law that created BJS in 1979 also created a formal advisory board for the agency, a provision that was later omitted in reauthorization language in 1984. We recommend that such an ongoing advisory group would be a useful vehicle for obtaining recommendations from diverse perspectives; specifically, we recommend that BJS establish an Advisory Group under the Federal Advisory Committee Act, the membership of which should include—at a minimum—leaders and practitioners from each of the major subject matters covered by BJS data, as well as those with statistical and other types of academic expertise (Recommendation 5.8). In the past, BJS has also made good use of ad hoc user and stakeholder workshops on targeted issues; the agency convened a very useful data user conference in February 2008 in partial support of this panel’s activities. We recommend that BJS continue to hold such ad hoc stakeholder workshops to suggest areas of immediate needs or to get input on contemporaneous topics of interest (Recommendation 5.7). The data extracts directly available from the BJS website tend to be selected tabulations and spreadsheets. As BJS works to enhance its website presence, it should explore methods for making more of its data available for direct tabulation, analysis, and—perhaps—mapping on the website. We also recommend that BJS take care in managing how some of its data collections (gathered by external data collection agents) are housed on non-BJS websites; BJS should articulate and describe the process by which links to
OCR for page 18
Ensuring the Quality, Credibility, and Relevance of U.S. Justice Statistics external sites are allowed and used, and should also articulate and justify the use of its insignia—which carries with it important quality connotations—on external websites (Recommendation 5.12). From the numerous references to BJS data and findings in legislative text and debate records, it is clear that BJS has a key and receptive audience in the U.S. Congress and its staff. However, from the types of legislative demands that are sometimes put on BJS (e.g., the PREA reporting requirements) and past difficulties BJS has experienced in securing funding, it is not quite as clear that the dialogue between BJS and Congress is effective. (At present, of course, a major reason for this problem in communication is that liaison with Congress is done principally by and through OJP; for instance, BJS directors do not directly testify to appropriations committees on agency needs.) Accordingly, we recommend that—regardless of the organizational structure of BJS and OJP—BJS should take a stronger role in cultivating a relationship with Congress: Recommendation 5.9: DOJ should take steps to ensure that congressional staff are aware of BJS data that could be used in developing legislation; DOJ and BJS should learn from congressional staff how their data are needed to inform/support legislation so that they can improve the utility of their current data and so that they can develop new datasets that could enhance policy development. From the technical standpoint, an important means for BJS to increase its public relevance is to find ways to address long-standing concerns about the agency’s data products: though they are generally held in high regard, they are frequently seen as lacking timeliness and, particularly for the NCVS, subnational geographic detail. One approach that we recommend to deal with this concern is that BJS evaluate each of its data programs to ascertain whether more timely estimates might be obtained by (a) making discrete data collections into more continuous operations and (b) issuing preliminary estimates, to be followed by final estimates (Recommendation 5.11). More fundamentally, and consistent with the expected principles and practices of a federal statistical agency, we believe that the cultivation of an active and continuous research program is essential to an agency’s progress. For BJS, such research should include work on statistical modeling to improve the temporal and spatial resolution of estimates (a program for which BJS could partner with other federal statistical agencies, most of which face similar pressures from their user constituencies). A vigorous research program could also suggest ways to better and more efficiently make use of state and local database resources, such as through steps to increase the automation and consistency of BJS’s National Corrections Reporting Program data (Recommendation 5.10). The presence of a solid research program would
OCR for page 19
Ensuring the Quality, Credibility, and Relevance of U.S. Justice Statistics also allow BJS to do something that sounds obvious—carefully study changes in the NCVS design before implementing them (Recommendation 5.13)—but that is difficult to accomplish in a climate of constrained resources. To carry out a research program and continually improve the agency’s data collection holdings, we recommend that BJS improve the technical skill mix of its staff, including mathematical statisticians, computer scientists, survey methodologists, and criminologists (Recommendation 5.15). PRIORITY SETTING AND CONSTRAINED RESOURCES A possible criticism of this report is that it tends to suggest adding to BJS’s inventory of collections rather than subtracting. The recommendations are generally geared to improvements within BJS’s various existing data collections—for instance, ensuring a high-quality independent measure of crime in the NCVS, emphasizing conceptual frameworks in BJS’s adjudication and law enforcement collections, and expanding its corrections series to study prisoner reentry into society. As a panel, we grappled with questions regarding this: How should these improvements be paid for? Should some existing series be cut to make ways for new ones? And, given the disproportionate share of BJS’s current resources consumed by the NCVS, should some smaller collections be stopped to free up additional resources for the NCVS or should some NCVS resources be steered to other areas? The reason why we rejected explicitly suggesting that some data series be cut in order to pay for others is certainly not that the current collections are perfect. The improvements we suggest in the recommendations are testimony to that. Rather, we believe that BJS cannot achieve its legislated goals by cutting programs. In our assessment, we think it can be stated as a fact: BJS has been given more responsibilities than can be achieved with current resources. The resources provided to BJS to carry out its work are not commensurate with the breadth—and importance—of the responsibilities assigned to the agency by its authorizing legislation. Because of this, the agency has for some years walked a tight line of small cuts of sample or measurement, short delays of publications, and temporary hiring freezes—each of these tolerable in itself, but cumulating over the years such that core functions have broken down. On a routine basis, BJS must make decisions about addressing certain responsibilities and not others, trading off periodicity and completeness of data collections, and balancing continuity of existing data collections against the need to comply with directives from Congress and DOJ. Such decisions are hardly unique to BJS—at some point, all organizations must make such trade-offs—but BJS’s mismatch between resources and responsibility makes the decisions particularly difficult.
OCR for page 20
Ensuring the Quality, Credibility, and Relevance of U.S. Justice Statistics Thus, in setting priorities, BJS directors have perforce had a short time horizon—responding to a certain set of demands even though those decisions may have negative long-term consequences for individual data collections and the health of the agency. Certainly, in the midst of year-to-year juggling of data series in order to keep production moving, longer-term investments in research and innovation become difficult or impossible to make. The most striking example of the consequences of this extremely tough climate is the current state of the NCVS: what was once, clearly, the best victimization survey in the world is now unable to satisfy its basic function of providing annual estimates of level and change in common-law crime. This decay happened gradually as BJS administrators were attempting to respond to immediate exigencies, aggravated by an overly broad mandate. Each single cut in sample size, or other cost-cutting measure, was justifiable given then-present alternatives. Cumulatively—as demonstrated most vividly by the declared “break in series” with the 2006 NCVS data—they lead to our conclusion in our interim report that “the [current] NCVS is not achieving and cannot achieve BJS’s legislatively mandated goals” (National Research Council, 2008b:Finding 3.1). Statistical infrastructure (data collections and record series) shares with physical infrastructure (such as roads, bridges, sewers, and cable) the fundamental problem that it lacks glamour and can be difficult to champion as a government spending priority. Yet it is undeniably important to the public knowledge and the public good; BJS’s purview includes topics important to general welfare, spanning the measurement of interpersonal violence, the function and magnitude of law enforcement and corrections, and the operations of the judicial branch of government. The problem is not that there are parts of BJS’s legal mandate that are unimportant or unworthy, but that its current resources do not permit it to cover its mandates as effectively as possible. Given this finding, we are loath to construct a list of series to terminate or reduce out of concern that the agency fail even more visibly to fulfill the charge given by its legislation. Another reason for not considering a specific ranking of data collections in order to suggest possible cuts is consistency with the approach we took in our interim report on options for the NCVS. In that report, we presented an array of possible options and described how specific design choices corresponded to particular goals for the survey. However, we made a point of “not suggest[ing] one single path as the ideal for a redesigned NCVS” (National Research Council, 2008b:4). Different people and decision makers do not necessarily put the exact same weights on the goals and objectives of a program such as the NCVS, and we did not want to presume “that our preferred set of NCVS goals is correct to the exclusion of all others” (National Research Council, 2008b:4). The same logic applies to considering other collections in the BJS portfolio: deeming one collection more worthy than
OCR for page 21
Ensuring the Quality, Credibility, and Relevance of U.S. Justice Statistics another involves complicated value judgments, not science. Specific constituencies for BJS data that are not represented on the panel or by groups who have spoken before the panel could make eloquent and compelling cases for their particular favored set of statistics; again, we do not wish to suggest that any weighting we could suggest is somehow paramount. Although hoping that tight fiscal constraints may be alleviated somewhat in coming years, we cannot assume infinite resources either; we have not interpreted our task as assembling a “wish list” for everything that a justice statistics agency could do, but rather think that out suggestions are a mix of short- and long-term, low- and higher-cost ideas for improving the statistical evidence with which crime and justice policy in the United States is developed. Though we do not explicitly rank BJS’s data collections, our suggested strategic goals provide BJS with a set of principles against which its data collection portfolio can be assessed. BJS, which marks its thirtieth anniversary in its present form this year, can rightly look back on a solid body of accomplishment. But our recommendations in this report suggest that there are many directions for continued improvement; much work remains to be done to ensure the quality, credibility, and relevance of statistics on justice in the United States—to achieve the BJS the country deserves.
OCR for page 22
Ensuring the Quality, Credibility, and Relevance of U.S. Justice Statistics This page intentionally left blank.