Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 14
Part II: Commentary This section comments on most of the topics in the principles and practices; the comments are offered to explain, illustrate, or further define the statement of principle in Part I. DEFINITION OF A FEDERAL STATISTICAL AGENCY A federal statistical agency is a unit of the federal government whose principal function is the compilation and analysis of data and the dissemination of information for statistical purposes. A statistical agency may be labeled a bureau, center, division, or office or similar title, so long as it is recognized as a distinct entity. Statistical agencies have been established for several reasons: (1) to develop new information for an area of public concern (e.g., the Bureau of Labor Statistics, the National Center for Health Statistics); (2) to conduct large statistical collection and dissemination operations specified by law (e.g., the U.S. Census Bureau); (3) to compile and analyze statistics from sets of administrative records for policy purposes and public use (e.g., the Statistics of Income Division in the Internal Revenue Service); and (4) to develop broad and consistent es- timates from a variety of statistical and administrative sources in accordance with a prespecified conceptual framework (e.g., the Bureau of Economic Analysis in the U.S. Department of Commerce). Once established, many statistical agencies engage in all of these functions to varying degrees. 

OCR for page 14
 PART ii: COMMENTARy This definition of a federal statistical agency does not include many statistical activities of the federal government because they are not per- formed by distinct units, or because they do not result in the dissemination of statistics to others—for example, statistics compiled by the U.S. Postal Service to set rates or by the U.S. Department of Defense to test weapons (see National Research Council, 1998b, 2002b, 2003b, 2006d, on statistics and testing for defense acquisition). Nor does it include agencies whose primary functions are the conduct or support of problem-oriented research, although their research may be based on information gathered by statistical means, and they may also sponsor important surveys, as do, for example, the National Institutes of Health, the Agency for Healthcare Research and Quality, and other agencies in the U.S. Department of Health and Human Services. Finally, this definition of a statistical agency does not usually include agencies whose primary function is policy analysis and planning (e.g., the Office of Tax Analysis in the U.S. Department of the Treasury, the Office of the Assistant Secretary for Planning and Evaluation in the U.S. Department of Health and Human Services). Such agencies may collect and analyze statistical information, and statistical agencies, in turn, may perform some policy-related analysis (e.g., produce reports on trends in after-tax income or child care arrangements of families). However, to maintain credibility as an objective source of accurate, useful information, statistical agencies must be separate from units that are involved in developing policy and assessing policy alternatives. The work of federal statistical agencies is coordinated through the In- teragency Council on Statistical Policy (ICSP), created by the U.S. Office of Management and Budget (OMB) in the 1980s and authorized in statute in the 1995 reauthorization of the Paperwork Reduction Act. The ICSP is chaired by OMB and currently includes representation from a total of fourteen agencies and units, which are housed in nine cabinet departments and three independent agencies (see Appendix A): • Bureau of Economic Analysis (Commerce Department) • Bureau of Justice Statistics (Justice Department) • Bureau of Labor Statistics (Labor Department) • Bureau of Transportation Statistics (Transportation Department) • Census Bureau (Commerce Department) • Economic Research Service (Agriculture Department) • Energy Information Administration (Energy Department)

OCR for page 14
 PRiNCiPLES AND PRACTiCES FOR A FEDERAL STATiSTiCAL AGENCy • National Agricultural Statistics Service (Agriculture Department) • National Center for Education Statistics (Education Department) • National Center for Health Statistics (Health and Human Services Department) • Office of Environmental Information (Environmental Protection Agency) • Office of Research, Evaluation, and Statistics (Social Security Administration) • Science Resources Statistics Division (National Science Foundation) • Statistics of Income Division (Treasury Department) Throughout the federal government, OMB recognizes more than 80 units and agencies that are not statistical agencies but that have annual budgets of $500,000 or more for statistical activities (U.S. Office of Man- agement and Budget, 2008c:Table 1). The principles for federal statistical agencies presented here should apply to other federal agencies that carry out statistical activities, and they may find many of the detailed practices pertinent as well. Similarly, the principles and practices may be relevant to statistical units in state and local government agencies, and international audiences may also find them useful. ESTABLISHMENT OF A FEDERAL STATISTICAL AGENCY One of the most important reasons for establishing a statistical agency is to provide information that will allow for an informed citizenry. A democ- racy depends on an informed electorate. A citizen has a right to information that comes from a trustworthy, credible source and is relevant, accurate, and timely. Timely information of high quality is also critical to policy analysts and decision makers in both the public and private sectors. (For more infor- mation on the purposes of official statistics, see the Fundamental Principles of Official Statistics of the United Nations Statistical Commission in Ap- pendix C; see also U.N. Economic Commission for Europe, 2003; U.N. Statistical Commission, 2003.) Federal statistical agencies serve the key functions of providing a broad array of information to the public and policy makers and of ensuring the necessary quality and credibility of the data. Commercial, nonprofit, and academic organizations in the private sec- tor also provide useful statistical information, including data they collect themselves and data they acquire from government agencies and other data

OCR for page 14
 PART ii: COMMENTARy collectors to which they add value. However, because the benefits of statisti- cal information are shared widely throughout society and because it is often difficult to collect payments for these benefits, private markets are not likely to provide all of the data that are needed for public and private decision making or to make data as widely available as needed for important public purposes. Government statistical agencies are established to ensure that a broad range of information is publicly available. (See National Research Council, 1999b and 2005b, for a discussion of the governmental role in providing public goods, or near public goods, such as research and data.) The United States government collected and published statistics long before any distinct federal statistical agency was formed (see Duncan and Shelton, 1978; Norwood, 1995). The U.S. Constitution mandated the conduct of a decennial census of population beginning in 1790, and the census enumeration was originally conducted by U.S. marshals as one of their many duties. Legislation providing for the compilation of statistics on agriculture, education, and income was enacted by Congress in the 1860s. The Bureau of Labor (forerunner of the Bureau of Labor Statistics) was established by law in 1884 as a separate agency with a general mandate to respond to widespread public demand for information on the conditions of industrial workers. The Census Bureau was established as a permanent agency in 1902 to conduct the decennial census and related statistical activities. Many federal statistical agencies that can trace their roots back to the 19th or early 20th century, such as the National Center for Education Sta- tistics and the National Center for Health Statistics, were organized in their current form following World War II. Several relatively new agencies have since been established, including the Energy Information Administration, the Bureau of Justice Statistics, and the Bureau of Transportation Statistics. In every case, the agency itself, in consultation with users of its information, has major responsibility for determining its specific statistical programs and for setting priorities. Initially, many of these agencies also had responsibili- ties for certain policy analysis functions for their department heads. More recently, policy analysis has generally been located in separate units that are not themselves considered to be statistical agencies, a separation that helps establish and maintain the credibility of statistical agencies as providers of data and analyses that are not designed for particular policy alternatives. A statistical agency has at least two roles: (1) provider of the statistical information and analysis needed to inform policy making and program assessment by its own department, and (2) source of national statistics for

OCR for page 14
 PRiNCiPLES AND PRACTiCES FOR A FEDERAL STATiSTiCAL AGENCy the public in its area of concern. It is sometimes difficult to keep these two roles distinct on policy-relevant statistics. An effective statistical agency, nevertheless, will frequently play a creative, not just reactive, role in the development of data needed for policy analysis. Sometimes federal statistical agencies play additional roles, such as monitor and consultant on statisti- cal matters to other units within the same department (see, e.g., National Research Council, 1985a) and collector of data on a reimbursable basis for other agencies. There is no set rule or guideline for when it is appropriate to establish a separate federal statistical agency, carry on statistical activities within the operating units of departments and independent agencies, or contract for statistical services from existing federal statistical agencies or other organi- zations. Establishment of a federal statistical agency should be considered when one or more of the following conditions prevail:1 • There is a need for information on an ongoing basis beyond the ca- pacity of existing operating units, possibly involving other departments and agencies. Such needs may require coordinating data from various sources, initiating new data collection programs to fill gaps, or developing regularly updated time series of estimates. • There is a need, as a matter of credibility, to ensure that major data series are independent of policy makers’ control. • There is a need to establish the functional separation of data on individuals and organizations that are collected for statistical purposes from data on individuals and organizations that may be used for administrative, regulatory, or law enforcement uses. Such separation, recommended by the Privacy Protection Study Commission (1977), bolsters a culture and practice of respect for privacy and protection of confidentiality. Functional separation is easier to maintain when the data to be used for statistical purposes are compiled and controlled by a unit that is separate from operat- ing units or department-wide data centers. The Confidential Information Protection and Statistical Efficiency Act of 2002 (CIPSEA) extended legal confidentiality protection to statistical data collections that may be carried 1The National Research Council (2001b:Ch. 6) cited a number of these reasons in recommending to the U.S. Department of Health and Human Services that it establish or identify a statistical unit to be assigned responsibility and authority for carrying out statistical functions and data collection for social welfare programs and the populations they serve; see also National Research Council and Institute of Medicine (2004).

OCR for page 14
 PART ii: COMMENTARy out by any federal agency, whether a statistical agency or other type of agency (see Appendix B). Nonetheless, functional separation of statistical data from other kinds of data is important because it makes promises of confidentiality protection more credible.2 • There is a need to emphasize the principles and practices of an ef- fective statistical agency, for example, professional practice, openness about the data provided, and wide dissemination of data. • There is a need to encourage research and development of a broad range of statistics in a particular area of public interest or of government activity or responsibility. • There is a need to consolidate compilation, analysis, and dissemina- tion of statistics in one unit to encourage high-quality performance, elimi- nate duplication, and streamline operations. PRINCIPLES FOR A FEDERAL STATISTICAL AGENCY Principle 1: A federal statistical agency must be in a position to provide objective information that is relevant to issues of public policy. A statistical agency supplies information not only for the use of man- agers and policy makers in the executive branch and for legislative design- ers and overseers in Congress, but also for all those who require objective statistical information on public issues, whether the information is needed for purposes of production, trade, consumption, or participation in civic affairs. Just as a free enterprise economic system depends on the availability of economic information to all participants, a democratic political system depends on—and has a fundamental duty to provide—wide access to in- formation on education, health, transportation, the economy, the environ- ment, criminal justice, and other social concerns. Federal statistical agencies are responsible for providing statistics on conditions in a variety of areas. The resulting information is used both inside and outside the government not only to delineate problems and sometimes to suggest courses of action, but also to evaluate the results of 2Under the guidance issued for CIPSEA (see Appendix B), OMB in 2007 recognized two new statistical units: the Office of Applied Studies within the Substance Abuse and Men- tal Health Services Administration of the U.S. Department of Health and Human Services, and the Microeconomic Surveys Section of the Federal Reserve Board of Governors.

OCR for page 14
0 PRiNCiPLES AND PRACTiCES FOR A FEDERAL STATiSTiCAL AGENCy government activity or lack of activity. The statistics provide much of the basis on which the government itself is judged. This role places a heavy re- sponsibility on federal statistical agencies for impartiality and objectivity. In order to provide information that is relevant to public issues, sta- tistical agencies need to reach out to users of the data. Federal statistical agencies usually are in touch with the primary users in their own depart- ments. Considerable energy and initiative are required to open avenues of communication more broadly to other current and potential users, includ- ing analysts and policy makers in other federal departments, state and local government agencies, academic researchers, private-sector organizations, organized constituent groups, the media, and Congress. Advisory com- mittees are recommended as a means to obtain the views of users, as well as people with relevant technical expertise (see, e.g., National Research Council, 1993a). Many agencies obtain advice from committees that are chartered under the Federal Advisory Committee Act—examples include the Advisory Committee on Agriculture Statistics for the National Agricul- tural Statistics Service, the Board of Scientific Counselors for the National Center for Health Statistics, and the Census Advisory Committee of Profes- sional Associations for the Census Bureau. The Federal Economic Statistics Advisory Committee (FESAC), chartered in November 1999, provides substantive and technical advice to three agencies—the Bureau of Economic Analysis, the Bureau of Labor Statistics, and the Census Bureau—thereby providing an important cross-cutting perspective on major economic sta- tistics programs (see http://www.bls.gov/bls/fesac.htm [December 2008]). Some agencies obtain advice from committees and working groups that are organized by an independent association, such as the American Statistical Association’s Committee on Energy Statistics for the Energy Information Administration. One frequently recommended method for alerting statistical agencies to emerging statistical information needs is for the agency’s own staff to en- gage in analysis of its data (Martin, 1981; Norwood, 1995; Triplett, 1991). For example, relevant analysis may use the agency’s data to examine cor- relates of key social or economic phenomena or to study the statistical error properties of the data. Such in-house analysis can lead to improvements in the quality of the statistics, to identification of new needs, to a reordering of priorities, and to closer cooperation and mutual understanding with policy analysis units. In its work for a policy analysis unit, a statistical agency de- scribes conditions and possibly measures progress toward some previously identified goal, but it refrains from making policy recommendations. The

OCR for page 14
 PART ii: COMMENTARy distinction between statistical analysis and policy analysis is not always clear, and a statistical agency will need to consider carefully the extent of policy- related activities that are appropriate for it to undertake. Principle 2: A federal statistical agency must have credibility with those who use its data and information. Users of a statistical agency’s data must be able to trust that the data were collected and analyzed in an objective, impartial manner and that they are as accurate and timely as the agency can make them. An agency should make every effort to provide accurate and credible statistics that will permit policy debates to be concerned about policy, not about the credibility of the data. Credibility is enhanced when an agency fully informs users of the strengths and weaknesses of the data, makes data available widely, and consults with users about priorities for data collection and analysis. When it does so, an agency is perceived to be working in the national interest, not the interest of a particular administration (Ryten, 1990). Principle 3: A federal statistical agency must have the trust of those whose information it obtains. The statistics programs of the federal government rely in large part on information supplied by individuals and by organizations outside the federal government, such as state and local governments, businesses, and other organizations. Some of this information is required by law or regula- tion (such as employers’ wage reports), some of it is related to administra- tion of government programs (such as information provided by benefit recipients), but much of it is obtained through the voluntary cooperation of respondents in statistical surveys. Even when response is mandatory, the cooperation of respondents reduces costs and likely promotes accuracy (see National Research Council, 1995b, 2004e). Important elements in encouraging such cooperation are that respondents believe that the data requested are important and legitimate for the government to collect, that they are being collected in an impartial, competent manner, and that the confidentiality of their responses will be protected. In brief, trust in a statistical agency must be maintained, and the agency must not be perceived as being swayed by political considerations. Respon- dent trust also depends on providing respondents with realistic promises of confidentiality that the agency can reasonably expect to honor and then

OCR for page 14
 PRiNCiPLES AND PRACTiCES FOR A FEDERAL STATiSTiCAL AGENCy scrupulously honoring those promises. Finally, respondent trust depends on adopting practices that respect personal privacy, such as taking steps to minimize the intrusiveness of questions and the time and effort required to participate in a survey. Principle 4: A federal statistical agency must have a strong position of independence within the government. A statistical agency must be able to provide credible information that may be used to evaluate the program and policies of its own department or the government as a whole. More broadly, a statistical agency must be a trustworthy source of objective, accurate information for decision makers, analysts, and others inside and outside the government who want to use statistics to understand present conditions, draw comparisons with the past, and help guide plans for the future.3 For these purposes, a strong position of independence for a statistical agency is essential. Statistical agency independence must be exercised in a broad frame- work. Legislative authority usually gives ultimate responsibility to the sec- retary of the department rather than the statistical agency head. In addition, an agency is subject to the normal budgetary processes and to various coor- dinating and review functions of OMB, as well as the legislative mandates, oversight, and informal guidance of Congress. Within this broad framework, a statistical agency must work to main- tain its credibility as an impartial purveyor of information. In the long run, the effectiveness of an agency depends on its maintaining a reputation for impartiality; thus, an agency must be continually alert to possible infringe- ments on its credibility and be prepared to argue strenuously against such infringements. An agency head’s independence can be strengthened by being ap- pointed for a fixed term by the President, with approval by the Senate, as is the case with the heads of the Bureau of Labor Statistics and the National Center for Education Statistics. For a fixed term, it is desirable that it not coincide with the presidential term so that professional considerations are more likely to be paramount in the appointment process. In contrast, the heads of the Bureau of Justice Statistics, the Census Bureau, and the Energy Information Administration are presidential appointees, but their terms 3See the Fundamental Principles of Official Statistics of the United Nations Statistical Commission in Appendix C.

OCR for page 14
 PART ii: COMMENTARy are not fixed and usually end with a change of administration. In some instances, heads of statistical agencies are career senior executives. It is also desirable that a statistical agency head have direct access to the secretary of the department or the head of the independent agency in which the statistical agency is located. Such access allows the head to inform new secretaries about the appropriate role of a statistical agency and present the case for new statistical initiatives to the secretary directly. Among the agency heads with presidential appointments, such direct access currently is provided by legislation only for the Bureau of Labor Statistics and the Energy Information Administration. It is desirable for a statistical agency to have its own funding appropria- tion from Congress and not be dependent on allocations from the budget of its parent department or agency, which may be subject to reallocation. These organizational aspects—appointment of the agency head by the President with approval by the Senate for a fixed term not coincident with that of the administration, direct access to the secretary of the agency’s department, and separate budgetary authority—are neither necessary nor sufficient for a strong position of independence for a statistical agency, but they facilitate such independence. In contrast, some agencies are under several layers of supervision within their departments (see Appendix A). Other characteristics related to independence are that a statistical agency has the following: • Authority for professional decisions over the scope, content, and frequency of data compiled, analyzed, or published within the framework set by its authorizing legislation. Most statistical agencies have such broad authority, limited by budgetary constraints, departmental requirements, OMB review, and congressional mandates. • Authority for selection and promotion of professional, technical, and operational staff. • Recognition by policy officials outside the statistical agency of its authority to release statistical information, including accompanying press releases and documentation, without prior clearance. • Authority to control information technology systems for data pro- cessing and analysis in order to securely maintain the integrity and confi- dentiality of data and reliably support timely and accurate production of key statistics. • Authority for the statistical agency head and qualified staff to speak about the agency’s statistics before Congress, with congressional staff, and before public bodies.

OCR for page 14
 PRiNCiPLES AND PRACTiCES FOR A FEDERAL STATiSTiCAL AGENCy • Adherence to fixed schedules in public release of important statisti- cal indicators to prevent even the appearance of manipulation of release dates for political purposes. • Maintenance of a clear distinction between statistical information and policy interpretations of such information by the president, the secre- tary of the department, or others in the executive branch. • Dissemination policies that foster regular, frequent release of major findings from an agency’s statistical programs to the public via the media, the Internet, and other means. Control over personnel actions, especially the selection and appoint- ment of qualified professional staff, including senior executive career staff, is an important aspect of independence. Agency staff reporting directly to the agency head should have formal education and deep experience in the substantive, methodological, operational, or management issues facing the agency as appropriate for their positions. In addition, professional qualifica- tions are of the utmost importance for statistical agency heads, whether the profession is that of statistician or the subject-matter field of the statistical agency (National Research Council, 1997b). Relevant professional associa- tions can be a source of valuable input on suitable candidates. The authority to ensure that information technology systems fulfill the specialized needs of the statistical agency is another important aspect of independence. A statistical agency must be able to vouch for the integrity, confidentiality, and impartiality of the information collected and main- tained under its authority so that it retains the trust of its data providers and data users. Such trust is fostered when a statistical agency has control over its information technology resources, and there is no opportunity or percep- tion that policy, program, or regulatory agencies could gain access to records of individual respondents. A statistical agency also needs control over its information technology resources to support timely and accurate release of official statistics, which are often produced under stringent deadlines. Authority to decide the scope and specific content of the data collected or compiled and to make decisions about technical aspects of data collec- tion programs is yet another important element of independence, although such authority can never be without limits. Congress frequently specifies particular data that it wishes to be collected (e.g., data on job openings and labor turnover by the Bureau of Labor Statistics, data on family farms by the Economic Research Service and National Agricultural Statistics Service) and, in the case of the decennial census, requires an opportunity to review

OCR for page 14
 PRiNCiPLES AND PRACTiCES FOR A FEDERAL STATiSTiCAL AGENCy As with descriptive analyses provided by the agency, substantive analyses should be designed to be relevant to policy by addressing topics of public interest and concern. However, such analyses should not include positions on policy options or be designed to reflect any particular policy agenda. These issues are discussed in Martin (1981), Norwood (1975), and Triplett (1991). Research on Methodology and Operations For statistical agencies to be innovative in methods for data collection, analysis, and dissemination, research on methodology and operational pro- cedures must be ongoing. Methodological research may be directed toward improving survey design, measuring error and, when possible, reducing it from such sources as nonresponse and reporting errors, reducing the time and effort asked of respondents, evaluating the best mix of interview modes (e.g., mail, telephone, personal interview) to cope with increasing nonre- sponse rates due to such phenomena as cell-phone-only households, devel- oping new and improved summary measures and estimation techniques, and developing innovative statistical methods for confidentiality protection. Research on operational procedures may be directed toward facilitating data collection in the field, improving the efficiency and reproducibility of data capture and processing, and enhancing the usability of Internet-based data dissemination systems. Much of current practice in statistical agencies was developed through research they conducted or obtained from other agencies. Federal statistical agencies, frequently in partnership with academic researchers, pioneered the applications of statistical probability sampling, the national economic accounts, input-output models, and other analytic methods. The U.S. Census Bureau pioneered the use of computers for processing the census, and research on data collection, processing, and dissemination operations continues to lead to creative uses of automated procedures and equipment in these areas. Several federal statistical agencies sponsor research using academic principles of cognitive psychology to improve the design of ques- tionnaires, the clarity of data presentation, and the ease of use of electronic data collection and dissemination tools such as the Internet. The history of the statistical agencies has shown repeatedly that methodological and opera- tions research can lead to large productivity gains in statistical activities at relatively low cost.

OCR for page 14
 PART ii: COMMENTARy An effective statistical agency actively partners with the academic com- munity for methodological research. It also seeks out academic and industry expertise for improving data collection, processing, and dissemination operations. For example, a statistical agency can learn techniques and best practices for improving software development processes from computer scientists (see National Research Council, 2003e, 2004d). Research on Policy Uses Much more needs to be known about how statistics are actually used in the policy-making process, both inside and outside the government. Re- search about how the information produced by a statistical agency is used in practice should contribute to future improvements in design, concepts, and format of data products. For example, public-use files of statistical microdata were developed in response to the growing analytic needs of government and academic researchers. Gaining an understanding of the variety of uses and users of an agency’s data is only a first step. More in-depth research on the policy uses of an agency’s information might, for example, explore the use of data in microsimulation or other economic models, or go further to examine how the information from such models and other sources is used in decision making (see National Research Council, 1991a, 1991b, 1997a, 2000b, 2001b, 2003a). Practice 9: Professional Advancement of Staff An effective federal statistical agency has personnel policies that encour- age the development and retention of a strong professional staff who are committed to the highest standards of quality work. There are several key elements of such a policy: • The required levels of technical and professional qualifications for positions in the agency are identified, and the agency adheres to these re- quirements in recruitment and professional development of staff. Position requirements take account of the different kinds of technical and other skills, such as supervisory skills, that are necessary for an agency to have a full range of qualified staff, including not only statisticians, but also experts in relevant subject-matter areas, data collection, processing, and dissemina- tion processes, and management of complex, technical operations.

OCR for page 14
 PRiNCiPLES AND PRACTiCES FOR A FEDERAL STATiSTiCAL AGENCy • Continuing technical education and training, appropriate to the needs of their positions, is provided to staff through in-house training pro- grams and opportunities for external education and training. • Position responsibilities are structured to ensure that staff have the opportunity to participate, in ways appropriate to their experience and ex- pertise, in research and development activities to improve data quality and cost-effectiveness of agency operations. • Professional activities, such as publishing in refereed journals and presentations at conferences, are encouraged and recognized, including presentations of technical work in progress with appropriate disclaimers. Participation in relevant statistical and other scientific associations is en- couraged to promote interactions with researchers and methodologists in other organizations. Such participation is also a mechanism for openness about the data provided. • Interaction with other professionals is increased through technical advisory committees, supervision of contract research and research consul- tants, fellowship programs of visiting researchers, exchange of staff with relevant statistical, policy, or research organizations, and opportunities for new assignments within the agency. • Accomplishment is rewarded by appropriate recognition and by af- fording opportunity for further professional development. The prestige and credibility of a statistical agency is enhanced by the professional visibility of its staff, which may include establishing high-level nonmanagement posi- tions for highly qualified technical experts. An effective statistical agency considers carefully the costs and benefits—monetary and nonmonetary—of using contractor organiza- tions, not only for data collection as most agencies do, but also to supple- ment in-house staff in other areas.8 Outsourcing can have benefits, such as: providing experts in areas in which the agency is unlikely to be able to attract highly qualified in-house staff (e.g., some information technology functions), enabling an agency to handle an increase in its workload that is expected to be temporary or that requires specialized skills, and allowing an agency to learn from best industry practices. However, outsourcing can also have costs, including that agency staff become primarily contract managers 8Only the Bureau of Labor Statistics and the Census Bureau maintain their own inter- viewing staff.

OCR for page 14
 PART ii: COMMENTARy and less qualified as technical experts and leaders in their fields. An effec- tive statistical agency maintains and develops a sufficiently large number of in-house staff, including mathematical statisticians, who are qualified to analyze the agency’s data and to plan, design, carry out, and evaluate its core operations so that the agency maintains the integrity of its data and its credibility in planning and fulfilling its mission. Statistical agencies should also maintain and develop staff with the expertise necessary for effective management of contractor resources. An effective statistical agency has policies and practices to instill the highest possible commitment to professional ethics among its staff, as well as procedures for monitoring contractor compliance with ethical standards. When an agency comes under pressure to act against its principles—for example, if it is asked to disclose confidential information for an enforce- ment purpose or to support an inaccurate interpretation of its data—it must be able to rely on its staff to resist such actions as contrary to the ethical principles of their profession. An effective agency refers its staff to such statements of professional practice as the guidelines published by the American Statistical Association (1999) and the International Statistical Institute (1985), as well as to the agency’s own statements about protec- tion of confidentiality, respect for privacy, standards for data quality, and similar matters. It endeavors in other ways to ensure that its staff are fully cognizant of the ethics that must guide their actions in order for the agency to maintain its credibility as a source of objective, reliable information for use by all. Practice 10: A Strong Internal and External Evaluation Program Statistical agencies that fully follow such practices as continual develop- ment of more useful data, openness about sources and limitations of the data provided, wide dissemination of data, commitment to quality and pro- fessional standards of practice, and an active research program will likely be in a good position to make continuous assessments of and improvements in the relevance and quality of their data collection systems. Yet even the best functioning agencies will benefit from an explicit program of internal and independent external evaluations, which frequently offer fresh perspectives. Such evaluations need to address not only specific agency programs, but also the agency’s portfolio of programs considered as a whole.

OCR for page 14
 PRiNCiPLES AND PRACTiCES FOR A FEDERAL STATiSTiCAL AGENCy Ealuating Quality Evaluation of data quality for a continuing survey or other kind of data collection program begins with regular monitoring of quality indicators. For surveys, such monitoring includes unit and item response rates, population coverage rates, and information on sampling error, such as coefficients of variation. (The American Community Survey provides these indicators on its web page, http://www.census.gov/acs/www/ [December 2008].) In ad- dition, in-depth assessment of quality on a wide range of dimensions—in- cluding sampling and nonsampling errors across time and among popula- tion groups and geographic areas—needs to be undertaken on a periodic basis (National Research Council, 2007b). Research on methods to improve data quality may cover such areas as alternative methods for imputing values for missing data and alternative question designs, using cognitive methods, to reduce respondent reporting errors. Methods for such research may include the use of “methods panels” (small samples of respondents with whom experiments are conducted by using alternative procedures and questionnaires), matching with adminis- trative records, simulations of sensitivity to alternative procedures, and the like. The goal of the research is the development of feasible, cost-effective improved procedures for implementation. In ongoing programs for which it is disruptive to implement improve- ments on a continuing basis, a common practice is to undertake major re- search and development activities at intervals of, say, 5 or 10 years or longer. Agencies should ensure, however, that the intervals between major research and development activities do not become so long that data collection pro- grams deteriorate in quality, relevance, and efficiency over time. Regular, well-designed program evaluations, with adequate budget sup- port, are key to ensuring that data collection programs do not deteriorate. Having a set schedule for research and development efforts will enable data collection managers to ensure that the quality and usefulness of their data are maintained and help prevent the locking into place of increasingly less optimal procedures over time. Ealuating Releance In addition to quality, it is important to assess the relevance of an agency’s data collection programs. The question in this instance is whether the agency is “doing the right thing” in contrast to whether the agency is

OCR for page 14
 PART ii: COMMENTARy “doing things right.” Relevance should be assessed not only for particu- lar programs or closely related sets of programs, but also for an agency’s complete portfolio to assist it in making the best choices among program priorities given the available resources. Keeping in close touch with stakeholders and important user constituencies—through such means as regular meetings, workshops, con- ferences, and other activities—is important to ensuring relevance. Customer surveys can be helpful on some aspects of relevance, although they typically provide only gross indicators of customer satisfaction, usually with regard to timeliness and ease of use of data products. As discussed in the next section, including other federal statistical colleagues, both as users and as collabora- tors, in this communication can also be valuable. Statistical agencies commonly find that it is difficult to discontinue or scale back a particular data series, even when it has largely outlived its usefulness relative to other series, because of objections by users who have become accustomed to it. In the face of limited resources, however, discon- tinuing a series is preferable to across-the-board cuts in all programs that reduce the accuracy and usefulness of the more relevant and less relevant data series alike. Regular internal and external reviews can help an agency not only reassess its priorities, but also develop the justification and support for changes to its portfolio. Types of Reiews Regular program reviews should include a mixture of internal and ex- ternal evaluation. Agency staff should set goals and timetables for internal evaluations, which should involve staff who do not regularly work on the program under review. Independent external evaluations should also be conducted on a regular basis, the frequency of which should depend on the importance of the data and on how quickly changes in such factors as respondent behavior and data collection technology may adversely affect a program. In a world in which people and organizations appear increasingly less willing to respond to surveys, it becomes urgent to continually monitor response and have more frequent evaluations than in a more stable environ- ment. In addition to program evaluations, agencies should seek outside re- views to examine priorities and quality practices across the entire agency. External reviews can take many forms. They may include recommen- dations from advisory committees that meet at regular intervals (typically every 6 months). However, advisory committees should never be the sole

OCR for page 14
0 PRiNCiPLES AND PRACTiCES FOR A FEDERAL STATiSTiCAL AGENCy source of outside review because the members of such committees rarely have the opportunity to become deeply familiar with agency programs. Administrations often develop evaluation mechanisms (see Appendix B) that may be helpful to an agency. External reviews can also take the form of a “visiting committee” using the National Science Foundation model or academic models (see, e.g., http://www.nsf.gov/od/oia/activities/cov/covs. jsp [December 2008]); or a special committee established by a relevant pro- fessional association (see, e.g., American Statistical Association, 1984); or a study by a panel of experts (see, e.g., National Research Council, 1985a, 1985b, 1986, 1993a, 1997b, 2000b, 2000c, 2003c, 2004c, 2004d, 2008c, 2008d). Practice 11: Coordination and Cooperation with Other Statistical Agencies The U.S. federal statistical system consists of many agencies in different departments, each with its own mission. Nonetheless, statistical agencies do not and should not conduct their activities in isolation. An effective statisti- cal agency actively explores ways to work with other agencies to meet current information needs, for example, by seeking ways to integrate the designs of existing data systems to provide new or more useful data than a single system can provide. An effective agency is also alert for occasions when it can provide technical assistance to other agencies—including not only other statistical agencies, but also program agencies in its department—as well as occasions when it can receive such assistance in turn. Efforts to standard- ize concepts and definitions, such as those for industries, occupations, and race and ethnicity, further contribute to effective coordination of statistical agency endeavors, as does the development of broad macro models, such as the system of national accounts (see, e.g., National Research Council, 2004a, 2004b; also see Appendix B). Initiatives for sharing data among sta- tistical agencies (including individual data and address lists when permitted by law and when sharing does not violate confidentiality promises) can be helpful for such purposes as achieving greater efficiency in drawing samples, evaluating completeness of population coverage, and reducing duplication among statistical programs, as well as reducing respondent burden. The responsibility for coordinating statistical work in the federal gov- ernment is specifically assigned to the Office of Information and Regulatory Affairs (OIRA) in OMB by the Paperwork Reduction Act (previously, by the Federal Reports Act and the Budget and Accounting Procedures Act—

OCR for page 14
 PART ii: COMMENTARy see Appendix B). The Statistical and Science Policy Office in OIRA, often working with the assistance of interagency committees, reviews concepts of interest to more than one agency; issues standard classification systems (of industries, metropolitan areas, etc.) and oversees their periodic revision; consults with other parts of OMB on statistical budgets; and, by reviewing statistical information collections as well as the statistical programs of the government as a whole, identifies gaps in statistical data, programs that may be duplicative, and areas in which interagency cooperation might lead to greater efficiency and added utility of data. The Statistical and Science Policy Office also is responsible for coordinating U.S. participation in in- ternational statistical activities.9 The Statistical and Science Policy Office encourages the use of admin- istrative data for statistical purposes, when feasible, and works to establish common goals and norms on major statistical issues, such as confidential- ity. It sponsors and heads the interagency Federal Committee on Statistical Methodology (FCSM), which issues guidelines and recommendations on statistical issues common to a number of agencies (see Federal Committee on Statistical Methodology, 1978a-2005; for the papers from the FCSM 2007 research conference, see http://www.fcsm.gov/events/papers2007 [December 2008]). It encourages the Committee on National Statistics at the National Academies to serve as an independent adviser and reviewer of federal statistical activities. The 1995 reauthorization of the Paperwork Reduction Act created a statutory basis for the Interagency Council on Statistical Policy (ICSP), formalizing an arrangement whereby statistical agency heads participated with OMB in activities to coordinate federal statistical programs (see Appendixes A and B). There are many forms of interagency cooperation and coordination. Some efforts are multilateral, some bilateral. Many result from common interests in specific subject areas, such as economic statistics, statistics on people with disabilities, or statistics on children or the elderly. U.S. Office of Management and Budget (2008c:Ch. 3) describes several interagency collaborative efforts, such as joint support for research that fosters new and innovative approaches to surveys, expansion and improvement of the coverage and features of FedStats, which provides access to statistics from 9The Statistical and Science Policy Office was renamed from the Statistical Policy Office to reflect added responsibilities with respect to the 2001 Information Quality Act standards and guidelines, OMB’s guidance on peer review planning and implementation, and evalua- tions of science underlying proposed regulatory actions.

OCR for page 14
 PRiNCiPLES AND PRACTiCES FOR A FEDERAL STATiSTiCAL AGENCy more than 100 government agencies at http://www.fedstats.gov [December 2008], and implementation of comparable measures of disability on major household surveys. A common type of bilateral arrangement is the agreement of a program agency to provide administrative data to a statistical agency to be used as a sampling frame, a source of classification information, or a summary compilation to check (and possibly revise) preliminary sample results. The Bureau of Labor Statistics, for example, benchmarks its monthly establish- ment employment reports to data supplied by state employment security agencies. Such practices improve statistical estimates, reduce costs, and eliminate duplicate requests for information from the same respondents. In other cases, federal statistical agencies engage in cooperative data collec- tion with state counterparts to let one collection system satisfy the needs of both. A number of such joint systems have been developed, notably by the Bureau of Labor Statistics, the National Agricultural Statistics Service, the National Center for Education Statistics, and the National Center for Health Statistics. Another example of a joint arrangement is the case in which one statisti- cal agency contracts with another to conduct a survey, compile special tabula- tions, or develop models. Such arrangements make use of the special skills of the supplying agency and facilitate use of common concepts and methods. The Census Bureau conducts many surveys for other agencies, both the National Center for Health Statistics and the National Agricultural Statistics Service receive funding from other agencies in their departments to support their survey work, and the Division of Science Resources Statistics receives funding from agencies in other departments to support several of its surveys (see U.S. Office of Management and Budget, 2008c:Table 2). The major federal statistics agencies are also concerned with interna- tional comparability of statistics. Under the leadership of OMB’s Statisti- cal and Science Policy Office, they contribute to the deliberations of the United Nations Statistical Commission, the Organisation for Economic Co-operation and Development, and other international organizations, participate in the development of international standard classifications and systems, and support educational activities that promote improved statistics in developing countries. Statistical agencies also learn from and contribute to the work of established statistical agencies in other countries in such areas as survey methodology, record linkage, confidentiality protec- tion techniques, and data quality standards. Several statistical agencies run educational programs for government statisticians in developing countries.

OCR for page 14
 PART ii: COMMENTARy Some statistical agencies have long-term cooperative relationships with international groups, for example, the Bureau of Labor Statistics with the International Labor Organization, the National Agricultural Statistics Service with the Food and Agriculture Organization, the National Center for Education Statistics with the International Indicators of Education Systems project of the Organisation for Economic Co-operation and De- velopment, and the National Center for Health Statistics with the World Health Organization. To be of most value, the efforts of statistical agencies to cooperate as partners with one another should involve the full range of their activities, including definitions, concepts, measurement methods, analytical tools, dissemination modes, and disclosure limitation techniques. Such efforts should also extend to policies and professional practices, so that agencies can respond effectively and with a coordinated voice to such government-wide initiatives as data quality guidelines, privacy impact assessments, perfor- mance rating criteria, institutional review board requirements, and others. Finally, coordination efforts should encompass the development of data, especially for emerging policy issues (National Research Council, 1999a). In some cases, it may be not only more efficient, but also productive of needed new data for agencies to fully integrate the designs of existing data systems, such as when one survey provides the sampling frame for a related survey. In other instances, cooperative efforts may identify ways for agencies to improve their individual data systems so that they are more useful for a wide range of purposes. Two of the more effective continuing cooperative efforts in this regard have been the Federal Interagency Forum on Aging-Related Statistics and the Federal Interagency Forum on Child and Family Statistics. The former was established in the mid-1980s by the National Institute on Aging, in cooperation with the National Center for Health Statistics and the Census Bureau. The forum’s goals include coordinating the development and use of statistical data bases among federal agencies, identifying information gaps and data inconsistencies, and encouraging cross-national research and data collection for the aging population. The forum was reorganized in 1998 to include six new member agencies and has grown over the years to include 15 agencies. The forum develops a periodic indicators chart book, which was first published in 2000 and was most recently issued in 2008 (Federal Interagency Forum on Aging-Related Statistics, 2008). The Federal Interagency Forum on Child and Family Statistics was formalized in a 1994 executive order to foster coordination and collabora-

OCR for page 14
 PRiNCiPLES AND PRACTiCES FOR A FEDERAL STATiSTiCAL AGENCy tion in the collection and reporting of federal data on children and families. Its membership currently includes 22 statistical and program agencies. The forum’s reports (e.g., Federal Interagency Forum on Child and Family Sta- tistics, 2007, 2008) describe the condition of America’s children, including changing population and family characteristics, the environment in which children are living, and indicators of well-being in the areas of economic security, health, behavior, social environment, and education. No single agency, whether a statistical or program agency, could have produced the forum reports alone. Working together in this way, federal sta- tistical agencies contribute to presenting data in a form that is more relevant to policy concerns and to a stronger statistical system overall.