3



Experience with Public Posting of Government Data

INTRODUCTION

The concept of publicly posting government-generated data to provide direct public access to information is not new. In response to calls for increased transparency and increased provision of information, several government agencies, including regulatory agencies responsible for protecting human health and safety, regularly post detailed data on the Internet. In some cases, the data are related to individual firms or facilities; in other cases, they are specific to commodities or products or to events. Although one could argue that government data are public by default and only under special circumstances should they be restricted, the committee began its assessment from a more neutral ground and by considering potential benefits of and concerns with releasing data, as this was the task given to the committee.

This chapter briefly summarizes several examples of public posting of detailed (disaggregated or establishment-specific) data. It also reviews some of the literature on the use and effects of data releases. Currently, there are no empirical data on the effects (both positive and adverse) of releasing establishment-specific FSIS data on the Internet. Therefore, the committee reviewed the existing evidence on the benefits and costs of public release of data by other government agencies. The review is meant to be illustrative rather than exhaustive. We focus on the posting of data that stem directly from regulatory activities (Category 1) but also briefly discuss the public posting of information derived from some prior analysis of data



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 35
3 Experience with Public Posting of Government Data INTRODUCTION The concept of publicly posting government-generated data to provide direct public access to information is not new. In response to calls for increased transparency and increased provision of information, several government agencies, including regulatory agencies responsible for protecting human health and safety, regularly post detailed data on the Internet. In some cases, the data are related to individual firms or facilities; in other cases, they are specific to commodities or products or to events. Although one could argue that government data are public by default and only under special circumstances should they be restricted, the committee began its assessment from a more neutral ground and by considering potential benefits of and concerns with releasing data, as this was the task given to the committee. This chapter briefly summarizes several examples of public posting of detailed (disaggregated or establishment-specific) data. It also reviews some of the literature on the use and effects of data releases. Currently, there are no empirical data on the effects (both positive and adverse) of releasing establishment-specific FSIS data on the Internet. Therefore, the committee reviewed the existing evidence on the benefits and costs of public release of data by other government agencies.   The review is meant to be illustrative rather than exhaustive. We focus on the posting of data that stem directly from regulatory activities (Category 1) but also briefly discuss the public posting of information derived from some prior analysis of data 35

OCR for page 35
(Category 2) and the posting of voluntarily provided data (Category 3). 20 In addition to the examples discussed in this chapter, there are numerous other examples of public release by government agencies of safety-related data on products or firms; some of these are briefly summarized in Box 3-1. Box 3-1 Examples of Sharing of Safety-Related Data on the Internet Airborne Contaminants. The Occupational Safety and Health Administration posts some of its compliance-monitoring information on airborne contaminants released from personal, area, and bulk samples in industrial sites. URL: http://www.osha.gov/opengov/healthsamples.html. Hospital Measures of Outcome of Care. Medicare publishes hospital-specific rates of outcome of care, which indicate what happened after patients with particular conditions were treated in the hospital. URL: http://data.medicare.gov/dataset/Hospital-Outcome-Of-Care-Measures/f24z-mvb9. Safety in the Transportation Industry. The Bureau of Transportation Statistics publishes multiple datasets on transportation accidents and exposure to safety risks (for example, measured in aviation incidents, accidents, or fatalities). URL: http://www.bts.gov/programs/safety/index.html. Safety of Nuclear Plants. The Nuclear Regulatory Commission posts plant-specific safety-inspection reports and licensees’ performance indicators. URL: http://www.nrc.gov/NRR/OVERSIGHT/ASSESS/index.html. Safety of Motor Vehicles and Equipment. The National Highway Traffic Safety Administration posts data on safety for the consumer, such as ratings of cars and tires (http://www.safercar.gov/Vehicle+Shoppers/5-Star+Safety+Ratings/2011-Newer+Vehicles and http://www.safercar.gov/Vehicle+Shoppers/Tires/Tires+Rating) and children’s car seats (http://www.nhtsa.gov/Safety/Ease-of-Use); a list of all vehicle, equipment, and tire safety-recall campaigns from 1966 to the present (http://www-odi.nhtsa.dot.gov/recalls/); consumer complaints related to the safety of motor vehicles and motor-vehicle equipment (http://www- odi.nhtsa.dot.gov/complaints/); and investigations of specific vehicles, tires, and equipment (http://www- odi.nhtsa.dot.gov/cars/problems/defect/defectsearch.cfm). 20 As described in Chapter 2, Category 1 data arise from the activities of agencies as part of their normal enforcement and compliance efforts. Category 2 data arise from the outcomes of enforcement and compliance efforts that have been interpreted by others for use by end users. Category 3 data are collected by agencies from voluntary programs not in conjunction with normal enforcement and compliance efforts but nonetheless intended to provide information.   36

OCR for page 35
EXAMPLES OF COLLECTION AND RELEASE OF DATA BY REGULATORY AGENCIES US Department of Labor As part of the broader Open Government initiatives of the Obama administration, various agencies of the US Department of Labor (DOL) have expanded direct public access to their inspection and enforcement data, which are posted on a comprehensive Web site. 21 The data underlying the site arise primarily from the enforcement activities of the agencies. Each agency offers different types of information and levels of detail to the public, which reflect differences in agency mission, nature of the regulatory process, sophistication of data collection, and administrative processes, such as case-review procedures. 22 The agencies provide a variety of information, including details about the inspected entity (such as industry, firm and establishment size, and single-plant vs. multiplant status), characteristics (such as time spent and type of inspection activity) and outcomes of the investigation (such as standards violated, severity of violations, and penalties assessed), and related administrative processes (appeals and their results). Accordingly, the data on the site are primarily in Category 1. The site is regularly expanded and improved. Prior updates have focused on making it easier for users to search by common criteria, such as company name or industry grouping. That potentially provides information about the compliance behavior of a specific employer or industry for a range of workplace laws. DOL is planning a number of future updates to increase usability, including display of data through maps and interactive “dashboards” and engaging public users of the data in finding “innovative ways of using DOL’s enforcement data to promote worker’s safety and protect worker’s rights”. 23 In addition to the information on the comprehensive DOL Web site, some of the individual agencies in DOL post detailed facility-specific safety data. For purposes of illustration, we focus here on the Mine Safety and Health Administration (MSHA). MSHA is responsible for the enforcement of health and safety standards for underground and surface metal and nonmetal mines. Compliance with detailed health and safety requirements is determined through physical inspection of mining facilities, interviews with mine operators and with workers and their representatives (in unionized mines), and review of administrative information. Inspectors also sample dust and air. The most extensive mine-level data available to the public are published on the MSHA Web site. 24 Those data originate in the electronic information systems maintained by the agency. The data are stored in 16 linked databases that provide information on inspections, citations, penalties, and abatement requirements. The site also provides mine-level data on fatalities and 21 See http://ogesdw.dol.gov/ (accessed June 7, 2011). 22 With respect to the latter dimension, agencies vary according to when the results of completed inspections and investigations are publicly posted. The Wage and Hour Division posts only cases that are considered “closed” (for example, all appeals of the investigators’ findings have been resolved). In contrast, the Mine Safety and Health Administration posts inspection data even when a mine operator or other party is appealing parts of a decision, such as penalties. 23 See http://ogesdw.dol.gov/coming_soon (accessed June 7, 2011). 24 See http://www.msha.gov/drs/drshome.htm (accessed June 7, 2011). 37

OCR for page 35
injuries, air sampling results, such mine-level characteristics as geology and type of mining technology, and detailed information on ownership and management of mining activities. The publicly available data span from 1983 to the present and are updated weekly. MSHA also provides information on closed and active inspections, including cases in which mine operators have contested penalties or abatement orders. Data can be searched by any of the characteristics of mine operation, ownership, inspection finding, and so on; the data can be downloaded as extracts; and data from the various databases can be combined by using a common mine-level identifier system. US Environmental Protection Agency In 2003, the US Environmental Protection Agency (EPA) launched its Enforcement and Compliance History Online (ECHO), 25 a Web-based platform that provides easy access to EPA and state data on environmental compliance and performance of over 800,000 individual facilities in the United States. The Web interface draws on an underlying dataset, the Integrated Data for Enforcement Analysis (IDEA), which integrates data from five enforcement and compliance history datasets. Users can use ECHO to search for facility-specific data by location (ZIP code) or other identifiers. It is designed primarily for situations in which a user is interested in information on a relatively small number of facilities, but users who want to review larger amounts of data can access the raw data from IDEA. The data are updated monthly. The content of ECHO and its user capabilities have evolved. ECHO now allows Web- based access to the following types of facility-specific data:  Inspection, violation, and enforcement data, including the number and dates of individual inspections, compliance status by quarter, and penalties imposed during the preceding 5 years.  Data on EPA enforcement cases.  Data on violations of the Safe Drinking Water Act, including the publication of a list of all water suppliers deemed to be “serious violators”.  Toxics Release Inventory (TRI) data, which are mandatory, self-reported releases of designated toxic chemicals by facility.  National Emissions Inventory data with information about estimates of air pollutants from point, nonpoint, and mobile sources in the United States, for example, data from state and local agencies, data on on-road sources from the Federal Highway Administration, and fuel-use data from the Department of Energy.  Detailed water-quality reports on facilities that have permits under the Clean Water Act and information on noncompliance with effluent limits. In addition to detailed facility-level data, ECHO includes more aggregated summary reports that provide information about trends and state-level analyses. 25 See http://www.epa-echo.gov/echo/about_data.html (accessed June 8, 2011). 38

OCR for page 35
EPA reports that in its first year ECHO provided information in response to over a million search requests. 26 It identifies members of the public, corporations, investors, and researchers as possible user groups. In addition, EPA notes that provision of data to the public creates an incentive for government agencies to improve the reporting of violations and for facilities to take steps to correct violations. Although EPA has worked continuously to enhance the usefulness of the data in ECHO, some of the reported data are in “raw” form and can be difficult for users to interpret. For example, the TRI data are reported in pounds released annually with no direct means of converting the releases to a more useful measure, such as associated health risk. Efforts have been made to convey health risk to end users, but the current information is not easy to find and is not detailed and quantitative enough for end users to use to estimate the risk to which a person might be exposed. Data disclosure, however, is likely to evolve and improve once shortcomings are identified. Food and Drug Administration The Food and Drug Administration (FDA) has several databases that are available to the public. They include data on inspections and enforcement (Category 1) and voluntarily reported information on actual or potential adverse events (Category 3). FDA also collects some microbiological sampling and testing data, but these are not generally available to the public. Although FDA has posted summary data for many years, it announced in May 2011 that it would disclose additional inspection information on FDA-regulated food products, including the compliance status of specific firms as determined by FDA inspectors during inspections and followup reviews for compliance. FDA made that information available to the public through a searchable database on the its Web site, which includes the names and addresses of inspected facilities, the dates of inspection, the types of FDA-regulated products involved, final inspection classification, and a summary of the most common inspection observations, although not in great detail. 27 The information is substantially equivalent to the information now provided by the US Department of Agriculture (USDA) Food Safety and Inspection Service (FSIS) on administrative actions (USDA FSIS, 2010; see Appendix C). FDA also provides access through its Web site 28 to facility-specific information, such as letters that warn firms that violations have been identified and must be corrected, and enforcement reports that contain information on actions, such as recalls, taken in connection with regulatory activities. Aggregated information about enforcement activities is also found on the FDA Web site. FDA justified the disclosure of the information on the grounds that it would help to provide the public with a rationale for the agency’s enforcement actions, help consumers and industry stakeholders to make informed choices in the marketplace, encourage industry compliance, and generally improve transparency of agency actions to be consistent with administration policies. In addition to Category 1 enforcement and compliance data, FDA collects other safety- related data from health-care professionals, public-health officials, consumers, and the food 26 See http://yosemite.epa.gov/opa/admpress.nsf/b1ab9f485b098972852562e7004dc686/e6bf84f19616f3b985256de30055a fcd?OpenDocument (accessed June 8, 2011). 27 See www.accessdata.fda.gov/scripts/inspsearch/ (accessed July 25, 2011). 28 See http://www.fda.gov/AboutFDA/Transparency/TransparencyInitiative/ucm254426.htm (accessed July 25, 2011). 39

OCR for page 35
industry. 29 For example, information on potential or actual adverse events is collected by FDA through the Reportable Food Registry (RFR) 30 for foods and through the Adverse Event Reporting System (AERS) 31 for drugs, biologics, and dietary supplements. Industry must report adverse events to the RFR. Raw data from the RFR are not released to the public, but FDA has posted two reports since RFR was implemented: a first-7-months report (September 2009–March 2010) and the annual report (September 2009–September 2010) with summary information aggregated in various forms, such as total entries by commodity or by commodity and hazard. FDA also posts quarterly data files from AERS on its Web site and summary statistics for each year. Although the information is not company-specific or facility-specific, the release of information about adverse events can affect individual firms and entire industries whose production is linked in some way to the events. State and Local Public-Health Agencies Regulation of restaurant hygiene falls under the jurisdiction of public-health officials in state, county, or city governments. In particular, local governments establish and implement food-safety standards for institutional food-service establishments, restaurants, retail food stores, and other retail food establishments; FDA, through its issuance of the Food Code, advises them on food-safety guidelines (FDA, 1993;,1997;, 2001; 2005; 2009), inspector training, and foodborne-illness risk factors (FDA, 2000; 2004; 2009). Those regulatory activities play a critical role in ensuring food safety. 32 29 FDA collaborates with the National Institutes of Health (NIH) to administer a database of federally and privately supported clinical trials at ClinicalTrials.gov. The database contains 108,486 trials sponsored by NIH, other federal agencies, and private industry. Studies listed in the database are conducted in all 50 states and in 174 countries. Users can access information on current clinical trials, including participant flow, baseline characteristics, outcome measures and statistical analyses, adverse-events information, administrative information, and study results when available. 30 The RFR is a new database administered by FDA. Required by Congress, it is an electronic portal for the food industry and public-health officials to report when there is a reasonable probability that an article of food will cause serious adverse health consequences. URL: www.fda.gov/Food/FoodSafety/FoodSafetyPrograms/RFR/default.htm (accessed July 25, 2011). 31 AERS contains over 4 million reports of adverse events from 1969 to the present. 32 Restaurant hygiene has been linked to foodborne disease, so restaurant inspection has been studied as a tool to reduce the occurrence outbreaks. For example, FDA (2000) checked 895 food establishments across the United States and found that restaurants and retail store delicatessens were in compliance with the five risk factors emphasized in the 1997 FDA Food Code only 60–74% of the time. In comparison, the average compliance record for other food establishments (hospitals, nursing homes, elementary schools, and other departments of retail food stores) ranged from 76 to 83%. Two followup reports (FDA, 2004, 2009) indicated that some of the risk factors identified in the 2000 report (such as improper food temperature, poor personal hygiene, and contaminated equipment) remained in need of attention despite some improvement (FDA, 2010). In 2007, the Center for Science in the Public Interest examined over 530 inspection reports in 20 cities and found that over 66% of restaurants had at least one high-risk violation (CSPI, 2008). Those statistics suggest that restaurant hygiene is an important contributor to outbreaks of foodborne disease. 40

OCR for page 35
Public access to restaurant hygiene-inspection outcomes (Category 1 data) varies greatly among regions and over time. The most traditional way to share data is “available on request”. In some cities (such as Pittsburgh and Washington, DC, before 2011), inspection outcomes are available only through the Freedom of Information Act (FOIA), which requires written requests and can take up to 6 months for receipt of a final report (CSPI, 2008). In other places (such as Atlanta and San Francisco), restaurants are required to keep copies of the most recent inspection reports and provide them on request by consumers. Alternatively, disaggregated inspection outcomes can be posted on an on-line searchable database; access to these data requires consumers to initiate an on-line search. Many states and large cities—including Virginia, Florida, Boston, Chicago, Denver, Philadelphia, and Washington, DC—have adopted on-line posting. Several jurisdictions have recently adopted methods that help to deliver restaurant hygiene-inspection results directly to consumers at the point of sale—the front door or window of the restaurant. For example, North Carolina, South Carolina, Tennessee, Las Vegas, St. Louis, Los Angeles, New York City, and some international cities, such as Beijing and Toronto, require storefront posting of hygiene information. The information can be in the form of a numerical score of the most recent inspection (usually of a total of 100 points) or broad categories (A–B–C or pass–conditional pass–closed) based on the numerical scores. The regulatory agency or some other body must define how the raw inspection results or scores will be used to define the relevant categories (for example, Category 2, for which the government agency interprets the disaggregated hygiene data and provides them in a more actionable form for consumers). REPORTING OF FOOD-SAFETY DATA BY NONREGULATORY AGENCIES As noted above, although many of the detailed data related to food safety are collected and reported by regulatory agencies as they engage in normal compliance and enforcement- related activities (Category 1 data) or are interpreted for consumers (Category 2 data), some agencies, such as the Centers for Disease Control and Prevention (CDC) and the USDA Agricultural Marketing Service (AMS), collect and report food-safety data that are generally reported to them by others, including consumers and health-care professionals. Because reporting is often voluntary, those data would typically fall into Category 3. The CDC and AMS data are not linked to individual firms or facilities and thereby differ from FSIS data. They have the potential to be of benefit to the public but can also affect related firms and industries. Centers for Disease Control and Prevention Much of the public-health surveillance for foodborne disease, including outbreaks, is conducted by state and local health departments. For multistate foodborne-illness outbreak investigations, CDC plays a prominent role in surveillance and investigation. CDC does not have authority to mandate that states report their surveillance data to it, but it has developed a system whereby state and local health departments voluntarily report outbreak data to it. CDC maintains aggregated and case-based disaggregated foodborne-illness surveillance data in multiple databases. No personal identifiers are maintained by CDC. Some of the data are publicly accessible, and others are available only through FOIA. 41

OCR for page 35
In 2009, CDC launched the Foodborne Outbreak Online Database (FOOD), 33 which is designed to allow the public direct access to state-level information on foodborne outbreaks. The database spans 1998–2008 and is updated periodically. FOOD enables the public to search and download data on reported outbreaks as an XML file. It does not identify specific establishments involved in outbreaks. A recent report suggested that the FOOD data have several limitations. 34 For example, state health departments may update the data at any time, so these entries are never considered “final”. The rigor with which state health departments collect and report data can vary widely and some users of the data have noted inconsistencies in the dataset. In addition, data are not updated in real time, so the most current data available are usually several years old. CDC also collects surveillance data through the Foodborne Diseases Active Surveillance Network (FoodNet) program. FoodNet is a partnership of 10 state and local health departments, CDC, FDA, and USDA that conducts population-based surveillance for laboratory-confirmed infections commonly implicated in foodborne disease. CDC releases annual summaries of FoodNet data in a published report but does not make the raw data publicly available, although they can be requested through FOIA. Agricultural Marketing Service The Monitoring Programs Office of USDA’s AMS is responsible for managing the Pesticide Data Program (PDP), 35 a voluntary program that was implemented in 1991 to test food commodities for pesticide residues. The PDP is based on a sampling plan with a rigorous statistical design to ensure the reliability of the data for use in exposure assessments. However, the pesticide databases are commodity-specific rather than establishment-specific. Every year, the AMS publishes on line a report summarizing the PDP data. It includes the study design for data collection for the relevant year, details about how data are reported, a summary of the results, the history of commodities tested, and the raw commodity-level data. Requests for PDP information are received from many parties, including other government agencies and various organizations, and the staff of the Monitoring Programs Office generates specific reports for these queries. However, users can also import the data into database- management software and conduct their own analyses. The PDP was not designed for the purpose of enforcing regulations. The major user of the PDP data is EPA, which uses them in its pesticide risk assessments and to estimate whether human exposure exceeds safety standards. 36 FDA is informed of residues that exceed tolerances 33 See http://wwwn.cdc.gov/foodborneoutbreaks/ (accessed June 8, 2011). 34 See http://www.aei.org/docLib/REG-2011-02-g.pdf (accessed June 8, 2011). 35 See http://www.ams.usda.gov/AMSv1.0/ams.fetchTemplateData.do?template=TemplateC&navID=PDPOviewBox2Link 1&rightNav1=PDPOviewBox2Link1&topNav=&leftNav=ScienceandLaboratories&page=PesticideDataProgram&r esultType=&acct=pestcddataprg (accessed on June 23, 2011). 36 Other groups also use the data to provide information to the lay public about pesticide residues in foods (see, for example, http://www.ewg.org/foodnews/), and researchers have conducted analyses with the PDP residue data (see, for example, Punzi et al., 2005; Baker et al., 2002; Kuchler et al., 1996). 42

OCR for page 35
or that have no tolerances. The program is voluntary (AMS has no regulatory authority to require participation in the program) and works with 12 state agencies that are responsible for sample collection and analysis. Five states were selected initially because they were diverse geographic areas; the list was later expanded to 12 states to increase the sampling data points (M. Lamont, USDA AMS, Manassas, VA, personal communication, July 20, 2011). REPORTED EFFECTS OF RELEASING ESTABLISHMENT-SPECIFIC DATA The overview above suggests that other government agencies have already had considerable experience with the release of detailed data. The academic literature also has examined the pros and cons of information disclosure in many contexts, including disclosure of establishment-specific regulatory information similar to the FSIS data and disclosure of product- specific information that may be traced back to manufacturers. The committee reviewed the many National Research Council and Institute of Medicine reports on data-sharing (for example, NRC, 1985; NRC and the Social Science Research Council, 1993; IOM, 1996; NRC, 2000; 2001, 2005; NAS, 2009; IOM and NRC, 2010). The reports have a somewhat different focus, and none addresses directly the issue of publicly releasing data gathered originally for regulatory purposes; for example, NRC (1985) focuses on data-sharing among researchers. However, many of the issues associated with data-sharing in other settings, as discussed in these National Research Council reports and related documents, do address benefits and concerns related to the process and point to conclusions that are similar to those discussed here and in Chapter 4. For example, the report Sharing Research Data (NRC, 1985) considers the benefits and costs of data-sharing among researchers. The noted benefits include promoting and improving research that leads to better decisions and improving measurement and data-collection methods. The costs include technical obstacles to sharing data and the costs of documentation and training. The entities that own and disclose data may go beyond regulatory agencies (to manufacturers and third-party certifiers), but lessons learned from the broader literature can help us to anticipate the specific potential effects of releasing establishment-specific FSIS data. As discussed in Chapter 2, the effectiveness of any transparency system is based on whether the information that it provides is “embedded” in the action cycle of users (such as consumers) and disclosers (such as businesses) of the information (Fung et al., 2007). On the positive side, the literature suggests that information disclosure may enable consumers to make more informed choices. Analysis of specific policies yields numerous examples of information affecting consumer choice. The public posting of hygiene inspection outcomes has resulted in increased sensitivity to restaurant hygiene (Jin and Leslie, 2003). In the health-care domain, substantial patient sorting has been observed in response to the disclosure of cardiac-surgery outcomes associated with hospitals and doctors (Dranove et al., 2003). Increased consumption of fiber-rich cereals has occurred on disclosure of the nutrition content of food products (Ippolito and Mathios, 1990). In addition, there is evidence that disclosure of firm-specific or facility-specific information can motivate firms to improve their performance, at least along the disclosed dimensions. For example, evidence shows that public posting of restaurant hygiene information led to better public-health outcomes in Los Angeles and Toronto (Jin and Leslie, 2003; Simon et al., 2005; Serapiglia et al., 2007). EPA TRI disclosure led to substantial improvements in environmental performance (Konar and Cohen, 1997). Likewise, for large Massachusetts water 43

OCR for page 35
suppliers, the mandatory public provision of information about violations of drinking-water standards resulted in a 30–44% reduction in total violations and a 40–57% reduction in more severe health violations (Bennear and Olmstead, 2008). A recent study of state voluntary site- cleanup programs revealed that public disclosure of contaminated sites is an efficient tool for promoting participation of property managers and developers in site remediation (Blackman et al., 2010). Patten (2002) further argues that release of establishment-level TRI data generated public-policy pressure that led to increased environmental disclosure by TRI firms. Those improvements can be in response to consumer pressures of the type discussed in the previous paragraph, pressures from input markets (such as investors and suppliers), and actual or threatened regulatory pressures (Fung et al., 2007). Release of establishment-level data has also generated research opportunities. For example, researchers have used establishment-level enforcement data to examine the effectiveness of inspection programs in a variety of contexts, including mine safety (Kniesner and Leeth, 2004), occupational safety (Bartel and Thomas, 1985; Gray and Jones, 1991; Weil, 1996; 2001), nuclear safety (Feinstein, 1989), seafood safety (Alberini et al., 2008), air and water pollution (Magat and Viscusi, 1990; Gray and Deily, 1996; Earnhart, 2004), and pharmaceutical production (Macher et al., 2011). Establishment-level data have also been used to study issues not directly related to enforcement, such as the link between air pollution and fetal or infant mortality (Agarwal et al., 2010), the effectiveness of nonregulatory programs (e.g., Arora and Cason, 1996), environmental justice (Daniels and Friedman, 1999; Dolinoy and Miranda, 2004), interjurisdictional pollution effects (Helland and Whitford, 2003), and the effects of physician prescription of drug combinations on competition among pharmaceutical firms (Lucarelli et al. 2010). The evidence reviewed suggests that public disclosure of establishment-specific data can have important social benefits. However, as with all regulatory interventions, some parties may be adversely affected by public data disclosure. Different parties have different perspectives on what constitutes an adverse effect. In fact, a negative for one party might be viewed as a positive by another or ultimately considered as a positive by the public at large. One potential adverse effect is related to the market. For example, a body of literature demonstrates that some firms suffered reductions in their stock prices immediately after public release of data (e.g., Hamilton, 1995; 2005). There are similar examples of the effect of food recalls on stock prices (e.g., Salin and Hooker, 2001; Thomsem and McKenzie, 2001). Konar and Cohen (1997) reported that firms more adversely affected by the release of TRI data were later more likely to reduce their toxic releases; this suggests that the adverse effects of the data release motivated firms to improve their performance (in response to pressure from investors in capital markets). Thus, the adverse effect on some firms may ultimately generate benefits for the broader community. Some researchers have raised concerns about potential adverse effects of disclosure due to misinterpretation or lack of understanding of the data. If that occurs, the disclosure might not have the intended effect. For example, the terrorist color-coded threat advisory system that was enacted shortly after the 9/11 attacks and was in effect until early 2011 provided vague information that tended to cause confusion, alarm, or eventually disregard by the public but little evidence of reduction of risk to the public (Fung et al., 2007). The implication, however, is not that the data should not be released but rather that the data should be provided in a meaningful and understandable form. Another concern raised in the literature is that information disclosure may encourage firms to improve on the reported outcomes but reduce performance regarding unreported 44

OCR for page 35
outcomes, especially when the omitted outcomes are unreported because of measurement difficulty rather than because of lack of importance. This type of distortionary behavior has been documented in a number of contexts. For example, Khanna et al. (1998) showed that release of TRI information reduced on-site releases of toxic substances but increased transfers of the same substances to off-site locations. As a result, in that particular case, the authors concluded that the overall effect on the amount of toxic waste generated was negligible. Similar examples can be found in the context of medical outcomes (Dranove et al., 2003) and school performance (Haney, 2000; Deere and Strayer, 2001; Jacob and Levitt, 2003; Hanushek and Raymond, 2005; Jacob, 2005; Cullen and Reback, 2006; Figlio and Getzler, 2006). The potential for distortionary behavior suggests that agencies contemplating public data disclosure should anticipate such responses by firms and design information collection and disclosure policies that will reduce that kind of behavior. Information disclosure not only has the potential to distort firm behavior but can add pressure on the people (such as inspectors) who generate data in the field. On the one hand, inspectors may be under closer scrutiny and thus pressured to do their jobs in a more precise and consistent way. For example, there is evidence that increased public attention after the Three Mile Island accident increased inspector detection rates (Feinstein, 1989); this suggests that public attention on inspection may motivate inspectors to do a better job. On the other hand, firms identifiable in the disclosure data have incentives to ask for leniency of the inspectors who are assigned to their facilities. Anecdotal evidence has shown inspector bribery after Los Angeles County adopted restaurant hygiene report cards, and data plots raise concern about leniency regarding the cutoffs of letter grades (90 for A and 80 for B, Jin and Leslie, 2003; 2005). The concern about inspector bias and potential bribery brings up issues regarding heterogeneity in inspector performance. In nuclear-safety inspection (Feinstein, 1989) and inspection of pharmaceutical manufacturing plants (Macher et al., 2011), researchers have found that inspector identity or inspector demographics, experience, or training is important in explaining inspection outcomes. That suggests that inspectors vary in their ability to detect or their preference in detecting violations. Although the committee is not aware of any published study that documents the effect of data disclosure on inspector behavior, public attention after data disclosure may highlight the existence of inspector heterogeneity and motivate the provision of additional training and standardization to enhance inspection consistency. Ironically, distorted firm and inspector behavior that occurs as a consequence of information disclosure suggests that the disclosed data are useful at least in the perception of primary data users. That highlights the importance of what to disclose, how to disclose it (including how to protect the identities of individual inspectors), and what additional support might be needed from FSIS to facilitate proper data use. The experience of reporting hospital outcomes may be informative. Given the complexity of raw data and consumer demand for easy- to-read information, hospital outcomes are often reported in averages. However, the precision of an average measure varies greatly with sample size, frequency of the measured event, and the pool of subjects that contribute to the sample. Researchers have shown that measurement problems can compromise the usefulness of disclosed data (Iezzoni, 1997; Kane and Staiger, 2002), but disclosure brings measurement issues to the forefront and thereby promotes research that can lead to improvements. Restaurant hygiene report cards provide another useful lesson: that many factors contribute to the effectiveness of data disclosure. For example, in addition to issuing grade cards, Los Angeles County adopted an easy-to-read format for the grade cards, inspected some 45

OCR for page 35
restaurants more frequently, provided restaurant inspectors with additional training, and put in more effort to educate restaurant owners and staff (LADPH, 2008). Several studies have found that sanitary conditions in restaurants could be improved through more frequent inspections and enhanced education efforts (Bader et al., 1978; Mathias et al., 1995; Cotterchio et al., 1998; Allwood et al., 1999; Cates et al., 2009; Hislop and Shaw, 2009). The Los Angeles restaurant hygiene report cards were motivated by a CBS 2 news program that revealed, through the use of hidden cameras, the unsanitary conditions in restaurant kitchens. That TV exposé 37 increased consumer awareness of restaurant hygiene, drew attention to the weakness of the existing system, and intensified political pressure for regulatory change. Another factor that potentially contributed to the success of Los Angeles grade cards is that the stringent inspection codes matched specific violations to defined numerical point deductions, which minimized the subjectivity of hygiene inspections. This system contributed to a more standardized evaluation among restaurants and inspectors, and increased consumer confidence in the grade cards. On-line posting has become a new norm for data disclosure because of its low cost and the ease of user access. Research is needed to examine the advantages and disadvantages of on- line posting relative to those of other methods of disclosure, such as posted restaurant report cards or published hospital rankings. The Internet facilitates posting of large amounts of data and allows user customization, but access to the Internet is probably skewed toward a set of the population (those of higher income and those who are better educated) and often requires expertise and effort by end users to analyze and interpret the data correctly. 38 However, the investment of time and expertise required to analyze and interpret large datasets appropriately is not peculiar to their release on the Internet, and the costs and knowledge necessary to obtain the same data through FOIA requests are potentially even greater barriers to the dissemination of the information. At a minimum, posting data on the Internet would make it easier for the public to know what kinds of information have been collected and are available and perhaps to gain some initial understanding of the quality, complexity, and potential usability of the data for specific purposes. Hence, the public may avoid the current costs of obtaining data through a FOIA request that would ultimately be unsuitable for their needs. However, posting on the internet may increase the potential for misinterpretation, if only by virtue of the fact that releasing data more broadly (via the Internet) will result in a higher number of users and uses. The experience of other federal agencies in posting data suggests the benefit of providing that information in formats and with 37 Behind the Kitchen Door: Joel Grover Investigation. First Broadcast November 1997. 38 The original discussion of the "digital divide" arose as a result of a survey conducted by the National Telecommunication and Information Administration (NTIA) of the US Department of Commerce in 1994 that showed differences in use of the emerging Internet by income and demographic characteristics. A second survey by NTIA in 1998 (with the subtitle "Defining the Digital Divide") provided further evidence of gaps in use. The surveys precipitated studies in the United States, the United Kingdom, and elsewhere that look into the causes and consequences of differential use (see, for example, Norris, P. 2001, Digital Divide, Civic Engagement, Information Poverty, and the Internet Worldwide. Cambridge, UK: Cambridge University Press). How much the gap has narrowed in recent years with respect to income, ethnicity, and age is controversial. 46

OCR for page 35
documentation that facilitate its analysis and interpretation. The committee believes that FSIS would be best suited to determine how to address misinterpretation of data on a case-by-case basis. SUMMARY A number of federal agencies (none with specific food-safety jurisdiction) release detailed data that are directly linked to the performance of individual facilities or firms or to the products that they produce. In many cases, the data originate in regulatory (compliance and enforcement) activities. A substantial body of literature documents the effects of the public release of data and their uses. The literature suggests that release of facility-specific performance data can have both benefits and costs (or unintended adverse consequences). Major benefits include enabling users to make more informed choices, motivating facilities to improve their performance, and provision of data for use in research studies of regulatory effectiveness and other performance-related issues. The possible costs of public disclosure of information include adverse effects on profitability, but it is precisely this possibility that creates an incentive for facilities to improve their performance. The literature has also raised concerns about some perhaps unintended consequences, including the potential for data misinterpretation, the incentive for establishments whose data are disclosed to “game the system”, 39 and potential pressure on inspector performance. Based on its review of the entirety of the extant literature, the committee concluded that the potential adverse impacts, while possible, were largely anecdotal or speculative, and are not backed up by any significant systematic evidence. On the other hand, the positive benefits are more credibly backed up by the scientific literature. Therefore, the current evidence of adverse effects is insufficient for predicting specific problems that would be inherent in the release of establishment-specific FSIS data. The committee believes that the potential for adverse effects is not necessarily insurmountable but highlights the need to pay careful attention to the design of an information-disclosure strategy. For example, potential adverse effects may be minimized if the disclosing entity (FSIS) is careful to ensure the integrity of the data and provides precise and appropriate definitions of what is being quantified, adequate documentation of context, a means by which to support analysis of the data by users, and precautionary measures to prevent the linking of portions of the data in ways that would allow users to deduce confidential information about particular establishments. It is clear that the most effective disclosure systems improve in quality, quantity, and scope as users gain a better understanding of how the data might be used. REFERENCES Agarwal, N., Banternghansa, C., and L.T.M Bui. 2010. Toxic exposure in America: Estimating fetal and infant health outcomes from 14 years of TRI reporting. Journal of Health Economics 29(4):557-74. Alberini, A., Lichtenberg, E., Mancini, D., and G. Galinato. 2008. Was it something I ate? Implementation of the FDA Seafood HACCP Program. American Journal of Agricultural Economics 90(1):28-41. 39 To adjust their strategies to minimize public disclosure. 47

OCR for page 35
Allwood, P.B., Lee, P., and P. Borden-Glass. 1999. The vital role of restaurant health inspections. Journal of Environmental Health 61(9):25-28. Arora, S., and T. N. Cason. 1996. Why do firms volunteer to exceed environmental regulations? Understanding participation in EPA's 33/50 program. Land Economics 72(4):413-32. Bader, M., Blonder, E., Henriksen, J., and W. Strong. 1978. A study of food service establishment sanitation inspection frequency. American Journal of Public Health 68(4):408-410. Baker, B.P., Benbrook, C.M., Groth, E. III, and K. Lutz Benbrook. 2002. Pesticide residues in conventional, integrated pest management (IPM)-grown and organic foods: insights from three US data sets. Food Additives and Contaminants 19(5):427-446. Behind the Kitchen Door: Joel Grover Investigation. First Broadcast November 1997. Blackman, A., Darley, S., Lyon, T.P., and K. Wernstedt. 2010. What Drives Participation in State Voluntary Cleanup Programs? Evidence from Oregon. Land Economics, 86(4): 785-799. Bartel, A.P., and L.G.Thomas. 1985. Direct and indirect effects of regulation: New look at OSHA’s impact. Journal of Law and Economics 28:1-25. Bennear, L.S., and S.M. Olmstead. 2008. The impacts of the 'right to know': Information disclosure and the violation of drinking water standards. Journal of Environmental Economics and Management. September 56(2):117-30. Cates, S.C., Muth, M.K., Karns, S.A., Penne, M.A., Stone, C.N., Harrison, J. E., and V. J. Radke. 2009. Certified kitchen managers: Do they improve restaurant inspection outcomes? Journal of Food Protection 72(2):384-391. Center for Science in the Public Interest (CSPI). 2008. Dirty Dining: Have Reservations? You Will Now. Available at www.cspinet.org/new/pdf/ddreport.pdf. Accessed June 8, 2011. Cotterchio, M., Gunn, J., Coffill, T., Tormey, P., and A.M. Barry. 1998. Effect of a manager training program on sanitary conditions in restaurants. Public Health Reports 113:353- 358. Cullen, J.B., and R. Reback. 2006. Tinkering toward accolades: school gaming under a performance accountability system. Pp. 1-34 in Improving School Accountability: Check-Ups or Choice, Advances in Applied Microeconomics 14, T. Gronberg and D. Jansen, eds. Amsterdam: Elsevier Science. Daniels, G., and S. Friedman. 1995. Spatial inequality and the distribution of industrial toxic releases: Evidence from the 1990 TRI. Social Science Quarterly 80(2):245-261. Deere, D., and W. Strayer. 2001. Putting Schools to the Test: School Accountability, Incentives and Behavior. Working paper 114. Department of Economics, Texas A&M University. Dolinoy, D.C. and M.L Miranda. 2004. GIS Modeling of air toxics releases from TRI-Reporting and Non-TRI-Reporting facilities: Impacts for environmental justice. Environmental Health Perspectives 112(17):1717-1724. Dranove, D., Kessler, D., McClellan, M., and M. Satterthwaite. 2003. Is more information better? The effects of "report cards" on health care providers. The Journal of Political Economy 111:555-588. Earnhart, D. 2004. Panel data analysis of regulatory factors shaping environmental performance. Review of Economics and Statistics 86(1):391-401. FDA (Food and Drug Administration). Food Code. 1993. U.S. Public Health Service, U.S. Dept. of Commerce. Technology Administration, National Technical Information Service. Pub. No. PB94-113941AS. Springfield, VA. 48

OCR for page 35
FDA. 1997. Food Code 1997. Available at http://www.fda.gov/Food/FoodSafety/RetailFoodProtection/FoodCode/FoodCode1997/de fault.htm. Accessed June 8, 2011. FDA. 2000. Report of the FDA Retail Food Program Database of Foodborne Illness Risk Factors. Available at http://www.fda.gov/Food/FoodSafety/RetailFoodProtection/FoodborneIllnessandRiskFac torReduction/RetailFoodRiskFactorStudies/ucm123544.htm. Accessed June 8, 2011. FDA. 2001. Food Code 2001. Available at http://www.fda.gov/Food/FoodSafety/RetailFoodProtection/FoodCode/FoodCode2001/de fault.htm. Accessed June 8, 2011. FDA. 2004. FDA Report on the Occurrence of Foodborne Illness Risk Factors in Selected Institutional Foodservice, Restaurant, and Retail Food Store Facility Types (2004). Available at http://www.fda.gov/Food/FoodSafety/RetailFoodProtection/FoodborneIllnessandRiskFac torReduction/RetailFoodRiskFactorStudies/ucm089696.htm. Accessed June 8, 2011. FDA. 2005. Food Code 2005. Available at http://www.fda.gov/Food/FoodSafety/RetailFoodProtection/FoodCode/FoodCode2005/de fault.htm. Accessed June 8, 2011. FDA. 2009. FDA Report on the Occurrence of Foodborne Illness Risk Factors in Selected Institutional Foodservice, Restaurant, and Retail Food Store Facility Types (2009). Available at http://www.fda.gov/Food/FoodSafety/RetailFoodProtection/FoodborneIllnessandRiskFac torReduction/RetailFoodRiskFactorStudies/ucm224321.htm. Accessed June 10, 2011. FDA. 2009. Food Code 2009. Available at http://www.fda.gov/Food/FoodSafety/RetailFoodProtection/FoodCode/FoodCode2009/. Accessed June 8, 2011. FDA. 2010. FDA Trend Analysis Report on the Occurrence of Foodborne Illness Risk Factors in Selected Institutional Foodservice, Restaurant, and Retail Food Store Facility Types (1998-2008). Available at http://www.fda.gov/Food/FoodSafety/RetailFoodProtection/FoodborneIllnessandRiskFac torReduction/RetailFoodRiskFactorStudies/ucm223293.htm. Accessed June 10, 2011. Feinstein, J. 1989. The Safety Regulation of U.S. Nuclear Power Plants: Violations, Inspections, and Abnormal Occurrences. Journal of Political Economy 97(1):115-154. Figlio, D., and L. Getzler. 2006. Accountability, Ability and Disability: Gaming the System. Available at http://papers.nber.org/papers/w9307.pdf. Accessed August 29, 2011. Fung, A., Graham, M., and D. Weil. 2007. Full Disclosure: The Perils and Promise of Transparency. New York: Cambridge University Press. Gerde, V.W., and J.M. Logsdon. 2001. Measuring environmental performance: Use of the Toxics Release Inventory (TRI) and other U.S. environmental databases. Business strategy and the environment 10:269–285. Gray, W.B., and C.A. Jones.1991. Longitudinal patterns of compliance with Occupational Safety and Health Administration health and safety regulations in the manufacturing sector. Journal of Human Resources 26(4):623–653. Gray, W.B., and M.E. Deiley. 1996. Compliance and enforcement: Air pollution regulation in the U.S. steel industry. Journal of Environmental Economics and Management 31(1):96- 111. 49

OCR for page 35
Hamilton, J. 2005. Regulation through Revelation: The Origins, Politics, and Impacts of the Toxics Release Inventory Program. New York: Cambridge University Press. Hamilton, J.T. 1995. Pollution as news: Media and stock market reactions to the toxics release inventory data. Journal of Environmental Economics and Management 28(1):98-113. Haney, W. 2000. The Myth of the Texas Miracle in Education. Available at http://epaa.asu.edu/ojs/article/view/432. Accessed August 29, 2011. Hanushek, E.A., and M.E. Raymond. 2005. School Accountability and Student Performance. Available at http://edpro.stanford.edu/hanushek/admin/pages/files/uploads/HanushekRaymond.pdf. Accessed August 29, 2011. Helland, E., and A.B. Whitford. 2003. Pollution incidence and political jurisdiction: evidence from the TRI. Journal of Environmental Economics and Management 46:403–424. Hislop, N., and K. Shaw. 2009. Food safety knowledge retention study. Journal of Food Protection 72(20):431-435. Iezzoni, L.I. 1997. Assessing quality using administrative data. Annals of Internal Medicine 127(8):666-674. Available at http://www.annals.org/content/127/8_Part_2/666.full.pdf+html. Accessed August 29, 2011. IOM (Institute of Medicine). 1996. Resource Sharing in Biomedical Research. Washington, DC: The National Academy Press. IOM. 2000. Protecting Data Privacy in Health Services Research. Washington, DC: The National Academy Press. IOM/NRC. 2010. Enhancing Food Safety: The Role of the Food and Drug Administration. Washington, DC: The National Academies Press. Ippolito, P., and A. Mathios. 1990. Information, advertising and health: A study of the cereal market. Rand Journal of Economics 21(3):459-480. Jacob, B.A. 2005. Accountability, incentives and behavior: The impact of high-stakes testing in the Chicago public schools. Journal of Public Economics 89:761-796. Available at http://faculty.smu.edu/millimet/classes/eco7321/papers/jacob.pdf. Accessed August 29, 2011. Jacob, B.A. and S.D. Levitt. 2003. Catching Cheating Teachers: The Results of an Unusual Experiment in Implementing Theory. Pp. 185-209 in Brookings-Wharton Papers on Urban Affairs 2003, W.G. Gale and J. Rothenberg Pack, eds. Washington, DC: Brookings Institution Press. Jin, G.Z., and P. Leslie. 2003. The effect of information on product quality: Evidence from restaurants hygiene grade cards. The Quarterly Journal of Economics 118(2):409-451. Jin, G.Z., and P. Leslie. 2005. The case in support of restaurant hygiene grade cards. Choices 20(2):97-102. Also available at http://www.choicesmagazine.org/2005-2/safety/2005-2- 02.htm. Accessed July 2011. Kane, T., and D. Staiger. 2002. The promise and pitfalls of using imprecise school accountability measures. Journal of Economic Perspectives 16:91-115. Khanna, M., Quimio, W., Rose, H., and D. Bojilova. 1998. Toxics release information: A policy tool for environmental protection. Journal of Environmental Economics and Management 36(3):243-66. 50

OCR for page 35
Kniesner, T. J., and J.D. Leeth. 2004. Data Mining Mining Data: MSHA Enforcement Efforts, Underground Coal Mine Safety, and New Health Policy Implications. Journal of Risk and Uncertainty 29(2): 83-111, 09. Konar, S., and M. A. Cohen. 1997. Information as Regulation: The Effect of Community Right to Know Laws on Toxic Emissions. Journal of Environmental Economics and Management 32(1):109-24. Kuchler, F., Chandran, R., and K. Ralston. 1996. The linkage between pesticide use and pesticide residues. American Journal of Alternative Agriculture 11(4):161-167. Los Angeles County Department of Public Health (LADPH). 2008. 10-year Review of Restaurant and Food Facility Grading Program. Available at http://file.lacounty.gov/bc/q1_2008/cms1_082885.pdf. Accessed June 8, 2011. Lucarelli, C., Nicholson, S., and M. Song. 2010. Bundling Among Rivals: A Case of Pharmaceutical Cocktails. NBER Working Paper Series. Available at http://www.nber.org/papers/w16321.pdf. Accessed July 10, 2011. Macher, J.T., Mayo, J.W., and J.A. Nickerson. 2011. Regulator heterogeneity and endogenous efforts to close the information asymmetry gap. Journal of Law and Economics 54:25-54. Magat, W.A., and W.K. Viscusi. 1990. Effectiveness of the EPA's regulatory enforcement: The case of industrial effluent standards. Journal of Law and Economics 33(2):331-60. Mathias, R.G., Sizto, R., Hazlewood, A., and W. Cocksedge. 1995. The effects of inspection frequency and food handler education on restaurant inspection violations. Canadian Journal of Public Health 86(1):46-50. National Academy of Sciences. 2009. Ensuring the Integrity, Accessibility, and Stewardship of Research Data in the Digital Age. Washington, DC: The National Academies Press. NRC (National Research Council). 1985. Sharing Research Data. Washington, DC: The National Academy Press. NRC and the Social Science Research Council. 1993. Private Lives and Public Policies: Confidentiality and Accessibility of Government Statistics. Washington, DC: The National Academy Press. NRC. 2000. Improving Access to and Confidentiality of Research Data: Report of a Workshop. Washington, DC: The National Academy Press. NRC. 2001. Studies of Welfare Populations: Data Collection and Research Issues. Washington, DC: The National Academy Press. NRC. 2005. Expanding Access to Research Data: Reconciling Risks and Opportunities. Washington, DC: The National Academies Press. Patten, D.M. 2002. Media exposure, public policy pressure, and environmental disclosure: An examination of the impact of TRI data availability. Accounting Forum 26(2):152-171. Punzi, J. S., Lamont, M., Haynes, D., and R.L. Epstein. 2005. USDA Pesticide Data Program: Pesticide residues on fresh and processed fruit and vegetables, grains, meats, milk, and drinking water. Outlooks on Pest Management 16(3):131-137. Salin, V., and N. H. Hooker. 2001. Stock market reaction to food recalls. Review of Agricultural Economics 23(1):33-46. Serapiglia, T., Kennedy, E., Thompson, S., and R. de Burger. 2007. Association of food premises inspection and disclosure program with retail-acquired foodborne illness and operator noncompliance in Toronto. Journal of Environmental Health 70(1):54-59. 51

OCR for page 35
Simon, P., Leslie, P., Run, G., Jin, G.Z., Reporter, R., Aguirre, A., and J.E. Fielding. 2005. Impact of restaurant hygiene grade cards on foodborne disease hospitalizations in Los Angeles County. Journal of Environmental Health 67:32-36. Thomsem, M.R., and A.M. McKenzie. 2001. Market incentives for safe foods: an examination of shareholder losses from meat and poultry recalls. American Journal of Economics 83(3):526-538. Weil, D. 1996. If OSHA is so bad why is compliance so good? RAND Journal of Economics 27(3):618-640. Weil, D. 2001. Assessing OSHA performance: Evidence from the construction industry. Journal of Policy Analysis and Management 20(4):651-674. USDA/FSIS. 2010. Quarterly Enforcement Report. Available at http://www.fsis.usda.gov/PDF/QER_Q1_FY11_Tables1-19.pdf. Accessed August 18, 2011. 52