Review of the Use of Process Control Indicators in the FSIS Public Health Risk-Based Inspection System A Letter Report (2009) / Chapter Skim
Currently Skimming:
Letter Report
Letter Report
Pages 1-60
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 1... ...
-- under the auspices of the Standing Committee on the Use of Public Health Data in FSIS Food Safety Programs -- established the Committee on Review of the Use of Process Control Indicators in the FSIS Public Health Risk-Based Inspection System to review criteria developed by FSIS for ranking establishments based on relative risk. The body of this letter report provides the committee's findings and recommendations regarding whether FSIS has adequately defined and identified indicators of process control that will be used to rank establishments and allocate agency inspection resources to protect public health.
|
From page 2... ...
• The statistical analysis that was conducted to find associations between proposed process control indicators -- lift analysis -- is a data-mining tool appropriate for use in finding initial associa tions among events that occur infrequently. However, the identi
|
From page 3... ...
. • Other proposed process control indicators also present limita tions.
|
From page 4... ...
Department of Agriculture (USDA) , the Institute of Medicine of the National Academies appointed the nine-member ad hoc Committee on Review of the Use of Process Control Indicators in the FSIS Public Health Risk-Based Inspection System.
|
From page 5... ...
• Noncompliance records • Enforcement actions • Class I and II recalls • Pulsed field gel electrophoresis (PFGE) patterns of Salmonella serovars of particular human health concern for isolates derived from the raw meat and poultry Salmonella verification testing program • System for Tracking E
|
From page 6... ...
Specifically, the committee will evaluate how FSIS is proposing to use its available data to develop a relative risk ranking of establishments described in the technical report Public Health RiskBased Inspection System for Processing and Slaughter, publicly posted at http://www.fsis.usda.gov/Regulations_&_Policies/National_Advisory_Committe e_on_Meat_&_Poultry/index.asp. BACKGROUND Public Health Risk-Based Inspection System for Processing and Slaughter The Food Safety and Inspection Service, the USDA agency responsible for ensuring the safety of meat, poultry, and egg products, has examined a number of strategies to develop a risk-based food safety system.
|
From page 7... ...
In its technical report Public Health Risk-Based Inspection System for Processing and Slaughter (hereafter referred to as PHRBIS)
|
From page 8... ...
was charged with reviewing the second component, the public health attribution system. Because the two components are closely related (e.g., the volume of production in an establishment influences the sampling plans for pathogen testing programs that FSIS proposes to use to indicate process control)
|
From page 9... ...
The descriptions of the algorithm, the scientific basis for the selection of the proposed process indicators, the analysis of data, and the use of the process control indicator algorithm as it is integrated in the overall inspection system were not clearly stated in the technical report PHRBIS that was provided to the committee. For example, FSIS uses the term "algorithm" to describe its decision-making tool to categorize plants into levels of inspection.
|
From page 10... ...
Definitions of Process Control and Process Control Indicators The FSIS (2008b) report does not adequately define various terms that are key to evaluating the proposed inspection system (e.g., algorithm, process control, process control indicator)
|
From page 11... ...
being applied to raw materials In its 2003 report Scientific Criteria to Ensure Safe Food, IOM evaluated the use of scientific criteria and standards in food safety regulations (IOM, 2003)
|
From page 12... ...
It also assumes that there is active monitoring of the process using appropriate metrics, which ideally would allow corrective actions to be taken before a critical safety limit is surpassed. Using this general definition of process control, the committee defined a process control indicator for the purpose of this review as: A measurable attribute that indicates whether a process maintains or surpasses an acceptable degree of risk or hazard control.
|
From page 13... ...
Such events are not true indicators of process control; rather, they demonstrate prior failures. This distinction is vital to understanding the algorithm, and therefore future improvements to it, and should be stated clearly along with the definition of process control indicator.
|
From page 14... ...
Finding 2: The committee finds that the technical report PHRBIS does not adequately define various terms that are key to a clear understanding of the proposed inspection system -- specifically, process control and process control indicators. RECOMMENDATION 2: Prior to analyzing the available data and selecting the indicators to develop a risk-based inspection system, the committee recommends that FSIS clearly define the terms (and their limitations)
|
From page 15... ...
. Although the committee was not specifically asked to comment on the number or thoroughness of the levels of inspection proposed, evaluation of FSIS's use of its available data as process control indicators to rank establishments required that the committee fully understand the details of the concepts and procedures associated with the proposed algorithm and decision criteria leading to the various levels of inspection.
|
From page 16... ...
Data Collection and Analysis of Proposed and New Indicators of Process Control FSIS used various statistical analyses to correlate proposed process control indicators with recognized process control indicators now in use (e.g., results of Salmonella verification testing) or with other proposed indicators that, based on FSIS regulatory definitions, record the presence of adulterants and therefore imply a failure of control in the system.
|
From page 17... ...
Some current predictors of process control may no longer be useful in the future. For example, the tendency and purpose of Salmonella verification testing is reduction in the frequency of contamination of raw meat and poultry products with Salmonella (i.e., fewer Salmonellapositive samples will be found over time)
|
From page 18... ...
Use of Lift Analysis Lift analysis is a data-mining tool that can identify associations between two variables separated in time. It is the central statistical analysis used by FSIS to estimate the predictability of loss of process control and is explained in detail in Appendix E of the technical report PHRBIS (FSIS, 2008b)
|
From page 19... ...
In conducting the lift analysis, it would seem particularly important to ensure that associations between attributes that are predictive be used in a manner that is consistent with their current use. It is not surprising that the Salmonella verification testing program was among the most effective indicators of process control examined by FSIS, since it is one of the few metrics evaluated that was specifically designed as a process control indicator (i.e., control of fecal contamination)
|
From page 20... ...
However, the identification of process control indicators requires more complex statistical analysis as well as data that have been collected for the purpose of identifying such indicators. The committee emphasizes that although mining or extracting data from currently existing data sets to design the inspection system is commendable and the use of lift statistics for data mining is justified, the system could be significantly improved if more complete statistical analyses were conducted in addition to the lift analysis and if additional data were collected for more useful predictors.
|
From page 21... ...
Microbiological Testing Several of the process control indicators included in the proposed algorithm are based on the results of microbiological testing programs. As discussed above, Salmonella verification testing appears to be particularly well suited as a predictor of loss of process control and an alert to take corrective actions prior to exceeding a public health limit.
|
From page 22... ...
. While this is a practical risk-management approach for the implementation of regulatory programs, its application in the algorithm suggests that FSIS has not fully considered the general concepts underlying process control indicators and the statistical basis for microbiological testing.
|
From page 23... ...
To assess compliance with the standards, FSIS implemented the Salmonella verification testing program in 1998, in which establishments are monitored by testing sampling sets for Salmonella at a specific frequency and comparing them with performance criteria based on product class (FSIS, 2008b)
|
From page 24... ...
. In the current Salmonella verification program, FSIS categorizes establishments producing those eight product classes into three categories by comparing their Salmonella verification results to the Salmonella prevalence rates within each class of product: • Category I -- Establishment achieved Salmonella prevalence rates <50 percent of the performance standard (based on the na tional estimate baseline for a given product)
|
From page 25... ...
. Although a decrease in the incidence of foodborne infections is often cited as evidence that the Salmonella verification testing program is an effective process control indicator, various factors may confound the value of that association.
|
From page 26... ...
. The level of inspection to which an establishment would be assigned would depend on its Salmonella testing results: • LOI 1: Establishment was below the Salmonella percent positive LOI 1 percentile cut point on most recent sample set, unan nounced sampling, or other Salmonella testing programs • LOI 2: Establishment was above the Salmonella percent positive LOI 1 percentile cut point on most recent sample set, unan nounced sampling, or other Salmonella testing programs and not in Salmonella Category III • LOI 3: Establishment is in Salmonella Category III FSIS proposes to determine the percentile cut point for the three levels by analyzing the number of Salmonella verification positive results for each specific class of product (e.g., broilers, ground beef)
|
From page 27... ...
Committee's Discussion Salmonella verification testing was designed to be a process control indicator in raw products, so its use in the algorithm is appropriate, as long as positive Salmonella test results occur with sufficient frequency and at high enough levels. As the prevalence of Salmonella decreases, alternatives to Salmonella testing should be sought.
|
From page 28... ...
Alternatively, FSIS should explore whether the Salmonella verification testing data would provide greater discriminatory power if they were quantitative instead of qualitative. Finding 6: The use of Salmonella verification testing results to rank establishments in different levels of inspection is justified, but could be enhanced by additional explanation and characterization.
|
From page 29... ...
This should include consideration of the magnitude of type I and type II errors, assumed pathogen concentration means and standard deviations, specificity and sensitivity of the microbiological protocols, and so forth. Listeria monocytogenes Testing Results for RTE Products Use and Scientific Evidence FSIS considers the presence of a pathogen in a ready-to-eat food product an indicator of a public health risk; therefore by definition, it indicates a potential loss of process control.
|
From page 30... ...
, and Intensified Verification Testing (IVT)
|
From page 31... ...
monocytogenes testing results in RTE foods to rank establishments in different levels of inspection has been justified based on the potential direct health risk of the pathogen, a valid riskmanagement decision criterion, particularly for specific production lots. However, the initial data analysis has not provided scientific support for use of this decision criterion to predict loss of process control or for its association with other indicators.
|
From page 32... ...
. • Conduct lift analysis and other appropriate analyses by product class to determine whether there is a correlation between L
|
From page 33... ...
coli O157:H7 in RTE products or ground beef or components, and any related FSA and follow-up sampling has been completed in the previous 120 days and all related enforcement actions are de ferred or in abeyance. • LOI 3: Establishment has had an FSIS positive test for E
|
From page 34... ...
While the occasional detection of a positive sample cannot definitively establish a loss of process control, the use of testing results for E coli O157:H7 for this purpose is a valid risk-management decision to operationalize the decision criterion for the testing program and to safeguard the public from a pathogen that is capable of infecting individuals at a low dose.
|
From page 35... ...
Finding 8: The use of E coli O157:H7 testing results for ground beef and its components to rank establishments in different levels of inspection has been justified based on the potential direct health risk of the pathogen; this is a valid risk-management decision criterion, particularly for specific production lots.
|
From page 36... ...
E coli O157:H7 and Salmonella Testing Results for RTE Products Use and Scientific Evidence In addition to testing raw ground beef, FSIS also tests specific RTE foods for the presence of E
|
From page 37... ...
Unlike the Salmonella verification testing program for raw meat and poultry products where it is used as an indicator of fecal contamination, Salmonella in RTE products is considered an adulterant. In summary, RTE foods are intended to be consumed without further processing, so any level of Salmonella or E
|
From page 38... ...
Finding 9: The use of Salmonella and E coli O157:H7 testing results in RTE foods to rank establishments in different levels of inspection has been justified based on the potential direct health risk of these pathogens; this is a valid risk-management decision criterion, particularly for specific production lots.
|
From page 39... ...
Because many NRs are not related to food safety or public health, FSIS performed the statistical analysis using the totality of NRs and also using exclusively health-related NRs to determine any improvement of predictability with use of only the selected NRs. Two sets of health-related NRs were created and analyzed separately: a group of nine FSIS experts with diverse backgrounds in the regulation of meat, poultry, and egg products assigned each NR a weight of 3, 2, 1, or 0 indicating the degree of loss of process control it represented, and the median score of each was used to identify those with a weight of three (W3NR)
|
From page 40... ...
. FSIS included a statistical analysis to justify the use of NRs as predictors of loss of process control in Appendix E of the technical report PHRBIS (FSIS, 2008b)
|
From page 41... ...
The lift analysis was performed using aggregated data, that is, data from all types of establishments, against the following variables: pathogen test results, consumer complaints, food safety recalls, enforcement actions, and RTE L monocytogenes alternatives (FSIS, 2008b)
|
From page 42... ...
monocytogenes. The significant association between NRs and Salmonella verification testing was the FSIS's basis for using selected health-related NRs as an indicator of process control and therefore including this indicator in the FSIS proposed algorithm.
|
From page 43... ...
The lift analysis performed by FSIS suggests that the likelihood of a positive Salmonella verification test is higher when specific NRs have previously been written for an establishment. The lift analysis shows only preliminary associations and will obviously be applicable only to
|
From page 44... ...
Finding 10: The use of selected NRs as process control indicators in a risk-based inspection system offers potential. However, because current NRs are written to document failure to comply with regulations, not all of them are predictors of loss of process control.
|
From page 45... ...
• Validate the use of NRs as predictors of positive Salmonella verification test results by conducting a pilot test to ensure that positives are not occurring simply as a result of the expected baseline load of Salmonella in the products or the variability of Salmonella load. • Investigate the utility of surrogate indicators (e.g., generic E
|
From page 46... ...
Enforcement Actions Use and Scientific Evidence Enforcement actions are taken by FSIS against an establishment that fails to comply with regulatory requirements. As with NRs, not every enforcement action is related to a food safety problem; some may be issued in response to regulatory requirements of a different nature (e.g., nutritional labeling)
|
From page 47... ...
. Nevertheless, FSIS justifies the use of enforcement actions as process control indicators because by definition they arise from a failure to abide by regulations, some of which are related to food safety (FSIS, 2008b)
|
From page 48... ...
Finding 11: The use of enforcement actions to rank establishments in different levels of inspection has been justified based on their suggesting a past loss of control, a valid risk-management decision criterion. However, the initial data analysis has not provided scientific support for use of this decision criterion to predict a loss of process control or for its association with other indicators.
|
From page 49... ...
as follows: • LOI 1: Establishment has not shipped adulterated or misbranded product (includes recalls related to human illness) ; or if it has, any related FSA and follow-up sampling was completed more than 120 days previously, any related enforcement actions are closed, and establishment meets all other criteria for LOI 1.
|
From page 50... ...
Finding 12: The use of public health-related recalls to rank establishments in different levels of inspection has been justified based on potential direct public health risk, a valid risk-management decision criterion. However, the initial data analysis has not provided scientific support for this decision criterion as being predictive of a loss of process control or for its association with other indicators.
|
From page 51... ...
: • LOI 1: An establishment has not been linked to an outbreak; or if it has, any related FSA and follow-up sampling has been com pleted more than 120 days previously, any related enforcement actions are closed, and establishment meets all other criteria for LOI 1.
|
From page 52... ...
Finding 13: The use of foodborne disease outbreaks to rank establishments in different levels of inspection has been justified based on their potential direct public health risk, a valid risk-management decision criterion. However, the initial data analysis has not provided scientific support for use of this decision criterion to predict loss of process control or for its association with other indicators.
|
From page 53... ...
, consumer complaints may often incorrectly associate a food with an adverse health effect. OTHER POTENTIAL INDICATORS OF PROCESS CONTROL Microbial Test Results The primary goal of process control for raw meat and poultry products is to limit the presence of fecal contamination, the source of enteric pathogenic microorganisms.
|
From page 54... ...
Furthermore, in the proposed algorithm, there are no identified process control indicators for RTE foods. RECOMMENDATION 16: FSIS should investigate the potential utility of industry data on generic E
|
From page 55... ...
Because of inherent problems in the use of NRs described above, the committee recommends redefining public health-related NRs and creating new ones where appropriate so that they reflect the current view of HACCP as a food safety control approach. This approach should identify true science-based indicators of process control.
|
From page 56... ...
The committee agrees with the general concept of using process control indicators as part of an algorithm to rank establishments in different levels of inspection. The committee recommends that FSIS continue the collection and analysis of data and, in consultation with stakeholders and expert panels, continue to improve its proposed risk-based inspection system so that it more effectively allocates inspection resources according to risk.
|
From page 57... ...
Also, because this new algorithm would bring about changes in inspection procedures, a parallel training program for the inspection force would likewise be necessary. The Committee on Review of the Use of Process Control Indicators in the FSIS Public Health Risk-Based Inspection System thanks FSIS for the opportunity to review the technical report Public Health Risk-Based Inspection System for Processing and Slaughter and hopes that its findings and recommendations are useful.
|
From page 58... ...
Microbiological testing in food safety management. New York: Klewer Academic/Plenum Publishers.
|
From page 59... ...
2004. Risk assessment of Listeria monocytogenes in ready-to-eat food: Technical report.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.