Introduction and Framework
Study Framework
The panel views its role as a technical one, to identify and assess measures that states can use to evaluate their progress toward important health objectives and to recommend actions to improve the utility of such measures. This report provides an assessment of measures that could be used over the next 3–5 years by states and the federal government to monitor progress in meeting agreed-upon health objectives. The report does not attempt to review all of the program options and policies to be considered in structuring PPG agreements between DHHS and states: such issues as funding levels, matching requirements, hold-harmless funding provisions, allocation of resources decisions, financial incentives, and the like are not covered. The panel's goal is to provide technically sound methods for assessing progress in meeting public health objectives and to provide states and others with practical and useful tools to advance their public health objectives.
Performance Measure: a quantitative indicator that can be used to track progress toward an objective, i.e., to detect change over time and difference in change across programs Objective: a specific level of measurable attainment between two points in time |
The panel's framework for assessing potential performance measures is simple: a public health program operating at the state level, with a certain size and structure (capacity), uses the resources provided by a federal funding program (process) to improve the health of the population it serves. The panel assumes that the effectiveness of a state program in using resources can most appropriately be evaluated by assessing the degree to which desired changes in health outcomes are achieved, together with a judgment of the degree to which those changes can be attributed to a program. When a firm causal link between the resources and processes used and the health outcome sought has not been established, as is often the case, or when the program resources are a small part of all the resources that contribute to the outcome, the panel believes that performance assessment must necessarily depend on a combination of health outcome, process, and program capacity measures. Furthermore, the panel suggests that performance measures be understood and adopted as the product of an evolutionary process, to be revised as additional empirical evidence is obtained and better methods of data collection are implemented.
Definitions
Public Health
In considering performance measures for public health programs, the panel was mindful of the concept of "core" public health functions developed by the Institute of Medicine (IOM) that is now widely accepted within the public health community. These core functions are assessment, policy development, and assurance (Institute of Medicine, 1988); see box. The IOM report also states that public health programs should include both disease prevention and health promotion, with "health" encompassing physical, mental, and environmental health.
The ten specific areas that the panel was asked by DHHS to examine with regard to performance measures are a subset of the full range of public health concerns. Many critical responsibilities of state and local public health agencies, such as maternal and child health, injury prevention, and environmental health, are not covered in this report, but the guidelines for assessing performance measures presented here can be applied to these other areas.
It is important to note that state public health departments are not always the designated recipients of federal funds. In the areas of substance abuse and mental health, for example, the grantee may well be the state department of human services. In many states, public health responsibilities are distributed among local districts. A PPG agreement in any given state, therefore, will need to clearly identify lines of responsibility and assure that the performance goals are reasonable given the organizational structure and resources available.
Public Health Assessment: "the regular and systematic collection, assembly, analysis and dissemination of information on the health of the community. This information includes statistics on health status, community health needs, and epidemiologic and other studies of health problems." Public Health Policy Development: "the development of comprehensive public health policies by promoting use of the scientific knowledge base in decision-making about public health and by leading in developing public health policy." Public Health Assurance: "assures that services necessary to achieve agreed upon goals are provided, either by encouraging actions by other entities (private or public sector), by requiring such actions through regulation, or by providing services directly" (Institute of Medicine, 1988:7–8). |
Outcomes, Risk Status, Process, and Capacity
Health outcome, risk status, process, and capacity measures are all needed at different times and in different situations to intelligently monitor both changes in the health status of defined populations and the performance of all public and private agencies in working toward specified health goals; see box for definitions adopted by the panel for this report. In some cases, actual health outcomes are impractical to measure as indicators of program performance because too much time is required between intervention and outcome or because many confounding factors affect the ultimate health outcome. In such cases, the panel recommends using an "intermediate" measure, risk status, for which there is general consensus that the result being measured is related to the health outcome.1
Meaningful analysis of performance requires determining whether desired health outcomes are achieved, whether specific agency commitments are carried out, and whether the agency has the capacity to conduct all the necessary processes. Outcomes are fundamental, and any process or capacity measure used to assess performance should be widely accepted as closely related to them. For example, if a state's PPG goal is to reduce its mortality rate from breast cancer, it can reduce the risk of such an adverse health outcome by increasing the number of mammograms it provides to women aged 50 and over. However, there are also
|
a series of process activities (e.g., health education programs, requirements that private insurers include coverage of, say, mammography, surgical and nonsurgical treatment, and postoperative follow-up care) and capacity indicators (e.g., number of trained staff and facilities offering mammography screening) that are believed to be related to the level of mortality from breast cancer and can be monitored over time. A detailed set of such measures could provide some understanding of what particular service mechanisms are present and may affect the trend in the outcomes of interest. The capacity of public agencies is important for any comprehensive and accurate assessment of program performance. Infrastructure activities, such as the maintenance of various public health data and surveillance systems, are as important as monitoring drinking water quality and conducting restaurant inspections in promoting the public health. The panel notes, in fact, that DHHS supported a major study of public health infrastructure, which is expected to provide infrastructure capacity measures for use in the PPG process (Lewin-VHI, Inc., 1997).
Assessment Guidelines
In considering how to assess the appropriateness of individual measures for tracking the performance of state public health agencies under the PPG process,
the panel reviewed materials developed by DHHS, state partners, and other professional groups (see Annotated Bibliography). The panel established guidelines for the assessment of proposed measures:
-
- Data to track any objective must meet reasonable statistical standards for accuracy and completeness;
- Data to track any objective must be available in a timely fashion, at appropriate periodicity, and at reasonable cost; and
- Data applied to a specific measure must be collected using similar methods and with a common definition throughout the population of interest. Comparisons of a measure across states are valid only if the definition and collection methodology are consistent across states.
4.
Measures should be valid, reliable, and responsive. Measures should, as much as possible, capture the essence of what they purport to measure (i.e., be unbiased and valid for their intended purpose), be reproducible (i.e., reliable), and be able to detect movement toward a desired objective (i.e., be responsive).
That a measure can be valid for one purpose but not for another is an important factor in performance measurement. For example, a state's infant mortality rate is usually considered a valid measure in assessing the actual change in a state's rate of infant death from one period to another, but changes in that rate may not be a valid measure of the performance of an individual public health agency: the agency may have no control over many factors that can affect infant mortality, such as changing socioeconomic conditions or the demographic characteristics of the population. Performance measures must also be reliable: have a high likelihood of yielding the same results on repeated trials and, therefore, low levels of random error in measurement. Similarly, performance measures should be known to be responsive to change at the level of change that one would like to detect.
1. |
Measures should be aimed at a specific objective and be result oriented. PPG measures must clearly specify a desired public health result, including identifying the population affected and the time frame involved. Process and capacity measures should clearly specify the health outcome, or long-term objective, to which they are thought to be related. |
2. |
Measures should be meaningful and understandable. Performance measures must be seen as important to both the general public and policy makers at all levels of government and they should be stated in nontechnical terms. |
3. |
Data should be adequate to support the measure. Adequate data on the populations of interest must be available for the use of measures and have the following characteristics: |
In an ideal world, each performance measure would fully satisfy all four guidelines; unfortunately, not many available health outcome measures can do so. For example, many factors not under a state agency's control can affect health outcomes, compromising the validity of measures of program effect. Consequently, the panel recommends that health outcome measures be used in conjunction with process and capacity measures to derive appropriately conservative inferences about the performance of a state agency. This approach will provide public officials and consumers with an opportunity to examine steps taken by agencies to achieve specific health outcomes and to better understand whether changes in the magnitude or direction of particular strategies should be considered. A combination of health outcome, process, and capacity measures should be used to identify what additional research is needed to establish more precisely the relations among program interventions and outcomes.
It is important that agencies that engage in performance monitoring specify the assumed relationship between any process or capacity measure proposed and the particular health outcome to which it is believed to be related and document, with empirical evidence and professional judgment, the assumed relationship. If states elect to implement new, experimental approaches to realize PPG objectives, they must collect the data necessary to document the effectiveness of those interventions.
One of the constraints of the PPG process, as currently formulated, is that the performance objectives must be judged capable of realization within 5 years. Yet many important public health objectives, such as lowered incidence of cancer and HIV infection, cannot be achieved over this short time period. However, it would be unwise to divert resources from those objectives simply because demonstrable results cannot be expected in the 5-year period. The panel recommends that DHHS and the states consider negotiating some items in their performance agreements that allow for longer term goals if relevant risk behaviors and process data can be used to measure progress toward the desired health outcomes.
References
Institute of Medicine (1988) The Future of Public Health. Committee for the Study of the Future of Public Health, Division of Health Care Services. Washington, D.C.: National Academy Press.
Lewin-VHI (1997) Strategies for Obtaining Public Health Infrastructure Data at Federal, State, and Local Levels. Report to the Public Health Service. Washington, D.C.: U.S. Department of Health and Human Services.