Cover Image

PAPERBACK
$66.00



View/Hide Left Panel
Click for next page ( 180


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 179
~ 1 Applying Effectiveness and Outcomes Research to Clinical Practice Albert G. Mulley Wide variations in medical practices for seemingly similar patients have called into question the adequacy of the knowledge base that supports clinical decision making (1~. Such variations have also fueled concerns about both the cost and the quality of medical care. The research community has responded with proposals for a new focus in clinical research on outcomes of patient care, and the National Center for Health Services Research has recently announced a new program to sponsor such research (2~. To meet its responsibility for ensuring the quality of care provided to Medicare ben- eficiaries, the Health Care Financing Administration launched a program within the Department of Health and Human Services. This Effectiveness Initiative will systematically gather information to improve our understanding of the relative effectiveness of alternative therapeutic approaches to conditions that commonly afflict Medicare beneficiaries (3~. My purpose is to examine the methodological issues associated with the application of effectiveness and outcomes research to clinical practice. What are these issues? The answer depends on how one defines effectiveness and outcomes research and how one distinguishes them from the clinical research that has informed, however well or poorly, clinical practice until now. WHAT IS DIFFERENT ABOUT EFFECTIVENESS AND OUTCOMES RESEARCH? First, the Effectiveness Initiative and outcomes research are motivated by concerns about the quality and cost of medical care. New knowledge alone may not be enough to declare success. The current wave of enthusiasm and support will be sustained only if the initiative has a demonstrable impact on quality, cost, or both. 179

OCR for page 179
180 EFFECTIVENESS AND OUTCOMES IN HEALTH CARE Second, the initiative insists on measurement of outcomes that are im- portant to patients. Five-year survival is too coarse a measure. Physiologi- cal measures are fine, but they are often irrelevant. However, detailed measures of health states and of subjective responses to those states will be just as irrelevant unless they can be communicated to the persons responsible for clinical and policy decisions. Third, effectiveness and outcomes research must recognize clinical practice as a source of information in the production of new knowledge rather than as a passive and not always attentive consumer of knowledge produced by a separate enterprise called clinical research. This distinction between more traditional research and the new initiative is evident in the use of claims data and other administrative data bases to capture more of the collective experience of clinical practice (3-5~. Each of these premises about how outcomes and effectiveness research are different from traditional research raises a different set of methodological issues. The first set of issues is related to the need for dissemination of information. How do we get new information about effectiveness and out- comes to the point where it can have a positive impact on quality or a restraining effect on inflation? It is no accident that each of the bills supporting outcomes research also has provisions for practice guidelines. The second premise, that the new research is different because of a focus on patient-oriented outcomes, complicates matters. Results of this new research must include the subjective responses of patients that determine their quality of life, as well as the trade-offs between quality and quantity that are acceptable to them. Communication of such subjective value judgments involves a set of methodological issues that must be addressed if we are to preserve the responsiveness of health care to the wants and needs of individual patients. The third premise raises pragmatic issues about the possible integration of research and practice, not to mention epistemological questions. Which elements of clinical investigation are essential for valid inferences to be drawn about effectiveness? When can collected experience in clinical practice be an acceptable, or even preferable, source of information? Methodologists will recognize this as a variant of the tension between the "internal validity" of a clinical study and the limits that requirements for internal validity place on the "external validity," or generalizability, of a study finding (6~. A CLINICAL EXAMPLE: OUTCOME PROBABILITIES IN DECISION MAKING These methodological issues will be less abstract in the context of a clinical example. A 72-year-old man, married and sexually active, has increasing symptoms of benign prostatic hyperplasia. He gets up twice

OCR for page 179
APPLICATION TO CLINICAL PRACTICE 181 each night to void, and during the day he voids frequently, with a sensation of urgency. The patient experiences a clinical process, and the result is a health outcome that we can define simply or in great detail, if we include the physical, psychological, and social dimensions. Any simple model that goes from patient, through process, to outcome is too deterministic. There is no single, discrete clinical process that is uniquely suited to a particular patient. The path followed through clinical practice is the result of a decision or series of decisions (in this case, whether to proceed with prostatectomy), with the outcomes contingent on each deci- sion being more or less uncertain (see Figure 1~. This may seem too obvi- ous to belabor so, but the critical link between "outcomes research" and effectiveness is the ability to make valid comparisons between outcomes produced by the alternative pathways. This may be forgotten by the outcomes researcher assembling a cohort from claims data procedure codes or the guideline developer whose frame of reference is a particular procedure rather than a particular condition. The irreducible uncertainty, or stochastic element of medicine in any individual case, may be forgotten by the quality assurance reviewer who equates a bad outcome with bad care. It is worth taking a closer look at the decision-making process (Figure 2~. The patient faces, with the help of his or her physician, a choice between two alternative treatment strategies. The first alternative is a bit risky because the eventual outcome is uncertain. Although there is a chance that it will produce the most valued outcome (in this case, relief from symptoms), there is also a chance that it will produce the outcome that is least valued (operative death). An intermediate outcome is also possible (for example, impotence). The second alternative is less risky: the only possible outcomes are the most valued and an intermediate health state that happens to be the Q r 1 ~ Clinical ~L~ Decision Clinical A,/..///,'/ Outcome Practice FIGURE 1 A health outcome can be viewed as the product of a clinical process that begins with a decision or series of decisions about the interventionist most likely to meet a particular patient's health care needs and wants.

OCR for page 179
182 EFFECTIVENESS AND OUTCOMES IN HEALTH CARE patient's current health state. A more elaborate model of the prostatectomy decision has been developed, but this simpler model suffices to illustrate the process (7~. What is needed to make this choice? First, the patient and physician need to know how likely each of the outcomes is. These probabilities can be depicted as pie diagrams, as seen in Figure 3, where our hypothetical 72- year-old is referred to as Patient A. Alternative 1 has a 90 percent chance of producing the most valued outcome, a 1 percent chance of catastrophe, such as operative death, and a 9 percent chance of a bad but not fatal result, such as incontinence or impotence. Alternative 2, which looked so good without these numbers, now looks less promising: there is only a 10 percent Q l ,~ 1 ~ A ~1 Alternative ~ <- Alternative 2 : FIGURE 2 An Abstract Representation of a Simple Clinical Decision. The cross- hatched triangle superimposed on the patient represents the health state that has prompted medical care and the current decision. The square node represents a choice between Alternatives 1 and 2. Alternative 1 offers a chance, indicated by the round node, of dramatic improvement (represented by the white triangle) but with a risk of death (the black triangle) or a serious complication (the diagonally hatched triangle). Alternative 2 offers a chance of improvement with the only other outcome the baseline symptom state.

OCR for page 179
APPLICATION TO CLINICAL PRACTICE 5% 10% Alternative 1 ( ~( ( Alternative 2 183 Or r Patient A Patient B Patient C )1oo% FIGURE 3 Outcome Probabilities. Probabilities of each of the outcomes in Figure 2 for both alternatives for three hypothetical patients (including the 72-year-old man cited In the text, here designated patient A). chance of improvement the odds are 9 to 1 that the health state that was bad enough to bring the patient to the doctor will persist. It can be said that knowledge is power because it confers the capacity to predict. Accurate estimation of outcome probabilities, as represented in these simple pie dia- grams, captures the essence of professional knowledge related to the practice of medicine. SOURCES OF PROBABILITIES AND A ROLE FOR OUTCOMES RESEARCH Where does this knowledge come from? The most obvious source o probabilities is the experience of previous patients. This constitutes the "clinical experience" of the provider that is so important to "clinical judgment." There are problems, however, with this source of information. First, there are problems with the way clinicians characterize individual patients. Sec- ond, clinical practice is not standardized. Interventions are not carefully defined and uniformly applied. Third, there is no routine mechanism to define outcomes with the appropriate level of detail or to aggregate and organize the information that could be derived from collective clinical experience. Without such systematic aggregation and analysis, the cognitive heuristics that we all use routinely may mislead the clinician's unaided, intuitive probability estimate (8~. Recognizing these problems, the profession relies heavily on published f

OCR for page 179
184 EFFECTIVENESS AND OUTCOMES IN HEALTH CARE clinical research when it is available. The randomized trial is the standard against which other clinical studies are measured. Information about patients entering the trial is systematically collected. The group is made homogeneous by applying exclusion and inclusion criteria. The alternative interventions are carefully defined and their elements carefully segregated. Outcomes, at least one or two of the more objective outcomes, are carefully catalogued. The scientific requirements of research designed to determine the effectiveness of one intervention relative to another, which is nothing more than the relative outcome probabilities, include: similarity of the initial states; the integrity of the interventions; and similarity of detection or measurement of outcomes. Unfortunately, clinical research that meets these requirements is the exception rather than the rule. In the case of benign prostatic hypertrophy there are no randomized trials. Studies published in English describe outcome prob- abilities for very few men with symptoms who elected not to have surgery, and there are many methodological problems that call the accuracy of these few data into question (9~. Even when well-conducted randomized trials are available, problems arise in using the results to estimate outcome probabilities. Clinicians may forget about differences between the circumstances of the clinical trial and the circumstances of clinical practice. They may also forget about the patients excluded from the clinical trial. These exclusions are not trivial; they com- monly represent more than 90 percent of the patients for whom the intervention would be used in practice. The exclusions are also important because different patients face different outcome probabilities, even when the care rendered is identical. Figure 3 represents different outcome probabilities for three hypothetical patients. Clearly, a choice made by or for one of these patients should be based on probabilities derived from the experience of similar patients. Any inference about the effectiveness of a particular intervention must adjust for different mixes of patients with different outcome probabilities. Outcomes research is an opportunity to integrate clinical research and clinical practice (Figure 4~. Obviously, there will still be a place for rigor- ously controlled trials. What outcomes research gives up in terms of inter- nal validity it more than makes up for in enhanced external validity and relevance to clinical practice. We need also to characterize patients, that is, determine disease severity, comorbidity, and other variables that affect prognosis. We need to characterize the processes of care. We must in addition describe and sort outcomes by the alternative care processes used and by patient type. Each of these tasks presents a challenging set of methodological issues that we must deal with if we are to realize the potential of outcomes research.

OCR for page 179
185 V REV ~7 ~ ~7 W ~ W . . . . . VV, VVV Vie ~VV ~V~ ~ ~ a ~ ~ ~ Imp <:mo A Is ~ - c can ~ , E Id ^ O ~ ,0 < ct O .o can . a ~ , ~ c, ~ ~c~ (_ ~ c~-~, ~ , ==~ ~ . . ( \< ~ (en ~ ~ q( i, . ~ ... . - .. ~c , in ~ , ~ ,~ ... . c . \ ..m _ ____ __ : (age L-'~__ ~ ('it '2-:~~~'' ( AW, . ; c- c, T: (- ~ C--,m ~ ., (--~- ) o ~ - 04 ~- ~ (-- ~- ~ ( -~( - ~} -'m ~, - (--N'~ ~ '- ~'< ~O au t~o. c~ ~ - - ~ - ~D o o u, u} ~ - ~ . c~ ~ ~ ~ - ~ ~ a' ~ ~ ~ i, au c) . ._ a, ~ ~ c) cd ~ . c) ~ . - ~ ~- ~ c~s -~ .O ~ o . ~ ~ ~ - ~ - ~ ~ ~ o . 3 o C,) oC ~ ~ ~ ._ {V ^ o o d_ ~ a' _ ~ 3 ~ a' o ~ C~ 3 ~ ~ ~ O =.D - ,= . e.= O ;~ C~ ~ ,4_ ~ ~ ~ ~ Cq ca ~ ~ a.) O - (~ C) C~ ca O ~ ~i = =^ C) ~ .= q) _'

OCR for page 179
186 EFFECTIVENESS AND OUTCOMES IN HEALTH CARE VALUE JUDGMENTS IN DECISION MAKING Outcomes and effectiveness research has the potential to improve dra- matically the clinician's ability to estimate clinically relevant outcome probabilities. Probabilities alone, however, are insufficient for informed decision making. Whether the pie diagrams in Figure 3 represent probabili- ties of outcomes for a health care decision or a simple game of roulette, information about the likelihood of the outcomes must be accompanied by information about their relative values in order to be helpful to a decision maker. The top bar in Figure 5 represents a scale on which we can register the value judgments of the hypothetical patient with prostate disease. It is anchored by the least and most desirable outcomes. The markings on the scale indicate that he prefers his current state to one that would be imposed by a complication of Alternative 1 (for example, impotence). This patient might, therefore, opt for the less risky Alternative 2. The bottom two scales display different value judgments of different hypothetical patients; these patients are similar enough to face the same outcome probabilities, but with different preferences. For the second patient, the same health state diminishes life's quality more; for him, Alternative 1 may be preferable despite the risks. For the third patient, Alternative 1 would almost certainly be the best Patient ~ ~ ., Least Valued Patient 2 ~ Least Valued Patient 3 ~ Least Valued Most Valued Most Valued Most Valued FIGURE 5 Value Judgments for Three Patients. Value judgments for three hypo- thetical patients, all of whom face the same outcome probabilities. Patient 1 values the baseline health state highly (the cross hatched triangle), relative to the state associated with a complication of alternative 1 (the diagonally hatched triangle [e.g., impotence]~. Patient 3 prefers the latter to the former.

OCR for page 179
APPLICATION TO CLINICAL PRACTICE 187 choice. The current health state is perceived as a serious hardship, and the state associated with a complication of Alternative 1 is not. How confident can a patient be about these value judgments? He or she may be more confident in making ~ determination about the goodness or badness of a state that he or she has experienced than one that must be imagined. Such imaginings may be helped by hearing about the experiences of other patients. Physicians can provide such vicarious experience, but it severely tests their communication skills. Furthermore, there is no system- atically collected body of experience on which to draw. VALUE JUDGMENTS AND THE ROLE OF OUTCOMES RESEARCH The assessment of values or preferences is extraordinarily difficult and raises a new set of methodological issues (10~. As indicated in Figure 5, preferences for the same health states vary widely among patients. This has been demonstrated in a number of important studies that used hypothetical case scenarios (11,12) and in a large patient interview study of men under- going surgery for prostate disease (13~. Varying medical practice to reflect accurately these differences is both appropriate and desirable. These value judgments also change over time and are influenced by the context of the decision or the scaling task used. Even when preferences are measured accurately, there are difficulties in communicating them to other patients who might benefit from them. At this interface between outcomes research and clinical practice, the methodological issues relate more to the physician- patient relationship and its effect on care and outcomes than to the scientific basis of medicine as defined by the biomedical model. New information about subjective responses to health states could also be of value to policymakers. It could help bridge the gap between the statistical person of cost-effectiveness analysis and the real patient when making coverage decisions or choosing those conditions for which restrictive boundary guidelines may be more or less appropriate. CONCLUSIONS The methodological issues in the application of effectiveness and out- comes research to clinical practice depend on the form that the new research takes in the coming years. Dissemination of results to decision makers will be of critical importance. Clinicians must be provided with information that will allow them to estimate accurately the outcome probabilities for different patients. Clinicians and their patients must be provided with information that will allow them to make informed value judgments about different potential health outcomes.

OCR for page 179
188 EFFECTIVENESS AND OUTCOMES IN HEALTH CARE A more ambitious view of outcomes research would see it take full ad- vantage of clinical practice as a source of information to inform future practice (Figure 6~. Aggregate outcomes of individual decisions would inform professional knowledge regarding outcome probabilities and would inform value judgments made by professionals and patients. In this case, the list of issues expands to include accurate baseline description of patients, measures to ensure the integrity of the therapeutic interventions used, and the unbiased monitoring and measurement of outcomes, including patients' subjective responses. Closing this loop of practice and research will require unprecedented cooperation between clinicians and investigators, but both have much to gain in the form of a more robust and relevant knowledge base for the practice of medicine and the delivery of health services. - _{'Profession~ ~ , Individual Clinical Decision Patients _, Values \ - / Clinical A///////,// : Practice ~ V Aggregate \ Outcome - FIGURE 6 A Model of Clinical Practice and Outcomes Research Functioning as a Feedback Loop. Aggregate outcomes of many individual clinical decisions serve as an information base that informs professional knowledge with outcome probabilities and simultaneously informs patients' and professionals' value judgments with previ- ous patients' subjective responses to those outcomes.

OCR for page 179
APPLICATION TO CLINICAL PRACTICE 189 REFERENCES 1. McPherson, K., Wennberg, J.E., Hovind, O.B., et al. Small-Area Variation in the Use of Common Surgical Procedures: An International Comparison of New England, England, and Norway. New England Journal of Medicine 307:1310-1314, 1982. 2. Patient Outcomes Assessment Research Program: Extramural Assessment Teams. NCHSR Program Note. Rockville, MD: U.S. Department of Health and Human Services. November, 1988.i 3. Roper, W.L., Winkenwerder, W.L., Hackbarth, G.M., et al. Effectiveness in Health Care: An Initiative to Evaluate and Improve Medical Practice. New England Journal of Medicine 319:1197-1202, 1988. 4. Wennberg, J.E., Roos, N.P., Sola, L., et al. Use of Claims Data Systems to Evaluate Health Care Outcomes. Mortality and Reoperation Following Prostatectomy. Journal of the American Medical Association 257:933-936, 1987. 5. Fisher, E.S. and Wennberg, J.E. Administrative Data in Effectiveness Stud- ies: The Prostatectomy Assessment. Pp. 80-89 in Effectiveness and Outcomes in Health Care. Heithoff, K.A. and Lohr, K.N., eds. Washington, D.C.: National Academy Press, 1990. 6. Rothman, K. Modern Epidemiology. Boston: Little, Brown, 1986. 7. Barry, M.J., Mulley, A.G., Fowler, F.J., et al. Watchful Waiting vs. Immediate Transurethral Resection for Symptomatic Prostatism: The Importance of Patients' Preferences. Journal of the American Medical Association 259:3010-3017, 1988. 8. Tversky, A. and Kahneman, D. The Framing of Decisions and the Psychology of Choice. Science 211:453 -458, 1981. 9. Barry, M.J. Medical Outcomes Research and Benign Prostatic Hyperplasia. In: The Prostate, in press. 10. Mulley, A.G. Assessing Patients' Utilities: Can the Ends Justify the Means? Medical Care 27~3) Supplement:S269-S281, 1989. 11. Sackett, D.L. and Torrance, G.W. The Utility of Different Health States as Perceived by the General Public. Journal of Chronic Diseases 31:697, 1978. 12. McNeil, B.J., Weichselbaum, R. and Pauker, S.G. Fallacy of the Five-Year Survival in Lung Cancer. New England Journal of Medicine 299:1397, 1978. 13. Fowler, F.J., Wennberg, J.E., Timothy, R.P., et al. Symptom Status and Quality of Life Following Prostatectomy. Journal of the American Medical Asso- ciation 259:3018-3022, 1988. iEditors' Note: Now the Patient Outcomes Research Teams of the Agency for Health Care Policy and Research.