Click for next page ( 491


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 490
APPENDIX B Selectec! Papers Guide to Comparative Clinical Trials Clifford S. Goodman* This guide is intended to help a reviewer evaluate reports of comparative experimen- tal clinical trials. Such trials are a mainstay of medical technology assessment, but their worth depends on the care with which they are designed, implemented, and analyzed. Experimental trials are prospective studies in that they entail the intentional application of a technology to an experimental group and then the observation of the effects of the tech- nology. Comparative experimental clinical trials are typically used to compare the safety and effectiveness of a resew technology with a standard treatment. Trials may have more than one experimental group. In the simple comparative trial, patients with a common condition are assigned to an experimental group or to a control group. The experimen- tal group receives the new technology, and the control group receives no treatment, a placebo, a standard treatment, or a variation (e.g., a different dosage) of the experimental treatment. After a designated time, each in- * National Research Council Fellow, National Academy of Sciences, Washington, D.C. 490 dividual in the experimental and control groups is assessed for a designated endpoint or outcome. Endpoints may be measured in qualitative terms (e.g., survived or died) or quantitative terms (e.g., blood pressure mea- surements) . The most definitive type of experimental clinical trial is the randomized controlled clinical trial (RCT). In an RCT, patients are randomly assigned to experimental and con- trol groups. Randomization reduces bias that might otherwise be introduced by prognostic and other selection factors not accounted for in the design of the trial. There are a number of design variations, which can be used in combination, to the simple comparative trial. Some of these are crossover, stratified, matched, and factorial designs. In a crossover trial, patients are systemati- cally switched from one treatment group to another during the trial, and outcomes in the same patient are contrasted. Switching may be determined by a time-dependent rule or a disease-state-dependent rule. In a self- controlled trial, which incorporates many of the same features of crossover studies, a sin-

OCR for page 490
APPENDIX B.: SELECTED PAPERS ale treatment under study is evaluated by comparison of patient status before and after treatment. Louis et al. (1984) describe im- portant factors in determining the effective- ness of crossover and self-controlled designs, e.g., crossover rules, and carry-over and se- quencing effects. In a stratified trial, patients are catego- rized according to characteristics which are thought to have prognostic significance (e. g., stage of disease), so as to isolate treatment ef- fects from those of the prognostic factors. Stratification may be used in designing a trial, or it may be applied in data analysis af- ter completion of the trial. Matching is an al- location process, used to gain statistical preci- sion, of sorting patients into pairs matched according to significant prognostic factors, and then randomly assigning one member of each pair to the treatment group and the other to the control group. (For trials involv- ing multiple treatment groups, patients can be matched into groups of the appropriate number.) In a factorial design trial, combi- nations of treatment factors are grouped and observed to determine independent and in- teractive effects of multiple treatments. For example, a 2 x 2 factorial design could be used to determine the effects of medication and dietary counseling for the treatment of hypertension in four treatment groups: medi- cation and counseling, medication only, counseling only, and neither medication nor counseling. Experimental design and the de- sign of clinical trials in particular are dis- cussed extensively in the literature, e.g., in Campbell and Stanley (1963), Cook and Campbell (1979), Chalmers et al. (1981), Friedman et al. (1981), Mosteller et al. (1980), Peto et al. (1976, 1977), and Shapiro and Louis (1983~. Useful observations of the effects of tech- nologies may be made under nonrandomized and nonexperimental conditions. Although this guide is written to accommodate the as- sessment of RCTs, other types of clinical trials are subject to most of the assessment criteria discussed here. Some trials use h~stor- ical control groups selected from hospital charts or computerized data bases, or stan- dard outcomes from reports in the literature (e.g., organ transplant survival curves or re- 491 jection rates following N years). In observa- tiortal studies (including most epidemiologi- cal studies), assignment of patients to treatment and control groups is generally not under the control of the investigator, making it difficult to control for prognostic factors which might affect observed outcomes. These may include prospective studies as well as retrospective studies (i.e., those in which the investigator identifies treatment and con- trol groups after their exposure and nonexpo- sure to the technology in question). Although lacking the rigor of RCTs, observational studies are valuable in formulating hypothe- ses and in ruling out certain explanations for observed effects of technologies. Observa- tional studies and those using historical con- trols may be useful in situations in which comparative experimental designs are impos- sible or precluded by ethical, financial, and other constraints. Examples of observational studies are cohort and case-control studies. CLINICAL TRIAL REPORTING No study can be adequately interpreted without information about the methods used in the design of the study and the analysis of the results. Instructive surveys of clinical trial reporting (e.g., by Chalmers et al., 1983; DerSimonian et al., 1982; Freiman et al., 1978; Lavori et al., 1983, and Louis et al., 1984) demonstrate the extent to which im- portant methodological elements are re- ported in clinical trials and their bearing on findings. DerSimonian et al. (1982) examined 67 clinical trials published in four prominent medical journals in 1979-1980 for 11 impor- tant aspects of trial design and analysis (e. g., method of randomization, blinding, and sta- tistical methods). Of the 11 items for each of the 67 trials published in the four journals, 56 percent were clearly reported, 10 percent were ambiguously mentioned, and 34 per- cent were not reported at all. The method of randomization was reported in only 19 per- cent of the papers, and statistical power to detect treatment effects was discussed in only 12 percent. Table 13-1 lists the percentage of articles that reported the 11 aspects of trial design and analysis in the four journals sur-

OCR for page 490
492 ASSESSING MEDICAL TECHNOLOGY Percentage of Clinical Trial Articles in Four Journals Reporting 11 TABLE B-1 0 _ Important Aspects of Design and Analysis Design and Whether Journalb (Number of articles) Analysis Aspects Reporteda NEJM (13) JAMA (14) BMJ (19) Lancet (21) Total (67) Eligibility criteria R 77 36 21 29 37 ? 23 50 58 48 46 O 0 14 21 24 16 Admission before R 85 64 63 29 57 allocation ? 8 7 5 14 9 O 8 29 32 57 34 Random allocation R 100 71 95 71 84 ? 0 0 0 0 0 O 0 29 5 29 16 Method of R 15 43 16 10 19 randomization ? 23 0 11 0 7 O 62 57 74 90 73 Patients' blindness R 62 71 37 57 55 to treatment ? 0 21 21 5 12 O 38 7 42 38 33 Blind assessment R 46 43 26 14 30 of outcome ? 23 36 21 29 27 O 31 21 53 57 43 Treatment R 92 71 58 48 64 complications ? 0 0 5 5 3 O 8 29 37 48 33 Loss to follow-up R 100 93 74 62 79 ? 0 7 11 5 7 O 0 00 16 33 15 Statistical analyses R 92 79 100 95 93 ? 0 0 0 0 0 O 8 21 0 5 8 Statistical methods ~ R 92 86 79 86 85 ? 0 0 5 0 1 O 8 14 16 14 13 - Power R 15 36 5 0 12 ? 0 0 5 5 3 O 85 64 89 95 85 Mean, all items R 71 63 52 46 56 ? 7 11 13 10 10 O 23 26 35 45 34 a R denotes item reported, ? item unclear, and O item omitted. b NEJM denotes the New England Journal of Medicine, JAMA the Journal of the American Medical Association. BMJ the British Medical Journal. SOURCE: R. DerSimonian et al. (1982~. veyed. Emerson et al. (1984) repeated the study on 84 clinical trials published in 1 vear in six journals, and they found that 58 per- cent were clearly reported, 5 percent were ambiguously mentioned, and 37 percent were not reported at all. Freiman et al. (1978) examined 71 pub- lished negative trials (i.e., those in which the outcomes of treatment groups were found to be no different than those of control groups) to determine whether investigators ade- quately addressed a particular element of trial design: power to detect important clini- cal differences between treatment groups.

OCR for page 490
APPENDIX B.: SELECTED PAPERS The study found that of 71 papers on medical randomized controlled trials that reported no significant differences among treatment groups, only four of the trials were large enough to ensure a reasonable chance, in this instance a power greater than 0. 90, of detect- ing a 25 percent improvement in patient out- comes. Only 30 percent of the trials had power greater than 0.90 for detecting a 50 percent improvement. Chalmers et al. (1983) provide evidence for the seriousness of bias introduced bv vari- ous methods of treatment assignment. Among 145 papers reporting controlled clini- cal trials of the treatment of acute myocar- dial infarction, they found significant differ- ences in bias associated with the method of treatment assignment. At least one prognos- tic factor was maldistributed (p < 0.05) in 14.0 percent of the blind randomized studies (57 papers), in 26.7 percent of the unblinded randomized studies (45 papers), and in 58.1 percent of the nonrandomized studies (43 pa- pers). Significant differences in outcome (case fatality rates) between experimental and control groups were reported in 8.8 per- cent of the blind randomized studies, in 24.4 percent of the unblinded randomized studies, and in 58.1 percent of the nonrandomized studies. These reporting rates among the three types of papers differed significantly (p 0.05). The major subjects addressed in this guide are basic descriptive material sample size selection of patients random allocation blinding treatments compliance withdrawals/loss to follow-up treatment complications tabulation of outcomes statistical methods and analyses power These aspects of clinical trials should be closely examined before one combines the results of smaller trials, or generalizes the results of trials to other populations. This guide draws upon work of others in the re- 493 porting of clinical trials, especially that of Chalmers et al. (1981) and DerSimonian et al. (1982), from whom permission has been granted to use the same or similar wording in places. Table B-2 summarizes the items re- ferred to in the text and may be used as a re- viewer's checklist. TABLE B-2 Checklist for Comparative Clinical Trials Check or complete multiple entries where applicablea 1. Basic descriptive material a. Authors b. Title c. Journal/publication d. Date/volume e. Trial type _ Randomized _ Matched _ Simple comparative _ Factorial Crossover Historical _ Stratified control Not reported Observational (specify) Other (specify) f. Sources of financial support _NIH VA _Drug company _ Other Not reported g. Biostatistician cited as author or evaluator Yes _ No _ Not reported h. Start and end dates for trial given _Yes No Peer reviewed Yes _No Unknown j. Statement of significant findings Major endpoints _ + + Statistically significant (treatment) + Trend (treatment) 0 No difference - Trend (control) - Statistically significant (control) Minor endpoints + + Statistically significant (treatment) + Trend (treatment) 0 No difference - Trend (control) - - Statistically significant (control) None Side effects + + Statistically significant + Trend 0 No side effects na 1.

OCR for page 490
494 TABLE B-2 Continued TABLE B-2 Continued 2. Sample size a. Expected control group endpoints Yes No Unclear na b. Improvement of clinical interest that should c. not be missed _Yes No _Unelear na Levels of risk given a: Yes No it: Yes No d. Prior estimate of numbers of patients required _Yes _No 3. Selection of patients a. Patient sources University Public Private _Clinie Industry Not reported b. Admission criteria description Yes _ No Unelear Rejection criteria description _Yes _No Unelear d. Number of patients actually entering trial given Yes No c. Unelear . na 8. ma e. Reject log reported _Yes No . na 4. Random allocation a. Method Envelope Pharmacy Telephone Not reported na Other (speeifyi b. Stratifieation/bloeking Yes No Blinding of random allocation described c. _na Yes No na d. Testing of randomization described c. Yes No na 5. Blinding a. Patients as to treatment assignment Yes No na b. Physicians as to treatment assignment Yes No na Physicians and patients as to trends of trial _Yes No na d. Biostatisticians/other evaluators _Yes No na e. Testing for blinding Physicians: Yes No na Patients: Yes No na 6. Treatments a. Description Yes b. Patient number and treatment Controls Group 1 ASSESSING MEDICAL TECHNOLOGY Group 2 Group 3 e. Placebo described Yes No Unclear na 7. Compliance a. Defined Yes No Unclear b. Accounted for all patients Yes Partial c. Biological equivalent _Yes _No Withdrawals/loss to follow-up a. Listed Yes No b. How analyzed Counted in original treatment group _Counted as end result at time of withdrawal Counted as in both ways above, and other ways Discarded Counted in new group Not reported na 9. Treatment complications Described Not described 10. Tabulation of outcomes Given _Not given 11. Statistical methods and analyses b. No Unclear Number Treatment na _na a. Statistical methods reported (speeifie-tests, techniques, computer programs, etc.) Yes No . Statistical analyses reported (beyond means, percentages, standard deviations) Yes No e. Test statistics given for endpoints Yes No _ Unelear d. Associated probability values given Yes No Unclear e. Confidence intervals given Yes _ No Unclear f. Regression or correlation analyses Yes No Unclear g. Statistical discussion of treatment complications given Yes No Unclear h. Appropriate retrospective analysis of subgroups given Yes No Unclear na 12. Power addressed for negative trials Yes _No a Yes means reported; unclear means inadequate, partial, or ambiguous information; no means not reported; na means inapplicability is reported or clearly implied. Adapted from Chalmers et al. (1981~. _na na na na

OCR for page 490
APPENDIX B.: SELECTED PAPERS Basic Descriptive Material When assessing a published report of a clinical trial, the reviewer should note cer- tain basic descriptive information, beginning with authors, title, journal, and date of pub- lication. The report of the clinical trial should in- clude a description of the trial design (e.g., RCT, stratified-blocking). Other basic de- scriptive information includes the sources of financial support for the trial and whether or not a biostatistician has participated in the study (as an author, consultant, or reviewer). Studies should list the starting and stopping dates of the trial so that the results can be in- terpreted in the light of other changes in ther- apy that may have occurred. The paper should include a statement of significant findings as the author understands and inter- prets them. It would be helpful for the reviewer of a published trial to know whether or not the report of the trial has been reviewed by peers. However, this often is not readily dis- cernable, even among published papers that have been so reviewed. Although many jour- nals use peer reviewers for most or all articles reporting scientific findings, these journals, as a matter of editorial policy, may not dis- close whether or not a particular article was subject to peer review. Sample Size The numbers of patients in the trial affect the ability of the trial to detect differences be- tween experimental and control groups. (The risks of making errors in detection of differ- ences a, the probability of the false-positive error, and d, the probability a false-negative error- are discussed below.) There should be evidence that a prior estimate of the numbers of patients required has been made. A paper should list the expected control group endpoints, the improvement of clini- cal interest that should not be missed, the chosen levels of risk (a and P), and the num- ber of patients required. Here is an example from a trial of cyclosporine in cadaveric renal transplantation: 495 Sample size was decided on the basis of a two-sided test of the hypothesis of equality of treatment groups for one-year graft survival. At the 5 per cent level of significance, the power of the test was set at 90 per cent for an expected difference between the two treatment groups of 20 per cent (55 vs. 75 per cent). The sample size was established at 100 patients per treatment group. Statistical analysis of the background variables was carried out to assess the balance between the two groups (Canadian Multicentre Transplant Study Group, 1983~. Regrettably, few studies do this (Altman, 1983~. Only 12 percent of the trials reviewed by DerSimonian et al. (1982) reported calcu- lations of power in planning sample size; Mosteller et al. (1980) found that less than 2 percent of the trials they reviewed did so. Selection of Patients The paper should provide a detailed de- scription of the criteria for admission and re- jection of patients to the trial, and should show that these criteria were applied before knowledge of the specific treatment assign- ment had been obtained. Without selection criteria, it is difficult to interpret and apply the findings of the trial. To the extent that study patients are not selected at random from a well-defined population (not to be confused with random allocation to treat- ment groups), doubt may exist as to whether trial findings may be generalized to that pop- ulation, as well as to others. Thus, a mere statement that a certain number of patients with a given diagnosis were randomized is in- sufficient. First, admission criteria should be given which describe who was eligible for the study, e.g., patients with a particular diag- nosis and treatment history, in a certain med- ical center, in a particular year, etc. Second, rejection criteria should be given which de- scribe reasons why those who might other- wise have been admitted were ruled out of the study, e.g., diagnosis not confirmed by pathology, other serious illnesses, patient re- fusals, etc. The number of patients actually entering the trial should be given. The description of the eligible patient pop- ulation rejected for the trial can be as impor- tant as the documentation of the subjects

OCR for page 490
496 studied. A log of those patients who are not allowed to enter the trial should be kept, in- cluding the reasons for their noneligibility. The primary use of such a reject log is to help identify bias in patient selection. An attempt should be made to compare the outcome of the rejected patients to the outcome of the trial subjects to detect any important selec- tion biases, especially in instances in which cooperative studies are being undertaken at different centers. Random Allocation Random allocation of patients to treat- ment groups is a major bias-reducing tech- nique in controlled clinical trials. The ob- served results of a trial may be affected (biased) by an uneven distribution to treat- ment groups of factors that affect prognosis such as age, disease state, or concurrent med- ical problems, as well as by the experimental treatment. (These prognostic factors may be referred to as confounding variables or co- variates.) Proper randomization is an indif- ferent yet objective procedure which, among other benefits, tends to spread prognostic fac- tors evenly among treatment groups. A randomized study should provide infor- mation about the method of random alloca- tion of patients, including information about the mechanism used to generate the random assignment and success in implementing it. A simple statement that a random assignment was made is insufficient, because some meth- ods of random assignment may be effective but are poorly implemented, and some that appear to be random have serious weak- nesses. In studies in which randomization was not possible, this should be noted. Although random number tables or coin flipping may be unbiased in and of them- selves, they may be used in ways which allow for the introduction of bias into a trial. Ran- domization should be verifiable as well as properly executed. Methods such as flipping coins, tossing dice, or drawing cards cannot be verified, and may lead investigators to in- terfere with the process. Some methods which are verifiable can also be too easily in- spected by study personnel, providing oppor- tunity for bias to influence acceptance and ASSESSING MEI)ICAL TECHNOLOGY therefore treatment distribution. Examples are allocation by birth date, chart number, alternate cases, and an open randomization table. Random numbers from one of the pub- lished random number tables or pseudo- random numbers generated by a well-studied computer method offer good sources of ran- domization. After being admitted into a trial, it is best that the patient is assigned to a treatment group by a central source. A pre- ferred method of randomization uses care- fully prepared, sealed, consecutively num- bered opaque envelopes. In the case of a drug trial, drugs should be prepackaged and num- bered for each patient before the time of ran- domization. Envelopes and packages should be returned to the biostatistician for verifica- tion of assignment. Whereas simple randomization tends to spread prognostic factors evenly among treatment groups in trials with large numbers of patients, small studies are more vulnerable to imbalances of prognostic factors. To en- hance the effect of randomization in studies with small sample sizes, the patient alloca- tion process may include stratification and blocking. Patients are first classified accord- ing to one or more important prognostic fac- tors (stratification), and then they are ran- domly assigned to experimental groups so that predetermined, appropriately fixed pro- portions of patients from each stratum re- ceive each treatment (blocking) (see, e.g., Lavori et al., 1983~. If used, the methods for stratification and blocking should be de- scribed. Randomization should be blinded in that the investigator must not be able to deduce which treatment is next in line when a pa- tient is accepted into the trial. It is especially important to blind the randomization process when the treatments are not blinded or trends in the study are known to the admit- ting investigator. An admitting investigator with a bias for or against a therapy that is thought to be next up for assignment may readily circumvent the patient in whom a suspected outcome might, in the view of the investigator, favor one treatment over an- other, or the investigator may delay admis- sion until some other patient has been admit-

OCR for page 490
APPENDIX B.: SELECTED PAPERS ted. The informed consent procedure is an opportunity to inject this bias. One method of testing randomization is the measurement of the prognostic factors of the groups being compared. Listing only de- mographic comparisons such as the usual age and sex distributions is usually insufficient. If the distribution among the treatment groups of prognostic factors is disproportion- ate, the cause may be chance or a previously unsuspected bias; thus, the distribution of known prognostic factors by treatment cate- gory should be shown in tabular form. These data are critical in both assessing the efficacy of the blinding of the randomization (which, if in serious doubt, will generally result in a trial's results being discarded) and removing the unwanted effects of chance variation by using stratified analysis of a trial's results. Analysis-of-variance modeling of prognostic factors may be used to reduce bias and to in- crease precision of estimates of effects (Lavori et al., 1983~. If the trial results are to be considered for use in combination with other trial results, the significance of known prognostic factors by treatment category may be important in deciding whether to do so. Blinding Blinding is a major bias-reducing tech- nique in clinical trials. Many papers report that the therapy given was concealed from the patient or the physician or both (double blinding). However, many reports stop after using the term blind or double blind and leave the reader uncertain of exactly what has been concealed from whom. These terms are not sufficiently descriptive because the roles in the trial may not be limited to the pa- tient and physician. Four or more parties may be involved with as many roles; each may be subject to hopes and prejudices about the trial. These are (1) the personts) making the random allocations to treatment groups, (2) the patients, (3) the physicians or other providers, and (4) the biostatistician or other evaluators. Sometimes the physician makes the random assignment to treatment groups and/or is the evaluator. Such multiple re- sponsibilities present further opportunities for bias which must be checked. 497 Persons making the random assignments who have knowledge of the assignments made for particular patients may have their own prejudices and hopes, which may bias the assignments. For scientific and ethical reasons, blinding of patients and physicians as to the ongoing results of the study is impor- tant. Of course, patients' attitudes toward their treatments may affect compliance, par- ticipation, and outcomes. The physician who gives or orders treatment naturally hopes for success and may treat patients differently, given knowledge of treatment assignments, such as providing extra attention to patients with the less-preferred treatment. If the treatments and randomization process are not adequately blinded, knowledge of the trial trends could lead the conscientious phy- sician to alter the intake of patients to the trial or to influence withdrawals from the trial. From an ethical standpoint, the physi- cian should no longer ask patients to join a study or to remain in it if the physician per- ceives an impressive trend. A data-monitor- ing committee, charged with studying the trial and notifying the investigators when a change in protocol should be considered or when the trial should be discontinued, is a proper inclusion in the informed consent pro- cedure (Chalmers, 1976~. Although such a committee would not dissolve the ethical considerations (which would be shifted in part to it), it would better enable the physi- cian to act consistently in randomization and treatment. Evaluators who are aware of the treatments given may bias their findings, de- spite conscious efforts to be fair. When neces- sary, the statistician-evaluator who has prop- erly participated in planning the trial may work with coded data. As for randomization, the methods of achieving blindness should be reported to give the reader important information for judging the adequacy of a trial's protection from bias. Although not all types of blindness are feasible for all trials, every reasonable at- tempt should be made to achieve as many types of blindness as possible. This aspect re- quires careful consideration and reporting by the authors. Five aspects of a trial which should be blinded (Chalmers et al., 1981) are as follows:

OCR for page 490
498 1. the randomization process (discussed above under Random Allocation); 2. patients as to treatment assignment; 3. physicians as to treatment assignment; 4. physicians and patients as to trends/on- going results of the trial; and 5. biostatisticians/other evaluators. In certain trials, patients and physicians should be blinded to the timing of interven- tions, e. g., the point during crossover trials at which patients switch from one treatment group to another. It is not sufficient to assume that a double- blind procedure is effective. In good studies the physicians and their patients are tested for blinding at the end of the study to deter- mine whether or not they have guessed treat- ment assignments. Treatments All experimental and ancillary treatment regimens must be described well enough to allow interpretation of the results and repli- cation in other studies or practice. This in- cludes the timing and amount of treatments in the trial and all other allowable treat- ments. If a trial used a placebo, it is insufficient merely to mention that a placebo was given. Identity of appearance and taste where ap- plicable should be documented, and evi- dence for physician and/or patient ability to distinguish between placebo and experimen- tal treatment should be noted. Compliance Objective methods of verifying that pa- tients are conforming to the protocol should be described. For example, in a drug trial, pill counts would be acceptable. When sub- jective (indirect) assessments of compliance are used, the validity of the subjective mea- sure should be addressed. Biological equiva- lent refers to a measure, where appropriate, of a therapeutic agent after absorption or in- jection, preferably in its active form. Exam- ples are pre- and postvagotomy measure- ments of gastric acid output in therapeutic trials of peptic ulcer. Blood or urine levels of ASSESSING MEDICAL TECHNOLOGY an active agent may also be used to measure compliance. Biological equivalent measure- ments are useful both as measures of compli- ance and in describing treatments. In some trials the assessment of compliance is self-evident, such as in certain trials com- paring surgical and medical treatments for a disease. Trials in which patients are to main- tain regimens on an outpatient basis, espe- cially over an extended period, present spe- cial problems in validating compliance. In some trials, a patient's compliance may be partial or temporary, as well as positive or negative. In any case, definitions for what constitutes compliance must be explicit, and compliance of all patients should be ac- counted for. WithdrawalslLoss to Follow-up In most reported trials, a number of sub- jects drop out or are withdrawn after the trial is under way. In trials with long-term follow- up, large trials, and trials with complicated protocols, some follow-up data are likely to be missing. Sometimes investigators cannot collect outcome data from all subjects be- cause some die, move away, decline to con- tinue to participate, or become lost from the study group for other reasons. Information should be available regarding what happened to all the patients treated. Dropouts should be listed by diagnosis, treat- ment, reason for withdrawal, and whether withdrawal occurred as a result of patient or investigator initiative. It is usually important to report outcome in this group after the time of withdrawal, and they should be consid- ered in the main analysis of the trial. When dropouts are properly reported, the reader can often assess the effect of missing data on the trial's conclusions; otherwise, the skepti- cal reader may conclude that the paper should be dismissed. Different kinds of with- drawals (i.e., in terms of prognostic charac- teristics) could bias the final makeup of each treatment group, thus diminishing the effi- cacy of the randomization procedure for ob- taining similar kinds of patients in each treat- ment group. Trials that do not mention withdrawals, or whose withdrawals exceed 5 percent, should

OCR for page 490
APPENDIX B.: SELECTED PAPERS be carefully scrutinized. Deletion of cases by the investigator after completion of the study raises strong concern over investigator bias and undermines any findings. Universal application of a rule regarding counting of withdrawals without considering the nature of a trial and its objectives may give misleading findings (Sackett and Gent, 1979~. Depending on the type of study, with- drawals are handled in different ways, for ex- ample, as follows. 1. Patients are considered as an end result for the group to which they were originally assigned regardless of what happens to them. 2. Patients can be counted as an end result at the time of withdrawal. 3. The results are analyzed with the drop- outs handled as in both 1 and 2 and in other ways if appropriate. For instance, it may be useful to characterize treatment groups in terms of patient-years of treatment, to which even withdrawals, under certain well- defined circumstances, would make contri- butions. 4. Patients can be ignored or eliminated from the study at the time of withdrawal, and thus not be counted as an end result. A1- though this is often done, it is rarely defensi- ble. 5. Patients may change groups, i.e., cross over, and be considered as an end result in the new group. Unless this is done as part of the planned protocol, it is not defensible. Treatment Complications The paper should provide information de- scribing the presence or absence of side ef- fects or complications after treatment. To de- termine the usefulness of a treatment, readers need to assess the nature and incidence of these side effects and their implications for patient care. The report should describe an active search for side effects or complications and discuss those that are found. If no side effects occur, this should be explicitly stated. As is done in the main analysis, statistical analysis should be made of side effects if the sample size warrants it, including compari- sons of percentages with a statistical test of significance and the observed probability. 499 Given no significant difference in side effects, the probability of the false-negative error (~) should be mentioned. Tabulation of Outcomes A good study will tabulate all events em- ployed as outcomes (endpoints) so that the reader can check the calculations and use the data more effectively in combining the results of different studies. Data of trial results should not be aggregated to a level that would preclude the reader from con- ducting secondary analyses. For all discrete endpoints that are spread over time, such as mortality or morbidity, even for some trials of short duration, life table or time series analysis should be carried out. Some papers present outcome data as crude rates, e.g., a 5-year death rate. This may be useful sum- mary information, but alone it may be inade- quate for illustrating the course of treatment effects. The data should be presented in a form that would allow the reader to repro- duce the survival curve or curves. Statistical Methods and Analysis The uncertainty associated with real- world sample sizes usually requires formal statistical inference to evaluate the effects ob- served in trials. When an author merely states that p was less than 0.05 without iden- tifying the statistical test, readers cannot sat- isfy themselves that the methods were appro- priate. The paper should include statistical analyses going beyond the computation of means, percentages, and standard devia- tions. The names of the specific statistical methods, i.e., tests, techniques, and com- puter programs (with program version) used for statistical analyses should be given. In the analysis of the data gathered in any clinical trial there are certain minimal procedures that are indicative of quality. These include, but are not limited to, significance of major endpoints, confidence intervals, and regres- sion or correlation analyses. The level of statistical significance is the probability of making a false-positive, or Type I, error, i.e., concluding that there is a

OCR for page 490
500 difference between the experimental and control groups when in fact there is none. The probability of a Type I error is known as a. When significance is reported, it should be given in such a way that the reader can make the actual calculations. Both the test statistic and its associated probability values should be stated. If one is given without the other, the reader may have trouble verifying or un- derstanding the statistical conclusions. Confidence intervals should be provided for the measurements used as trial endpoints. Confidence intervals provide information that adds to the accept-reject findings of a hypothesis test. The confidence limits define the interval in which one can be reasonably confident (e.g., greater than 90 percent) that the true difference between treatments lies. If that interval includes zero, the null hy- pothesis of no true difference cannot be re- jected. However, the location and width of the interval may suggest the direction of a true difference and the ability of a larger sample size to reject the null hypothesis. Con- fidence intervals that encompass clinically unrealistic measurements raise questions about the assumed distribution of measure- ments and should be discussed. Regression or correlation analyses should be carried out for trials when it is of interest to know how treatment and outcome vari- ables change or do not change together, such as when the critical response is a function of drug dosage or predetermined, quantifiable clinical factors. When a trial is over, it is tempting and sometimes useful to select for analysis sub- groups that were not stratified at the trial's outset. However, investigators and reviewers should realize that such post hoc study is sub- ject to selection bias just as is any retrospec- tive study, and that no rigorous conclusions can be drawn from them. Retrospective stud- ies are useful to suggest new studies; may point out inadequacies arising from the ran- dom allocation process, dropouts, or compli- ance; and may help to estimate their effect on outcomes. Although many papers state the specific objectives of the study, it is often very diffi- cult to find results in the paper that apply di- rectly to the specific objectives. A clear pre- sentation of results should be made. ASSESSING MEDICAL TECHNOLOGY Power The probability of making a false-nega- tive, or Type II, error, i.e., of not detecting a difference between the experimental and control treatments when in fact one does ex- ist, is known as ,B. Power, generally defined as (1 - hi, is the probability of avoiding Type II error, i.e., detecting that true differ- ence. As discussed above, the paper should provide information describing the determi- nation of sample size before the trial, which would enable the detection of clinically im- portant differences. Although confidence limits portray the un- certainty of a treatment effect, discussion of power denotes the strength of the conclusion. As illustrated in the study by Freiman et al. (1978) referenced above under Clinical Trial Reporting, small sample size frequently leads to trials with little power to detect differences among treatment groups. If the difference between the experimental and control groups is not statistically significant, then the false- negative error and its probability should be addressed. A well-designed trial with high power that detects no statistically significant difference between treatments can be con- vincing. But if no statistically significant dif- ference is found and the power is low, or not discussed, the reader cannot dismiss the pos- sib~lity that the study was not large enough to detect an important treatment effect. For a negative trial, it would be informative to esti- mate the number of patients that would have been required to document as significant the observed difference between treatment and control groups, assuming that that difference were to hold up with the larger sample size. REFERENCES Altman, D. G. 1983. Size of clinical trials. British Medical Journal 286:1842-1843. Campbell, D. T., and J. C. Stanley. 1963. Experi- mental and Quasi-Experimental Design for Research. Chicago: Rand McNally. Canadian Multicentre Transplant Study Group. 1983. A randomized clinical trial of cyclosporine in cadaveric renal transplantation. New England Jour- nal of Medicine 309:809-815. Chalmers T. C., and discussants. 1976. How to turn off an experiment. In J. D. Cooper and H. D. Ley, eds., Ethical Safeguards in Research on Hu-

OCR for page 490
Government Payers for Health Care Donald A. Young* Between 1965 and 1980 the government replaced direct payment by consumers as the dominant source of the dollars used to pur- chase personal health care services and sup- plies. In 198O, a total of $217.9 billion was spent for personal health care in the United States. Government programs spent $86.4 billion and provided 39.7 percent of personal health care expenditures. Federal funds pro- vided $62.5 billion, more than two-thirds of the public outlay. This compares dramati- cally with the situation in 1965 when the fed- eral government paid only 10.1 percent of the bills for personal health care services and consumers paid 51.7 percent of the share. While the total expenditures for medical ser- vices has risen rapidly in this 15-year period, the percentage of the total outlay paid by the state and local governments and by private health insurance has remained relatively sta- ble. The dramatic increase in total governmen- tal expenditures as well as the increasing share paid by the federal government makes governmental bodies significant parties with interest in health services delivery, the use of evaluative information, and making sound policy decisions regarding payment for medi- cal services. Although many governmental agencies ex- pend funds for health services and supplies through an array of public programs, the Medicare and Medicaid programs are domi- nant, accounting in 1980 for $60.6 billion in personal expenditures, two-thirds of all pub- lic spending for personal health care, and fi- nancing nearly 28 percent of all personal health care expenditures. Other significant contributors to public spending for personal health care include veterans medical care, $5.8 billion; Defense Department medical care, $4.2 billion; worker's compensation, $3.9 billion; and outlays by state and local * Executive Director, Prospective Payment Assess- ment Commission, Washington, D.C. 554 governments for hospital care, in addition to that provided to Medicaid recipients, $6.0 billion. Numerous other programs account for the remainder of the governmental medi- cal care expenditures. A brief overview of the largest governmen- tal programs that provide medical services and benefits is followed by a more compre- hensive examination of the Medicare pro- gram. THE MEDICARE AND MEDICAID PROGRAMS The Health Care Financing Administra- tion (HCFA), through its Medicare and Medicaid programs, helps pay medical ex- penses of 50 million poor, elderly, disabled, and blind Americans. A total of 28 million people are Medicare beneficiaries and 23 mil- lion people are Medicaid beneficiaries. In 1980 $60.6 billion was paid for health ser- vices used by Medicare and Medicaid benefi- ciaries. This makes HCFA the single largest payer of health care services. The Medicare program is a health insur- ance program. Like other public and private insurance programs, its purpose is to reduce the economic risk to beneficiaries of the cost of illness. The costs of the program are paid through Social Security tax payments, fed- eral general revenues, and individual cost- sharing provisions. Although the program is administered by the Health Care Financing Administration, an agency of the Depart- ment of Health and Human Services (DHHS), the day-by-day claims processing and payment functions are carried out by fis- cal agents under contract to HCFA. The con- tractors are generally public insurance orga- nizations such as Blue Cross/Blue Shield or private commercial insurance companies. The processes of claims review and payment are, therefore, similar to the process used by these groups in the conduct of their private business. The Medicare program differs from pri-

OCR for page 490
APPENDIX B.: SELECTED PAPERS vate health insurance, however, in that cov- ered benefits are determined by Congress in statutory authority rather than contract and are subject to change by lawmakers and to in- terpretation by the executive branch of the government, which is charged with adminis- tering the program. Beneficiaries do not have the option of selecting from a range of benefit packages designed to meet their specific needs. Benefits available to one beneficiary are generally available to all beneficiaries subject to medical need for the services. In addition, in public and private insurance plans, premiums taken in plus administrative costs must equal or exceed over a period of time payments paid out. Because the Medi- care program is funded by Social Security taxes and general revenues, there is no direct relationship between funds contributed bv beneficiaries and funds paid for services pro- vided to beneficiaries. The Medicaid program differs in a number of respects from the Medicare program. The primary difference is that Medicaid is a vol- untary, state-administered program. The federal government participates by sharing with the states the cost of providing care. In return, the government requires that a cer- tain minimum level of services be made avail- able as well as other requirements. Signifi- cant flexibility is given to the states in determining eligibility for medical assis- tance, benefits made available, and reim- bursement amounts to be paid. Providers who participate in the program must accept Medicaid-determined reimbursements as payment in full and cannot bill beneficiaries. There is no beneficiary cost-sharing except for nominal copayments for a limited num- ber of services. States may process claims themselves or contract with private organiza- tions, and nearly half of the states currently contract out all or part of the claims process- ing functions. DEPARTMENT OF DEFENSE In 1980, the Department of Defense ex- pended $4.2 billion for medical care for ac- tive duty personnel as well as retirees and military dependents. The greatest amount of this expenditure was for the direct provision 555 of services in facilities owned and operated by the military. For military dependents, re- tirees and their dependents, and some other eligibility groups unable to obtain care in a medical facility, the federal government pro- vides a medical benefits program, the Civil- ian Health and Medical Program of the Uni- formed Services (CHAMPUS). It became ef- fective December 7, 1956, and was amended in 1966 to include coverage for retired uni- formed service personnel and their depen- dents as well as dependents of active duty personnel. CHAMPUS is provided by law (Title 10, United States Code, Chapter 55) and is operated in accordance with policies and procedures set forth by the Department of Defense in regulations. Although it is not a health insurance pro- gram, CHAMPUS is similar in many respects to health insurance and especially to the Medicare program. Authorized medical ser- vices and supplies are cost shared by the gov- ernment from money appropriated by the Congress to the Department of Defense for this purpose. The uniformed services to which CHAMPUS applies are the Army, Navy, Marine Corps, Air Force, Coast Guard, Commissioned Corps of the United States, Public Health Service, and Commis- sioned Corps of the National Oceanic and At- mospheric Administration. Beneficiaries are encouraged, and in some circumstances required, to obtain medical care from uniformed services medical facili- ties, i.e., military (and Public Health Ser- vice) hospitals. Beneficiaries do, however, have the option of obtaining needed medical care from civilian sources when care is not available close to their homes or in emer- gency situations. For most medical care ob- tained from civilian sources, CHAMPUS re- quires that the beneficiary pay part of the expense through deductibles and cost- sharing. CHAMPUS program benefits are very similar to those provided by Medicare, and CHAMPUS also relies on contractors to receive and process the claims for service. VETERANS ADMINISTRATION The Veterans Administration (VA) health care system furnishes services to eligible vet-

OCR for page 490
556 erans in 172 medical centers, 226 outpatient clinics, 92 nursing homes, and 16 domiciliar- ies. During 1980, VA treated approximately 1.25 million hospital inpatients; 15.8 million outpatient medical care visits were furnished directly by VA staff, and an additional 2.2 million visits were authorized by the VA payable to non-VA physicians authorized to render care on a fee-for-service basis. In addition, under an agreement with the De- partment of Defense, approximately 224,000 dependents of veterans were eligible to re- ceive care under the Civilian Health and Medical Program of the Veterans Adminis- tration (CHAMPVA). The care was fur- nished in non-VA facilities. HEALTH CARE FINANCING ADMINISTRATION HCFA was established in 1977 to combine health financing and quality assurance pro- grams into a single agency. It is responsible for the Medicare program, federal participa- tion in the Medicaid program, and a variety of other health care quality assurance pro- grams. As its mission statement indicates, HCFA views its responsibility to be much broader than simply paying medical bills. The mission of HCFA is to administer the Medicare and Medicaid programs and re- lated provisions of the Social Security Act in a manner that (1) promotes the timely and eco- nomic delivery of appropriate quality health care to eligible beneficiaries, (2) promotes beneficiary awareness of the services for which they are eligible and improves the ac- cessibility of those services, and (3) promotes efficiency and quality within the total health care delivery system. To accomplish this mis- sion, HCFA provides operational direction and policy guidance for the nationwide ad- ministration of the Medicare and Medicaid health care financing programs; the Profes- sional Standards Review Organization (PSRO) and related quality assurance pro- grams designed to promote quality, safety, and appropriateness of health care services provided under Medicare and Medicaid; quality control programs designed to ensure the financial integrity of Medicare and Medi- ASSESSING MEDICAL TECHNOLOGY caid funds; and various policy planning, re- search, and demonstration activities. Medicare and Medicaid, along with other third-party payers, have an interest in con- taining administration and program costs and promoting efficiency in the delivery of services to beneficiaries while maintaining the availability of high-quality, medically necessary services. To make decisions regard- ing benefits, HCFA must have up-to-date medical, scientific, and health services re- search information. To understand how ap- propriate information influences benefit de- cisions in these programs, it is first necessary to review the authority, structure, and pro- cesses of the Medicare and Medicaid pro- grams as they relate to benefit decision mak- ing. Medicare The Medicare program was established by Congress in 1965 with the enactment of Title XVIII of the Social Security Act and became effective on July 1, 1966. In 1972, major changes were made in the program's provi- sions, and the name of the Medicare program was officially changed to Health Insurance for the Aged and Disabled. The program pro- vides payment for certain medical services for persons 65 years of age or over, disabled beneficiaries, and persons with end-stage re- nal disease. The program currently covers 24.9 million aged and 3.1 million disabled in- dividuals. In the title and opening sections of the Medicare statute, Congress indicated clearly that Medicare was to be an insurance pro- gram providing basic protection against the costs of medical care rather than a health ser- vices delivery program. In addition to stress- ing the insurance nature of the program, the opening sections of the statute prohibit any federal interference in the practice of medi- cine or the manner in which medical services are provided, guarantees beneficiaries free choice of qualified providers, and allows in- dividuals the option of obtaining other health insurance protection. The Medicare program consists of two sep- arate but complementary insurance pro-

OCR for page 490
APPENDIX B.: SELECTED PAPERS grams, a Hospital Insurance Program, known as Part A, and a Supplementary Med- ical Insurance Program, known as Part B. All persons age 65 or over who qualify for Social Security cash benefits, and individuals who have been receiving Social Security disability benefits for 24 months or more are automati- cally enrolled in Part A. Part A is financed by a payroll tax shared equally by employers and employees. Although Part A is called hospital insurance, covered benefits include medical services furnished in institutional settings including hospitals, skilled nursing facilities, or provided by a home health agency. Such institutions are termed pro- viders by Medicare and must be certified as qualified providers of services and have signed an agreement to participate in the program. The Medicare law includes limits, based on the concept of a benefit period, on the services which may be covered in the var- ious settings. The law also established cost- sharing by the individual through deductible and coinsurance payments. Part A providers of services are reimbursed directly by the pro- gram (for all reasonable costs) and generally cannot bill beneficiaries other than for appli- cable cost-sharing. The Supplementary Medical Insurance Program (SMI), or Part B of Medicare, is vol- untary for individuals who elect to be cov- ered. It is financed from premium payments by enrollees together with contributions from appropriated general revenue funds. (Be- cause of limits on premiums, the federal con- tribution has been increasing more rapidly than the premium. Currently, premiums fi- nance about 30 percent of the program costs, with the remaining 70 percent coming from general revenues.) Medicare Part B covers medical services and supplies furnished by physicians or others in connection with phy- sicians services, outpatient hospital services, and home health services. Physicians' services covered under the program include visits to the home, office, hospital, and other institu- tions. The program also pays for certain drugs and biologicals that cannot be self- administered, diagnostic x-ray and labora- tory tests, purchase or rental of durable med- ical equipment, ambulance services, pros- thetic devices, and certain medical supplies. 557 In contrast to Part A institutional costs re- imbursement, benefits paid under Part B are usually reimbursed on a fee or charge basis. After the beneficiary pays an annual deducti- ble, Medicare will pay 80 percent of the rea- sonable charge for most covered services for that year. Physicians and other suppliers, however, are allowed to charge beneficiaries an additional amount if the Medicare pay- ment is less than their usual charge. CLAIMS PROCESSING The separation of the Medicare program into an institutional (provider) component (Part A) and a noninstitutional (medical ser- vices) component (Part B) was patterned af- ter a program alignment used by Blue Cross/ Blue Shield Associations in paying for services to their subscribers. In order to keep the fed- eral health insurance program closely linked to the private sector, Congress decided that most claims-processing and administrative functions for both Part A and Part B of Medi- care should be handled by public or private insurance organizations (commercial or Blue Cross/Blue Shield) acting as fiscal agents for the Medicare program. The fiscal agents responsible for the ad- ministration of hospital insurance or Part A benefits are termed intermediaries. Institu- tional providers (hospitals, skilled nursing fa- cilities, home health agencies) were initially allowed to select the intermediary of their choice; however, this is slowly changing. In- termediaries act as the link between the pro- vider and the Health Care Financing Admin- istration, which is responsible for the administration of the Medicare program. The major role of the intermediaries is to re- view and pay claims for the costs of providing care to beneficiaries. The intermediary makes these payments to providers for cov- ered items and services on the basis of reason- able cost determinations following policies set by HCFA. Under the SMI (Part B), the fiscal agents are called carriers. Carriers are selected on a geographical basis by the secretary of the De-

OCR for page 490
558 partment of Health and Human Services; physicians and others furnishing Part B ser- vices have no say in selecting the carrier to process claims for these services. Since Part B services are reimbursed primarily on a rea- sonable charge (as opposed to reasonable cost) basis, one of the major functions of car- riers is to determine the reasonable charges in their respective areas for each medical care service paid for under the program. Carriers are also responsible for reviewing and paying claims to or on behalf of beneficiaries for the services provided. The functions performed by Medicare in- termediaries and carriers in the adjudication of claims is similar for both their private and their government business. These functions include, in addition to claims review and processing, utilization review, beneficiary hearing and appeals, professional relations, and statistical activities. The final decisions regarding payment for services in their pri- vate insurance business is determined by a contract with beneficiaries or their represen- tatives. The final decision in their Medicare business is determined by statutory author- ity, regulations promulgated by DHHS and program instructions, and guidance pre- pared by HCFA to implement regulations and statutory authority. In the absence of HCFA instructions concerning a specific ser- vice, authority is vested in carriers and inter- mediaries to make the benefit decisions. Medicare intermediaries and carriers are re- imbursed for their administrative costs under the basic principle of no profit or no loss. Contractors are not at risk with respect to program benefit payments as these payments are entirely underwritten by the program. Contractors, however, are regularly evalu- ated as to their capability and efficiency in administering the program and are subject to loss of contract for poor performance. Medicaid The Medicaid program was also enacted by Congress in 1965. Title XIX of the Social Security Act, Grants to States for Medical As- sistance Programs, succeeded earlier, wel- fare-linked medical care programs. Under the Medicaid program, states may enter into ASSESSING MEDICAL TECHNOLOGY an agreement with the secretary of DHHS to finance health care services for certain cate- gories of low-income individuals, primarily those eligible to receive cash payment under the Aid to Families with Dependent Children (AFDC) program and the Supplemental Se- curity Income (SSI) program for the aged, blind, and disabled (categorically needy). In addition, many states have exercised the op- tion to extend coverage to "medically needy" individuals who meet the AFDC or SSI cate- gorical criteria but whose incomes are slightly above the welfare standards or indi- viduals who have incurred substantial medi- cal expenses. An estimated 22.5 million indi- viduals are Medicaid recipients. The federal share of program costs is related to state per capita income, ranging from 50 percent in the highest per capita income states to 77 per- cent in the lowest per capita states. The fed- eral contribution is referred to as Federal Fi- nancial Participation (FFP). Federal law mandates that states cover hospital, physician, skilled nursing facility, family planning, home health, laboratory, x-ray, rural health clinic, and nurse midwife services for all eligible recipients, and early and periodic screening, diagnosis, and treat- ment (EPSDT) services for children under 21. States may also provide a variety of op- tional services, including intermediate care facility services, prescription drugs, dental care, eyeglasses, and other services. States determine the scope of services of- fered and the reimbursement rate for these services subject to federal guidelines. They also exercise a great amount of control over the income eligibility level for Medicaid. The Omnibus Budget Reconciliation Act of 1981 further extended the states' flexibility in these matters. All of these variations in benefits of- fered, income standards, and levels of reim- bursement mean that Medicaid programs differ greatly from state to state. States are responsible for claims processing and other administrative functions for their Medicaid programs, although the federal government shares in the cost of these func- tions. Some states administer their Medicaid programs directly; others contract with the private sector to perform various functions. Fiscal agent contracts are currently used by

OCR for page 490
APPENDIX B.: SELECTED PAPERS the majority of states to process and pay claims for some or all services. Fiscal agents are reimbursed on either a cost-reimburse- ment or fixed-price basis. In some cases the state contracts with the same fiscal agent re- sponsible for processing Medicare claims. COVERAGE AND REIMBURSEMENT UNDER MEDICARE PROGRAM This discussion will outline the current au- thority, criteria, and process by which deci- sions are made to pay for certain medical pro- cedures and services within the Medicare program. The Medicare program pays for some or all of the cost of certain medical ser- vices furnished to eligible beneficiaries. In this regard, the Medicare program is similar to the insurance programs of other third- party payers such as Blue Cross and Blue Shield and commercial insurance companies. Individuals with such insurance plans are en- titled to certain benefits under the conditions of their particular policy. Individuals, their employers, or other groups frequently negoti- ate with the insurance agent a package of benefits to be included in the policy, and sub- scribers are frequently given the opportunity to select from a range of different benefit packages the policy best suited to their needs. tastes, and income. In the Medicare program, the benefits available to eligible beneficiaries are called covered services. Services not covered are not paid for with Medicare funds. The Medicare program differs from other insurance pro- grams in that there is no negotiation between individual beneficiaries or their representa- tives regarding the content of a benefit pack- age or selection of alternative benefit pack- ages. Rather, all eligible beneficiaries may have partial or full payment made for those services which are covered if their medical condition and level of care is judged to war- rant it. In the following discussion, a distinction is made between issues related to coverage of services and issues related to reimbursement for services. Reimbursement, in terms of the Medicare program, deals with determining the methods and amounts of payment for ser- vices which are covered. Reimbursement is- 559 sues become important only after it has been determined that a service is covered as a ben- efit. For example a new device may be devel- oped which monitors the rhythm of the heart in a new way. After review, it may be deter- mined that the device performs this function safely and effectively, and since heart rhythm monitors are covered services, the de- vice is also covered. The question then be- comes one of reimbursement. Should the level of reimbursement for the use of the de- vice be the same as for devices previously used or is there reason for a different level of reimbursement? This discussion will focus on issues related to coverage rather than reim- bursement of services, that is with determin- in~ the services to be paid for as benefits un- der the Medicare program rather than the method or level of reimbursement. * Services covered by the Medicare program are determined by Medicare statute, regula- tions developed in keeping with the statute, program instructions included in a series of manuals used by those administering the pro- gram on a day-to-day basis, and interpreta- tions of policy in response to specific inqui- r~es. Medicare Statute The Medicare law specifically provides coverage for broad categories of benefits, for example, hospital benefits, skilled nursing fa- cility benefits, home health benefits, physi- cians' services, ambulance services, labora- tory services, durable medical equipment, and others. The Medicare statute also, to some degree, defines these broad categories of benefits. For example, physicians' cervices means ". . . professional services performed by physicians, including surgery, consulta- tion, a home, office, and institutional call. . . ." The statute also lists some specific items which are covered such as diagnostic x-ray tests, surgical dressings, iron lungs, oxygen tents, wheelchairs, and others. In addition, the statute places some limitations, of a gen- eral and categorical nature, on the services * This paper does not cover the new prospective payment system. See Chapter 5 of this book.

OCR for page 490
560 that can be covered when furnished by cer- tain practitioners, such as dentists, chiro- practors, and podiatrists. In addition to indi- cating what is covered, the law expressly excludes some categories and types of services from coverage, such as cosmetic surgery, per- sonal comfort items, custodial care, and rou- tine physical checkups. The Medicare law does not, however, fur- nish an all-inclusive list of specific items, ser- vices, treatment procedures, or technologies covered. Thus, except for the listed examples of medical and other health services, the stat- ute does not explicitly include or exclude cov- erage of most medical devices, surgical pro- cedures, or diagnostic or therapeutic services. The apparent intention of Congress, at the time the act was passed, was that Medicare should generally cover services ordinarily furnished by hospitals, skilled nursing facili- ties, and physicians licensed to practice medi- cine. However, it is also apparent that the Congress understood that questions as to cov- erage of specific items and services would in- variably~arise and would require a specific coverage decision by those administering the program. Thus, the Medicare law states: Notwithstanding any other provisions of this ti- tle, no payment may be made under Medicare for any expenses incurred for items or services . . . which are not reasonable and necessary for the diagnosis or treatment of illness or injury or to improve the functioning of a malformed body member. This a key provision. First, by reason of the words "notwithstanding any other provision of this title . . ." this is an overriding exclu- sion, and may be applicable in a given situa- tion despite the other provisions for coverage in the statute. Second, it provides the secre- tary of the Department of Health and Hu- man Services considerable discretion and flexibility to respond to changes in the way health care is furnished, especially to the de- velopment and application of new medical practices, procedures, and devices. Medicare Regulations The regulations implementing the reason- able and necessary section of the Medicare ASSESSING MEDICAL TECHNOLOGY law are also quite general ~42 CFR 405.310(k)~. The term reasonable and neces- sary is not further defined in the regulation, nor does the regulation spell out a process for how this term is to be applied. The regula- tions do, however, contain a variety of spe- cific exclusions and limitations on the bene- fits covered by Medicare. These exclusions include such things as routine physical ex- ams, eyeglasses, cosmetic surgery, and dental services which were spelled out in the Medi- care statute. Program Instructions and Policy The clearest formal operational definition of reasonable and necessary is contained in program instructions prepared by HCFA and sent to the fiscal agents (carriers and interme- diaries) responsible for processing Medicare claims for services and administering the pro- gram on a day-by-day basis. This statement of policy translates the statutory and regula- tory terms reasonable and necessary into a test of whether the item, service, or proce- dure in question is 1. generally accepted as safe and effective, or proven to be safe and effective, 2. not experimental; 3. medically necessary; or 4. furnished in accordance with accepted standards of medical practice in an appropri- ate setting. Over the years, this test has been applied to many items and services resulting in a large collection of informal policy statements and accumulated decisions serving as precedents for current policy work. These policy state- ments are continuously undergoing change as medical services and procedures evolve and new research findings emerge. Decisions on Individual Claims The claim review process for the Medicare program is designed to identify services which may not be covered or about which there may be a question of medical necessity or reasonableness. Medicare's contractors (carriers and intermediaries) have discretion within the statutory, regulatory, and pro-

OCR for page 490
APPENDIX B.: SELECTED PAPERS gram instruction guidelines to decide cover- age issues identified in the claims review pro- cess. All contractors have nurses on their claims review staffs and all have physicians available to provide advice and to consult with other physician specialists and peer groups in the community, in order to resolve coverage issues on the basis of sound medical judgment and information. Medicare contractors are currently pro- cessing 200 million individual claims for ser- vice each year. Most of these are paid with- out serious questions being raised about whether the items and services are covered under Medicare. When questions are raised, they relate primarily to whether the service was medically necessary in the particular case and was furnished in an appropriate manner and setting, rather than to the broader issue of general coverage. However, at times an issue arises as to whether a proce- dure or item should be covered under any cir- cumstance. These services are usually new procedures or new applications for existing procedures, although occasionally questions will arise regarding potentially outmoded services and items. Such questions are re- ferred to the HCFA central office for further review and evaluation. Although the claims review process is the major source of questions about coverage of procedures, items, and services, inquiries also come to HCFA from physicians and pro- fessional groups and with increasing fre- quency from manufacturers of medical equipment and devices. HCFA examines the question and is able to answer many referrals based on the statute, regulations, and exist- ing policies and definitions concerning cov- ered services. A few questions raise important new issues which HCFA cannot resolve with- out seeking additional professional and medi- cal expertise. If medical consultation appears necessary, HCFA will review the medical and scientific literature related to the service in question, gather appropriate articles and background material, and present the question to a panel of physicians employed by HCFA and other components of the department. The task of the panel is to sharpen and clarify the ques- tions that need to be answered. For example, 561 HCFA was asked recently if plasmapheresis (apheresis) was a covered service. After re- view and discussion, and with assistance from physician members on the panel from the Public Health Service (PHS), it was clear that apheresis had potential application for many diseases and conditions and that evalu- ations of this procedure were necessary based on the specific indications for its use. Cur- rently, HCFA covers apheresis for a limited number of indications with additional evalu- ation under way. When the panel confirms the need for further expert medical opinion and evaluation, HCFA refers the question with the background information to PHS. USE OF EVALUATIVE DATA IN COVERAGE DECISIONS The criteria currently used to determine if an item or service is reasonable and necessary in terms of the Medicare program are as un- specific as safe and effective. HCFA may in- terpret the meaning of safe and effective in a very different manner from other groups. For example, in considering whether to approve new medical devices, the Food and Drug Ad- ministration (FDA) also uses the terms safe and effective. The process and specific crite- ria used by the FDA in evaluating the safety and effectiveness of a medical device for pur- poses of market approval, however, are very different from those used by HCFA to deter- mine safety and effectiveness for the purposes of providing Medicare payment. It is possible that a new device may be ap- proved by the FDA based on a limited amount of research data focused on short- term safety and effectiveness of the device rather than longer-term safety and effective- ness in terms of improved health outcome necessary for Medicare coverage. Hence, cer- tain devices or procedures may be approved by FDA but not covered as benefits under the Medicare program. Because both agencies use the terms safe and effective, the public may be confused by the seeming inconsis- tency. The issue is further complicated by the lan- guage describing a service as either generally accepted as safe and effective or proven as safe and effective. There is no commonly ac-

OCR for page 490
562 cepted definition of these terms. For many new items, services, and procedures it may not be possible to make a decision based on general acceptability by the medical profes- sion because the service has usually been pro- vided by only a small number of physicians. Hence, the coverage decision will rest on medical evidence or judgments proving safety and efficacy. But even here, there is no clear agreement as to what constitutes an ac- ceptable level of proof. There are similar difficulties in determin- ing if a procedure is still experimental. There are no accepted definitions or operational measures to indicate when a procedure or service has moved from a clear research phase, to an investigational phase, to ac- cepted medical practice. In actuality, these stages overlap and research and investigation continue at the time a new procedure is gain- ing acceptance by practicing physicians. For example, studies of the diffusion of comput- erized axial tomography scanning indicate the diffusion of the procedure, and the growth of the investigative literature base proceeded in parallel. At what point has the procedure become generally accepted? The question is complicated further because the decision to pay is usually yes or no. It is gener- ally not possible to pay for a service only in certain institutions or when performed by certain specially qualified physicians. To date, HCFA and its medical and scien- tific advisors and contractors have not explic- itly considered criteria beyond safety, effi- ciency, and research status in determining Medicare coverage policy. It is reasonable to assume, however, that other considerations such as economic, ethical, and social issues are at least implicitly considered as some pro- cedures are evaluated for Medicare coverage. For example, coverage questions referred to HCFA for detailed evaluation frequently concern services or devices that have the po- tential for high program costs if covered. In such cases, economic considerations or issues of distribution of services and access to ser- vices may be an implicit factor in the decision to refer the issue to HCFA for evaluation or to initiate a thorough evaluation. Such con- siderations may implicitly affect the coverage decision if for no other reason than that a ASSESSING MEDICAL TECHNOLOGY greater burden of proof may be required be- fore a decision is made to cover the service as a Medicare benefit. To date, however, no safe and effective procedures have been de- nied coverage based on cost or social consid- erations. Current Status of Coverage Decision Making HCFA, its medical advisors, and the Medi- care contractors have wide discretion in mak- ing coverage decisions concerning individual items and services. Although leaving signifi- cant room for flexibility and individual con- siderations and judgments by the medical profession, the lack of more explicit coverage criteria has also resulted at times in inconsis- tency from claim to claim or service to ser- vice. HCFA, in preparing national coverage instructions, which are binding on contrac- tors, may also inconsistently apply the crite- ria in evaluating certain coverage questions. As noted above, a more rigorous burden of proof may be required for newly introduced services or costly services compared with es- tablished services which may be less safe or effective or more costly. TECHNOLOGICAL INNOVATION, COVERAGE DECISIONS, AND MEDICAL PRACTICE There is a belief that the technological re- search and development capabilities are ex- ceeding the capacity of the health care deliv- ery system and the individual practitioner to evaluate the medical research findings and appropriately apply them to patient care needs. Although drugs and some medical de- vices are subjected to scientific scrutiny by the FDA before marketing and wide avail- ability, other medical procedures, devices, and services are accepted and widely applied by the medical community with little evi- dence regarding relative safety or effective- ness. For example, gastric freezing in the treatment of peptic ulcer disease and internal mammary artery ligation for coronary artery disease were widely used by medical practi- tioners before clinical studies demonstrated their lack of effectiveness. With the publica-

OCR for page 490
APPENDIX B.: SELECTED PAPERS tion of evaluation studies, these procedures subsequently disappeared from the therapeu- tic armamentarium. In addition, many medical devices and procedures are evaluated as individual items rather than in comparison with existing, al- ternative approaches to achieve similar medi- cal outcomes. This is especially true for diag- nostic studies in which new findings may be quickly applied and new tests are added to the array of those already available rather than replacing existing studies. HCFA thus far has directed the bulk of its medical coverage evaluation resources to in- dividual new devices and procedures. But in- novations are also occurring in the patterns of health services delivery. Data are being gath- ered concerning the appropriate minimum numbers of procedures and distribution of services such as open heart surgery, the growth in numbers and appropriateness of coronary care and intensive care unit beds, home health and day care services, and so- called unnecessary surgery. Recently, there has also been a significant effort by the medi- cal profession to move services from the tradi- tional hospital setting to settings outside the hospital. Ambulatory surgical centers and free-standing cardiac rehabilitative facilities are examples of such movement of services. Medical and scientific evaluative informa- tion is also necessary to determine coverage policies in these areas. For services that have long been accepted by the medical community, but frequently are unproved as to effectiveness, Medicare coverage policy has proceeded on adminis- trative rather than medical judgments. For example, concerning home health or rehabil- itation services, administrative decisions are usually made in terms of the Medicare stat- ute, regulations, and policy. Only on occa- sion do new medical research findings or technological innovations lead to a change in policy. The failure to use medical and health ser- vices research for assistance in determining coverage policy in the service area is under- standable. Questions regarding the appropri- ate applications of new technologies or the existing patterns of health services delivery are complex and highly value-laden. Fur- 563 thermore, there are very significant differ- ences in delivery of medical care services in different areas of the country. A physician evaluating alternative approaches and select- ing those services that best serve the needs of an individual patient will draw upon very different information and values than will a policy analyst evaluating information to de- termine if a service qualifies as reasonable and necessary and, therefore, will be paid for by the Medicare program. Evaluative information is necessary for the physician and patient to select the proper mix of services. It is also necessary for the public policy official charged with the responsible administration of a publicly funded pro- gram. The absence of a sound information base as well as the potential conflict between the needs of an individual patient and the needs of third-party payers to exercise a fidu- ciary responsibility in behalf of all beneficia- ries is the source of a major conflict surround- ing benefit coverage decision making. Frequently a test, procedure, or service is considered necessary by a physician if it is likely to make any difference at all in the di- agnostic process or therapeutic outcome. The economic concepts of marginal gain and marginal cost may not be applied by practi- tioners in the care of individual patients, par- ticularly when third-party payers such as Medicare are picking up most of the bill. In this case, the apparent costs to the individual approach zero and the service or test is or- dered even if its value also may approach zero. For example, in evaluating a patient with coronary artery disease, many different tests and procedures are available. Are all the tests or only certain selected ones necessary for an individual patient? There are no clear research findings to answer the question, and because physicians are trained to acquire all the possible data available to minimize un- certainty, the tests are ordered and usually , paid for by the Medicare or other third-party payers. When either physicians or third-party pay- ers turn to the medical and health services de- livery research data for guidance on ques- tions similar to this, they find it may fail to provide the information needed. Sound in- formation and consensus on the safety and ef-

OCR for page 490
564 fectiveness of alternative methods of diagno- sis, treatment, and delivery of services fre- quently do not exist as part of an accepted body of knowledge. Many procedures and services commonly used and accepted in medical practice have not been evaluated by means of carefully planned, well-designed, controlled clinical studies. Nevertheless, Medicare generally pays for these commonly accepted procedures and services when or- dered by a licensed physician. It is the new procedures or new applications for accepted procedures that are currently subject to eval- uation by the Medicare program. The medical profession contends that an assessment of the risks and costs as well as the benefits of services and procedures has been central to the exercise of good medical judg- ment for decades and that such analysis and judgments are better made, and are being re- sponsibly made, within the medical profes- ASSESSING MEDICAL TECHNOLOGY sign. An alternative view holds that an indi- vidual physician frequently does not have available all the information needed to make a sound decision regarding the safety and ef- fectiveness of complex new procedures, al- though from the physician's own experience with the procedure it would appear to be working out well. Such might have been the case with carotid artery ligation or gastric freezing. One view placing high weight on the judg- ment of individual physicians might be that if a physician orders any procedure or service Medicare should pay for it. An opposite view could require that payment be made only for those services that have been evaluated with evidence as to safety and efficacy. In prac- tice, the Medicare program looks to the judg- ment, experience, and opinion of physicians and to sound scientific evidence.