1
Introduction

Abstract: This chapter presents the objectives and context for this report and describes the approach that the Institute of Medicine (IOM) Committee on Standards for Systematic Reviews of Comparative Effectiveness Research used to undertake the study. The committee’s charge was two-fold: first, to assess potential methodological standards that would assure objective, transparent, and scientifically valid systematic reviews (SRs) of comparative effectiveness research and, second, to recommend a set of methodological standards for developing and reporting such SRs. A companion IOM committee was charged with developing standards for trustworthy clinical practice guidelines.

Healthcare decision makers in search of the best evidence to inform clinical decisions have come to rely on systematic reviews (SRs). Well-conducted SRs systematically identify, select, assess, and synthesize the relevant body of research, and will help make clear what is known and not known about the potential benefits and harms of alternative drugs, devices, and other healthcare services. Thus, SRs of comparative effectiveness research (CER) can be essential for clinicians who strive to integrate research findings into their daily practices, for patients to make well-informed choices about their own care, for professional medical societies and other organizations that



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 17
1 Introduction Abstract: This chapter presents the objectives and context for this report and describes the approach that the Institute of Medicine (IOM) Committee on Standards for Systematic Reviews of Com- parative Effectiveness Research used to undertake the study. The committee’s charge was two-fold: first, to assess potential meth- odological standards that would assure objective, transparent, and scientifically valid systematic reviews (SRs) of comparative effectiveness research and, second, to recommend a set of method- ological standards for developing and reporting such SRs. A com - panion IOM committee was charged with developing standards for trustworthy clinical practice guidelines. Healthcare decision makers in search of the best evidence to inform clinical decisions have come to rely on systematic reviews (SRs). Well-conducted SRs systematically identify, select, assess, and synthesize the relevant body of research, and will help make clear what is known and not known about the potential benefits and harms of alternative drugs, devices, and other healthcare services. Thus, SRs of comparative effectiveness research (CER) can be essential for clinicians who strive to integrate research findings into their daily practices, for patients to make well-informed choices about their own care, for professional medical societies and other organizations that 17

OCR for page 17
18 FINDING WHAT WORKS IN HEALTH CARE develop CPGs, and for payers and policy makers.1 A brief overview of the current producers and users of SRs is provided at the end of the chapter. SRs can also inform medical coverage decisions and be used to set agendas and funding for primary research by highlight- ing gaps in evidence. Although the importance of SRs is gaining appreciation, the quality of published SRs is variable and often poor (Glasziou et al., 2008; Hopewell et al., 2008b; Liberati et al., 2009; Moher et al., 2007). In many cases, the reader cannot judge the quality of an SR because the methods are poorly documented (Glenton et al., 2006). If methods are described, they may be used inappropriately, such as in meta-analyses (Glenny et al., 2005; Laopaiboon, 2003). One cannot assume that SRs, even when published in well-regarded journals, use recommended methods to minimize bias (Bassler et al., 2007; Colliver et al., 2008; Roundtree et al., 2008; Song et al., 2009; Steinberg and Luce, 2005; Turner et al., 2008). Many SRs fail to assess the quality of the included research (Delaney et al., 2007; Mallen et al., 2006; Tricco et al., 2008) and neglect to report funding sources (Lundh et al., 2009; Roundtree et al., 2008). A plethora of conflicting approaches to evidence hierarchies and grading schemes for bodies of evidence is a further source of confusion (Glasziou et al., 2004; Lohr, 2004; Schünemann et al., 2003). In its 2008 report, Knowing What Works in Health Care: A Road- map for the Nation, the Institute of Medicine (IOM) recommended that methodological standards be developed for SRs that focus on research on the effectiveness of healthcare interventions and for CPGs (IOM, 2008). The report concluded that decision makers would be helped significantly by development of standards for both SRs and CPGs, especially with respect to transparency, minimizing bias and conflict of interest, and clarity of reporting. The IOM report was soon followed by a congressional mandate in the Medicare Improve- ments for Patients and Providers Act of 20082 for two follow-up IOM studies: one, to develop standards for conducting SRs, and the other to develop standards for CPGs. The legislation directs the IOM to recommend methodological standards to ensure that SRs and CPGs “are objective, scientifically valid, and consistent.” In response to this congressional directive, the IOM entered into a contract with the Agency for Healthcare Research and Qual- ity (AHRQ) in July 2009 to produce both studies at the same time. 1 The IOM Committee on Standards for Developing Trustworthy Clinical Practice Guidelines defines CPGs as “statements that include recommendations intended to optimize patient care that are informed by an SR of evidence and an assessment of the benefits and harms of alternative care options.” 2 Public Law 110-275, Section 304.

OCR for page 17
19 INTRODUCTION The IOM appointed two independent committees to undertake the projects. The 16-member3 Committee on Standards for Systematic Reviews of Comparative Effectiveness Research included experts in biostatistics and epidemiology, CER, CPG development, clinical trials, conflict of interest, clinical care and delivery of healthcare services, consumer perspectives, health insurance, implementation science, racial and ethnic disparities, SR methods, and standards of evidence. Brief biographies of the SR committee members are presented in Appendix I. This report presents the findings and rec- ommendations of the SR committee. A companion report, Clinical Practice Guidelines We Can Trust, presents the findings and recom- mendations of the Committee on Standards for Developing Trust- worthy Clinical Practice Guidelines. COMMITTEE CHARGE The charge to the SR committee was two-fold: first, to assess potential methodological standards that would assure objective, transparent, and scientifically valid SRs of CER, and second, to recommend a set of methodological standards for developing and reporting such SRs (Box 1-1). WHAT IS COMPARATIVE EFFECTIVENESS RESEARCH? In recent years, various terms such as evidence-based medi- cine, health technology assessment, clinical effectiveness research, and comparative effectiveness research have been used to describe healthcare research that focuses on generating or synthesizing evi - dence to inform real-world clinical decisions (Luce et al., 2010). While the legislation that mandated this study used the term clinical effectiveness research, the committee could not trace the ancestry of the phrase and was uncertain about its meaning separate from the phrase comparative effectiveness research in general use by clinicians, researchers, and policy makers. Thus, this report adopts the more commonly used terminology—comparative effectiveness research and defines CER as proposed in the IOM report, Initial National Priorities for Comparative Effectiveness Research (IOM, 2009, p. 42): CER is the generation and synthesis of evidence that com- pares the benefits and harms of alternative methods to pre- vent, diagnose, treat, and monitor a clinical condition or 3 One member stepped down from the committee in July 2010.

OCR for page 17
20 FINDING WHAT WORKS IN HEALTH CARE BOX 1-1 Charge to the Committee on Standards for Systematic Reviews of Comparative Effectiveness Research An ad hoc committee will conduct a study to recommend methodologi- cal standards for systematic reviews (SRs) of comparative effectiveness research on health and health care. The standards should ensure that the reviews are objective, transparent, and scientifically valid, and require a common language for characterizing the strength of the evidence. Deci- sion makers should be able to rely on SRs of comparative effectiveness to know what is known and not known and to describe the extent to which the evidence is applicable to clinical practice and particular patients. In this context, the committee will: 1. Assess whether, if widely adopted, any existing set of standards would assure that SRs of comparative effectiveness research are objective, transparent, and scientifically valid. 2. Recommend a set of standards for developing and reporting SRs of comparative effectiveness research. to improve the delivery of care. The purpose of CER is to assist consumers, clinicians, purchasers, and policy makers to make informed decisions that will improve health care at both the individual and population levels. Research that is compatible with the aims of CER has six defin- ing characteristics (IOM, 2009): 1. The objective is to inform a specific clinical decision. 2. It compares at least two alternative interventions, each with the potential to be “best practice.” 3. It addresses and describes patient outcomes at both a popu- lation and a subgroup level. 4. It measures outcomes that are important to patients, includ- ing harms as well as benefits. 5. It uses research methods and data sources that are appropri- ate for the decision of interest. 6. It is conducted in settings as close as possible to the settings in which the intervention will be used. Body of Evidence for Systematic Reviews of Comparative Effectiveness Research The body of evidence for an SR of CER includes randomized controlled trials (RCTs) and observational studies such as cohort

OCR for page 17
21 INTRODUCTION studies, cross-sectional studies, case-control studies, registries, and SRs themselves (Box 1-2). RCTs have an ideal design to answer questions about the comparative effects of different interventions across a wide variety of clinical circumstances. However, to be appli- cable to real-world clinical decision making, SRs should assess well- BOX 1-2 Types of Comparative Effectiveness Research Studies Experimental study: A study in which the investigators actively intervene to test a hypothesis. •    ontrolled trials are experimental studies in which a group receives C the intervention of interest while one or more comparison groups receive an active comparator, a placebo, no intervention, or the standard of care, and the outcomes are compared. In head-to-head trials, two active treatments are compared. •   n a randomized controlled trial (RCT), participants are randomly I allocated to the experimental group or the comparison group. Cluster randomized trials are RCTs in which participants are randomly as- signed to the intervention or comparison in groups (clusters) defined by a common feature, such as the same physician or health plan. Observational study: A study in which investigators simply observe the course of events. •  In cohort studies, groups with certain characteristics or receiving   certain interventions (e.g., premenopausal woman receiving chemo- therapy for breast cancer) are monitored over time to observe an outcome of interest (e.g., loss of fertility). •  In case-control studies, groups with and without an event or out-   come are examined to see whether a past exposure or characteristic is more prevalent in one group than in the other. •   n cross-sectional studies, the prevalence of an exposure of inter- I est is associated with a condition (e.g., prevalence of hysterectomy in African American versus white women) and is measured at a specific time or time period. Systematic review (SR): A scientific investigation that focuses on a spe- cific question and that uses explicit, planned scientific methods to identify, select, assess, and summarize the findings of similar but separate studies. It may or may not include a quantitative synthesis (meta-analysis) of the results from separate studies. •  A meta-analysis is an SR that uses statistical methods to combine   quantitatively the results of similar studies in an attempt to allow inferences to be made from the sample of studies and be applied to the population of interest. SOURCE: Adapted from Last (1995).

OCR for page 17
22 FINDING WHAT WORKS IN HEALTH CARE designed research on the comparative effectiveness of alternative treatments that includes a broad range of participants, describes results at both the population and subgroup levels, and measures outcomes (both benefits and harms) that are important to patients, and reflects results in settings similar to those in which the interven- tion is used in practice. Many RCTs lack these features (IOM, 2009). As a result, in certain situations and for certain questions, decision makers find it limiting to use SRs that are confined to RCTs. Observational research is particularly useful for identifying an intervention’s potential for unexpected effects or harms because many adverse events are too rare to be observed during typical RCTs or do not occur until after the trial ends (Chou et al., 2010; Reeves BOX 1-3 Four Examples of the Use of Observational Studies in Systematic Reviews of Comparative Effectiveness Research Important outcomes are not captured in randomized controlled trials (RCTs) More than 50 RCTs of triptans focused on the speed and degree of migraine pain relief related to a few isolated episodes of headache. These trials provided no evidence about two outcomes important to patients: the reliability of migraine relief from episode to episode over a long period of time, and the overall effect of use of the triptan on work productivity. The best evidence for these outcomes came from a time-series study based on employment records merged with pre- scription records comparing work days lost before and after a triptan became available. Although the study did not compare one triptan with another, the study provided data that a particular triptan improved work productivity—information that was not available in RCTs. Available trials of antipsychotic medications for schizophrenia included a narrow spectrum of participants and only evaluated short-term outcomes In a systematic review (SR) of antipsychotic medications, 17 short- term efficacy trials evaluated a relatively narrow spectrum of patients with schizophrenia, raising a number of questions: Is the effect size observed in the RCTs similar to that observed in practice? Do groups of patients excluded from the trials respond as frequently and as well as those included in the trials? Are long-term outcomes similar to short- term outcomes? For a broad spectrum of patients with schizophrenia who are initiating treatment with an atypical antipsychotic medication, which drugs have better persistency and sustained effectiveness for longer term follow-up (e.g., 6 months to 2 years)? Given the many questions not addressed by RCTs, these review authors determined that they would examine and include observational studies. Meta- analyses of RCTs were conducted where appropriate, but most of the data were summarized qualitatively.

OCR for page 17
23 INTRODUCTION et al., 2008). Moreover, observational studies may provide evidence about the performance of an intervention in everyday practice or about outcomes that were not evaluated in available RCTs (Box 1-3). Despite their potential advantages, however, observational studies are at greater risk of bias compared to randomized studies for deter- mining intervention effectiveness. STUDY SCOPE This report presents methodological standards for SRs that are designed to inform everyday healthcare decision making, especially for patients, clinicians and other healthcare providers, and develop - Participants in trials comparing percutaneous coronary intervention (PCI) versus coronary artery bypass graft (CABG) differed from patients seen in community practices An SR of PCI versus CABG for coronary disease identified 23 relevant RCTs. At the outset, cardiothoracic surgical experts raised concerns that the trials enrolled patients with a relatively narrow spectrum of disease (generally single- or two-vessel disease) relative to patients receiving the procedures in current practice. Thus, the review included 96 articles reporting findings from 10 large cardiovascular registries. The registry data confirmed that the choice between the two proce- dures in the community varied substantially with extent of coronary disease. For patients similar to those enrolled in the trials, mortality results in the registries reinforced the findings from trials (i.e., no dif- ference in mortality between PCI and CABG). At the same time, the registries reported that the relative mortality benefits of PCI versus CABG varied markedly with extent of disease, raising caution about extending trial conclusions to patients with greater or lesser disease than those in the trial population. Paucity of trial data on using a commonly prescribed drug for a specific indication, that is, heparin for burn injury In an SR on heparin to treat burn injury, the review team determined very early in its process that observational data should be included. Based on preliminary, cursory reviews of the literature and input from experts, the authors determined that there were few (if any) RCTs on the use of heparin for this indication. Therefore, they decided to include all types of studies that included a comparison group before running the main literature searches. SOURCES: Adapted from Norris et al. (2010), including Bravata et al. (2007); Helfand and Peterson (2003); McDonagh et al. (2008); and Oremus et al. (2006).

OCR for page 17
24 FINDING WHAT WORKS IN HEALTH CARE ers of CPGs. The focus is on the development and reporting of com- prehensive, publicly funded SRs of the comparative effectiveness of therapeutic medical or surgical interventions. The recent health reform legislation underscores the imperative for establishing SR standards, calling for a new research institute similar to the national program envisioned in Knowing What Works. The Patient Protection and Affordable Care Act of 20104 created the nation’s first nonprofit, public–private Patient-Centered Outcomes Research Institute (PCORI). It will be responsible for establishing and implementing a research agenda—including SRs of CER—to help patients, clinicians, policy makers, and purchasers in making informed healthcare decisions. As this report was being developed, the plans for PCORI were underway. An initial task of the newly appointed PCORI governing board is to establish a standing meth- odology committee charged with developing and improving the science and methods of CER. The IOM committee undertook its work with the intention to inform the PCORI methodology commit - tee’s own standards development. The IOM committee also views other public sponsors of SRs of CER as key audiences for this report, including the AHRQ Effective Health Care Program, Medicare Evi- dence Development & Coverage Advisory Committee (MEDCAC), Drug Effectiveness Research Project (DERP), National Institutes of Health, Centers for Disease Control and Prevention, and U.S. Pre- ventive Services Task Force. See Table 1-1 for a brief overview of the statutory requirements for PCORI. Outside the Scope of the Study As noted earlier, this report focuses on methods for producing comprehensive, publicly funded SRs of the comparative effective - ness of therapeutic interventions. The report’s recommended stan- dards are not intended for SRs initiated and conducted for purely academic purposes. Nor does the report address SR methods for synthesizing research on diagnostic tests, disease etiology or prog- nosis, systems improvement, or patient safety practices. The evi- dence base and expert guidance for SRs on these topics is consider- ably less advanced. For example, while the Cochrane Collaboration issued its fifth edition of its handbook for SRs of interventions in 2008 (Higgins and Green, 2008), a Cochrane diagnostics handbook is still under development (Cochrane Collaboration Diagnostic Test 4 Public Law 111-148.

OCR for page 17
25 INTRODUCTION TABLE 1-1 Statutory Requirements for the Patient-Centered Outcomes Research Institute Topic Provisions Purpose •    o assist patients, clinicians, policy makers, and purchasers  T in making informed health decisions by identifying and analyzing: o National research priorities o New clinical evidence and evidentiary gaps o Relevance of evidence and economic effects Organization •    onprofit corporation N •    ot an agency or establishment of the U.S. government N Funding •    iscal years (FYs) 2010–2012: Direct appropriations of $10  F million, $50 million, and $150 million per year, respectively •    Ys 2013–2019: Trust fund with annual inflow of $150  F million in appropriations plus annual per-capita charges per enrollee from Medicare, health insurance, and self-insured health plans •    fter FY 2019: No funds available from trust fund A Oversight •    ublic–private board of governors; 19 members include  P Agency for Healthcare Research and Quality (AHRQ) and National Institutes of Health (NIH) designees •    ethodology committee to develop and update science- M based methodological standards; include AHRQ and NIH Research •    ill award contracts for peer-reviewed research W •    uthorized to enter into contracts with outside entities to  A manage funding and conduct research; preference given to AHRQ and NIH if research is authorized by their governing statutes Dissemination •    ake research findings available within 90 days M and •    HRQ, in consultation with NIH, will broadly disseminate  A transparency research findings •    rovide public comment periods for major actions P •    stablish publicly available resource database  E SOURCE: Clancy and Collins (2010). Accuracy Working Group, 2011). AHRQ methods guidance for SRs of diagnostics and prognosis is also underway. Finally, the utility of an SR is only as good as the body of indi- vidual studies available. A considerable literature documents the shortcomings of reports of individual clinical trials and observational research (Altman et al., 2001; Glasziou et al., 2008; Hopewell et al., 2008b; Ioannidis et al., 2004; Plint et al., 2006; von Elm et al., 2007). This report will emphasize that the quality of individual studies must be scrutinized during the course of an SR. However, it is beyond the scope of this report to examine the many quality-scoring systems that

OCR for page 17
26 FINDING WHAT WORKS IN HEALTH CARE have been developed to measure the quality of individual research studies (Brand, 2009; Hopewell et al., 2008a; Moher et al., 2010). Relationship with the Committee on Standards for Developing Trustworthy Clinical Practice Guidelines The boundaries of this study were defined in part by the work of the companion CPG study (Box 1-4). A coordinating group5 for the two committees met regularly to consider the interdependence of SRs and CPGs and to minimize duplication of effort. The coordinat- ing group agreed early on that SRs are critical inputs to the guideline development process. It also decided that the SR committee would limit its focus to the development of SRs, starting with the formu- lation of the research question and ending with the completion of a final report—while paying special attention to the role of SRs in supporting CPGs. At the same time, the CPG committee would work under the assumption that guideline developers have access to high- quality SRs (as defined by the SR committee’s recommended standards) that address their specific research questions, and would discuss what steps in an SR are particularly important for a CPG. In Chapter 2 of this report, the SR committee addresses how the SR and CPG teams may interact when an SR is being conducted to inform a specific CPG. CONCEPTUAL FRAMEWORK Fundamentals of Systematic Reviews Experts agree on many of the key attributes of a high-quality SR (CRD, 2009; Higgins and Green, 2008; Owens et al., 2010). The objec- tive of an SR is to answer a specific research question by using an explicit, preplanned protocol to identify, select, assess, and summa - rize the findings of similar but separate studies. SRs often include— but do not require—a quantitative synthesis (meta-analysis). The SR process can be summarized in six steps: Step 1: Initiate the process, organize the review team, develop a process for gathering user and stakeholder input, formulate the research question, and implement procedures for minimiz- 5 The six-member coordinating group included the chair, vice chair, and one other individual from each committee.

OCR for page 17
27 INTRODUCTION BOX 1-4 Charge to the Committee on Standards for Developing Trustworthy Clinical Practice Guidelines An ad hoc committee will conduct a study to recommend standards for developing clinical practice guidelines and recommendations. The stan- dards should ensure that clinical practice guidelines are unbiased, scientifi- cally valid, and trustworthy and also incorporate separate grading systems for characterizing quality of available evidence and strength of clinical rec- ommendations. In this context, the committee should: 1. Assess whether, if widely adopted, any existing set of standards would assure the development of unbiased, scientifically valid, and trustworthy clinical practice guidelines. 2. Endorse an existing set of standards for developing clinical practice guidelines. If the committee judges current standards to be inad- equate, it will develop a new set of standards. 3. Determine best practices for promoting voluntary adoption of the standards. ing the impact of bias and conflict of interests (see standards in Chapter 2). Step 2: Develop the review protocol, including the context and rationale for the review and the specific procedures for the search strategy, data collection and extraction, qualitative synthesis and quantitative data synthesis (if a meta-analysis is done), report- ing, and peer review (see standards in Chapter 2). Step 3: Systematically locate, screen, and select the studies for review (see standards in Chapter 3). Step 4: Appraise the risk of bias in the individual studies and extract the data for analysis (see standards in Chapter 3). Step 5: Synthesize the findings and assess the overall quality of the body of evidence (see standards in Chapter 4). Step 6: Prepare a final report and have the report undergo peer review (see standards in Chapter 5). SRs of CER can be narrow in scope and consist of simple compar- isons, such as drug X versus drug Y. They can also address broader topics including comparisons of the effectiveness of drugs versus surgery for a condition, or “watchful waiting” when it is a reason-

OCR for page 17
34 FINDING WHAT WORKS IN HEALTH CARE inadequate to implement all of them, or if some seem inappropriate for the particular task or question at hand. Transparency in report- ing the methods actually used and the reasoning behind the choices are among the most important of the standards recommended by the committee. CURRENT LANDSCAPE This section provides a brief overview of the major producers, users, and other stakeholders involved in SRs. Producers of Systematic SRs A number of public- and private-sector organizations produce SRs. As noted earlier, the committee focused much of its review on the methods of AHRQ, the Cochrane Collaboration, and CRD. However, many other organizations play a key role in sponsoring, conducting, and disseminating SRs. Some of the key U.S. and inter- national organizations are described below. U.S. Organizations In the United States, the federal government funds a number of SRs, primarily through the AHRQ EPCs (Table 1-3). Private orga- nizations also conduct SRs of CER, including the Blue Cross and Blue Shield Association’s Technology Evaluation Center, the ECRI Institute, and Hayes, Inc. (Table 1-4). International Organizations The U.S. SR enterprise is part of a larger international effort focused on SRs. Many international organizations have advanced and highly sophisticated SR programs that not only produce SRs, but also focus on how best to conduct SRs. Table 1-5 describes sev- eral leading international SR organizations. Users and Stakeholders This report uses the terms “users” and “stakeholders” to refer to individuals and organizations that are likely to consult a specific SR to guide decision making or who have a particular interest in the outcome of an SR. Table 1-6 lists examples of user and stake- holder organizations that use SRs to inform decision making. The

OCR for page 17
35 INTRODUCTION TABLE 1-3 Examples of U.S. Governmental Organizations That Produce Systematic Reviews Organization Description Agency for Healthcare In 1997, AHRQ established 12 Evidence-based Research and Quality Practice Centers (EPCs) to promote evidence- (AHRQ) Effective Health based practice in everyday care. AHRQ awards Care Program 5-year contracts to EPCs to develop evidence reports and technology assessments. Currently, there are 14 EPCs in university and private settings. The U.S. Department of Veterans Affairs, the U.S. Preventive Services Task Force, and the Centers for Medicare & Medicaid Services use EPC reviews. Centers for Disease The CDC supports two programs for systematic Control and Prevention reviews, the Guide to Community Preventive (CDC) Services, initiated in 1996 and focusing on synthesizing evidence related to public health interventions, and the HIV/AIDS Prevention Research Synthesis, established in 1996 to review and summarize HIV behavioral prevention research literature. Substance Abuse and Since 1997 SAMHSA has provided information Mental Health Services about the scientific basis and practicality of Administration interventions that prevent or treat mental health (SAMHSA) and substance abuse disorders through the National Registry of Evidence-based Programs and Practices. SOURCES: Adapted from GAO (2009), IOM (2008). report focuses on four major categories of users and stakeholders: (1) consumers, including patients, families, and informal (or unpaid) caregivers; (2) clinicians, including physicians, nurses, and other healthcare professionals; (3) payers; and (4) policy makers, including guideline developers and other SR sponsors. ORGANIZATION OF THE REPORT Chapter Objectives This introductory chapter has described the background, charge to the committee, study scope, conceptual framework, current land- scape, and methods for this report. Chapter 2 through Chapter 5 present the committee’s review of and recommended standards for the basic steps in an SR. Chapter 6 provides a summary of the com- mittee’s conclusions and recommendations.

OCR for page 17
36 FINDING WHAT WORKS IN HEALTH CARE TABLE 1-4 Examples of Private U.S. Organizations That Produce Systematic Reviews Organization Description Blue Cross and Blue BCBSA founded TEC in 1985 to provide decision Shield Association makers with objective assessments of comparative (BCBSA), Technology effectiveness. TEC serves a wide range of clients in Evaluation Center both the private and public sectors, including Kaiser (TEC) Permanente and the Centers for Medicare & Medicaid Services. TEC is a designated Evidence-based Practice Center (EPC), and its products are publicly available. ECRI Institute The ECRI Institute is a nonprofit organization that provides technology assessments and cost-effectiveness analyses to ECRI Institute members and clients, includ- ing hospitals; health systems; public and private payers; U.S. federal and state government agencies; and min- istries of health, voluntary-sector organizations, asso- ciations, and accrediting agencies. Its products and methods are generally not available to the public. The ECRI Institute is a designated EPC and is also a Col- laborating Center for the World Health Organization. Hayes, Inc. Hayes, Inc., is a for-profit organization, established in 1989, to develop health technology assessments for health organizations, including health plans, managed-care companies, hospitals, and health networks. Hayes, Inc., produces several professional products, including the Hayes Briefs, the Hayes Directory, and the Hayes Outlook. Its products and methods are generally not available to the public. SOURCE: Adapted from IOM (2008). Chapter 2, Standards for Initiating a Systematic Review, focuses on the early steps in an SR that define the objectives of the review and influence its ultimate relevance to clinical deci- sions: establishing the review team, ensuring user and stake- holder input, managing bias and conflict of interest, and formu- lating the research topic and review protocol. Chapter 3, Standards for Finding and Assessing Individual Studies, focuses on a central step in the SR process: the iden- tification, collection, screening, and appraisal of the individual studies that make up an SR’s body of evidence. Chapter 4, Standards for Synthesizing the Body of Evidence, focuses on considerations in the synthesis and assessment of the body of evidence that are key to ensuring objectivity, transpar- ency, and scientific rigor.

OCR for page 17
37 INTRODUCTION TABLE 1-5 Examples of International Organizations That Produce Systematic Reviews Organization Description Cochrane Founded in 1993, the Cochrane Collaboration is an Collaboration independent, nonprofit multinational organization that produces systematic reviews (SRs) of healthcare interventions. Cochrane SRs are prepared by researchers who work with one or more of 52 Cochrane Review Groups that are overseen by an elected Steering Committee. Editorial teams oversee the preparation and maintenance of the SRs and the application of quality standards. Cochrane’s global contributors and centers are funded by government agencies and private sources; its central infrastructure is supported by subscriptions to The Cochrane Library. Commercial funding of review groups is not allowed. Cochrane review abstracts and plain-language summaries are free; complete SRs are available via subscription. The Cochrane Database of SRs includes more than 6,000 protocols and SRs. Centre for CRD is part of the National Institute for Health Research Reviews and (NIHR) and a department of the University of York in Dissemination the UK. Founded in 1994, CRD produces SRs of health (CRD) interventions, SR methods research, and guidance for conducting SRs. CRD also produces the Database of Abstracts of Reviews of Effects (DARE), the National Health Service Economic Evaluation Database, and the Health Technology Assessment Database, which are used internationally by health professionals, policy makers, and researchers. An international prospective registry of SRs utilizing existing database infrastructure is also under development. The DARE includes over 19,000 records of SRs of health care interventions, including more than 10,000 critical abstracts, which summarize the methods and findings of published reviews—highlighting their strengths and weaknesses. Approximately 1,200 new critical abstracts are added to DARE annually. CRD is funded primarily through NIHR with some funding from other government agencies. To avoid conflict of interest, CRD has a policy not to undertake research for or receive funds from the pharmaceutical or medical devices industries. Campbell The Campbell Collaboration is an international research Collaboration network that produces SRs of the effects of social interventions. It was established in 2000 and has five Coordinating Groups: Social Welfare, Crime and Justice, Education, Methods, and Users. The Coordinating Groups oversee the production, scientific merit, and relevance of the SRs. Final SRs are published in the peer-reviewed monograph series, Campbell Systematic Reviews. The International Secretariat is hosted by the Norwegian Centre for the Health Services. continued

OCR for page 17
38 FINDING WHAT WORKS IN HEALTH CARE TABLE 1-5 Continued Organization Description National NICE was established in 1999 as part of the U.K.’s National Institute Health Service (NHS). It provides guidance to NHS, sets for Health quality standards, and manages a national database to and Clinical improve health and prevent and treat ill health. NICE Excellence commissions SRs on new and existing technologies from (NICE) independent academic centers. NICE then uses the SRs to make recommendations to NHS on how a technology should be used in NHS. SOURCES: Information on the Cochrane Collaboration was adapted from IOM (2008). Information on CRD and the Campbell Collaboration: The Campbell Collaboration (2010); CRD (2010); NICE (2010). TABLE 1-6 Examples of Organizations That Use Systematic Reviews Organization Description Drug DERP is a collaboration of public and private organizations, Effectiveness including 13 state programs, which develops reports Review Project assessing the comparative effectiveness and safety of drugs (DERP) within particular drug classes. Evidence-based Practice Centers (EPCs) conduct evidence reviews for DERP. State Medicaid programs have used this information to develop their drug formularies. Medicare The Centers for Medicare & Medicaid (CMS) established the Evidence Medicare Coverage Advisory Committee (now the Medicare Development & Evidence Development & Coverage Advisory Committee Coverage [MedCAC]) in 1998 to provide independent expert advice Advisory to CMS on specific clinical topics. MedCAC reviews and Committee evaluates the medical literature and technology assessments (MedCAC) on medical items and services that are under evaluation at CMS, including systematic reviews (SRs) produced by the EPCs and other producers of SRs. MedCAC can be an integral part of the national coverage determination process. MedCAC is advisory in nature; CMS is responsible for all final decisions. NIH Consensus CDP produces consensus statements on the effects of Development healthcare interventions. CDP convenes independent Program panels of researchers, health professionals, and public (CDP) representatives who consider the literature reviews conducted by EPCs, as well as expert testimony. Topics are chosen based on their public health importance, prevalence, controversy, potential to reduce gaps between knowledge and practice, availability of scientific information, and potential impact on healthcare costs.

OCR for page 17
39 INTRODUCTION TABLE 1-6 Continued Organization Description Performance Performance measurement organizations track and evaluate measurement provider performance by measuring providers’ actual organizations clinical practices against the recommended practices. To conduct this work, these organizations typically establish standards of care based on SRs, against which the performance of providers can be assessed. Examples of performance measurement organizations include the AQA Alliance and the National Quality Forum. Professional Many professional medical societies have instituted processes medical and directed resources to developing clinical practice societies guidelines on the basis of systematic reviews. Examples of societies with well-established guideline development procedures include the American College of Cardiology/ American Heart Association, American College of Chest Physicians, American Academy of Neurology, and American Academy of Pediatrics. U.S. Preventive The USPSTF consists of a panel of private-sector experts Services Task that makes recommendations about which preventive Force services should be incorporated routinely into primary (USPSTF) medical care. Its evidence-based recommendations are regarded as the “gold standard” for clinical preventive services. USPSTF is supported by an EPC, which conducts systematic reviews on relevant clinical prevention topics. SOURCE: Adapted from IOM (2008). Chapter 5, Standards for Reporting Systematic Reviews, focuses on the components of an SR final report that are fundamental to its eventual utility for patients, clinicians, and others. Chapter 6, Improving the Quality of Systematic Reviews: Discussion, Conclusions, and Recommendations, presents the committee’s conclusions and recommendations for advancing the science underlying SR methods and for providing a more supportive environment for the conduct of SRs. REFERENCES Altman, D. G., K. F. Schulz, D. Moher, M. Egger, F. Davidoff, D. Elbourne, P. C. Gøtzsche, and T. Lang. 2001. The revised CONSORT statement for reporting randomized trials: Explanation and elaboration. Annals of Internal Medicine 134(8):663–694. Bassler, D., I. Ferreira-Gonzalez, M. Briel, D. J. Cook, P. J. Devereaux, D. Heels-Ansdell, H. Kirpalani, M. O. Meade, V. M. Montori, A. Rozenberg, H. J. Schünemann, and G. H. Guyatt. 2007. Systematic reviewers neglect bias that results from trials stopped early for benefit. Journal of Clinical Epidemiology 60(9):869–873.

OCR for page 17
40 FINDING WHAT WORKS IN HEALTH CARE Boyd, C. M., J. Darer, C. Boult, L. P. Fried, L. Boult, and A. W. Wu. 2005. Clinical practice guidelines and quality of care for older patients with multiple comorbid diseases: Implications for pay for performance. JAMA 294(6):716–724. Brand, R. A. 2009. Standards of reporting: The CONSORT, QUORUM, and STROBE guidelines. Clinical Orthopaedics and Related Research 467(6):1393–1394. Bravata, D. M., K. M. McDonald, A. L. Gienger, V. Sundaram, M. V. Perez, R. Varghese, J. R. Kapoor, R. Ardehali, M. C. McKinnon, C. D. Stave, D. K. Owens, and M. Hlatky. 2007. Comparative effectiveness of percutaneous coronary interventions and coronary artery bypass grafting for coronary artery disease. Rockville, MD: AHRQ. The Campbell Collaboration. 2010. About us. http://www.campbellcollaboration. org/about_us/index.php (accessed September 22, 2010). Chou, R., N. Aronson, D. Atkins, A. S. Ismaila, P. Santaguida, D. H. Smith, E. Whitlock, T. J. Wilt, and D. Moher. 2010. Assessing harms when comparing medical in- terventions: AHRQ and the Effective Health-Care Program. Journal of Clinical Epidemiology 63(5):502–512. Clancy, C., and F. S. Collins. 2010. Patient-Centered Outcomes Research Institute: The intersection of science and health care. Science Translational Medicine 2(37): 37cm18. Cochrane Collaboration Diagnostic Test Accuracy Working Group. 2011. Handbook for DTA reviews. http://srdta.cochrane.org/handbook-dta-reviews (accessed March 15, 2011). Colliver, J. A., K. Kucera, and S. J. Verhulst. 2008. Meta-analysis of quasi-experimental research: Are systematic narrative reviews indicated? Medical Education 42(9):858–865. CRD (Centre for Reviews and Dissemination). 2009. Systematic reviews: CRD’s guidance for undertaking reviews in health care. York, U.K.: York Publishing Services, Ltd. CRD. 2010. About CRD. http://www.york.ac.uk/inst/crd/about_us.htm (accessed September 22, 2010). Delaney, A., S. M. Bagshaw, A. Ferland, K. Laupland, B. Manns, and C. Doig. 2007. The quality of reports of critical care meta-analyses in the Cochrane Database of Sys- tematic Reviews: An independent appraisal. Critical Care Medicine 35(2):589–594. Edwards, P., M. Clarke, C. DiGuiseppi, S. Pratap, I. Roberts, and R. Wentz. 2002. Identification of randomized controlled trials in systematic reviews: Accuracy and reliability of screening records. Statistics in Medicine 21:1635–1640. GAO (Government Accountability Office). 2009. Program evaluation: A variety of rigor- ous methods can help identify effective interventions. Vol. GAO-10-30. Washington, DC: GAO. Glasziou, P., J. Vandenbroucke, and I. Chalmers. 2004. Assessing the quality of re- search. BMJ 328(7430):39–41. Glasziou, P., E. Meats, C. Heneghan, and S. Shepperd. 2008. What is missing from descriptions of treatment in trials and reviews? BMJ 336(7659):1472–1474. Glenny, A. M., D. G. Altman, F. Song, C. Sakarovitch, J. J. Deeks, R. D’Amico, M. Bradburn, and A. J. Eastwood. 2005. Indirect comparisons of competing inter- ventions. Health Technology Assessment 9(26):1–134. Glenton, C., V. Underland, M. Kho, V. Pennick, and A. D. Oxman. 2006. Summaries of findings, descriptions of interventions, and information about adverse ef - fects would make reviews more informative. Journal of Clinical Epidemiology 59(8):770–778. Helfand, M., and K. Peterson. 2003. Drug class review on the triptans: Drug Effectiveness Review Project. Portland, OR: Oregon Evidence-based Practice Center. Higgins, J. P. T., and S. Green, eds. 2008. Cochrane handbook for systematic reviews of interventions. Chichester, UK: John Wiley & Sons.

OCR for page 17
41 INTRODUCTION Hopewell, S., M. Clarke, D. Moher, E. Wager, P. Middleton, D. G. Altman, and K. F. Schulz. 2008a. CONSORT for reporting randomized controlled trials in journal and conference abstracts: Explanation and elaboration. PLoS Medicine 5(1):e20. Hopewell, S., L. Wolfenden, and M. Clarke. 2008b. Reporting of adverse events in systematic reviews can be improved: Survey results. Journal of Clinical Epidemiol- ogy 61(6):597–602. Ioannidis, J. P., J. W. Evans, P. C. Gøtzsche, R. T. O’Neill, D. Altman, K. Schulz, and D. Moher. 2004. Better reporting of harms in randomized trials: An extension of the CONSORT Statement. Ann Intern Med 141:781–788. IOM (Institute of Medicine). 2001. Crossing the quality chasm: A new health system for the 21st century. Washington, DC: National Academy Press. IOM. 2008. Knowing what works in health care: A roadmap for the nation. Edited by J. Eden, B. Wheatley, B. J. McNeil, and H. Sox. Washington, DC: The National Academies Press. IOM. 2009. Initial national priorities for comparative effectiveness research. Washington, DC: The National Academies Press. Laopaiboon, M. 2003. Meta-analyses involving cluster randomization trials: A review of published literature in health care. Statistical Methods in Medical Research 12(6):515–530. Last, J. M., ed. 1995. A dictionary of epidemiology, 3rd ed. New York: Oxford University Press. Liberati, A., D. G. Altman, J. Tetzlaff, C. Mulrow, P. C. Gøtzsche, J. P. A. Ioannidis, M. Clarke, P. J. Devereaux, J. Kleijnen, and D. Moher. 2009. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: Explanation and elaboration. Annals of Internal Medi- cine 151(4):W1–W30. Lohr, K. N. 2004. Rating the strength of scientific evidence: Relevance for quality im - provement programs. International Journal for Quality in Health Care 16(1):9–18. Luce, B. R., M. Drummond, B. Jönsson, P. J. Neumann, J. S. Schwartz, U. Siebert, and S. D. Sullivan. 2010. EBM, HTA, and CER: Clearing the confusion. Milbank Quarterly 88(2):256–276. Lundh, A., S. L. Knijnenburg, A. W. Jorgensen, E. C. van Dalen, and L. C. M. Kremer. 2009. Quality of systematic reviews in pediatric oncology—A systematic review. Cancer Treatment Reviews 35(8):645–652. Mallen, C., G. Peat, and P. Croft. 2006. Quality assessment of observational stud - ies is not commonplace in systematic reviews. Journal of Clinical Epidemiology 59(8):765–769. McDonagh, M., K. Peterson, S. Carson, R. Fu, and S. Thakurta. 2008. Drug class review: Atypical antipsychotic drugs. Update 3. Portland, OR: Oregon Evidence-based Practice Center. Moher, D., B. Pham, T. P. Klassen, K. F. Schulz, J. A. Berlin, A. R. Jadad, and A. Liberati. 2000. What contributions do languages other than English make on the results of meta-analyses? Journal of Clinical Epidemiology 53 (9):964–972. Moher, D., B. Pham, M. L. Lawson, and Klassen T. P. 2003. The inclusion of reports of randomised trials published in languages other than English in systematic reviews. Health Technology Assessment 7 (41):1–90. Moher, D., J. Tetzlaff, A. C. Tricco, M. Sampson, and D. G. Altman. 2007. Epidemiology and reporting characteristics of systematic reviews. PLoS Medicine 4(3):447–455. Moher, D., K. F. Schulz, I. Simera, and D. G. Altman. 2010. Guidance for developers of health research reporting guidelines. PLoS Med 7(2):e1000217.

OCR for page 17
42 FINDING WHAT WORKS IN HEALTH CARE Morrison, A., K. Moulton, M. Clark, J. Polisena, M. Fiander, M. Mierzwinski-Urban, S. Mensinkai, T. Clifford, and B. Hutton. 2009. English-language restriction when conducting systematic review-based meta-analyses: Systematic review of published stud- ies. Ottawa, CA: Canadian Agency for Drugs and Technologies in Health. NICE (National Institute for Health and Clinical Excellence). 2010. About NICE. http:// www.nice.org.uk/aboutnice/about_nice.jsp (accessed October 27, 2010). Norris, S., D. Atkins, W. Bruening, S. Fox, E. Johnson, R. Kane, S. C. Morton, M. Oremus, M. Ospina, G. Randhawa, K. Schoelles, P. Shekelle, and M. Viswanathan. 2010. Selecting observational studies for comparing medical interventions. In Methods guide for comparative effectiveness reviews. http://effectivehealthcare.ahrq. gov/ehc/products/196/454/MethodsGuideNorris_06042010.pdf (accessed No- vember 8, 2010). Oremus, M., M. Hanson, R. Whitlock, E. Young, A. Gupta, A. Dal Cin, C. Archer, and P. Raina. 2006. The uses of heparin to treat burn injury. Evidence Report/ Technology Assessment No. 148. AHRQ Publication No. 07-E004. Rockville, MD: AHRQ. Owens, D. K., K. N. Lohr, D. Atkins, J. R. Treadwell, J. T. Reston, E. B. Bass, S. Chang, and M. Helfand. 2010. AHRQ Series Paper 5: Grading the strength of a body of evidence when comparing medical interventions: AHRQ and the Effective Health-Care Program. Journal of Clinical Epidemiology 63 (5):513–523. Pham, H. H., D. Schrag, A. S. O’Malley, B. Wu, and P. B. Bach. 2007. Care patterns in Medicare and their implications for pay for performance. New England Journal of Medicine 356(11):1130–1139. Plint, A. C., D. Moher, A. Morrison, K. Schulz, D. G. Altman, C. Hill, and I. Gaboury. 2006. Does the CONSORT checklist improve the quality of reports of randomised controlled trials? A systematic review. Medical Journal of Australia 185(5):263– 267. Reeves, B. C., J. J. Deeks, J. Higgins, and G. A. Wells. 2008. Chapter 13: Including non - randomized studies. In Cochrane handbook for systematic reviews of interventions, edited by J. P. T. Higgins and G. S. West. Chichester, UK: John Wiley & Sons. Roundtree, A. K., M. A. Kallen, M. A. Lopez-Olivo, B. Kimmel, B. Skidmore, Z. Ortiz, V. Cox, and M. E. Suarez-Almazor. 2008. Poor reporting of search strategy and conflict of interest in over 250 narrative and systematic reviews of two biologic agents in arthritis: A systematic review. Journal of Clinical Epidemiology 62(2):128–137. Schünemann, H., D. Best, G. Vist, and A. D. Oxman. 2003. Letters, numbers, symbols and words: How to communicate grades of evidence and recommendations. Canadian Medical Association Journal 169(7):677–680. Slone Survey. 2006. Patterns of medication use in the United States. Boston, MA: Slone Epidemiology Center. Song, F., Y. K. Loke, T. Walsh, A. M. Glenny, A. J. Eastwood, and D. G. Altman. 2009. Methodological problems in the use of indirect comparisons for evalu - ating healthcare interventions: Survey of published systematic reviews. BMJ 338:b1147. Steinberg, E. P., and B. R. Luce. 2005. Evidence based? Caveat emptor! [editorial]. Health Affairs (Millwood) 24(1):80–92. Tinetti, M. E., S. T. Bogardus, Jr., and J. V. Agostini. 2004. Potential pitfalls of disease- specific guidelines for patients with multiple conditions. New England Journal of Medicine 351(27):2870–2874. Tricco, A. C., J. Tetzlaff, M. Sampson, D. Fergusson, E. Cogo, T. Horsley, and D. Moher. 2008. Few systematic reviews exist documenting the extent of bias: A systematic review. Journal of Clinical Epidemiology 61(5):422–434.

OCR for page 17
43 INTRODUCTION Turner, E. H., A. M. Matthews, E. Linardatos, R. A. Tell, and R. Rosenthal. 2008. Selec - tive publication of antidepressant trials and its influence on apparent efficacy. New England Journal of Medicine 358(3):252–260. Vogeli, C., A. Shields, T. Lee, T. Gibson, W. Marder, K. Weiss, and D. Blumenthal. 2007. Multiple chronic conditions: Prevalence, health consequences, and implications for quality, care management, and costs. Journal of General Internal Medicine 22(Suppl. 3):391–395. von Elm, E., D. G. Altman, M. Egger, S. J. Pocock, P. C. Gotzsche, and J. P. Vandenbroucke. 2007. The strengthening the reporting of observational studies in epidemiology (STROBE) statement: Guidelines for reporting observational studies. Annals of Internal Medicine 147(8):573–577.

OCR for page 17