Summary1

Healthcare decision makers in search of the best evidence to inform clinical decisions have come to rely on systematic reviews (SRs) of comparative effectiveness research (CER) to learn what is known and not known about the potential benefits and harms of alternative drugs, devices, and other healthcare services. An SR is a scientific investigation that focuses on a specific question and uses explicit, prespecified scientific methods to identify, select, assess, and summarize the findings of similar but separate studies. It may include a quantitative synthesis (meta-analysis), depending on the available data. Although the importance of SRs is increasingly appreciated, the quality of published SRs is variable and often poor. In many cases, the reader is unable to judge the quality of an SR because the methods are poorly documented, and even if methods are described, they may be used inappropriately, for example, in meta-analyses. Many reviews fail to assess the quality of the underlying research and also neglect to report funding sources. A plethora of conflicting approaches to evidence hierarchies and grading schemes for bodies of evidence is a further source of confusion.

In the 2008 report, Knowing What Works in Health Care: A Roadmap for the Nation, the Institute of Medicine (IOM) recommended that methodological standards be developed for both SRs and clini-

1

This summary does not include references. Citations for the findings presected in the Summary appear in the subsequent chapters.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 1
Summary1 Healthcare decision makers in search of the best evidence to inform clinical decisions have come to rely on systematic reviews (SRs) of comparative effectiveness research (CER) to learn what is known and not known about the potential benefits and harms of alternative drugs, devices, and other healthcare services. An SR is a scientific investigation that focuses on a specific question and uses explicit, prespecified scientific methods to identify, select, assess, and summarize the findings of similar but separate studies. It may include a quantitative synthesis (meta-analysis), depending on the available data. Although the importance of SRs is increas- ingly appreciated, the quality of published SRs is variable and often poor. In many cases, the reader is unable to judge the quality of an SR because the methods are poorly documented, and even if meth- ods are described, they may be used inappropriately, for example, in meta-analyses. Many reviews fail to assess the quality of the underlying research and also neglect to report funding sources. A plethora of conflicting approaches to evidence hierarchies and grad- ing schemes for bodies of evidence is a further source of confusion. In the 2008 report, Knowing What Works in Health Care: A Road- map for the Nation, the Institute of Medicine (IOM) recommended that methodological standards be developed for both SRs and clini - 1 This summary does not include references. Citations for the findings presected in the Summary appear in the subsequent chapters. 1

OCR for page 1
2 FINDING WHAT WORKS IN HEALTH CARE cal practice guidelines (CPGs). The report was followed by a con- gressional mandate in the Medicare Improvements for Patients and Providers Act of 2008 for two follow-up IOM studies: one to develop standards for conducting SRs, and the other to develop standards for CPGs. This is the report of the IOM Committee on Standards for Systematic Reviews of Comparative Effectiveness Research. A companion report by the IOM Committee on Standards for Devel- oping Trustworthy Clinical Practice Guidelines is being released in conjunction with this report. The charge to this IOM committee was twofold: first, to assess potential methodological standards that would assure objective, transparent, and scientifically valid SRs of CER and, second, to recommend a set of methodological standards for developing and reporting such SRs (Box S-1). The boundaries of this study were defined in part by the work of the companion CPG study. The SR committee limited its focus to the development of SRs. At the same time, the CPG committee worked under the assumption that guide - line developers have access to and use high-quality SRs (as defined by the standards recommended in this report). This report presents methodological standards for SRs that are designed to inform everyday healthcare decision making, especially for patients, clinicians and other healthcare providers, and devel - BOX S-1 Charge to the Committee on Standards for Systematic Reviews of Comparative Effectiveness Research An ad hoc committee will conduct a study to recommend methodologi- cal standards for systematic reviews (SRs) of comparative effectiveness research (CER) on health and health care. The standards should ensure that the reviews are objective, transparent, and scientifically valid, and re- quire a common language for characterizing the strength of the evidence. Decision makers should be able to rely on SRs of comparative effectiveness to determine what is known and not known and to describe the extent to which the evidence is applicable to clinical practice and particular patients. In this context, the committee will: 1. Assess whether, if widely adopted, any existing set of standards would assure that SRs of comparative effectiveness research are objective, transparent, and scientifically valid. 2. Recommend a set of standards for developing and reporting SRs of CER.

OCR for page 1
3 SUMMARY opers of CPGs. The focus is on the development and reporting of comprehensive, publicly funded SRs of the comparative effective - ness of therapeutic medical or surgical interventions. The recent health reform legislation underscores the imperative for establish - ing standards to ensure the highest quality SRs. The Patient Pro- tection and Affordable Care Act of 2010 (ACA) created the nation’s first nonprofit, public–private Patient-Centered Outcomes Research Institute (PCORI). PCORI will be responsible for establishing and implementing a research agenda—including SRs of CER—to help patients, clinicians and other healthcare providers, purchasers, and policy makers make informed healthcare decisions. As this report was being developed, planning for PCORI was underway. An initial task of the newly appointed governing board of the institute is to establish a standing methodology committee charged with develop- ing and improving the science and methods of CER. The IOM committee undertook its work with the intention to inform the PCORI methodology committee’s own standards devel- opment. The IOM committee also views other public sponsors of SRs of CER as key audiences for this report, including the Agency for Healthcare Research and Quality (AHRQ) Effective Health Care Program, the Centers for Medicaid and Medicare Coverage Advisory Committee, the Drug Effectiveness Research Project, the National Institutes of Health (NIH), the Centers for Disease Control and Prevention (CDC), and the U.S. Preventive Services Task Force. PURPOSE OF SETTING STANDARDS Organizations establish standards to set performance expecta- tions and to promote accountability for meeting these expectations. For SRs in particular, the principal objective of setting standards is to minimize bias in identifying, selecting, and interpreting evidence. For the purposes of this report, the committee defined an SR “stan- dard” as a process, action, or procedure that is deemed essential to producing scientifically valid, transparent, and reproducible SRs. A standard may be supported by scientific evidence, by a reasonable expectation that the standard helps achieve the anticipated level of quality in an SR, or by the broad acceptance of the practice by authors of SRs. The evidence base for many of the steps in the SR process is sparse, especially with respect to linking characteristics of SRs to clinical outcomes, the ultimate test of quality. The committee devel- oped its standards and elements of performance based on available research evidence and expert guidance from the AHRQ Effective

OCR for page 1
4 FINDING WHAT WORKS IN HEALTH CARE Health Care Program; the Centre for Reviews and Dissemination (CRD) (University of York, United Kingdom); the Cochrane Collabo- ration; the Grading of Recommendations Assessment, Development, Evaluation (GRADE) Working Group2; and the Preferred Reporting Items for Systematic Reviews and Meta-Analyses group (PRISMA). The committee faced a difficult task in proposing a set of stan- dards where in general the evidence is thin and expert guidance varies. Yet the evidence that is available does not suggest that high- quality SRs can be done quickly and cheaply. SRs conducted with methods prone to bias do indeed often miss the boat, leading to clinical advice that may in the end harm patients. All of the commit - tee’s recommended standards are based on current evidence, expert guidance, and thoughtful reasoning, and are actively used by many experts, and thus are reasonable “best practices” for reducing bias and for increasing the scientific rigor of SRs of CER. However, all of the recommended standards must be considered provisional pend- ing better empirical evidence about their scientific validity, feasibil - ity, efficiency, and ultimate usefulness in medical decision making. The committee recommends 21 standards with 82 elements of performance, addressing the entire SR process, from the initial steps of formulating the topic, building a review team, and establishing a research protocol, to finding and assessing the individual studies that make up the body of evidence, to producing qualitative and quantitative syntheses of the body of evidence, and, finally, to devel- oping the final SR report. Each standard is articulated in the same format: first, a brief statement of the step in the SR process (e.g., in Chapter 3, Standard 3.1. Conduct a comprehensive systematic search for evidence) followed by a series of elements that are essential com- ponents of the standard. These “elements” are steps that should be taken for all publicly funded SRs of CER. Collectively the standards and elements present a daunting task. Few, if any, members of the committee have participated in an SR that fully meets all of them. Yet the evidence and experience are strong enough that it is impossible to ignore these standards or hope that one can safely cut corners. The standards will be especially valu- able for SRs of high-stakes clinical questions with broad population impact, where the use of public funds to get the right answer justi- fies careful attention to the rigor with which the SR is conducted. Individuals involved in SRs should be thoughtful about all of the 2GRADE was a primary source for Chapter 4 only. PRISMA was a primary source for Chapter 5 only.

OCR for page 1
5 SUMMARY standards and elements, using their best judgment if resources are inadequate to implement all of them, or if some seem inappropriate for the particular task or question at hand. Transparency in report- ing the methods actually used and the reasoning behind the choices are among the most important of the standards recommended by the committee. Initiating the SR Process The first steps in the SR process define the focus and methods of the SR and influence its ultimate utility for clinical decisions. Cur- rent practice falls far short of expert guidance; well-designed, well- executed SRs are the exception. (Note that throughout this report reference to “expert guidance” refers to the published methodologi- cal advice of the AHRQ Effective Health Care Program, CRD, and the Cochrane Collaboration.) The committee recommends eight standards for initiating the SR process, minimizing potential bias in the SR’s design and execution. The standards address the creation of the SR team, user and stakeholder input, managing bias and conflict of interest (COI), topic formulation, and development of the SR protocol (Box S-2). The SR team should include individuals with appropriate expertise and perspectives. Creating a mechanism for users and stakeholders—consumers, clinicians, payers, and mem - bers of CPG panels—to provide input into the SR process at multiple levels helps to ensure that the SR is focused on real-world health- care decisions. However, a process should be in place to reduce the risk of bias and COI from stakeholder input and in the SR team. The importance of the review questions and analytic framework in guiding the entire review process demands a rigorous approach to formulating the research questions and analytic framework. Requir- ing a research protocol that prespecifies the research methods at the outset of the SR process helps to prevent the effects of author bias, allows feedback at an early stage in the SR, and tells readers of the review about protocol changes that occur as the SR develops. Finding and Assessing Individual Studies The committee recommends six standards for identifying and assessing the individual studies that make up an SR’s body of evi - dence, including standards addressing the search process, screening and selecting studies, extracting data, and assessing the quality of individual studies (Box S-3). The objective of the SR search is to iden- tify all the studies (and all the relevant data from the studies) that

OCR for page 1
6 FINDING WHAT WORKS IN HEALTH CARE BOX S-2 Recommended Standards for Initiating a Systematic Review Standard 2.1 Establish a team with appropriate expertise and experi- ence to conduct the systematic review Required elements: 2.1.1 Include expertise in the pertinent clinical content areas 2.1.2 Include expertise in systematic review methods 2.1.3 Include expertise in searching for relevant evidence 2.1.4 Include expertise in quantitative methods 2.1.5 Include other expertise as appropriate Standard 2.2 Manage bias and conflict of interest (COI) of the team conducting the systematic review Required elements: 2.2.1 Require each team member to disclose potential COI and professional or intellectual bias 2.2.2 Exclude individuals with a clear financial conflict 2.2.3 Exclude individuals whose professional or intellectual bias would diminish the credibility of the review in the eyes of the intended users Standard 2.3 Ensure user and stakeholder input as the review is de- signed and conducted Required element: 2.3.1 Protect the independence of the review team to make the final decisions about the design, analysis, and reporting of the review Standard 2.4 Manage bias and COI for individuals providing input into the systematic review Required elements: 2.4.1 Require individuals to disclose potential COI and profes- sional or intellectual bias 2.4.2 Exclude input from individuals whose COI or bias would di- minish the credibility of the review in the eyes of the intended users Standard 2.5 Formulate the topic for the systematic review Required elements: 2.5.1 Confirm the need for a new review may pertain to the research question and analytic framework. The search should be systematic, use prespecified search parameters, and access an array of information sources that provide both pub- lished and unpublished research reports. Screening and selecting

OCR for page 1
7 SUMMARY 2.5.2 Develop an analytic framework that clearly lays out the chain of logic that links the health intervention to the outcomes of interest and defines the key clinical questions to be ad- dressed by the systematic review 2.5.3 Use a standard format to articulate each clinical question of interest 2.5.4 State the rationale for each clinical question 2.5.5 Refine each question based on user and stakeholder input Standard 2.6 Develop a systematic review protocol Required elements: 2.6.1 Describe the context and rationale for the review from both a decision-making and research perspective 2.6.2 Describe the study screening and selection criteria (inclu- sion/exclusion criteria) 2.6.3 Describe precisely which outcome measures, time points, interventions, and comparison groups will be addressed 2.6.4 Describe the search strategy for identifying relevant evidence 2.6.5 Describe the procedures for study selection 2.6.6 Describe the data extraction strategy 2.6.7 Describe the process for identifying and resolving dis- agreement between researchers in study selection and data extraction decisions 2.6.8 Describe the approach to critically appraising individual studies 2.6.9 Describe the method for evaluating the body of evi- dence, including the quantitative and qualitative synthesis strategies 2.6.10 Describe and justify any planned analyses of differential treatment effects according to patient subgroups, how an intervention is delivered, or how an outcome is measured 2.6.11 Describe the proposed timetable for conducting the review Standard 2.7 Submit the protocol for peer review Required element: 2.6.9 Provide a public comment period for the protocol and pub- licly report on disposition of comments Standard 2.8 Make the final protocol publicly available, and add any amendments to the protocol in a timely fashion studies should use methods that address the pervasive problems of SR author bias, errors, and inadequate documentation of the study selection process in SRs. Study methods should be reported in suf - ficient detail so that searches can be replicated and appraised. Qual -

OCR for page 1
8 FINDING WHAT WORKS IN HEALTH CARE BOX S-3 Recommended Standards for Finding and Assessing Individual Studies Standard 3.1 Conduct a comprehensive systematic search for evidence Required elements: 3.1.1 Work with a librarian or other information specialist trained in performing systematic reviews to plan the search strategy 3.1.2 Design the search strategy to address each key research question 3.1.3 Use an independent librarian or other information specialist to peer review the search strategy 3.1.4 Search bibliographic databases 3.1.5 Search citation indexes 3.1.6 Search literature cited by eligible studies 3.1.7 Update the search at intervals appropriate to the pace of generation of new information for the research question be- ing addressed 3.1.8 Search subject-specific databases if other databases are unlikely to provide all relevant evidence 3.1.9 Search regional bibliographic databases if other databases are unlikely to provide all relevant evidence Standard 3.2 Take action to address potentially biased reporting of research results Required elements: 3.2.1 Search grey-literature databases, clinical trial registries, and other sources of unpublished information about studies 3.2.2 Invite researchers to clarify information about study eligibil- ity, study characteristics, and risk of bias 3.2.3 Invite all study sponsors and researchers to submit unpub- lished data, including unreported outcomes, for possible inclusion in the systematic review 3.2.4 Handsearch selected journals and conference abstracts 3.2.5 Conduct a web search 3.2.6 Search for studies reported in languages other than English if appropriate Standard 3.3 Screen and select studies Required elements: 3.3.1 Include or exclude studies based on the protocol’s prespeci- fied criteria 3.3.2 Use observational studies in addition to randomized clinical trials to evaluate harms of interventions

OCR for page 1
9 SUMMARY 3.3.3 Use two or more members of the review team, working independently, to screen and select studies 3.3.4 Train screeners using written documentation; test and retest screeners to improve accuracy and consistency 3.3.5 Use one of two strategies to select studies: (1) read all full- text articles identified in the search or (2) screen titles and abstracts of all articles and then read the full texts of articles identified in initial screening 3.3.6 Taking account of the risk of bias, consider using observa- tional studies to address gaps in the evidence from random- ized clinical trials on the benefits of interventions Standard 3.4 Document the search Required elements: 3.4.1 Provide a line-by-line description of the search strategy, in- cluding the date of every search for each database, web browser, etc. 3.4.2 Document the disposition of each report identified including reasons for their exclusion if appropriate Standard 3.5 Manage data collection Required elements: 3.5.1 At a minimum, use two or more researchers, working inde- pendently, to extract quantitative and other critical data from each study. For other types of data, one individual could extract the data while the second individual independently checks for accuracy and completeness. Establish a fair pro- cedure for resolving discrepancies—do not simply give final decision-making power to the senior reviewer 3.5.2 Link publications from the same study to avoid including data from the same study more than once 3.5.3 Use standard data extraction forms developed for the spe- cific SR 3.5.4 Pilot-test the data extraction forms and process Standard 3.6 Critically appraise each study Required elements: 3.6.1 Systematically assess the risk of bias, using predefined criteria 3.6.2 Assess the relevance of the study’s populations, interven- tions, and outcome measures 3.6.3 Assess the fidelity of the implementation of interventions

OCR for page 1
10 FINDING WHAT WORKS IN HEALTH CARE ity assurance and control are essential when data are extracted from individual studies from the collected body of evidence. A thorough and thoughtful assessment of the validity and relevance of each eli- gible study helps ensure scientific rigor and promote transparency. Synthesizing the Body of Evidence The committee recommends four standards for the qualitative and quantitative synthesis and assessment of an SR’s body of evi- dence (Box S-4). The qualitative synthesis is an often undervalued component of an SR. Many SRs lack a qualitative synthesis alto- gether or simply recite the facts about the studies without examining them for patterns or characterizing the strengths and weaknesses BOX S-4 Recommended Standards for Synthesizing the Body of Evidence Standard 4.1 Use a prespecified method to evaluate the body of evidence Required elements: 4.1.1 For each outcome, systematically assess the following char- acteristics of the body of evidence: • Risk of bias • Consistency • Precision • Directness • Reporting bias 4.1.2 For bodies of evidence that include observational research, also systematically assess the following characteristics for each outcome: • Dose–response association • Plausible confounding that would change the observed effect • Strength of association 4.1.3 For each outcome specified in the protocol, use consistent language to characterize the level of confidence in the esti- mates of the effect of an intervention Standard 4.2 Conduct a qualitative synthesis Required elements: 4.2.1 Describe the clinical and methodological characteristics of the included studies, including their size, inclusion or exclu- sion of important subgroups, timeliness, and other relevant factors

OCR for page 1
11 SUMMARY of the body of evidence as a whole. If the SR is to be comprehen- sible, it should use consistent language to describe the quality of evidence for each outcome and incorporate multiple dimensions of study quality. For readers to have a clear understanding of how the evidence applies to real-world clinical circumstances and specific patient populations, SRs should describe—in easy-to-understand language—the clinical and methodological characteristics of the individual studies, including their strengths and weaknesses and their relevance to particular populations and clinical settings. It should also describe how flaws in the design or execution of the individual studies could bias the results. The qualitative synthesis is more than a narrative description or set of tables that simply detail how many studies were assessed, the reasons for excluding other 4.2.2 Describe the strengths and limitations of individual studies and patterns across studies 4.2.3 Describe, in plain terms, how flaws in the design or execu- tion of the study (or groups of studies) could bias the results, explaining the reasoning behind these judgments 4.2.4 Describe the relationships between the characteristics of the individual studies and their reported findings and patterns across studies 4.2.5 Discuss the relevance of individual studies to the popula- tions, comparisons, cointerventions, settings, and outcomes or measures of interest Standard 4.3 Decide if, in addition to a qualitative analysis, the system- atic review will include a quantitative analysis (meta-analysis) Required element: 4.3.1 Explain why a pooled estimate might be useful to decision makers Standard 4.4 If conducting a meta-analysis, then do the following: Required elements: 4.4.1 Use expert methodologists to develop, execute, and peer review the meta-analyses 4.4.2 Address the heterogeneity among study effects 4.4.3 Accompany all estimates with measures of statistical uncertainty 4.4.4 Assess the sensitivity of conclusions to changes in the proto- col, assumptions, and study selection (sensitivity analysis) NOTE: The order of the standards does not indicate the sequence in which they are carried out.

OCR for page 1
12 FINDING WHAT WORKS IN HEALTH CARE studies, the range of study sizes and treatments compared, or the quality scores of each study as measured by a risk of bias tool. Meta-analysis is the statistical combination of results from mul - tiple individual studies. Many published meta-analyses have com - bined the results of studies that differ greatly from one another. The assumption that a meta-analysis is an appropriate step in an SR should never be made. The decision to conduct a meta-analysis is neither purely analytical nor statistical in nature. It will depend on a number of factors, such as the availability of suitable data and the likelihood that the analysis could inform clinical decision mak- ing. Ultimately, authors should make this subjective judgment in consultation with the entire SR team, including both clinical and methodological perspectives. If appropriate, the meta-analysis can provide reproducible summaries of the individual study results and offer valuable insights into the patterns in the study results. A strong meta-analysis features and clearly describes its subjective components, scrutinizes the individual studies for sources of het- erogeneity, and tests the sensitivity of the findings to changes in the assumptions, the set of included studies, the outcome metrics, and the statistical models. The Final Report Authors of all publicly sponsored SRs should produce a detailed final report. The committee recommends three standards for produc- ing the SR final report: (1) including standards for documenting the SR process; (2) responding to input from peer reviewers, users, and stakeholders; and (3) making the final report publicly available (Box S-5). The committee’s standards for documenting the SR process drew heavily on the PRISMA checklist. The committee recommends adding items to the PRISMA checklist to ensure that the report of an SR describes all of the steps and judgments required by the commit- tee’s standards (Boxes S-2, S-3, and S-4). RECOMMENDATIONS The evidence base supporting many elements of SRs is incom- plete and, for some steps, nonexistent. Research organizations such as the AHRQ Effective Health Care Program, CRD, and the Cochrane Collaboration have published standards, but none of these are universally accepted and consistently applied during planning, conducting, reporting, and peer review of SRs. Furthermore, the SR enterprise in the United States lacks both adequate funding and

OCR for page 1
13 SUMMARY coordination; many organizations conduct SRs, but do not typically work together. Thus, the committee concludes that improving the quality of SRs will require improving not only the science support - ing the steps in the SR process (Boxes S-2, S-3, and S-4), but also pro - viding a more supportive environment for the conduct of SRs. The committee proposes a framework for improving the quality of the science underpinning SRs and supporting the environment for SRs. The framework has several broad categories: strategies for involving the right people, methods for conducting reviews, methods for syn- thesizing and evaluating evidence, and methods for communicating and using results. The standards and elements form the core of the committee’s conclusions, but the standards themselves do not indicate how the standards should be implemented, nor do the standards address issues of improving the science for SRs or for improving the envi- ronment that supports the development and use of an SR enter- prise. In consequence, the committee makes the following two recommendations: Recommendation 1: Sponsors of SRs of CER should adopt appropriate standards for the design, conduct, and reporting of SRs and require adherence to the standards as a condition for funding. SRs of CER in the United States are now commissioned and conducted by a vast array of private and public entities, some sup - ported generously with adequate funding to meet the most exacting standards, others supported less generously so that the authors must make compromises at every step of the review. The committee rec- ognizes that its standards and elements are at the “exacting” end of the continuum, some of which are within the control of the review team whereas others are contingent on the SR sponsor’s compliance. However, high-quality reviews require adequate time and resources to reach reliable conclusions. The recommended standards are an appropriate starting point for publicly funded reviews in the United States (including PCORI, federal, state, and local funders) because of the heightened attention and potential clinical impact of major reviews sponsored by public agencies. The committee also recog- nizes that authors of SRs supported by public funds derived from nonfederal sources (e.g., state public health agencies) will see these standards as an aspirational goal rather than as a minimum require - ment. SRs that significantly deviate from the standards should clearly explain and justify the use of different methods.

OCR for page 1
14 FINDING WHAT WORKS IN HEALTH CARE BOX S-5 Recommended Standards for Reporting Systematic Reviews Standard 5.1 Prepare final report using a structured format Required elements: 5.1.1 Include a report title* 5.1.2 Include an abstract* 5.1.3 Include an executive summary 5.1.4 Include a summary written for the lay public 5.1.5 Include an introduction (rationale and objectives)* 5.1.6 Include a methods section. Describe the following: • Research protocol* • Eligibility criteria (criteria for including and excluding studies in the systematic review)* • Analytic framework and key questions • Databases and other information sources used to iden- tify relevant studies* • Search strategy* • Study selection process* • Data extraction process* • Methods for handling missing information* • Information to be extracted from included studies* • Methods to appraise the quality of individual studies* • Summary measures of effect size (e.g., risk ratio, differ- ence in means)* • Rationale for pooling (or not pooling) results of included studies • Methods of synthesizing the evidence (qualitative and meta-analysis*) • Additional analyses, if done, indicating which were prespecified* Recommendation 2: The Patient-Centered Outcomes Research Institute and the Department of Health and Human Services (HHS) agencies (directed by the secretary of HHS) should col- laborate to improve the science and environment for SRs of CER. Primary goals of this collaboration should include • Developing training programs for researchers, users, consumers, and other stakeholders to encourage more effective and inclusive contributions to SRs of CER; • Systematically supporting research that advances the methods for designing and conducting SRs of CER;

OCR for page 1
15 SUMMARY 5.1.7 Include a results section; organize the presentation of re- sults around key questions; describe the following (repeat for each key question): • Study selection process* • List of excluded studies and reasons for their exclusion* • Appraisal of individual studies’ quality* • Qualitative synthesis • Meta-analysis of results, if performed (explain rationale for doing one)* • Additional analyses, if done, indicating which were prespecified* • Tables and figures 5.1.8 Include a discussion section. Include the following: • Summary of the evidence* • Strengths and limitations of the systematic review* • Conclusions for each key question* • Gaps in evidence • Future research needs 5.1.9 Include a section describing funding sources* and COI Standard 5.2 Peer review the draft report Required elements: 5.2.1 Use a third party to manage the peer review process 5.2.2 Provide a public comment period for the report and publicly report on disposition of comments Standard 5.3 Publish the final report in a manner that ensures free public access * Indicates items from the PRISMA checklist. (The committee endorses all of the PRISMA checklist items.) • Supporting research to improve the communication and use of SRs of CER in clinical decision making; • Developing effective coordination and collaboration between U.S. and international partners; • Developing a process to ensure that standards for SRs of CER are regularly updated to reflect current best practice; and • Using SRs to inform priorities and methods for pri- mary CER. This recommendation conveys the committee’s view of how best to implement its recommendations to improve the science and sup-

OCR for page 1
16 FINDING WHAT WORKS IN HEALTH CARE port the environment for SRs of comparative effectiveness research, which is clearly in the public’s interest. PCORI is specifically named because of its statutory mandate to establish and carry out a CER research agenda. As noted above, it is charged with creating a meth - odology committee that will work to develop and improve the sci- ence and methods of SRs of CER and to regularly update such stan - dards. PCORI is also required to assist the Comptroller General in reviewing and reporting on compliance with its research standards, the methods used to disseminate research findings, the types of training conducted and supported in CER, and the extent to which CER research findings are used by healthcare decision makers. The HHS agencies are specifically named because AHRQ, NIH, CDC, and other sections of HHS are major funders and producers of SRs. In particular, the AHRQ EPC program has been actively engaged in coordinating high-quality SRs and in developing SR methodol- ogy. The committee assigns these groups with responsibility and accountability for coordinating and moving the agenda ahead. The committee found compelling evidence that having high- quality SRs based on rigorous standards is a topic of international concern, and that individual colleagues, professional organiza- tions, and publicly funded agencies in other countries make up a large proportion of the world’s expertise on the topic. Nonetheless, the committee followed the U.S. law that brought this report into being, which suggests a management approach appropriate to the U.S. environment. A successful implementation of the final recom- mendation should result in an enterprise in the United States that participates fully and harmonizes with the international develop- ment of SRs, serving in some cases in a primary role, in others as a facilitator, and in yet others as a participant. The new enterprise should recognize that this cannot be entirely scripted and managed in advance—structures and processes must allow for innovation to arise naturally from those individuals and organizations in the United States already fully engaged in the topic.