6
Improving the Quality of Systematic Reviews: Discussion, Conclusions, and Recommendations1

Abstract: The committee recommends that sponsors of systematic reviews (SRs) of comparative effectiveness research (CER) should adopt appropriate standards for the design, conduct, and reporting of SRs and require adherence to the standards as a condition for funding. The newly created Patient-Centered Outcomes Research Institute and agencies of the U.S. Department of Health and Human Services should collaborate to improve the science and environment for SRs of CER. Although the recommended SR standards presented in this report are based on the best available evidence and current practice of respected organizations, many of the standards should be considered provisional pending more methods research. This chapter presents a framework for improving the quality of the science underpinning SRs in several broad categories: involving the right people, methods for conducting reviews, methods for grading and synthesizing evidence, and methods for communicating and using results.

Systematic reviews (SRs) should be at the center of programs developing a coordinated approach to comparative effectiveness

1

This chapter does not include references. Citations for the findings presented appear in the preceding chapters.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 223
6 Improving the Quality of Systematic Reviews: Discussion, Conclusions, and Recommendations1 Abstract: The committee recommends that sponsors of systematic reviews (SRs) of comparative effectiveness research (CER) should adopt appropriate standards for the design, conduct, and reporting of SRs and require adherence to the standards as a condition for funding. The newly created Patient-Centered Outcomes Research Institute and agencies of the U.S. Department of Health and Human Services should collaborate to improve the science and environment for SRs of CER. Although the recommended SR standards presented in this report are based on the best available evidence and current practice of respected organizations, many of the standards should be considered provisional pending more methods research. This chapter presents a framework for improv- ing the quality of the science underpinning SRs in several broad categories: involving the right people, methods for conducting re- views, methods for grading and synthesizing evidence, and meth- ods for communicating and using results. Systematic reviews (SRs) should be at the center of programs developing a coordinated approach to comparative effectiveness 1 This chapter does not include references. Citations for the findings presented ap - pear in the preceding chapters. 223

OCR for page 223
224 FINDING WHAT WORKS IN HEALTH CARE research (CER), both for setting priorities among individual CER studies and for appropriately focusing studies during their design. The committee recognizes that fully implementing all of the SR standards proposed in this report will be costly, resource intensive, and time consuming. Further, as previous chapters make clear, the evidence base supporting many elements of SRs is incomplete and, for some steps, nonexistent. Finally, the committee is fully aware that there is little direct evidence linking high-quality SRs to clinical guidance that then leads to improved health. Nonetheless, designing and conducting new comparative effectiveness studies without first being fully informed about the state of the evidence from an SR risks even higher costs and waste by conducting studies that are poorly designed or redundant. Research organizations such as the Agency for Healthcare Research and Quality (AHRQ) Effective Health Care Program, Centre for Reviews and Dissemination (CRD) (University of York), and the Cochrane Collaboration have published standards, but none of these are generally accepted and consistently applied during planning, conducting, reporting, and peer reviewing of SRs. Furthermore, the environment supporting development of a robust SR enterprise in the United States lacks both adequate funding and coordination; many organizations conduct SRs, but do not typically work together. Thus the committee concludes that improving the quality of SRs will require advancing not only the science support - ing the steps in the SR process and linking SRs to improved health, but also providing a more supportive environment for the conduct of SRs. In this chapter the committee outlines some of the principal issues that must be addressed in both of these domains. Throughout the chapter and in its final recommendations, the committee refers to the newly established Patient-Centered Out- comes Research Institute (PCORI) and, in particular, its Methodol - ogy Committee, as a potentially appropriate organization to provide comprehensive oversight and coordination of the development of the science and to promote the environment for SRs in support of CER in the United States. The committee views PCORI as an unusually timely development—albeit untested—that should help advance the field of SRs as an essential component of its overall mission, building on the strengths of well-established programs in the United States (e.g., AHRQ, National Institutes of Health [NIH]) and internation - ally (e.g., Cochrane Collaboration, National Institute for Health and Clinical Excellence in the United Kingdom), that either produce or rely on SRs for policy purposes. Nonetheless, while the committee views PCORI as relevant and promising, PCORI is by no means the only way to achieve the stated aims. Other agencies, working

OCR for page 223
225 IMPROVING THE QUALITY OF SYSTEMATIC REVIEWS individually, are able to contribute to advancing the field as well. However, the committee believes that collaborative relationships among agencies, both public and private, would be most effective at contributing to progress. Furthermore, the committee recognizes that U.S. developments are only part of a substantial international effort focused on how best to conduct SRs, an effort that in some countries is advanced and highly sophisticated. Given the potential for duplication of efforts, the need to ensure that gaps in the infor- mation base are appropriately addressed, and the need for efficient use of limited available resources, the coordination across multiple organizations within the United States and throughout the world will have clear benefits and should be viewed as essential. IMPROVING THE SCIENCE OF SYSTEMATIC REVIEWS Establishing a process for ongoing development of the research agenda in SRs must be an important part of the path forward. Although the committee believes the recommended standards and elements of performance presented in this report are founded on the best available evidence and current practice of respected organiza- tions, all SR standards should be considered provisional pending additional experience and research on SR methods. The committee recognizes that each of its recommended standards could be exam- ined in appropriately designed research, with the expectation that some items would be validated, some discarded, and some added. Future research to develop methods that promote efficiency and scientific rigor is especially important. A detailed description of research that might be conducted on each step, however, is outside the committee’s scope of work and would require substantial time and resources. We also note that some of the needed work may be more appropriately categorized as program development and evalu- ation than research. The committee promotes the goal of developing a coordinated approach to improving the science of SR, embedded in a program of innovation, implementation, and evaluation that improves the qual- ity of SRs overall. PCORI is an appropriate organization to provide comprehensive oversight and coordination of the development of the science of SRs in support of CER, in cooperation with agencies of the U.S. Department of Health and Human Services (HHS). It could, also function as an important U.S. collaborator with interna - tional organizations similarly focused (e.g., Cochrane Collaboration, Campbell Collaboration, and CRD). Among other goals, such a coor- dinated program would support a description of methods currently

OCR for page 223
226 FINDING WHAT WORKS IN HEALTH CARE employed, methodological research, and comparative studies of alternative approaches, working with international partners to effi- ciently advance the research agenda. By supporting innovation, the incorporation of a feedback loop into design and reporting of trials and observational studies, and the appropriate and intentional (not accidental or wasteful) replication of methodological research and SRs of methods, PCORI will contribute to ensuring that standards are evidence based. SR methods research will also help to identify gaps in the literature and, through the application of the findings of empirical, “meta-epidemiologic” approaches (i.e., investigations of how particular features of study design or study populations relate to the validity and applicability of primary studies), will provide information about how well standards are being applied. In this section, the committee proposes a framework for improv- ing the quality of the science underpinning SRs in several broad categories: strategies for involving the right people, methods for conducting the SR, methods for synthesizing and evaluating evi- dence, and methods for communicating and using results. Strategies for Involving the Right People2 Successful execution and effective use of an SR is a collaborative activity requiring a wide range of experience and expertise among the contributors (the review team). The committee believes that involving people with sufficient experience in each step of the SR process has not received enough attention. While noting that a typi - cal review will require people with certain expertise in specific steps, the committee resisted proposing a standard that a particular menu of experts and stakeholders must be a part of every SR, regardless of topic and purpose. On the other hand, the committee believes that current practice, particularly among groups with modest resources, probably underestimates and undervalues the need for certain kinds of expertise, with the result that SRs vary enormously in quality and credibility. In contrast to authors carrying out the traditional literature review, review teams need formal education and training, which should include hands-on experience and mentoring. There is also wide variability in the involvement of consumers and other users and stakeholders in SRs. Finally, the committee recognizes that 2 See Chapter 2 for a discussion of the individuals who should be included on the review team and the importance of involving users and stakeholders in the SR process.

OCR for page 223
227 IMPROVING THE QUALITY OF SYSTEMATIC REVIEWS depth of experience in participating in SRs varies among individuals in a given field, so that the mere presence of an individual with gen- eral expertise in a relevant domain does not ensure that the issues will be covered adequately in the review. As an example, not all biostatisticians are fluent in methods of SRs, even though they may be experts in other areas. Similarly, generalist librarians and other information specialists may require special training or experience in conducting SRs, including special knowledge of bibliographic database-specific search terms, to design and execute the search strategy appropriately. Little descriptive information is available about how the issues of personnel and expertise are handled in various SR enterprises, and the evidence base comparing different approaches is inadequate. For example, we believe comparative studies of models involving consumers and patients are needed. As another example, research on the effects of conflict of interest and bias is provocative, but the topic needs to be addressed more systematically using sophisticated research methods. The committee recognizes that performing such research will present challenges, beginning with defining appropri- ate outcome measures in these methodologic investigations. Methods for Conducting the Systematic Review Developing a review protocol, locating and screening studies, and collecting and appraising the data (the subject of Chapters 2 and 3) are many specific steps along the pathway to completing an SR. Some steps, such as the use of different databases and sen - sitive search filters to identify relevant literature, are supported with empirical data, but many other steps have not been exam- ined in research. The committee believes an entity such as PCORI should systematically support research that examines key steps in the methods involved in conducting an SR. The committee’s crite- ria (Table 1-2) might be a useful framework to identify topics for further research. For example, how do alternative approaches to some individual steps affect the scientific rigor and efficiency of SR? In addition, we have data on how particular steps in a review are potentially influenced by bias (e.g., reporting biases), but not on whether the bias is of concern in an individual review. The challenge will be not only to identify topics that can be researched, but also to set priorities among them. For example, those standards that have a substantial effect on cost (e.g., dual extraction) might be initially considered higher priority.

OCR for page 223
228 FINDING WHAT WORKS IN HEALTH CARE Methods for Synthesizing and Evaluating Evidence3 Quantitative synthesis of empirical data is a highly developed and active topic of research. Research ranges from the theoretical (with emphasis on the statistics) to comparisons of different model- ing approaches. Despite ongoing research in the field, many out- standing questions remain, particularly related to the synthesis of complex multivariate data structures. The committee recommends a range of approaches to answering these questions, including theo- retical, empirical, and simulation studies as appropriate. Qualitative synthesis (i.e., a narrative description of available evidence without drawing conclusions based on statistical inference) has received less attention in research than quantitative synthesis, although the committee recommends that it be part of all SRs. Fail- ing to perform a qualitative synthesis is problematic because the evidence available to answer specific SR questions often does not lend itself to quantitative methods. There is no empirical research to guide synthesis when a qualitative synthesis is the only approach possible. Even when a quantitative review is conducted, we need to understand what perspectives and judgments should be considered in undertaking qualitative synthesis that require authors to be reflec- tive, critical, and as objective as possible in their presentation and interpretation of the data. Because an important goal of qualitative synthesis is communication to users and stakeholders, research in this area might also focus on effectiveness of communication, or perceived objectivity. Furthermore, although a formal approach to assessing the qual- ity of a body of evidence is recommended, there is little, if any, research testing the reliability and validity of existing approaches to evidence assessment (e.g., the Grading of Recommendations Assess- ment, Development, Evaluation, or GRADE model). Careful consid- eration of how to define validity for an SR (e.g., defining a reference standard) will be an important challenge in this research. Finally, the field clearly needs to develop a common lexicon and set of symbols for summarizing the quality of evidence, a process that will need to be coordinated among groups using SRs to develop clinical guid- ance where there is further variation in lexicon and symbols. The committee believes that PCORI and its Methodology Com- mittee should invest in research on quantitative and qualitative syntheses and grading of the body of evidence for SRs. This work 3 See Chapter 4 for discussions on qualitative synthesis, quantitative synthesis, and evaluating the quality of a body of evidence.

OCR for page 223
229 IMPROVING THE QUALITY OF SYSTEMATIC REVIEWS should be done in close collaboration with other groups commis- sioning and doing SRs, including the U.S. Public Health Service (e.g., AHRQ, NIH, U.S. Preventive Services Task Force, and the Commu- nity Guide), professional organizations and associations, and exist - ing international organizations (e.g., the Cochrane Collaboration). Methods for Communicating and Using the Results4 The committee placed high value on public availability of, and transparency in reporting, SRs, but was not able to cite specific research supporting a particular format. Research on how to most effectively communicate the results of an SR to various users (e.g., clinicians, clinical guidelines panels, consumers, healthcare organi - zations, payers, both public and private, etc.) is limited, and more would be useful. The committee also notes that in current practice, the process of conducting some SRs is often formally separated from processes in which they are actually used. Although appropriate objectivity and freedom from undue influence need to be main- tained, the committee believes that research examining the utility of connecting the SR with its intended users (e.g., clinical guidelines groups, practicing clinicians, and patients), as well as effectiveness and impact of current collaborative efforts, would be timely. IMPROVING THE ENVIRONMENT FOR SUPPORTING SYSTEMATIC REVIEWS Developing the science of SRs is not enough to address all the issues that the committee identified as important to improving the quality of SRs to inform CER. A number of environmental factors will critically influence whether the quality of SRs can be improved. Some are best described as infrastructure (e.g., training, registries), but others have to do with SRs as required elements for a culture aimed at improving CER overall. The committee believes there is a need for greater collabora- tion among multiple stakeholder groups, including PCORI, gov - ernment agencies (especially AHRQ and NIH), the Cochrane Col- laboration, medical professional societies, researchers, healthcare delivery organizations, patient interest groups, and others. Such multidisciplinary and multiorganizational collaborations have the potential to improve the rigor, transparency, and efficiency of SRs; encourage standardization of methods and processes; set priorities 4 See Chapter 5 for a discussion on preparing final reports of SRs.

OCR for page 223
230 FINDING WHAT WORKS IN HEALTH CARE for selection of clinical topics of interest to clinicians and patients; reduce unintentional duplication of efforts; provide a shared fund - ing source for the generation of high-quality evidence reviews; more effectively manage conflicts of interest; and facilitate implementation of reviews. Developing effective collaborations, however, requires a transformation in current thinking and structural approaches to con- ducting SRs. The importance of including international collaborators in discussions of priorities for methodologic research, in particular, cannot be overstated; there is deep expertise and effective leadership in the SR field inside and outside U.S. borders. The committee also underscores that its recommended stan- dards and elements of performance for publicly funded SRs are provisional, subject to change as the science of SRs advances and les- sons are learned from applying the standards in practice. A mecha - nism is needed to monitor the progress of the science and update the standards periodically to reflect current best practice. As in the preceding section on developing the science, the com - mittee found that dividing issues into four general categories was a useful way to organize our conclusions: (1) strategies for involving the right people; (2) methods for conducting SRs; (3) methods for synthesizing and evaluating evidence; and (4) methods for commu - nicating and using results. Strategies for Involving the Right People The committee believes the environment must be improved to allow and encourage people with sufficient training and experience to be engaged in an SR. Training and professional development must be well established, supported, and recognized by the research community before aspiring researchers will feel secure in choosing careers in SR. Rewards and promotion systems for faculty and sci- entists in academic institutions need to recognize that the conduct of SRs and the research on SR methods are inherently collaborative efforts. Substantive intellectual contributions to such collaborative efforts need to be recognized in meaningful ways that will attract, not discourage, participation by top scientists. Training targeted to the specific skills needed for SRs needs to be addressed in any national program supporting CER. This research is often multidis- ciplinary, which training curriculums must take into account. This may require innovation as many disciplines narrowly focus their pre- and post-doctoral training. Support for the training of users and stakeholders—such as consumers, patients, clinicians, payers, representatives from the

OCR for page 223
231 IMPROVING THE QUALITY OF SYSTEMATIC REVIEWS insurance industry—in the design, conduct, and use of SRs will be essential if representatives from these groups are to contribute effectively to the choice of the clinical scenario for the review and otherwise fully participate in the conduct of a review and its dis- semination. Finally, involving the right people requires providing an environment in which a transparent and robust approach to managing conflict of interest and bias is developed, implemented, and rigorously evaluated for all who participate in an SR. Key CER studies often involve proprietary interests, which involve confiden - tiality and legal issues. Promoting thorough, transparent analysis will require consideration of these interests, potentially including changes to rules and regulations. Methods for Conducting Systematic Reviews and for Synthesizing and Evaluating Evidence The science of conducting an SR, from design through review, synthesis, and evaluation, can only thrive in an environment in which all aspects are supported in a culture valuing the contribution of SRs to improvements in health care. The committee noted many specific ways in which the environment could provide such support. These include establishing a registry for SR protocols (under devel- opment by CRD at the University of York in the United Kingdom),5 providing a repository for data extracted during the conduct of SRs (being explored by the Tufts Medical Center Evidence-based Prac- tice Center),6 publicly posting protocols and reviews, using public mechanisms to ensure timely updating of protocols and reviews, guaranteeing access to data from primary studies for use in SRs, and ensuring that SRs are a required part of planning, designing, and conducting future primary CER. Establishing a collaborative methodologic research infrastructure will also be valuable to advancing the science of SRs. Some aspects of methodology might be amenable to rigorous study through the vari- ous organizations that fund SRs. For example, a study comparing 5 CRD is developing a registry of SR protocols—focused initially on SRs of the ef - fectiveness of health interventions—with the support of the UK National Institute of Health Research, the Canadian Institute of Health Research, and the International Network of Agencies for Health Technology Appraisal. 6 The Tufts Medical Center Evidence-based Practice Center, Boston, Massachusetts, with support from AHRQ and the National Library of Medicine, is currently explor- ing methods for improving the efficiency and quality of SRs through the creation of an electronic SR data repository.

OCR for page 223
232 FINDING WHAT WORKS IN HEALTH CARE structures for presenting qualitative reviews might randomize SRs being performed by several organizations, measuring acceptability, efficiency, or transparency. The details of study designs are beyond the scope of this report, but the committee believes that a coordi- nated and collaborative approach to reviews that are already being conducted could offer rich opportunities for efficiently advancing research, particularly if this planning is done prospectively so that reviews are updated in a timely manner. Methods for Communicating and Using the Results The committee believes that developing an environment that supports the understanding and use of SRs is critical if the enterprise is to improve CER. Terminology should be consistent, and conven- tions and standards for publication uniform and well defined. When publicly funded SRs are intended to be used in support of clinical guidance, these reviews should be formally linked with guidelines committees that also meet rigorous standards. The use and useful- ness of SRs commissioned as part of a guidelines process should be evaluated once the guideline is implemented, with a feedback loop into future reviews on similar topics and methods used to conduct the review. CONCLUSIONS AND RECOMMENDATIONS The committee explored a wide range of topics in its delib - erations. The standards and elements of performance form the core of our conclusions, but the standards themselves do not indicate how the standards should be implemented, nor do the standards address issues of improving the science for SRs or improving the environment that supports the development and use of an SR enterprise. In consequence, the committee makes the following two recommendations. Recommendation 1: Sponsors of SRs of CER should adopt stan- dards for the design, conduct, and reporting of SRs and require adherence to the standards as a condition for funding. SRs of CER in the United States are now commissioned and conducted by a vast array of private and public entities, some sup - ported generously with adequate funding to meet the most exacting standards, others supported less generously with the result that compromises must be made at every step of the review. Regardless

OCR for page 223
233 IMPROVING THE QUALITY OF SYSTEMATIC REVIEWS of the level of funding, all sponsors of SRs of CER should adopt standards for the planning, conducting, and reporting of SRs to ensure that a minimal level of quality is met, and should make the adopted standards publicly available. The committee recognizes that its recommended standards are provisional, subject to change as the science of SRs advances and lessons are learned from applying the standards in real-world situations. Also, its standards and elements of performance are at the “exacting” end of the continuum, some of which are within the control of the review team whereas others are contingent on the SR sponsor’s compliance. However, high-quality reviews require adequate time and resources to reach reliable con- clusions. The recommended standards are an appropriate starting point for publicly funded reviews in the United States (including PCORI, federal, state, and local funders) because of the heightened attention and potential clinical impact of major reviews sponsored by public agencies. The committee also recognizes that a range of SRs are supported by public funds derived from nonfederal sources (e.g., state public health agencies) and private sources where these standards will be seen as an aspiration rather than as a minimum bar. Application of the standards to reviews embedded within other programs that may be publicly funded (e.g., highly focused reviews conducted by individual investigators as part of research grants) also presents difficult operational issues. On the whole, however, the committee feels strongly that the standards (and their successor standards) should serve as a benchmark for all SRs of CER. They could even, for example, be used to inform other topic areas (e.g., risk assessment, epidemiologic research) where standards are also being developed. SRs that significantly deviate from the standards should clearly explain and justify the use of different methods. Recommendation 2: PCORI and the Department of Health and Human Services (HHS) agencies (directed by the Secretary of HHS) should collaborate to improve the science and envi- ronment for SRs of CER. Primary goals of this collaboration should include •  Developing training programs for researchers, users, con- sumers, and other stakeholders to encourage more effec- tive and inclusive contributions to SRs of CER; •  Systematically supporting research that advances the methods for designing and conducting SRs of CER; •  Supporting research to improve the communication and use of SRs of CER in clinical decision making;

OCR for page 223
234 FINDING WHAT WORKS IN HEALTH CARE • Developing effective coordination and collaboration be- tween U.S. and international partners; • Developing a process to ensure that standards for SRs of CER are regularly updated to reflect current best practice; and • Using SRs to inform priorities and methods for primary CER. This recommendation conveys the committee’s view of how best to implement its recommendations to improve the science and support the environment for SRs of CER, which is clearly in the public’s interest. PCORI is specifically named because of its statu - tory mandate to establish and carry out a CER research agenda. As noted above, it is charged with creating a methodology committee that will work to develop and improve the science and methods of SRs of CER and to update such standards regularly. PCORI is also required to assist the Comptroller General in reviewing and report- ing on compliance with its research standards, the methods used to disseminate research findings, the types of training conducted and supported in CER, as well as the extent to which CER research findings are used by healthcare decision makers. The HHS agencies are specifically named because AHRQ, NIH, the Centers for Dis - ease Control and Prevention, and other divisions of HHS are major funders and producers of SRs. In particular, the AHRQ Effective Health Care Program has been actively engaged in coordinating high-quality SRs and developing SR methodology. The committee assigns these groups with responsibility and accountability for coor- dinating and moving the agenda ahead. The committee found compelling evidence that having high- quality SRs based on rigorous standards is a topic of international concern, and that individual colleagues, professional organiza- tions, and publicly funded agencies in other countries make up a large proportion of the world’s expertise on the topic. Nonetheless, the committee necessarily follows the U.S. law that facilitated this report, which suggests a management approach appropriate to the U.S. environment is useful. A successful implementation of our final recommendation should result in a U.S. enterprise that participates fully and harmonizes with the international development of SRs, serving in some cases in a primary role and in others as a facilitator or participant. The new enterprise should also fully understand that this cannot be entirely scripted and managed in advance—structures and processes must allow for innovation to arise naturally from among U.S. individuals and organizations already fully engaged in the topic.