Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 195
5 Standards for Reporting Systematic Reviews Abstract: Authors of publicly sponsored systematic reviews (SRs) should produce a detailed, comprehensive final report. The com- mittee recommends three related standards for documenting the SR process, responding to input from peer reviewers and other users and stakeholders, and making the final report publicly avail- able. The standards draw extensively from the Preferred Report- ing Items for Systematic Reviews and Meta-Analyses (PRISMA) checklist. The committee recommends several reporting items in addition to the PRISMA requirements to ensure that the final report (1) describes all of the steps and judgments required by the standards in the previous chapters and (2) focuses on informing patient and clinical decision making. High-quality systematic review (SR) reports should accurately document all of the steps and judgments in the SR process using clear language that is understandable to users and stakeholders. A report should provide enough detail that a knowledgeable reader could reproduce the SR. The quality of a final report has profound implications for patients and clinicians. Too often the information that researchers report in published SRs does not adequately reflect 195

OCR for page 195
196 FINDING WHAT WORKS IN HEALTH CARE their study methods (Devereaux et al., 2004).1 If SRs are poorly reported, patients and clinicians have difficulty determining whether an SR is trustworthy enough to be used to guide decision making or the development of clinical practice guidelines (Moher et al., 2007). High-quality SR reports summarize the methodological strengths and weaknesses of the SR and include language designed to help nonexperts interpret and judge the value of the SR (AHRQ, 2010b; CRD, 2010a; Higgins and Green, 2008; Liberati et al. 2009; Moher et al. 2009). However, according to an extensive literature, many published SRs inadequately document important aspects of the SR process (Delaney et al., 2005, 2007; Golder et al., 2008; McAlister et al., 1999; Moher et al., 2007; Mulrow, 1987; Roundtree et al., 2008; Sacks et al., 1987). A seminal study conducted by Mulrow, for exam- ple, assessed 50 review articles published in four leading medical journals and found that many reviews failed to report the methods of identifying, selecting, and validating information, and choos- ing areas for future research (Mulrow, 1987). More recently, Moher and colleagues (2007) evaluated 300 SRs indexed in MEDLINE dur- ing November 2004. They concluded that information continues to be poorly reported, with many SRs failing to report key compo- nents of SRs, such as assessing for publication bias, aspects of the searching and screening process, and funding sources. Other stud- ies have found that SRs published in journals often inadequately report search strategies, validity assessments of included studies, and authors’ conflicts of interest (Delaney et al., 2005; Golder et al., 2008; Roundtree et al., 2008). Authors of all publicly sponsored SRs must produce a detailed final report, which is typically longer and more detailed than the version submitted for journal publication. The sponsor typically publishes the final report on its website, where it stands as the definitive documentation of the review. The standards recom- mended by the committee apply to this definitive comprehensive final report. The committee recommends three standards for pro- ducing a comprehensive SR final report (Box 5-1), including stan- dards for documenting the SR process, responding to input from peer reviewers and other users and stakeholders, and making the final reports publicly available. Each standard includes elements of performance that the committee deems essential. The evidence base for developing standards for the final report is sparse. In addition, most evaluations of the quality of published SRs have focused on 1 See Chapter 3 for a review of the literature on reporting bias and dearth of ad - equate documentation in most SRs of comparative effectiveness.

OCR for page 195
197 STANDARDS FOR REPORTING SYSTEMATIC REVIEWS BOX 5-1 Recommended Standards for Reporting Systematic Reviews Standard 5.1 Prepare the final report using a structured format Required elements: 5.1.1 Include a report title* 5.1.2 Include an abstract* 5.1.3 Include an executive summary 5.1.4 Include a summary written for the lay public 5.1.5 Include an introduction (rationale and objectives)* 5.1.6 Include a methods section. Describe the following: • Research protocol* • Eligibility criteria (criteria for including and excluding studies in the sysematic review)* • Analytic framework and key questions • Databases and other information sources used to identify relevant studies* • Search strategy* • Study selection process* • Data extraction process* • Methods for handling missing information* • Information to be extracted from included studies* • Methods to appraise the quality of individual studies* • Summary measures of effect size (e.g., risk ratio, difference in means)* • Rationale for pooling (or not pooling) results of included studies • Methods of synthesizing the evidence (qualitative and meta-analysis*) • Additional analyses, if done, indicating which were prespecified* 5.1.7 Include a results section. Organize the presentation of results around key questions. Describe the following (re- peat for each key question): • Study selection process* • List of excluded studies and reasons for their exclusion* • Appraisal of individual studies’ quality* • Qualitative synthesis • Meta-analysis of results, if performed (explain ratio- nale for doing one)* • Additional analyses, if done, indicating which were prespecified* • Tables and figures 5.1.8 Include a discussion section. Include the following: • Summary of the evidence* continued

OCR for page 195
198 FINDING WHAT WORKS IN HEALTH CARE BOX 5-1 Continued • Strengths and limitations of the systematic review* • Conclusions for each key questions* • Gaps in evidence • Future research needs 5.1.9 Include a section describing funding sources* and COI Standard 5.2 Peer review the draft report Required elements: 5.2.1 Use a third party to manage the peer review process 5.2.2 Provide a public comment period for the report and pub- licly report on disposition of comments Standard 5.3 Publish the final report in a manner that ensures free public access * Indicates items from the PRISMA checklist. (The committee endorses all of the PRISMA checklist items.) journal articles rather than SR reports. The committee developed the standards by first reviewing existing expert guidance, particularly the Preferred Reporting Items for Systematic Reviews and Meta- Analyses (PRISMA) checklist (Liberati et al., 2009). However, PRISMA is focused on journal articles, not comprehensive final reports to public sponsors. The committee recommended including items that were not on the PRISMA checklist because it believed that the report of an SR should describe all the steps and judg- ments required by the committee’s standards in Chapters 2 through 4 to improve the transparency of the SR process and to inform patient and clinical decision making. The committee also took into account the legislatively mandated reporting requirements for the Patient-Centered Outcomes Research Institute (PCORI), as specified by the 2010 Patient Protection and Affordable Care Act (ACA). Box 5-2 describes the ACA reporting requirements for research funded by PCORI. See Appendix G for the Agency for Healthcare Research and Quality (AHRQ), Centre for Reviews and Dissemination (CRD), and Cochrane Collaboration guidance on writing an SR final report. Appendix H contains the PRISMA checklist.

OCR for page 195
199 STANDARDS FOR REPORTING SYSTEMATIC REVIEWS BOX 5-2 Requirements for Research Funded by the Patient-Centered Outcomes Research Institute The 2010 Patient Protection and Affordable Care Act created the Patient- Centered Outcomes Research Institute (PCORI), a nonprofit corporation intended to advance comparative effectiveness research. The act stipulates that research funded by PCORI, including systematic reviews, adhere to the following reporting and publication requirements: • For each research study, the following information should be post- ed on PCORI’s website: o A research protocol, including measures taken, methods of re- search and analysis, research results, and other information the institute determines appropriate. o The research findings conveyed in a manner that is comprehen- sible and useful to patients and providers in making healthcare decisions. o Considerations specific to certain subpopulations, risk factors, and comorbidities, as appropriate. o The limitations of the research and what further research may be needed as appropriate. o The identity of the entity and the investigators conducting the research. o Conflicts of interest, including the type, nature, and magnitude of the interests. • PCORI is required to: o Provide a public comment period for systematic reviews to in- crease public awareness, and to obtain and incorporate public input and feedback on research findings. o Ensure there is a process for peer review to assess a study’s sci- entific integrity and adherence to methodological standards. o Disseminate research to physicians, healthcare providers, pa- tients, payers, and policy makers. SOURCE: The Patient Protection and Affordable Care Act, Public Law 111-148, 111th Cong., Subtitle D, § 6301 (March 23, 2010). REPORTING GUIDELINES Over the past decade, several international, multidisciplinary groups have collaborated to develop guidelines for reporting the methods and results of clinical research (reporting guidelines). Reporting guidelines exist for many types of health research (Ioannidis et al., 2004; Liberati et al., 2009; Moher et al., 1999, 2001a,b,

OCR for page 195
200 FINDING WHAT WORKS IN HEALTH CARE 2009; Stroup et al., 2000). These guideline initiatives were under- taken out of concern that reports on health research were poorly documenting the methods and results of the research studies (IOM, 2008). Detailed reporting requirements are also seen as a line of defense against reporting bias.2 For SRs to be trustworthy enough to inform healthcare decisions, accurate, thorough, and transpar- ent reporting are essential. The adoption of reporting guidelines furthers this goal. Examples of reporting guidelines include the Consolidated Standards of Reporting Trials (CONSORT) statement for reporting randomized clinical trials (Ioannidis et al., 2004; Moher et al., 2001b), and the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement for reporting obser- vational studies in epidemiology (von Elm et al., 2007). The major reporting guideline for SRs and meta-analyses is PRISMA (Liberati et al., 2009; Moher et al., 2009), an update to the 1999 Quality of Reporting of Meta-analyses (QUOROM) statement (Moher et al., 1999). In 2006, the Enhancing Quality and Transparency of Health Research (EQUATOR) Network was launched to coordinate ini- tiatives to promote transparent and accurate reporting of health research and to assist in the development of reporting guidelines (EQUATOR Network, 2010). See Box 5-3 for a historical overview of reporting guidelines. The methodological quality of SRs (i.e., how well the SR is con- ducted) is distinct from reporting quality (i.e., how well reviewers report their methodology and results) (Shea et al., 2007). Whether reporting guidelines improve the underlying methodological qual - ity of research studies is unknown. However, incomplete documen- tation of the SR process makes it impossible to evaluate its method- ological quality, so that it is impossible to tell whether a step in the SR process was performed correctly but not reported (poor reporting quality), completed inadequately, or not completed at all and there- fore not reported (poor methodological quality). At present, the evidence that reporting guidelines improve the quality of reports of SRs and meta-analyses is weak. The few obser- vational studies that have addressed the issue have serious flaws. For example, Delaney and colleagues (2005) compared the quality of reports of meta-analyses addressing critical care, including top- ics related to shock, resuscitation, inotropes, and mechanical ven - tilation, published before and after the release of the QUOROM statement (the precursor to PRISMA). They found that reports of meta-analyses published after QUOROM were of higher quality 2 See Chapter 3 for a discussion on reporting bias.

OCR for page 195
201 STANDARDS FOR REPORTING SYSTEMATIC REVIEWS BOX 5-3 A History of Reporting Guidelines for Comparative Effectiveness Research In 1993 the Standards for Reporting Trials (SORT) group met to ad- dress inadequate reporting of randomized controlled trials (RCTs). This group developed the concept of a structured reporting guideline, and pro- posed a checklist of essential items for reporting RCTs. Five months later the Asilomar Working group met independently to discuss challenges in reporting RCTs and developed a reporting checklist. The Consolidated Standards of Reporting Trials (CONSORT) statement was developed in 1996 and consolidated the recommendation from both groups. The CON- SORT statement consists of a checklist of reporting items, such as the back- ground, methods, results, discussion, and conclusion sections, as well as a flow diagram for documenting participants through the trial. Many journals have adopted the CONSORT statement. It has been extended to address a number of specific issues in the reporting of RCTs (e.g., reporting of harms, noninferiority and equivalence RCTs, cluster RCTs). Following the success of the CONSORT statement, two interna- tional groups of review authors, methodologists, clinicians, medical edi - tors, and consumers developed standard formats for reporting systematic reviews (SRs) and meta-analyses: Quality of Reporting of Meta-analyses (QUOROM) and Meta-analysis of Observational Studies in Epidemiology (MOOSE). The statements consist of checklists of items to include in reports and flow diagrams for documenting the search process. However, unlike CONSORT, reporting guidelines for SRs and meta-analyses have not been widely adopted by prominent journals. In 2009, the Preferred Reporting Items for Systematic Reviews and Meta- analyses (PRISMA) statement was published to update the QUOROM state- ment. According to its developers, PRISMA reflects the conceptual and practi- cal advances made in the science of SRs since the development of QUOROM. These conceptual advances include the following: completing an SR is an iterative process; the conduct and reporting of research are distinct processes; the assessment of risk of bias requires both a study-level assessment (e.g., adequacy of allocation concealment) and outcome-level assessment (i.e., reli- ability and validity of the data for each outcome); and the importance of ad- dressing reporting bias. PRISMA decouples several checklist items that were a single item on the QUOROM checklist and links other items to improve the consistency across the SR report. PRISMA was funded by the Canadian Insti- tutes of Health Research; Universita di Modena e Reggio Emilia, Italy; Cancer Research U.K.; Clinical Evidence BMJ Knowledge; The Cochrane Collabora- tion; and GlaxoSmithKline, Canada.a It has been endorsed by a number of or- ganizations and journals, including the Centre for Reviews and Dissemination, Cochrane Collaboration, British Medical Journal, and Lancet.b a The following Institute of Medicine committee members were involved in the devel- opment of PRISMA: Jesse Berlin, Kay Dickersin, and Jeremy Grimshaw. b See the following website for a full list of organizations endorsing PRISMA: http:// www.prisma-statement.org/endorsers.htm (accessed July 14, 2010). SOURCES: Begg et al. (1996); Ioannidis et al. (2004); IOM (2008); Liberati et al. (2009); Moher et al. (1999, 2001a, 2001b, 2007, 2009); Stroup et al. (2000).

OCR for page 195
202 FINDING WHAT WORKS IN HEALTH CARE than reports published before and were more likely to describe whether a comprehensive literature search was conducted; the cri- teria for screening the studies; and the methods used to combine the findings of relevant studies (Delaney et al., 2005). Mrkobrada and colleagues (2008) evaluated 90 SRs published in 2005 in the field of nephrology. They found that only a minority of journals (4 out of 48) recommended adherence to SR reporting guidelines. The four journals that endorsed or adopted reporting guidelines published SRs of significantly higher methodological quality than the other journals, and were more likely to report assessing methodological quality of included studies and taking precautions to avoid bias in study selection. Neither of these studies, however, assessed whether the journals endorsing QUOROM published higher quality reviews than the other journals prior to the adoption of QUOROM. In addi - tion, journals that endorse reporting guidelines, such as QUOROM, may merely recommend that authors comply with the reporting items, but may not require authors to show compliance by submit- ting a checklist stating whether or not they adhered to each item as a condition of accepting the SR for review. As a result, whether the reporting improvements were due to QUOROM or other develop- ments in the field is unclear. No controlled trials have evaluated the effectiveness of PRISMA on improving the reporting of SRs (Liberati et al., 2009). In light of this history of reporting guidelines for medical jour- nals, the committee decided to develop reporting guidelines spe- cifically for the final report to the sponsor of an SR. The committee intends for its reporting requirements to improve the documentation of SR final report study methodology and results, and to increase the likelihood that SR final reports will provide enough information for patients and clinicians to determine whether an SR is trustworthy enough to be used to guide decision making. SYSTEMATIC REVIEWS PUBLISHED IN JOURNALS The committee recognizes that a journal publishing SRs will choose the level of documentation that is most appropriate for its readers. It also recognizes that its reporting requirements for final reports to public sponsors of SRs are quite detailed and compre- hensive, and will produce manuscripts that are too long and too detailed for most journals to publish in full. Ideally, all published SRs (both final reports to sponsors and journal publications) should follow one reporting standard. With the advent of electronic-only appendixes to journal articles, journals can now require authors to

OCR for page 195
203 STANDARDS FOR REPORTING SYSTEMATIC REVIEWS meet the committee’s full reporting guidelines (i.e., journals can post any reporting items not included in the actual journal publication in an online appendix). Alternatively, journals can publish a link to the website of the full SR report to the public sponsor, explaining what information readers would find only at the sponsor’s website. RECOMMENDED STANDARD FOR PREPARING THE FINAL REPORT The committee recommends the following standard for preparing the final report: Standard 5.1—Prepare the final report using a structured format Required elements: 5.1.1 Include a report title 5.1.2 Include an abstract 5.1.3 Include an executive summary 5.1.4 Include a summary written for the lay public 5.1.5 Include an introduction (rationale and objectives) 5.1.6 Include a methods section 5.1.7 Include a results section. Organize the presenta- tion of results around key questions 5.1.8 Include a discussion section 5.1.9 Include a section describing funding sources and COI Rationale All SR reports to public sponsors should use a structured for- mat to help guide the readers to relevant information, to improve the documentation of the SR process, and to promote consistency in reporting. More than 150 journals have adopted the PRISMA requirements (PRISMA, 2010). Because of this support, the commit- tee used the PRISMA checklist as its starting point for developing its reporting standards. However, PRISMA is focused on journal articles, which are usually subject to length restrictions in the print version of the article, and the committee’s reporting standards are directed at comprehensive, final reports to public sponsors (e.g., AHRQ, PCORI), which typically do not have word limits. Most of the committee’s additions and revisions to PRISMA were necessary to make the standards for the final report consistent with all of the steps and judgments in the SR process required by the standards for

OCR for page 195
204 FINDING WHAT WORKS IN HEALTH CARE performing an SR, as recommended in Chapters 2 through 4 of this report. In addition, the committee added several items to PRISMA because of the committee’s focus on setting standards for public agencies that sponsor SRs of comparative effectiveness research (CER), which place a strong emphasis on generating evidence to inform patient and clinical decision making. Therefore, the committee’s reporting recommendations build on PRISMA, but incorporate the following revisions: greater specificity in reporting the data collection and study selection process, and eight new checklist items. The checklist items are as follows: (1) an executive summary, (2) a summary written for the lay public, (3) an analytic framework and description of the chain of logic for how the intervention may improve a health outcome, (4) rationale for pool - ing (or not pooling) results across studies, (5) results of the qualita - tive synthesis, including findings of differences in responses to the intervention for key subgroups (this requirement reflects a specific characteristic of CER: the search for evidence to help patients and clinicians tailor the decisions to the characteristics and needs of the individual patient), (6) tables and figures summarizing the results, (7) gaps in evidence, and (8) future research needs. The following sections present the committee’s recommenda- tions for the key components of a final SR report: title, abstract and summaries, introduction, methods, results, discussion, and funding and conflict-of-interest (COI) sections of SR reports (see Box 5-1 for a complete list of all required reporting elements). Report Title The title should identify the report as an SR, a meta-analysis, or both (if appropriate). This may improve the indexing and iden - tification of SRs in bibliographic databases (Liberati et al., 2009). The title should also reflect the research questions addressed in the review in order to help the reader understand the scope of the SR. PRISMA provides the following example of a clear title: “Recur- rence Rates of Video-assisted Thoracoscopic versus Open Surgery in the Prevention of Recurrent Pneumothoraces: A Systematic Review of Randomized and Nonrandomized Trials” (Barker et al., 2007; Liberati et al., 2009). Abstract, Executive Summary, and Plain-Language Summary The SR final report should include a structured abstract orga- nized under a series of headings corresponding to the background,

OCR for page 195
205 STANDARDS FOR REPORTING SYSTEMATIC REVIEWS methods, results, and conclusions (Haynes et al., 1990; Mulrow et al., 1988). A structured abstract helps readers to quickly determine the scope, processes, and findings of a review without reading the entire report. Structured abstracts also give the reader more com- plete information than unstructured abstracts (Froom and Froom, 1993; Hartley, 2000; Hartley et al., 1996; Pocock et al., 1987). In SR final reports, the abstract should address, as applicable: background; objectives; data sources; study eligibility criteria (inclusion/exclu - sion criteria), participants, and interventions; study appraisal and synthesis methods; results; appraisal of the body of evidence; limita- tions; conclusions and implications of key findings; and SR registra- tion number3 (Liberati et al., 2009). See Box 5-4 for an example of a structured abstract. The final report should also include an executive summary. Many users and stakeholders find concise summaries that highlight the main findings and allow for rapid scanning of results very useful (Lavis et al., 2005; Oxman et al., 2006). Because the length of abstracts is often limited they may not provide enough information to satisfy decision makers. The committee’s recommendation to include an executive summary and abstract in final reports is consistent with guidance from AHRQ and CRD (AHRQ, 2009a; CRD, 2009). SR reports, including their abstracts and executive summaries, are often written in language that is too technical for consumers and patients to use in decision making. This is especially problematic for SRs of CER studies because one of the major goals of CER is to help patients and consumers make healthcare decisions (IOM, 2009). To improve the usability of SRs for patients and consumers, the com- mittee recommends that final reports include summaries written in nontechnical language (the plain-language summary) (see Box 5-5 for an example). The plain-language summary should include background information about the healthcare condition, population, intervention, and main findings. The committee believes the plain- language summary should explain the shortcomings of the body of evidence, so the public can form a realistic appreciation of the limita- tions of the science. Developing plain-language summaries requires specialized knowledge and skills. An important resource in this area is the John M. Eisenberg Clinical Decisions and Communications Science Center at Baylor College of Medicine in Houston, Texas. The Center, with AHRQ funding, translates SRs of CER conducted by the 3 An SR registration number is the unique identification number assigned to a proto- col in an electronic registry. See Chapter 2 for a discussion on protocol publication.

OCR for page 195
212 FINDING WHAT WORKS IN HEALTH CARE TABLE 5-2 Topics to Include in the Results Section (repeat for each key question) Results Topic Include Study •    umbers of studies that were screened, assessed for  N selection eligibility, and included in the review •     flow chart that shows the number of studies that  A remain after each stage of the selection process •    rovide a citation for each included study  P Excluded •    xcluded studies that experts might expect to see  E studies included and reason for their exclusion Appraisal of •    ummarize the threats to validity in each study and, if  S individual available, any outcome-level assessment of the effects of studies bias •    ummarize the relevance of the studies to the  S populations, interventions, and outcome measures •    ummarize the fidelity of the implementation of  S interventions Qualitative •    ummarize clinical and methodological characteristics of  S synthesis the included studies, such as: o Number and characteristics of study participants, including factors that may impact generalizability of results to real-world settings (e.g., comorbidities in studies of older patients or race/ethnicity in conditions where disparities exist) o Clinical settings o Interventions o Primary and secondary outcome measures o Follow-up period •    bserved patterns of threats to validity across studies,  O strengths, and weaknesses of the evidence, and confidence in the results •    escription of the overall body of evidence across the  D following domains: o Risk of bias o Consistency o Precision o Directness o Reporting bias o Dose–response association o Plausible confounding that would change the observed effect o Strength of association •    indings of differences in responses to the intervention  F for key subgroups (e.g., by age, race, gender, socioeconomic status, and/or clinical findings)

OCR for page 195
213 STANDARDS FOR REPORTING SYSTEMATIC REVIEWS TABLE 5-2 Continued Results Topic Include Meta-analysis •    ustification for why a pooled estimate might be more  J (if performed) useful to decision makers than the results of each study individually •    xamination of how heterogeneity in the treatment’s  E effects may be due to clinical differences in the study population or methodological differences in the studies’ design •    esults of each meta-analysis, including a measure  R of statistical uncertainty and the sensitivity of the conclusions to changes in the protocol, assumptions, and study selection Additional •    f done, results of additional analyses (e.g., subgroup  I analyses analyses, meta-regression), indicating whether the analysis was prespecified or exploratory Tables and •    n evidence table summarizing the characteristics of  A figures included studies •    raphic displays of results (e.g., forest plots to  G summarize quantitative findings, GRADE summary tables) NOTE: GRADE = Grading of Recommendations Assessment, Development and Evaluation. than SRs funded through other sources (Lexchin et al., 2003; Yank et al., 2007). Identifying the sources of funding and the role of the spon- sor (including whether the sponsor reserved the right to approve the content of the report) in the final report improves the transparency and is critical for the credibility of the report (Liberati et al., 2009). Currently, many peer-reviewed publications fail to provide com- plete or consistent information regarding the authors’ biases and COI (Chimonas et al., 2011; McPartland, 2009; Roundtree et al., 2008). A recent study of payments received by physicians from orthopedic device  companies  identified  41  individuals  who  each  received  $1  million or more in 2007. In 2008 and 2009, these individuals pub - lished a total of 95 articles relating to orthopedics. Fewer than half the articles disclosed the authors’ relationships with the orthopedic device manufacturers, and an even smaller number provided infor- mation on the amount of the physicians’ payments (Chimonas et al., 2011). Requiring authors to disclose any potential outside influences on their judgment, not just industry relationships, improves the

OCR for page 195
214 FINDING WHAT WORKS IN HEALTH CARE TABLE 5-3 EPICOT Format for Formulating Future Research Recommendations EPICOT Component Issues to Consider Example E Evidence What is the current One systematic review evidence? dominated by a large randomized controlled study conducted in hospital setting P Population Diagnosis, disease stage, Primary care patients comorbidity, risk factor, sex, with confirmed stroke or age, ethnic group, specific transient ischemic attack (mean age ≥ 75 years, inclusion or exclusion criteria, clinical setting female–male ratio 1:1, time since last cerebrovascular event ≥ 1 year) I Intervention Type, frequency, dose, Intensive blood pressure duration, prognostic factor lowering C Comparison Placebo, routine care, No active treatment or alternative treatment/ placebo management O Outcomes Which clinical or patient- Major vascular events related outcomes will the (stroke, myocardial researcher need to measure, infarction, vascular death); improve, influence, or adverse events, risk of accomplish? Which methods discontinuation of treatment of measurement should be because of adverse events used? T Time stamp Date of literature search or February 2006 recommendation SOURCE: Brown et al. (2006). transparency and trustworthiness of the review. The ACA contains a similar requirement for authors of research funded by PCORI.7 RECOMMENDED STANDARD FOR REPORT REVIEW The committee recommends one overarching standard for review by scientific peers, other users and stakeholders, and the public: Standard 5.2–-Peer review the draft report Required elements: 5.2.1 Use a third party to manage the peer review process 7 The Patient Protection and Affordable Care Act at § 6301(h)(3)(B).

OCR for page 195
215 STANDARDS FOR REPORTING SYSTEMATIC REVIEWS BOX 5-6 Reporting Funding and Conflict of Interest: Selected Examples Source of Funding “PRISMA was funded by the Canadian Institutes of Health Research; Uni- versita’ di Modena e Reggio Emilia, Italy; Cancer Research UK; Clinical Evidence BMJ Knowledge; the Cochrane Collaboration; and GlaxoSmith- Kline, Canada. AL is funded, in part, through grants of the Italian Ministry of University (COFIN–PRIN 2002 prot. 2002061749 and COFIN–PRIN 2006 prot. 2006062298). DGA is funded by Cancer Research UK. DM is funded by a University of Ottawa Research Chair.” Role of Funders “None of the sponsors had any involvement in the planning, execution, or write-up of the PRISMA documents. Additionally, no funder played a role in drafting the manuscript.” Potential Conflicts of Interest “ The authors have declared that no competing interests exist.” SOURCE: Moher et al. (2009). 5.2.2 Provide a public comment period for the report and publicly report on disposition of comments Rationale SR final reports should be critically reviewed by peer reviewers to ensure accuracy and clarity and to identify any potential meth- odological flaws (e.g., overlooked studies, methodological errors). The original protocol for the SR (including any amendments) should be made available to the peer reviewers. A small body of empirical evidence suggests that the peer review process improves the qual- ity of published research by making the manuscripts more readable and improving the comprehensiveness of reporting (Goodman et al., 1994; Jefferson et al., 2002; Weller, 2002). In addition, the critical assessment of manuscripts by peer reviewers is an essential part of the scientific process (ICMJE, 2010). Journals rely on the peer review process to establish when a study is suitable for publication and to improve the quality of reporting and compliance with reporting guidelines (ICMJE, 2010). Some version of peer review is recom-

OCR for page 195
216 FINDING WHAT WORKS IN HEALTH CARE mended in the guidance from all the major producers of SRs (CRD, 2009; Higgins and Green, 2008; Slutsky et al., 2010). Peer review of research funded through PCORI will be required, either directly through a process established by PCORI or by an appropriate medi- cal journal or other entity.8 The evidence is unclear on how to select peer reviewers, the qualifications that are important for peer reviewers to possess, and what type of training and instructions improve the peer review process (Callaham and Tercier, 2007; Jefferson et al., 2002; Schroter et al., 2004, 2006). In the context of publicly funded SRs, the com- mittee recommends that peer reviewers include a range of relevant users and stakeholders, such as practicing clinicians, statisticians and other methodologists, and consumers. This process can be used to gather input from perspectives that were not represented on the review team (e.g., individuals with diverse clinical specialties). The committee also recommends that the public be given an opportunity to comment on SR reports as part of the peer review process. Allowing public comments encourages publicly funded research that is responsive to the public’s interests and concerns and is written in language that is understandable and usable for patient and clinical decision making. Requiring a public comment period is also consistent with the ACA, which directs PCORI to obtain public input on research findings,9 as well as guidance from AHRQ and Cochrane (Higgins and Green, 2008; Whitlock et al., 2010). The review team should be responsive to the feedback provided by the peer reviewers and the public, and publicly report how it revised the SR in response to the comments. The authors should document the major comments and input received; how the final report was or was not modified accordingly; and the rationale for the course of action. The authors’ response to this feedback can be organized into general topic areas of response, rather than respond - ing to each individual comment. Requiring authors to report on the disposition of comments holds the review authors accountable for responding to the peer reviewers’ comments and improves the public’s confidence in the scientific integrity and credibility of the SR (Whitlock et al., 2010). A neutral third party should manage and oversee the entire peer review process. The main role of the third party should be to pro- vide an independent judgment about the adequacy of the authors’ responses (Helfand and Balshem, 2010). This recommendation is 8 The Patient Protection and Affordable Care Act at § 6301(d)(7). 9 The Patient Protection and Affordable Care Act at § 6301(h).

OCR for page 195
217 STANDARDS FOR REPORTING SYSTEMATIC REVIEWS consistent with the rules governing PCORI that allow, but do not require, the peer review process to be overseen by a medical journal or outside entity.10 It also furthers SR authors’ accountability for responding to reviewers’ feedback and it is consistent with AHRQ guidance (Helfand and Balshem, 2010; Whitlock et al., 2010). The National Academies has an office that manages the review of all Academies studies. A monitor and coordinator, chosen by the report review office from the membership of the Academies, oversee the response to external review. They must approve the response to review before release of the report. RECOMMENDED STANDARD FOR PUBLISHING THE FINAL REPORT The committee recommends one standard for publishing the final report: Standard 5.3—Publish the final report in a manner that en- sures free public access Rationale The final report should be publicly available. PCORI will be required to post research findings on a website accessible to clinicians, patients, and the general public no later than 90 days after receipt of the research findings and completion of the peer review process.11 This requirement should be extended to all publicly funded SRs of effectiveness research. Publishing final reports is consistent with leading guidance (AHRQ, 2010c; CRD, 2009; Higgins and Green, 2008) and this committee’s criteria of transparency and credibility. Public sponsors should not prevent the SR team from publishing the SR in a peer-reviewed journal and should not interfere with the journal’s peer review process. Ideally, the public sponsor will cooperate with the journal to ensure timely, thorough peer review, so that journal publication and posting on the sponsor’s website can take place simultaneously. In any case, posting an SR final report on a government website should not qualify as a previous publication, in the same way that journals have agreed that publication of an abstract describing clinical trial results in clinicaltrials.gov (which is required by federal law) does not count as prior publication (ICMJE, 10 The Patient Protection and Affordable Care Act at § 6301(d)(7). 11 The Patient Protection and Affordable Care Act at § 6301(d)(8)(A).

OCR for page 195
218 FINDING WHAT WORKS IN HEALTH CARE 2009). In addition, public sponsors should encourage the review team to post the research results in international SR registries, such as the one being developed by the CRD (CRD, 2010b). REFERENCES AHRQ (Agency for Healthcare Research and Quality). 2009a. AHRQ Evidence-based Practice Centers partners guide. Gaithersburg, MD: AHRQ. AHRQ. 2009b. Cancer, clinical utility of family history. http://www.ahrq.gov/clinic/ tp/famhist2tp.htm (accessed September 20, 2010). AHRQ. 2010a. Consumers and patients. http://www.ahrq.gov/consumer/compare. html (accessed September 10, 2010). AHRQ. 2010b. About the Eisenberg Center. http://www.effectivehealthcare.ahrq.gov/index. cfm/who-is-involved-in-the-effective-health-care-program1/about-the-eisenberg- center/ (accessed December 27, 2010).AHRQ. 2010c. Search for guides, reviews, and reports. http://www.effectivehealthcare.ahrq.gov/index.cfm/search-for-guides- reviews-and-reports/ (accessed July 2, 2010). AHRQ. 2010c. Search for guides, reviews, and reports. http://www.effectivehealthcare. ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/ (accessed July 2, 2010). Akl, E., A. D. Oxman, J. Herrin, G. E. Vist, I. Terrenato, F. Sperati, C. Costiniuk, D. Blank, and H. Schunemann. In press. Using alternative statistical formats for presenting risks and risk reduction. Cochrane Database of Systematic Reviews. Barker, A., E. C. Maratos, L. Edmonds, and M. Lim. 2007. Recurrence rates of video- assisted thoracoscopic versus open surgery in the prevention of recurrent pneu- mothoraces: A systematic review of randomised and non-randomised trials. Lancet 370(9584):329–335. Begg, C., M. Cho, S. Eastwood, R. Horton, D. Moher, I. Olkin, R. Pitkin, D. Rennie, K. F. Schulz, D. Simel, and D. F. Stroup. 1996. Improving the quality of reporting of randomized controlled trials: The CONSORT statement. JAMA 276(8):637–639. Brown, P., K. Brunnhuber, K. Chalkidou, I. Chalmers, M. Clarke, M. Fenton, C. Forbes, J. Glanville, N. J. Hicks, J. Moody, S. Twaddle, H. Timimi, and P. Young. 2006. How to formulate research recommendations. BMJ 333(7572):804–806. Callaham, M. L., and J. Tercier. 2007. The relationship of previous training and ex - perience of journal peer reviewers to subsequent review quality. PLoS Medicine 4(1):e40. Chimonas, S., Z. Frosch, and D. J. Rothman. 2011. From disclosure to transparency: The use of company payment data. Archives of Internal Medicine.171(1):81–86. Clarke, L., M. Clarke, and T. Clarke. 2007. How useful are Cochrane reviews in iden- tifying health needs? Journal of Health Services & Research Policy 12(2):101–103. CRD (Centre for Reviews and Dissemination). 2009. Systematic reviews: CRD’s guidance for undertaking reviews in health care. York, UK: York Publishing Services, Ltd. CRD. 2010a. Database of Abstracts of Reviews of Effects (DARE). http://www.crd.york. ac.uk/crdweb/Home.aspx?DB=DARE (accessed December 27, 2010). CRD. 2010b. Register of ongoing systematic reviews. http://www.york.ac.uk/inst/crd/ projects/register.htm (accessed June 17, 2010). Delaney, A., S. Bagshaw, A. Ferland, B. Manns, K. Laupland, and C. Doig. 2005. A sys- tematic evaluation of the quality of meta-analyses in the critical care literature. Critical Care 9(5):R575–R582.

OCR for page 195
219 STANDARDS FOR REPORTING SYSTEMATIC REVIEWS Delaney, A., S. M. Bagshaw, A. Ferland, K. Laupland, B. Manns, and C. Doig. 2007. The quality of reports of critical care meta-analyses in the Cochrane Database of Systematic Reviews: An independent appraisal. Critical Care Medicine 35(2): 589–594. Devereaux, P. J., P. Choi, S. El-Dika, M. Bhandari, V. M. Montori, H. J. Schünemann, A. X. Garg, J. W. Busse, D. Heels-Ansdell, W. A. Ghali, B. J. Manns, and G. H. Guyatt. 2004. An observational study found that authors of randomized controlled trials frequently use concealment of randomization and blinding, despite the failure to report these methods. Journal of Clinical Epidemiology 57(12):1232–1236. Docherty, M., and R. Smith. 1999. The case for structuring the discussion of scientific papers. BMJ 318(7193):1224–1225. EQUATOR Network. 2010. Welcome to the EQUATOR Network website: The resource centre for good reporting of health research studies. http://www.equator-network. org/home/ (accessed June 29, 2010). Froom, P., and J. Froom. 1993. Deficiencies in structured medical abstracts. Journal of Clinical Epidemiology 46(7):591–594. Glenton, C. 2002. Developing patient-centered information for back pain patients. Health Expectations 5(4):319–329. Glenton, C. 2010. Presenting the results of Cochrane systematic reviews to a consumer audience: A qualitative study. Medical Decision Making 30(5):566–577. Glenton, C., E. Nilsen, and B. Carlsen. 2006a. Lay perceptions of evidence-based information: A qualitative evaluation of a website for back pain sufferers. BMC Health Services Research 6(1):34. Glenton, C., V. Underland, M. Kho, V. Pennick, and A. D. Oxman. 2006b. Summa - ries of findings, descriptions of interventions and information about adverse effects would make reviews more informative. Journal of Clinical Epidemiology 59(8):770–778. Golder, S., Y. Loke, and H. M. McIntosh. 2008. Poor reporting and inadequate searches were apparent in systematic reviews of adverse effects. Journal of Clinical Epide- miology 61(5):440–448. Goodman, S. N., J. Berlin, S. W. Fletcher, and R. H. Fletcher. 1994. Manuscript quality before and after peer review and editing at Annals of Internal Medicine. Annals of Internal Medicine 121(1):11–21. Hartley, J. 2000. Clarifying the abstracts of systematic literature reviews. Bulletin of the Medical Library Association 88(4):332–337. Hartley, J., M. Sydes, and A. Blurton. 1996. Obtaining information accurately and quickly: Are structured abstracts more efficient? Journal of Information Science 22(5):349–356. Haynes, R. B., C. D. Mulrow, E. J. Huth, D. G. Altman, and M. J. Gardner. 1990. More informative abstracts revisited. Annals of Internal Medicine 113(1):69–76. Helfand, M., and H. Balshem. 2010. AHRQ Series Paper 2: Principles for develop - ing guidance: AHRQ and the Effective Health-Care Program. Journal of Clinical Epidemiology 63(5):484–490. Higgins, J. P. T., and S. Green, eds. 2008. Cochrane handbook for systematic reviews of interventions. Chichester, UK: John Wiley & Sons. ICMJE (International Committee of Medical Journal Editors). 2009. Obligation to reg- ister clinical trials. http://www.icmje.org/publishing_10register.html (accessed September 9, 2010). ICMJE. 2010. Uniform requirements for manuscripts submitted to biomedical journals: Ethi - cal considerations in the conduct and reporting of research: Peer review. http://www. icmje.org/ethical_3peer.html (accessed July 12, 2010).

OCR for page 195
220 FINDING WHAT WORKS IN HEALTH CARE Ioannidis, J. P., J. W. Evans, P. C. Gøtzsche, R. T. O’Neill, D. Altman, K. Schulz, and D. Moher. 2004. Better reporting of harms in randomized trials: An extension of the CONSORT Statement. Annals of Internal Medicine 141(10):781–788. IOM (Institute of Medicine). 2008. Knowing what works in health care: A roadmap for the nation. Edited by J. Eden, B. Wheatley, B. McNeil, and H. Sox. Washington, DC: The National Academies Press. IOM. 2009. Initial national priorities for comparative effectiveness research. Washington, DC: The National Academies Press. Jefferson, T., P. Alderson, E. Wager, and F. Davidoff. 2002. Effects of editorial peer review: A systematic review. JAMA 287(21):2784–2786. Lavis, J., H. Davies, A. D. Oxman, J. L. Denis, K. Golden-Biddle, and E. Ferlie. 2005. Towards systematic reviews that inform health care management and policy- making. Journal of Health Services & Research Policy 10(Suppl 1):35–48. Lexchin, J., L. A. Bero, B. Djubegovic, and O. Clark. 2003. Pharmaceutical indus - try sponsorship and research outcome and quality: Systematic review. BMJ 326(7400):1167–1170. Liberati, A., D. G. Altman, J. Tetzlaff, C. Mulrow, P. Gotzsche, J. P. Ioannidis, M. Clarke, P. J. Devereaux, J. Kleijnen, and D. Moher. 2009. The PRISMA Statement for reporting systematic reviews and meta-analysis of studies that evaluate health care interventions: Explanation and elaboration. Annals of Internal Medi- cine 151(4):W11–W30. Lipkus, I. M. 2007. Numeric, verbal, and visual formats of conveying health risks: Suggested best practices and future recommendations. Medical Decision Making 27(5):696–713. McAlister, F. A., H. D. Clark, C. van Walraven, S. E. Straus, F. Lawson, D. Moher, and C. D. Mulrow. 1999. The medical review article revisited: Has the science improved? Annals of Internal Medicine 131(12):947–951. McPartland, J. M. 2009. Obesity, the endocannabinoid system, and bias arising from pharmaceutical sponsorship. PLoS One 4(3):e5092. Moher, D., D. J. Cook, S. Eastwood, I. Olkin, D. Rennie, and D. F. Stroup. 1999. Im - proving the quality of reports of meta-analyses of randomised controlled trials: The QUOROM statement. Lancet 354(9193):1896–1900. Moher, D., A. Jones, and L. Lepage. 2001a. Use of the CONSORT statement and qual - ity of reports of randomized trials: A comparative before-and-after evaluation. JAMA 285:1992–1995. Moher, D., K. F. Schulz, D. Altman, and the CONSORT Group. 2001b. The CONSORT statement: Revised recommendations for improving the quality of reports of parallel-group randomized trials. JAMA 285(15):1987–1991. Moher, D., J. Tetzlaff, A. C. Tricco, M. Sampson, and D. G. Altman. 2007. Epidemiol - ogy and reporting characteristics of systematic reviews. PLoS Medicine 4(3):447– 455. Moher, D., A. Liberati, J. Tetzlaff, and D. G. Altman. 2009. Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. PLoS Medi- cine 6(7):1–6. Mrkobrada, M., H. Thiessen-Philbrook, R. B. Haynes, A. V. Iansavichus, F. Rehman, and A. X. Garg. 2008. Need for quality improvement in renal systematic reviews. Clinical Journal of the American Society of Nephrology 3(4):1102–1114. Mulrow, C. D. 1987. The medical review article: State of the science. Annals of Internal Medicine 106(3):485–488. Mulrow, C. D., S. B. Thacker, and J. A. Pugh. 1988. A proposal for more informative abstracts of review articles. Annals of Internal Medicine 108(4):613–615.

OCR for page 195
221 STANDARDS FOR REPORTING SYSTEMATIC REVIEWS Odierna, D. H., and L. A. Bero. 2009. Systematic reviews reveal unrepresentative evidence for the development of drug formularies for poor and nonwhite popu - lations. Journal of Clinical Epidemiology 62(12):1268–1278. Oxman, A. D., H. Schunemann, and A. Fretheim. 2006. Improving the use of research evidence in guideline development: Synthesis and presentation of evidence. Health Research Policy and Systems 4:20. Pocock, S., M. D. Hughes, and R. J. Lee. 1987. Statistical problems in the reporting of clinical trials. A survey of three medical journals. New England Journal of Medicine 317(7):426–432. PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses). 2010. PRISMA endorsers. http://www.prisma-statement.org/endorsers.htm (ac- cessed September 13, 2010). Roberts, D., and S. Dalziel. 2006. Antenatal corticosteroids for accelerating fetal lung maturation for women at risk of preterm birth. In Cochrane Database of Systematic Reviews. http://www.cochrane.org/cochrane-reviews/sample-review#PLS (ac- cessed October 6, 2010). Roundtree, A. K., M. A. Kallen, M. A. Lopez-Olivo, B. Kimmel, B. Skidmore, Z. Ortiz, V. Cox, and M. E. Suarez-Almazor. 2008. Poor reporting of search strategy and conflict of interest in over 250 narrative and systematic reviews of two biologic agents in arthritis: A systematic review. Journal of Clinical Epidemiology 62(2):128–137 Sacks, H. S., D. Reitman, V. A. Ancona-Berk, and T. C. Chalmers. 1987. Meta–analysis of randomized controlled trials. New England Journal of Medicine 316(8):450– 455. Santesso, N., L. Maxwell, P. S. Tugwell, G. A. Wells, A. M. Connor, M. Judd, and R. Buchbinder. 2006. Knowledge transfer to clinicians and consumers by the Cochrane Musculoskeletal Group. The Journal of Rheumatology 33(11):2312–2318. Schroter, S., N. Black, S. Evans, J. Carpenter, F. Godlee, and R. Smith. 2004. Ef- fects of training on quality of peer review: Randomised controlled trial. BMJ 328(7441):657–658. Schroter, S., L. Tite, A. Hutchings, and N. Black. 2006. Differences in review quality and recommendations for publication between peer reviewers suggested by authors or by editors. JAMA 295(3):314–317. Schünemann, H. J., E. Ståhl, P. Austin, E. Akl, D. Armstrong, and G. H. Guyatt. 2004. A comparison of narrative and table formats for presenting hypothetical health states to patients with gastrointestinal or pulmonary disease. Medical Decision Making 24(1):53–60. Schwartz, L. M., S. Woloshin, and H. G. Welch. 2009. Using a drug facts box to com- municate drug benefits and harms. Annals of Internal Medicine 150(8):516–527. Shea, B., L. M. Bouter, J. Peterson, M. Boers, N. Andersson, Z. Ortiz, T. Ramsay, A. Bai, V. K. Shukla, and J. M. Grimshaw. 2007. External validation of a measurement tool to assess systematic reviews (AMSTAR). PLoS ONE 2(12):e1350. Slutsky, J., D. Atkins, S. Chang, and B. A. Collins Sharp. 2010. AHRQ Series Paper 1: Comparing medical interventions: AHRQ and the Effective Health-Care Pro - gram. Journal of Clinical Epidemiology 63(5):481–483. Stroup, D. F., J. Berlin, S. Morton, I. Olkin, G. D. Williamson, D. Rennie, D. Moher, B. J. Becker, T. A. Sipe, S. B. Thacker, and the Meta-analysis of Observational Studies in Epidemiology Group. 2000. Meta-analysis of observational studies in epidemiology: A proposal for reporting. JAMA 283(15):2008–2012. Trevena, L. J., A. Barratt, P. Butow, and P. Caldwell. 2006. A systematic review on communicating with patients about evidence. Journal of Evaluation in Clinical Practice 12(1):13–23.

OCR for page 195
222 FINDING WHAT WORKS IN HEALTH CARE von Elm, E., D. G. Altman, M. Egger, S. J. Pocock, P. C. Gotzsche, J. P. Vandenbroucke, and the Strobe Initiative. 2007. The Strengthening the Reporting of Observa - tional Studies in Epidemiology (STROBE) statement: Guidelines for reporting observational studies. Annals of Internal Medicine 147(8):573–577. Weller, A. C. 2002. Editorial peer review: Its strengths and weaknesses. Medford, NJ: American Society for Information Science and Technology. http://books.google. com/books?hl=en&lr=&id=lH3DVnkbTFsC&oi=fnd&pg=PR12&dq=ICMJE+P EER+REVIEW&ots=rpXA8rZOna&sig=nCltaFOsxGxyrVSoGCeWWhRNLIE# v=onepage&q=ICMJE%20PEER%20REVIEW&f=false (accessed July 12, 2010). Whitlock, E. P., S. A. Lopez, S. Chang, M. Helfand, M. Eder, and N. Floyd. 2010. AHRQ Series Paper 3: Identifying, selecting, and refining topics for comparative effectiveness systematic reviews: AHRQ and the Effective Health-Care Program. Journal of Clinical Epidemiology 63(5):491–501. Wills, C. E., and M. Holmes-Rovner. 2003. Patient comprehension of information for shared treatment decision making: State of the art and future directions. Patient Education and Counseling 50(3):285–290. Yank, V., D. Rennie, and L. A. Bero. 2007. Financial ties and concordance between results and conclusions in meta-analyses: Retrospective cohort study. BMJ 335(7631):1202–1205.