5
Standards for Reporting Systematic Reviews
Abstract: Authors of publicly sponsored systematic reviews (SRs) should produce a detailed, comprehensive final report. The committee recommends three related standards for documenting the SR process, responding to input from peer reviewers and other users and stakeholders, and making the final report publicly available. The standards draw extensively from the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) checklist. The committee recommends several reporting items in addition to the PRISMA requirements to ensure that the final report (1) describes all of the steps and judgments required by the standards in the previous chapters and (2) focuses on informing patient and clinical decision making.
High-quality systematic review (SR) reports should accurately document all of the steps and judgments in the SR process using clear language that is understandable to users and stakeholders. A report should provide enough detail that a knowledgeable reader could reproduce the SR. The quality of a final report has profound implications for patients and clinicians. Too often the information that researchers report in published SRs does not adequately reflect
their study methods (Devereaux et al., 2004).1 If SRs are poorly reported, patients and clinicians have difficulty determining whether an SR is trustworthy enough to be used to guide decision making or the development of clinical practice guidelines (Moher et al., 2007). High-quality SR reports summarize the methodological strengths and weaknesses of the SR and include language designed to help nonexperts interpret and judge the value of the SR (AHRQ, 2010b; CRD, 2010a; Higgins and Green, 2008; Liberati et al. 2009; Moher et al. 2009). However, according to an extensive literature, many published SRs inadequately document important aspects of the SR process (Delaney et al., 2005, 2007; Golder et al., 2008; McAlister et al., 1999; Moher et al., 2007; Mulrow, 1987; Roundtree et al., 2008; Sacks et al., 1987). A seminal study conducted by Mulrow, for example, assessed 50 review articles published in four leading medical journals and found that many reviews failed to report the methods of identifying, selecting, and validating information, and choosing areas for future research (Mulrow, 1987). More recently, Moher and colleagues (2007) evaluated 300 SRs indexed in MEDLINE during November 2004. They concluded that information continues to be poorly reported, with many SRs failing to report key components of SRs, such as assessing for publication bias, aspects of the searching and screening process, and funding sources. Other studies have found that SRs published in journals often inadequately report search strategies, validity assessments of included studies, and authors’ conflicts of interest (Delaney et al., 2005; Golder et al., 2008; Roundtree et al., 2008).
Authors of all publicly sponsored SRs must produce a detailed final report, which is typically longer and more detailed than the version submitted for journal publication. The sponsor typically publishes the final report on its website, where it stands as the definitive documentation of the review. The standards recommended by the committee apply to this definitive comprehensive final report. The committee recommends three standards for producing a comprehensive SR final report (Box 5-1), including standards for documenting the SR process, responding to input from peer reviewers and other users and stakeholders, and making the final reports publicly available. Each standard includes elements of performance that the committee deems essential. The evidence base for developing standards for the final report is sparse. In addition, most evaluations of the quality of published SRs have focused on
1 |
See Chapter 3 for a review of the literature on reporting bias and dearth of adequate documentation in most SRs of comparative effectiveness. |
BOX 5-1 Recommended Standards for Reporting Systematic Reviews Standard 5.1 Prepare the final report using a structured format Required elements:
|
journal articles rather than SR reports. The committee developed the standards by first reviewing existing expert guidance, particularly the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) checklist (Liberati et al., 2009). However, PRISMA is focused on journal articles, not comprehensive final reports to public sponsors. The committee recommended including items that were not on the PRISMA checklist because it believed that the report of an SR should describe all the steps and judgments required by the committee’s standards in Chapters 2 through 4 to improve the transparency of the SR process and to inform patient and clinical decision making. The committee also took into account the legislatively mandated reporting requirements for the Patient-Centered Outcomes Research Institute (PCORI), as specified by the 2010 Patient Protection and Affordable Care Act (ACA). Box 5-2 describes the ACA reporting requirements for research funded by PCORI. See Appendix G for the Agency for Healthcare Research and Quality (AHRQ), Centre for Reviews and Dissemination (CRD), and Cochrane Collaboration guidance on writing an SR final report. Appendix H contains the PRISMA checklist.
BOX 5-2 Requirements for Research Funded by the Patient-Centered Outcomes Research Institute The 2010 Patient Protection and Affordable Care Act created the Patient-Centered Outcomes Research Institute (PCORI), a nonprofit corporation intended to advance comparative effectiveness research. The act stipulates that research funded by PCORI, including systematic reviews, adhere to the following reporting and publication requirements:
SOURCE: The Patient Protection and Affordable Care Act, Public Law 111-148, 111th Cong., Subtitle D, § 6301 (March 23, 2010). |
REPORTING GUIDELINES
Over the past decade, several international, multidisciplinary groups have collaborated to develop guidelines for reporting the methods and results of clinical research (reporting guidelines). Reporting guidelines exist for many types of health research (Ioannidis et al., 2004; Liberati et al., 2009; Moher et al., 1999, 2001a,b,
2009; Stroup et al., 2000). These guideline initiatives were undertaken out of concern that reports on health research were poorly documenting the methods and results of the research studies (IOM, 2008). Detailed reporting requirements are also seen as a line of defense against reporting bias.2 For SRs to be trustworthy enough to inform healthcare decisions, accurate, thorough, and transparent reporting are essential. The adoption of reporting guidelines furthers this goal. Examples of reporting guidelines include the Consolidated Standards of Reporting Trials (CONSORT) statement for reporting randomized clinical trials (Ioannidis et al., 2004; Moher et al., 2001b), and the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement for reporting observational studies in epidemiology (von Elm et al., 2007). The major reporting guideline for SRs and meta-analyses is PRISMA (Liberati et al., 2009; Moher et al., 2009), an update to the 1999 Quality of Reporting of Meta-analyses (QUOROM) statement (Moher et al., 1999). In 2006, the Enhancing Quality and Transparency of Health Research (EQUATOR) Network was launched to coordinate initiatives to promote transparent and accurate reporting of health research and to assist in the development of reporting guidelines (EQUATOR Network, 2010). See Box 5-3 for a historical overview of reporting guidelines.
The methodological quality of SRs (i.e., how well the SR is conducted) is distinct from reporting quality (i.e., how well reviewers report their methodology and results) (Shea et al., 2007). Whether reporting guidelines improve the underlying methodological quality of research studies is unknown. However, incomplete documentation of the SR process makes it impossible to evaluate its methodological quality, so that it is impossible to tell whether a step in the SR process was performed correctly but not reported (poor reporting quality), completed inadequately, or not completed at all and therefore not reported (poor methodological quality).
At present, the evidence that reporting guidelines improve the quality of reports of SRs and meta-analyses is weak. The few observational studies that have addressed the issue have serious flaws. For example, Delaney and colleagues (2005) compared the quality of reports of meta-analyses addressing critical care, including topics related to shock, resuscitation, inotropes, and mechanical ventilation, published before and after the release of the QUOROM statement (the precursor to PRISMA). They found that reports of meta-analyses published after QUOROM were of higher quality
2 |
See Chapter 3 for a discussion on reporting bias. |
BOX 5-3 A History of Reporting Guidelines for Comparative Effectiveness Research In 1993 the Standards for Reporting Trials (SORT) group met to address inadequate reporting of randomized controlled trials (RCTs). This group developed the concept of a structured reporting guideline, and proposed a checklist of essential items for reporting RCTs. Five months later the Asilomar Working group met independently to discuss challenges in reporting RCTs and developed a reporting checklist. The Consolidated Standards of Reporting Trials (CONSORT) statement was developed in 1996 and consolidated the recommendation from both groups. The CONSORT statement consists of a checklist of reporting items, such as the background, methods, results, discussion, and conclusion sections, as well as a flow diagram for documenting participants through the trial. Many journals have adopted the CONSORT statement. It has been extended to address a number of specific issues in the reporting of RCTs (e.g., reporting of harms, noninferiority and equivalence RCTs, cluster RCTs). Following the success of the CONSORT statement, two international groups of review authors, methodologists, clinicians, medical editors, and consumers developed standard formats for reporting systematic reviews (SRs) and meta-analyses: Quality of Reporting of Meta-analyses (QUOROM) and Meta-analysis of Observational Studies in Epidemiology (MOOSE). The statements consist of checklists of items to include in reports and flow diagrams for documenting the search process. However, unlike CONSORT, reporting guidelines for SRs and meta-analyses have not been widely adopted by prominent journals. In 2009, the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) statement was published to update the QUOROM statement. According to its developers, PRISMA reflects the conceptual and practical advances made in the science of SRs since the development of QUOROM. These conceptual advances include the following: completing an SR is an iterative process; the conduct and reporting of research are distinct processes; the assessment of risk of bias requires both a study-level assessment (e.g., adequacy of allocation concealment) and outcome-level assessment (i.e., reliability and validity of the data for each outcome); and the importance of addressing reporting bias. PRISMA decouples several checklist items that were a single item on the QUOROM checklist and links other items to improve the consistency across the SR report. PRISMA was funded by the Canadian Institutes of Health Research; Universita di Modena e Reggio Emilia, Italy; Cancer Research U.K.; Clinical Evidence BMJ Knowledge; The Cochrane Collaboration; and GlaxoSmithKline, Canada.a It has been endorsed by a number of organizations and journals, including the Centre for Reviews and Dissemination, Cochrane Collaboration, British Medical Journal, and Lancet.b
SOURCES: Begg et al. (1996); Ioannidis et al. (2004); IOM (2008); Liberati et al. (2009); Moher et al. (1999, 2001a, 2001b, 2007, 2009); Stroup et al. (2000). |
than reports published before and were more likely to describe whether a comprehensive literature search was conducted; the criteria for screening the studies; and the methods used to combine the findings of relevant studies (Delaney et al., 2005). Mrkobrada and colleagues (2008) evaluated 90 SRs published in 2005 in the field of nephrology. They found that only a minority of journals (4 out of 48) recommended adherence to SR reporting guidelines. The four journals that endorsed or adopted reporting guidelines published SRs of significantly higher methodological quality than the other journals, and were more likely to report assessing methodological quality of included studies and taking precautions to avoid bias in study selection. Neither of these studies, however, assessed whether the journals endorsing QUOROM published higher quality reviews than the other journals prior to the adoption of QUOROM. In addition, journals that endorse reporting guidelines, such as QUOROM, may merely recommend that authors comply with the reporting items, but may not require authors to show compliance by submitting a checklist stating whether or not they adhered to each item as a condition of accepting the SR for review. As a result, whether the reporting improvements were due to QUOROM or other developments in the field is unclear. No controlled trials have evaluated the effectiveness of PRISMA on improving the reporting of SRs (Liberati et al., 2009).
In light of this history of reporting guidelines for medical journals, the committee decided to develop reporting guidelines specifically for the final report to the sponsor of an SR. The committee intends for its reporting requirements to improve the documentation of SR final report study methodology and results, and to increase the likelihood that SR final reports will provide enough information for patients and clinicians to determine whether an SR is trustworthy enough to be used to guide decision making.
SYSTEMATIC REVIEWS PUBLISHED IN JOURNALS
The committee recognizes that a journal publishing SRs will choose the level of documentation that is most appropriate for its readers. It also recognizes that its reporting requirements for final reports to public sponsors of SRs are quite detailed and comprehensive, and will produce manuscripts that are too long and too detailed for most journals to publish in full. Ideally, all published SRs (both final reports to sponsors and journal publications) should follow one reporting standard. With the advent of electronic-only appendixes to journal articles, journals can now require authors to
meet the committee’s full reporting guidelines (i.e., journals can post any reporting items not included in the actual journal publication in an online appendix). Alternatively, journals can publish a link to the website of the full SR report to the public sponsor, explaining what information readers would find only at the sponsor’s website.
RECOMMENDED STANDARD FOR PREPARING THE FINAL REPORT
The committee recommends the following standard for preparing the final report:
Standard 5.1—Prepare the final report using a structured format
Required elements:
5.1.1 |
Include a report title |
5.1.2 |
Include an abstract |
5.1.3 |
Include an executive summary |
5.1.4 |
Include a summary written for the lay public |
5.1.5 |
Include an introduction (rationale and objectives) |
5.1.6 |
Include a methods section |
5.1.7 |
Include a results section. Organize the presentation of results around key questions |
5.1.8 |
Include a discussion section |
5.1.9 |
Include a section describing funding sources and COI |
Rationale
All SR reports to public sponsors should use a structured format to help guide the readers to relevant information, to improve the documentation of the SR process, and to promote consistency in reporting. More than 150 journals have adopted the PRISMA requirements (PRISMA, 2010). Because of this support, the committee used the PRISMA checklist as its starting point for developing its reporting standards. However, PRISMA is focused on journal articles, which are usually subject to length restrictions in the print version of the article, and the committee’s reporting standards are directed at comprehensive, final reports to public sponsors (e.g., AHRQ, PCORI), which typically do not have word limits. Most of the committee’s additions and revisions to PRISMA were necessary to make the standards for the final report consistent with all of the steps and judgments in the SR process required by the standards for
performing an SR, as recommended in Chapters 2 through 4 of this report. In addition, the committee added several items to PRISMA because of the committee’s focus on setting standards for public agencies that sponsor SRs of comparative effectiveness research (CER), which place a strong emphasis on generating evidence to inform patient and clinical decision making.
Therefore, the committee’s reporting recommendations build on PRISMA, but incorporate the following revisions: greater specificity in reporting the data collection and study selection process, and eight new checklist items. The checklist items are as follows: (1) an executive summary, (2) a summary written for the lay public, (3) an analytic framework and description of the chain of logic for how the intervention may improve a health outcome, (4) rationale for pooling (or not pooling) results across studies, (5) results of the qualitative synthesis, including findings of differences in responses to the intervention for key subgroups (this requirement reflects a specific characteristic of CER: the search for evidence to help patients and clinicians tailor the decisions to the characteristics and needs of the individual patient), (6) tables and figures summarizing the results, (7) gaps in evidence, and (8) future research needs.
The following sections present the committee’s recommendations for the key components of a final SR report: title, abstract and summaries, introduction, methods, results, discussion, and funding and conflict-of-interest (COI) sections of SR reports (see Box 5-1 for a complete list of all required reporting elements).
Report Title
The title should identify the report as an SR, a meta-analysis, or both (if appropriate). This may improve the indexing and identification of SRs in bibliographic databases (Liberati et al., 2009). The title should also reflect the research questions addressed in the review in order to help the reader understand the scope of the SR. PRISMA provides the following example of a clear title: “Recurrence Rates of Video-assisted Thoracoscopic versus Open Surgery in the Prevention of Recurrent Pneumothoraces: A Systematic Review of Randomized and Nonrandomized Trials” (Barker et al., 2007; Liberati et al., 2009).
Abstract, Executive Summary, and Plain-Language Summary
The SR final report should include a structured abstract organized under a series of headings corresponding to the background,
methods, results, and conclusions (Haynes et al., 1990; Mulrow et al., 1988). A structured abstract helps readers to quickly determine the scope, processes, and findings of a review without reading the entire report. Structured abstracts also give the reader more complete information than unstructured abstracts (Froom and Froom, 1993; Hartley, 2000; Hartley et al., 1996; Pocock et al., 1987). In SR final reports, the abstract should address, as applicable: background; objectives; data sources; study eligibility criteria (inclusion/exclusion criteria), participants, and interventions; study appraisal and synthesis methods; results; appraisal of the body of evidence; limitations; conclusions and implications of key findings; and SR registration number3 (Liberati et al., 2009). See Box 5-4 for an example of a structured abstract.
The final report should also include an executive summary. Many users and stakeholders find concise summaries that highlight the main findings and allow for rapid scanning of results very useful (Lavis et al., 2005; Oxman et al., 2006). Because the length of abstracts is often limited they may not provide enough information to satisfy decision makers. The committee’s recommendation to include an executive summary and abstract in final reports is consistent with guidance from AHRQ and CRD (AHRQ, 2009a; CRD, 2009).
SR reports, including their abstracts and executive summaries, are often written in language that is too technical for consumers and patients to use in decision making. This is especially problematic for SRs of CER studies because one of the major goals of CER is to help patients and consumers make healthcare decisions (IOM, 2009). To improve the usability of SRs for patients and consumers, the committee recommends that final reports include summaries written in nontechnical language (the plain-language summary) (see Box 5-5 for an example). The plain-language summary should include background information about the healthcare condition, population, intervention, and main findings. The committee believes the plainlanguage summary should explain the shortcomings of the body of evidence, so the public can form a realistic appreciation of the limitations of the science. Developing plain-language summaries requires specialized knowledge and skills. An important resource in this area is the John M. Eisenberg Clinical Decisions and Communications Science Center at Baylor College of Medicine in Houston, Texas. The Center, with AHRQ funding, translates SRs of CER conducted by the
3 |
An SR registration number is the unique identification number assigned to a protocol in an electronic registry. See Chapter 2 for a discussion on protocol publication. |
BOX 5-4 Example of a Structured Abstract: Clinical Utility of Cancer Family History Collection in Primary Care Objectives: This systematic review aimed to evaluate, within unselected populations, the:
Data sources: MEDLINE, EMBASE, CINAHL Cochrane Central, Cochrane Database of Systematic Reviews, and PsycINFO were searched from 1990 to June 2008. Cancer guidelines and recommendations were searched from 2002 forward and systematic reviews from 2003 to June 2008. Review methods: Standard systematic review methodology was employed. Eligibility criteria included English studies evaluating breast, colorectal, ovarian, or prostate cancers. Study designs were restricted to systematic review, experimental and diagnostic types. Populations were limited to those unselected for cancer risk. Interventions were limited to collection of cancer FHx; primary and/or secondary prevention interventions for breast, colorectal, ovarian, and prostate cancers. Results:
|
EPCs into short, easy-to-read guides and tools that can be used by consumers, clinicians, and policy makers (AHRQ, 2010b).
Advice about the best method of presenting the research results for a consumer audience has a substantial body of evidence to support it (Akl et al., in press; Glenton, 2002; Glenton et al., 2006a; Glenton et al., 2006b; Lipkus, 2007; Santesso et al., 2006; Schünemann et al., 2004; Schwartz et al., 2009; Trevena et al., 2006; Wills and Holmes-Rovner, 2003). For example, Glenton (2010) conducted a series of semi-structured interviews with members of the public and
Conclusions: Our review indicates a very limited evidence base with which to address all four of the research questions:
SOURCE: AHRQ (2009b). |
found that summarizing SR results using both qualitative statements and numbers in tables improves consumer comprehension (Glenton, 2010). Other research has found that consumer comprehension is improved if authors use frequencies (e.g., 1 out of 100) rather than percentages or probabilities; use a consistent numeric format to summarize research results; and use absolute risk rather than relative risk (Akl et al., in press; Lipkus, 2007; Wills and Holmes-Rovner, 2003). The recommendation to include a plain-language summary follows guidance from AHRQ and Cochrane (AHRQ, 2010a; Higgins
BOX 5-5 Example of a Plain-Language Summary: Antenatal Corticosteroids for Accelerating Fetal Lung Maturation for Women at Risk of Preterm Birth Corticosteroids given to women in early labor help the babies’ lungs to mature and so reduce the number of babies who die or suffer breathing problems at birth. Babies born very early are at risk of breathing difficulties (respiratory distress syndrome) and other complications at birth. Some babies have developmental delay and some do not survive the initial complications. In animal studies, corticosteroids are shown to help the lungs to mature and so it was suggested these drugs may help babies in preterm labor too. This review of 21 trials shows that a single course of corticosteroid, given to the mother in preterm labor and before the baby is born, helps to develop the baby’s lungs and reduces complications like respiratory distress syndrome. Furthermore, this treatment results in fewer babies dying and fewer common serious neurological and abdominal problems, e.g. cerebroventricular haemorrhage and necrotising enterocolitis, that affect babies born very early. There does not appear to be any negative effects of the corticosteroid on the mother. Long-term outcomes on both baby and mother are also good. SOURCE: Roberts and Dalziel (2006). |
and Green, 2008). Also consistent with the requirement is that PCORI convey the research findings so patients can understand and apply them to their personal circumstances.4
Introduction to the Final Report
The introduction section of an SR final report should describe the research questions as well as the rationale for undertaking the review. The description should address the perspectives of both patients and clinicians, the current state of knowledge, and what the SR aims to add to the body of knowledge. It should indicate whether the review is new or an update of an existing one. If it is an update, the authors should state why the update is needed and describe in general terms how the evidence base has changed since the previous
review (e.g., three new, large randomized controlled trials have been published in the past 2 years).
Methods Section
Detailed reporting of methods is important because it allows the reader to assess the reliability and validity of the review. Table 5-1 lists and describes the topics that should be included in the methods section.
Results Section
The results section should logically lay out the key findings from the SR and include all the topics described in Table 5-2.
Discussion Section
The discussion should include a summary of the main findings; the strength of evidence; a general interpretation of the results for each key question; the strengths and limitations of the study; and gaps in evidence, including future research needs. The discussion should draw conclusions only if they are clearly supported by the evidence (Docherty and Smith, 1999; Higgins and Green, 2008). At the same time, the discussion should provide an interpretation of the data that are useful to users and stakeholders. The peer review process often improves the quality of discussion sections and can provide an evaluation of whether the authors went beyond the evidence in their interpretation of the results (Goodman et al., 1994).
Future research is particularly important for authors to discuss because most SRs identify significant gaps in the body of evidence (Clarke et al., 2007). The ACA language specifies that reports funded by PCORI should “include limitations of the research and what further research may be needed as appropriate.”5 Policy makers and research funders rely on well-written discussions of future research needs to set research agendas and funding priorities. When information gaps are reported clearly, SRs can bring attention to future research needs. Odierna and Bero, for example, used Drug Effectiveness Review Project SRs to identify the need for better drug studies in non-white and economically disadvantaged populations (Odierna and Bero, 2009).
TABLE 5-1 Topics to Include in the Methods Section
Methods Topic |
Include |
Research protocol |
|
Eligibility criteria (for including and excluding studies in the SR) |
|
Analytic framework and key questions |
|
Databases and other information sources |
|
Search strategy |
|
Study selection |
|
Data extraction |
|
Missing information |
|
Information to be extracted |
|
Appraisal of individual studies |
|
Summary measures |
|
Data pooling across studies |
|
Methods Topic |
Include |
Synthesizing the results |
|
Additional analyses |
|
NOTE: PICO(TS) = population, intervention, comparator, outcome, timing, and setting. |
Unfortunately, many SRs are not explicit when recommending future research and not specific enough about recommending types of participants, interventions, or outcomes that need additional examination (Clarke et al., 2007). The EPICOT acronym is a helpful guide for organizing the discussion of future research needs: Evidence, Population, Intervention, Comparison, Outcomes, and Time Stamp (see Table 5-3 for an example recommending that further evidence be collected on the efficacy and adverse effects of intensive blood-pressure lowering in representative populations) (Brown et al., 2006). This tool indicates that recommendations on future research needs should be specific on all of the PICO elements that are required in SR topic formulation (see Chapter 2). The discussion should also report the strength of existing evidence on the topic, using consistent language when discussing different studies (see Chapter 4) and the date of the most recent literature search or recommendation.
Funding and Conflict-of-Interest Section
The final report should describe the sources of funding for the SR; the role of the funder in carrying out the review (including approval of the content); the review authors’, contributing users’, and stakeholders’ biases and COIs; and how any potential conflicts were managed (See Box 5-6 for examples of how to report funding and COI statements).6 The sponsor of an SR can have a significant impact on the SR process and resulting conclusions. SRs funded by industry, for example, are more likely to favor the sponsor’s product
6 |
See Chapter 2 for an overview of COI and bias in the review team, and a discussion of the role of the sponsor in the SR process. |
TABLE 5-2 Topics to Include in the Results Section (repeat for each key question)
Results Topic |
Include |
Study selection |
|
Excluded studies |
|
Appraisal of individual studies |
|
Qualitative synthesis |
|
Results Topic |
Include |
Meta-analysis (if performed) |
|
Additional analyses |
|
Tables and figures |
|
NOTE: GRADE = Grading of Recommendations Assessment, Development and Evaluation. |
than SRs funded through other sources (Lexchin et al., 2003; Yank et al., 2007). Identifying the sources of funding and the role of the sponsor (including whether the sponsor reserved the right to approve the content of the report) in the final report improves the transparency and is critical for the credibility of the report (Liberati et al., 2009).
Currently, many peer-reviewed publications fail to provide complete or consistent information regarding the authors’ biases and COI (Chimonas et al., 2011; McPartland, 2009; Roundtree et al., 2008). A recent study of payments received by physicians from orthopedic device companies identified 41 individuals who each received $1 million or more in 2007. In 2008 and 2009, these individuals published a total of 95 articles relating to orthopedics. Fewer than half the articles disclosed the authors’ relationships with the orthopedic device manufacturers, and an even smaller number provided information on the amount of the physicians’ payments (Chimonas et al., 2011). Requiring authors to disclose any potential outside influences on their judgment, not just industry relationships, improves the
TABLE 5-3 EPICOT Format for Formulating Future Research Recommendations
EPICOT Component |
Issues to Consider |
Example |
E Evidence |
What is the current evidence? |
One systematic review dominated by a large randomized controlled study conducted in hospital setting |
P Population |
Diagnosis, disease stage, comorbidity, risk factor, sex, age, ethnic group, specific inclusion or exclusion criteria, clinical setting |
Primary care patients with confirmed stroke or transient ischemic attack (mean age ≥ 75 years, female–male ratio 1:1, time since last cerebrovascular event ≥ 1 year) |
I Intervention |
Type, frequency, dose, duration, prognostic factor |
Intensive blood pressure lowering |
C Comparison |
Placebo, routine care, alternative treatment/management |
No active treatment or placebo |
O Outcomes |
Which clinical or patient-related outcomes will the researcher need to measure, improve, influence, or accomplish? Which methods of measurement should be used? |
Major vascular events (stroke, myocardial infarction, vascular death); adverse events, risk of discontinuation of treatment because of adverse events |
T Time stamp |
Date of literature search or recommendation |
February 2006 |
SOURCE: Brown et al. (2006). |
transparency and trustworthiness of the review. The ACA contains a similar requirement for authors of research funded by PCORI.7
RECOMMENDED STANDARD FOR REPORT REVIEW
The committee recommends one overarching standard for review by scientific peers, other users and stakeholders, and the public:
BOX 5-6 Reporting Funding and Conflict of Interest: Selected Examples Source of Funding “PRISMA was funded by the Canadian Institutes of Health Research; Universita’ di Modena e Reggio Emilia, Italy; Cancer Research UK; Clinical Evidence BMJ Knowledge; the Cochrane Collaboration; and GlaxoSmithKline, Canada. AL is funded, in part, through grants of the Italian Ministry of University (COFIN–PRIN 2002 prot. 2002061749 and COFIN–PRIN 2006 prot. 2006062298). DGA is funded by Cancer Research UK. DM is funded by a University of Ottawa Research Chair.” Role of Funders “None of the sponsors had any involvement in the planning, execution, or write-up of the PRISMA documents. Additionally, no funder played a role in drafting the manuscript.” Potential Conflicts of Interest “The authors have declared that no competing interests exist.” SOURCE: Moher et al. (2009). |
5.2.2 |
Provide a public comment period for the report and publicly report on disposition of comments |
Rationale
SR final reports should be critically reviewed by peer reviewers to ensure accuracy and clarity and to identify any potential methodological flaws (e.g., overlooked studies, methodological errors). The original protocol for the SR (including any amendments) should be made available to the peer reviewers. A small body of empirical evidence suggests that the peer review process improves the quality of published research by making the manuscripts more readable and improving the comprehensiveness of reporting (Goodman et al., 1994; Jefferson et al., 2002; Weller, 2002). In addition, the critical assessment of manuscripts by peer reviewers is an essential part of the scientific process (ICMJE, 2010). Journals rely on the peer review process to establish when a study is suitable for publication and to improve the quality of reporting and compliance with reporting guidelines (ICMJE, 2010). Some version of peer review is recom-
mended in the guidance from all the major producers of SRs (CRD, 2009; Higgins and Green, 2008; Slutsky et al., 2010). Peer review of research funded through PCORI will be required, either directly through a process established by PCORI or by an appropriate medical journal or other entity.8
The evidence is unclear on how to select peer reviewers, the qualifications that are important for peer reviewers to possess, and what type of training and instructions improve the peer review process (Callaham and Tercier, 2007; Jefferson et al., 2002; Schroter et al., 2004, 2006). In the context of publicly funded SRs, the committee recommends that peer reviewers include a range of relevant users and stakeholders, such as practicing clinicians, statisticians and other methodologists, and consumers. This process can be used to gather input from perspectives that were not represented on the review team (e.g., individuals with diverse clinical specialties).
The committee also recommends that the public be given an opportunity to comment on SR reports as part of the peer review process. Allowing public comments encourages publicly funded research that is responsive to the public’s interests and concerns and is written in language that is understandable and usable for patient and clinical decision making. Requiring a public comment period is also consistent with the ACA, which directs PCORI to obtain public input on research findings,9 as well as guidance from AHRQ and Cochrane (Higgins and Green, 2008; Whitlock et al., 2010).
The review team should be responsive to the feedback provided by the peer reviewers and the public, and publicly report how it revised the SR in response to the comments. The authors should document the major comments and input received; how the final report was or was not modified accordingly; and the rationale for the course of action. The authors’ response to this feedback can be organized into general topic areas of response, rather than responding to each individual comment. Requiring authors to report on the disposition of comments holds the review authors accountable for responding to the peer reviewers’ comments and improves the public’s confidence in the scientific integrity and credibility of the SR (Whitlock et al., 2010).
A neutral third party should manage and oversee the entire peer review process. The main role of the third party should be to provide an independent judgment about the adequacy of the authors’ responses (Helfand and Balshem, 2010). This recommendation is
consistent with the rules governing PCORI that allow, but do not require, the peer review process to be overseen by a medical journal or outside entity.10 It also furthers SR authors’ accountability for responding to reviewers’ feedback and it is consistent with AHRQ guidance (Helfand and Balshem, 2010; Whitlock et al., 2010). The National Academies has an office that manages the review of all Academies studies. A monitor and coordinator, chosen by the report review office from the membership of the Academies, oversee the response to external review. They must approve the response to review before release of the report.
RECOMMENDED STANDARD FOR PUBLISHING THE FINAL REPORT
The committee recommends one standard for publishing the final report:
Standard 5.3—Publish the final report in a manner that ensures free public access
Rationale
The final report should be publicly available. PCORI will be required to post research findings on a website accessible to clinicians, patients, and the general public no later than 90 days after receipt of the research findings and completion of the peer review process.11 This requirement should be extended to all publicly funded SRs of effectiveness research. Publishing final reports is consistent with leading guidance (AHRQ, 2010c; CRD, 2009; Higgins and Green, 2008) and this committee’s criteria of transparency and credibility. Public sponsors should not prevent the SR team from publishing the SR in a peer-reviewed journal and should not interfere with the journal’s peer review process. Ideally, the public sponsor will cooperate with the journal to ensure timely, thorough peer review, so that journal publication and posting on the sponsor’s website can take place simultaneously. In any case, posting an SR final report on a government website should not qualify as a previous publication, in the same way that journals have agreed that publication of an abstract describing clinical trial results in clinicaltrials.gov (which is required by federal law) does not count as prior publication (ICMJE,
2009). In addition, public sponsors should encourage the review team to post the research results in international SR registries, such as the one being developed by the CRD (CRD, 2010b).
REFERENCES
AHRQ (Agency for Healthcare Research and Quality). 2009a. AHRQ Evidence-based Practice Centers partners guide. Gaithersburg, MD: AHRQ.
AHRQ. 2009b. Cancer, clinical utility of family history. http://www.ahrq.gov/clinic/tp/famhist2tp.htm (accessed September 20, 2010).
AHRQ. 2010a. Consumers and patients. http://www.ahrq.gov/consumer/compare.html (accessed September 10, 2010).
AHRQ. 2010b. About the Eisenberg Center. http://www.effectivehealthcare.ahrq.gov/index.cfm/who-is-involved-in-the-effective-health-care-program1/about-the-eisenbergcenter/ (accessed December 27, 2010).
AHRQ. 2010c. Search for guides, reviews, and reports. http://www.effectivehealthcare.ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/ (accessed July 2, 2010).
AHRQ. 2010c. Search for guides, reviews, and reports. http://www.effectivehealthcare.ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/ (accessed July 2, 2010).
Akl, E., A. D. Oxman, J. Herrin, G. E. Vist, I. Terrenato, F. Sperati, C. Costiniuk, D. Blank, and H. Schunemann. In press. Using alternative statistical formats for presenting risks and risk reduction. Cochrane Database of Systematic Reviews.
Barker, A., E. C. Maratos, L. Edmonds, and M. Lim. 2007. Recurrence rates of video-assisted thoracoscopic versus open surgery in the prevention of recurrent pneumothoraces: A systematic review of randomised and non-randomised trials. Lancet 370(9584):329–335.
Begg, C., M. Cho, S. Eastwood, R. Horton, D. Moher, I. Olkin, R. Pitkin, D. Rennie, K. F. Schulz, D. Simel, and D. F. Stroup. 1996. Improving the quality of reporting of randomized controlled trials: The CONSORT statement. JAMA 276(8):637–639.
Brown, P., K. Brunnhuber, K. Chalkidou, I. Chalmers, M. Clarke, M. Fenton, C. Forbes, J. Glanville, N. J. Hicks, J. Moody, S. Twaddle, H. Timimi, and P. Young. 2006. How to formulate research recommendations. BMJ 333(7572):804–806.
Callaham, M. L., and J. Tercier. 2007. The relationship of previous training and experience of journal peer reviewers to subsequent review quality. PLoS Medicine 4(1):e40.
Chimonas, S., Z. Frosch, and D. J. Rothman. 2011. From disclosure to transparency: The use of company payment data. Archives of Internal Medicine.171(1):81–86.
Clarke, L., M. Clarke, and T. Clarke. 2007. How useful are Cochrane reviews in identifying health needs? Journal of Health Services & Research Policy 12(2):101–103.
CRD (Centre for Reviews and Dissemination). 2009. Systematic reviews: CRD’s guidance for undertaking reviews in health care. York, UK: York Publishing Services, Ltd.
CRD. 2010a. Database of Abstracts of Reviews of Effects (DARE). http://www.crd.york.ac.uk/crdweb/Home.aspx?DB=DARE (accessed December 27, 2010).
CRD. 2010b. Register of ongoing systematic reviews. http://www.york.ac.uk/inst/crd/projects/register.htm (accessed June 17, 2010).
Delaney, A., S. Bagshaw, A. Ferland, B. Manns, K. Laupland, and C. Doig. 2005. A systematic evaluation of the quality of meta-analyses in the critical care literature. Critical Care 9(5):R575–R582.
Delaney, A., S. M. Bagshaw, A. Ferland, K. Laupland, B. Manns, and C. Doig. 2007. The quality of reports of critical care meta-analyses in the Cochrane Database of Systematic Reviews: An independent appraisal. Critical Care Medicine 35(2): 589–594.
Devereaux, P. J., P. Choi, S. El-Dika, M. Bhandari, V. M. Montori, H. J. Schünemann, A. X. Garg, J. W. Busse, D. Heels-Ansdell, W. A. Ghali, B. J. Manns, and G. H. Guyatt. 2004. An observational study found that authors of randomized controlled trials frequently use concealment of randomization and blinding, despite the failure to report these methods. Journal of Clinical Epidemiology 57(12):1232–1236.
Docherty, M., and R. Smith. 1999. The case for structuring the discussion of scientific papers. BMJ 318(7193):1224–1225.
EQUATOR Network. 2010. Welcome to the EQUATOR Network website: The resource centre for good reporting of health research studies. http://www.equator-network.org/home/ (accessed June 29, 2010).
Froom, P., and J. Froom. 1993. Deficiencies in structured medical abstracts. Journal of Clinical Epidemiology 46(7):591–594.
Glenton, C. 2002. Developing patient-centered information for back pain patients. Health Expectations 5(4):319–329.
Glenton, C. 2010. Presenting the results of Cochrane systematic reviews to a consumer audience: A qualitative study. Medical Decision Making 30(5):566–577.
Glenton, C., E. Nilsen, and B. Carlsen. 2006a. Lay perceptions of evidence-based information: A qualitative evaluation of a website for back pain sufferers. BMC Health Services Research 6(1):34.
Glenton, C., V. Underland, M. Kho, V. Pennick, and A. D. Oxman. 2006b. Summaries of findings, descriptions of interventions and information about adverse effects would make reviews more informative. Journal of Clinical Epidemiology 59(8):770–778.
Golder, S., Y. Loke, and H. M. McIntosh. 2008. Poor reporting and inadequate searches were apparent in systematic reviews of adverse effects. Journal of Clinical Epidemiology 61(5):440–448.
Goodman, S. N., J. Berlin, S. W. Fletcher, and R. H. Fletcher. 1994. Manuscript quality before and after peer review and editing at Annals of Internal Medicine. Annals of Internal Medicine 121(1):11–21.
Hartley, J. 2000. Clarifying the abstracts of systematic literature reviews. Bulletin of the Medical Library Association 88(4):332–337.
Hartley, J., M. Sydes, and A. Blurton. 1996. Obtaining information accurately and quickly: Are structured abstracts more efficient? Journal of Information Science 22(5):349–356.
Haynes, R. B., C. D. Mulrow, E. J. Huth, D. G. Altman, and M. J. Gardner. 1990. More informative abstracts revisited. Annals of Internal Medicine 113(1):69–76.
Helfand, M., and H. Balshem. 2010. AHRQ Series Paper 2: Principles for developing guidance: AHRQ and the Effective Health-Care Program. Journal of Clinical Epidemiology 63(5):484–490.
Higgins, J. P. T., and S. Green, eds. 2008. Cochrane handbook for systematic reviews of interventions. Chichester, UK: John Wiley & Sons.
ICMJE (International Committee of Medical Journal Editors). 2009. Obligation to register clinical trials. http://www.icmje.org/publishing_10register.html (accessed September 9, 2010).
ICMJE. 2010. Uniform requirements for manuscripts submitted to biomedical journals: Ethical considerations in the conduct and reporting of research: Peer review. http://www.icmje.org/ethical_3peer.html (accessed July 12, 2010).
Ioannidis, J. P., J. W. Evans, P. C. Gøtzsche, R. T. O’Neill, D. Altman, K. Schulz, and D. Moher. 2004. Better reporting of harms in randomized trials: An extension of the CONSORT Statement. Annals of Internal Medicine 141(10):781–788.
IOM (Institute of Medicine). 2008. Knowing what works in health care: A roadmap for the nation. Edited by J. Eden, B. Wheatley, B. McNeil, and H. Sox. Washington, DC: The National Academies Press.
IOM. 2009. Initial national priorities for comparative effectiveness research. Washington, DC: The National Academies Press.
Jefferson, T., P. Alderson, E. Wager, and F. Davidoff. 2002. Effects of editorial peer review: A systematic review. JAMA 287(21):2784–2786.
Lavis, J., H. Davies, A. D. Oxman, J. L. Denis, K. Golden-Biddle, and E. Ferlie. 2005. Towards systematic reviews that inform health care management and policy-making. Journal of Health Services & Research Policy 10(Suppl 1):35–48.
Lexchin, J., L. A. Bero, B. Djubegovic, and O. Clark. 2003. Pharmaceutical industry sponsorship and research outcome and quality: Systematic review. BMJ 326(7400):1167–1170.
Liberati, A., D. G. Altman, J. Tetzlaff, C. Mulrow, P. Gotzsche, J. P. Ioannidis, M. Clarke, P. J. Devereaux, J. Kleijnen, and D. Moher. 2009. The PRISMA Statement for reporting systematic reviews and meta-analysis of studies that evaluate health care interventions: Explanation and elaboration. Annals of Internal Medicine 151(4):W11–W30.
Lipkus, I. M. 2007. Numeric, verbal, and visual formats of conveying health risks: Suggested best practices and future recommendations. Medical Decision Making 27(5):696–713.
McAlister, F. A., H. D. Clark, C. van Walraven, S. E. Straus, F. Lawson, D. Moher, and C. D. Mulrow. 1999. The medical review article revisited: Has the science improved? Annals of Internal Medicine 131(12):947–951.
McPartland, J. M. 2009. Obesity, the endocannabinoid system, and bias arising from pharmaceutical sponsorship. PLoS One 4(3):e5092.
Moher, D., D. J. Cook, S. Eastwood, I. Olkin, D. Rennie, and D. F. Stroup. 1999. Improving the quality of reports of meta-analyses of randomised controlled trials: The QUOROM statement. Lancet 354(9193):1896–1900.
Moher, D., A. Jones, and L. Lepage. 2001a. Use of the CONSORT statement and quality of reports of randomized trials: A comparative before-and-after evaluation. JAMA 285:1992–1995.
Moher, D., K. F. Schulz, D. Altman, and the CONSORT Group. 2001b. The CONSORT statement: Revised recommendations for improving the quality of reports of parallel-group randomized trials. JAMA 285(15):1987–1991.
Moher, D., J. Tetzlaff, A. C. Tricco, M. Sampson, and D. G. Altman. 2007. Epidemiology and reporting characteristics of systematic reviews. PLoS Medicine 4(3):447–455.
Moher, D., A. Liberati, J. Tetzlaff, and D. G. Altman. 2009. Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. PLoS Medicine 6(7):1–6.
Mrkobrada, M., H. Thiessen-Philbrook, R. B. Haynes, A. V. Iansavichus, F. Rehman, and A. X. Garg. 2008. Need for quality improvement in renal systematic reviews. Clinical Journal of the American Society of Nephrology 3(4):1102–1114.
Mulrow, C. D. 1987. The medical review article: State of the science. Annals of Internal Medicine 106(3):485–488.
Mulrow, C. D., S. B. Thacker, and J. A. Pugh. 1988. A proposal for more informative abstracts of review articles. Annals of Internal Medicine 108(4):613–615.
Odierna, D. H., and L. A. Bero. 2009. Systematic reviews reveal unrepresentative evidence for the development of drug formularies for poor and nonwhite populations. Journal of Clinical Epidemiology 62(12):1268–1278.
Oxman, A. D., H. Schunemann, and A. Fretheim. 2006. Improving the use of research evidence in guideline development: Synthesis and presentation of evidence. Health Research Policy and Systems 4:20.
Pocock, S., M. D. Hughes, and R. J. Lee. 1987. Statistical problems in the reporting of clinical trials. A survey of three medical journals. New England Journal of Medicine 317(7):426–432.
PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses). 2010. PRISMA endorsers. http://www.prisma-statement.org/endorsers.htm (accessed September 13, 2010).
Roberts, D., and S. Dalziel. 2006. Antenatal corticosteroids for accelerating fetal lung maturation for women at risk of preterm birth. In Cochrane Database of Systematic Reviews. http://www.cochrane.org/cochrane-reviews/sample-review#PLS (accessed October 6, 2010).
Roundtree, A. K., M. A. Kallen, M. A. Lopez-Olivo, B. Kimmel, B. Skidmore, Z. Ortiz, V. Cox, and M. E. Suarez-Almazor. 2008. Poor reporting of search strategy and conflict of interest in over 250 narrative and systematic reviews of two biologic agents in arthritis: A systematic review. Journal of Clinical Epidemiology 62(2):128–137
Sacks, H. S., D. Reitman, V. A. Ancona-Berk, and T. C. Chalmers. 1987. Meta–analysis of randomized controlled trials. New England Journal of Medicine 316(8):450–455.
Santesso, N., L. Maxwell, P. S. Tugwell, G. A. Wells, A. M. Connor, M. Judd, and R. Buchbinder. 2006. Knowledge transfer to clinicians and consumers by the Cochrane Musculoskeletal Group. The Journal of Rheumatology 33(11):2312–2318.
Schroter, S., N. Black, S. Evans, J. Carpenter, F. Godlee, and R. Smith. 2004. Effects of training on quality of peer review: Randomised controlled trial. BMJ 328(7441):657–658.
Schroter, S., L. Tite, A. Hutchings, and N. Black. 2006. Differences in review quality and recommendations for publication between peer reviewers suggested by authors or by editors. JAMA 295(3):314–317.
Schünemann, H. J., E. Ståhl, P. Austin, E. Akl, D. Armstrong, and G. H. Guyatt. 2004. A comparison of narrative and table formats for presenting hypothetical health states to patients with gastrointestinal or pulmonary disease. Medical Decision Making 24(1):53–60.
Schwartz, L. M., S. Woloshin, and H. G. Welch. 2009. Using a drug facts box to communicate drug benefits and harms. Annals of Internal Medicine 150(8):516–527.
Shea, B., L. M. Bouter, J. Peterson, M. Boers, N. Andersson, Z. Ortiz, T. Ramsay, A. Bai, V. K. Shukla, and J. M. Grimshaw. 2007. External validation of a measurement tool to assess systematic reviews (AMSTAR). PLoS ONE 2(12):e1350.
Slutsky, J., D. Atkins, S. Chang, and B. A. Collins Sharp. 2010. AHRQ Series Paper 1: Comparing medical interventions: AHRQ and the Effective Health-Care Program. Journal of Clinical Epidemiology 63(5):481–483.
Stroup, D. F., J. Berlin, S. Morton, I. Olkin, G. D. Williamson, D. Rennie, D. Moher, B. J. Becker, T. A. Sipe, S. B. Thacker, and the Meta-analysis of Observational Studies in Epidemiology Group. 2000. Meta-analysis of observational studies in epidemiology: A proposal for reporting. JAMA 283(15):2008–2012.
Trevena, L. J., A. Barratt, P. Butow, and P. Caldwell. 2006. A systematic review on communicating with patients about evidence. Journal of Evaluation in Clinical Practice 12(1):13–23.
von Elm, E., D. G. Altman, M. Egger, S. J. Pocock, P. C. Gotzsche, J. P. Vandenbroucke, and the Strobe Initiative. 2007. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: Guidelines for reporting observational studies. Annals of Internal Medicine 147(8):573–577.
Weller, A. C. 2002. Editorial peer review: Its strengths and weaknesses. Medford, NJ: American Society for Information Science and Technology. http://books.google.com/books?hl=en&lr=&id=lH3DVnkbTFsC&oi=fnd&pg=PR12&dq=ICMJE+PEER+REVIEW&ots=rpXA8rZOna&sig=nCltaFOsxGxyrVSoGCeWWhRNLIE#v=onepage&q=ICMJE%20PEER%20REVIEW&f=false (accessed July 12, 2010).
Whitlock, E. P., S. A. Lopez, S. Chang, M. Helfand, M. Eder, and N. Floyd. 2010. AHRQ Series Paper 3: Identifying, selecting, and refining topics for comparative effectiveness systematic reviews: AHRQ and the Effective Health-Care Program. Journal of Clinical Epidemiology 63(5):491–501.
Wills, C. E., and M. Holmes-Rovner. 2003. Patient comprehension of information for shared treatment decision making: State of the art and future directions. Patient Education and Counseling 50(3):285–290.
Yank, V., D. Rennie, and L. A. Bero. 2007. Financial ties and concordance between results and conclusions in meta-analyses: Retrospective cohort study. BMJ 335(7631):1202–1205.