| ||||||||||||||||||||||||||||||
|
|
|||||||||||||||||||||||||||||
| Copyright © 2009. National Academy of Sciences. All rights reserved. Terms of Use and Privacy Statement |
Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 81
Knowing what Works in Health Care: A Roadmap for the Nation
4
Systematic Reviews: The Central Link Between Evidence and Clinical Decision Making
If, as is sometimes supposed, science consisted in nothing but the laborious accumulation of facts, it would soon come to a standstill, crushed, as it were, under its own weight. Two processes are thus at work side by side, the reception of new material and the digestion and assimilation of the old…. The work which deserves, but I am afraid does not always receive, the most credit is that in which discovery and explanation go hand in hand, in which not only are new facts presented, but their relation to old ones is pointed out.
J. W. Strutt Lord Rayleigh
Address to the British Association for the Advancement of Science
(Rayleigh, 1884, p. 1)
More than a decade has passed since it was first shown that patients have been harmed by failure to prepare scientifically defensible reviews of existing research evidence. There are now many examples of the dangers of this continuing scientific sloppiness. Organizations and individuals concerned about improving the effectiveness and safety of health care now look to systematic reviews of research—not individual studies—to inform their judgments.
Iain Chalmers
Academia’s Failure to Support Systematic Reviews
(Chalmers, 2005)
Abstract: This chapter provides the committee’s findings and recommendations for conducting systematic evidence reviews under the aegis of a proposed national clinical effectiveness assessment program (“the Program”). The chapter reviews the origins of systematic review methods and describes the fundamental components of systematic reviews and the shortcomings of current efforts. Under the status quo, the quality of the reviews is variable, methods are poorly documented, and findings are often unreliable. The committee recommends that the Program establish evidence-based, methodological standards for systematic reviews, including standard terminology for characterizing the strength of evidence and a standard reporting format for systematic reviews. Once Program stan-
OCR for page 82
Knowing what Works in Health Care: A Roadmap for the Nation
dards are established, the Program should fund only those reviewers who commit to and consistently meet the standards. The committee found that the new science of systematic reviews has made great strides, but more methodological research is needed. Investing in the science of research synthesis will increase the quality and the value of the evidence provided in systematic reviews. It is not clear whether there are sufficient numbers of qualified researchers to conduct high-quality reviews. The capacity of the workforce should be assessed and expanded, if needed.
Systematic reviews are central to scientific inquiry into what is known and not known about what works in health care (Glasziou and Haynes, 2005; Helfand, 2005; Mulrow and Lohr, 2001; Steinberg and Luce, 2005). In 1884, J. W. Strutt Lord Rayleigh, who later won a Nobel prize in physics, observed that the synthesis and explanation of past discoveries are integral to future progress (Rayleigh, 1884). Yet, more than a century later, Antman and colleagues (1992) and Lau and colleagues (1992) clearly demonstrated that this message was still largely ignored, with the potential for great harm to patients. In a series of meta-analyses examining the treatment of myocardial infarction, the researchers concluded that clinicians need better access to syntheses of the results of existing studies to formulate clinical recommendations. Today, systematic reviews of the available evidence remain an often undervalued scientific discipline.
This chapter has three principal objectives: (1) to describe the fundamental components of a systematic review, (2) to present the committee’s recommendations for conducting systematic evidence reviews under the aegis of a proposed national clinical effectiveness assessment program (“the Program”), and (3) to highlight the key challenges in producing high-quality systematic reviews.
BACKGROUND
What Is a Systematic Review?
A systematic review is a scientific investigation that focuses on a specific question and uses explicit, preplanned scientific methods to identify, select, assess, and summarize similar but separate studies (Haynes et al., 2006; West et al., 2002). It may or may not include a quantitative synthesis of the results from separate studies (meta-analysis). A meta-analysis quantitatively combines the results of similar studies in an attempt to allow inference from the sample of studies included to the population of interest. This report uses the term “systematic review” to describe reviews that incorporate meta-analyses as well as reviews that present the study data descriptively rather than inferentially.
OCR for page 83
Knowing what Works in Health Care: A Roadmap for the Nation
Individual studies rarely provide definitive answers to clinical effectiveness questions (Cook et al., 1997). If it is conducted properly, a systematic review should make obvious the gap between what is known about the effectiveness of a particular service and what clinicians and patients want to know (Helfand, 2005). As such, systematic reviews are also critical to the development of an agenda for further primary research because they reveal where the evidence is insufficient and new information is needed (Neumann, 2006). Without systematic reviews, researchers may miss promising leads or pursue questions that have already been answered (Mulrow et al., 1997). In addition, systematic reviews provide an essential bridge between the body of research evidence and the development of clinical guidance.
Key U.S. Producers and Users of Systematic Reviews
This section briefly describes the variety of contexts in which key U.S. organizations produce or use systematic reviews (Table 4-1). The ultimate purposes of systematic reviews vary and include health coverage decisions, practice guidelines, regulatory approval of new pharmaceuticals or medical devices, clinical research or program planning. Within the federal government, the users include the Agency for Healthcare Research and Quality (AHRQ), the Centers for Medicare & Medicaid Services (CMS), the Medicare Evidence Development and Coverage Advisory Committee (MedCAC), the Centers for Disease Control and Prevention (CDC), the U.S. Food and Drug Administration (FDA), the Substance Abuse and Mental Health Administration (SAMHSA), the U.S. Preventive Services Task Force (USPSTF), and the Veterans Health Administration (VHA).
AHRQ plays a lead role in producing systematic reviews through its program of Evidence-based Practice Centers (EPCs) as a part of its Effective Health Care Program. EPCs produce systematic reviews for professional medical societies and several federal agencies, including CMS and the National Institutes of Health (NIH) Consensus Development Conferences, as well as a variety of other public and private requestors, such as the USPSTF and the American Heart Association. The reviews cover a broad range of topics, including the effectiveness and safety of health care interventions, emergency preparedness, research methods, and approaches to improving the quality and delivery of health care.1 The AHRQ Effective Health Care Program produces comparative effectiveness studies on surgical procedures, medical devices, and medical therapies in 10 priority areas (Slutsky, 2007).
The CDC conducts or sponsors systematic effectiveness reviews to evaluate and make recommendations on population-based and public
1
See Table 3-3 in Chapter 3 for a list of recent EPC studies.
OCR for page 84
Knowing what Works in Health Care: A Roadmap for the Nation
TABLE 4-1 Key U.S. Producers and Users of Systematic Reviews
Component
Government Agencies
AHRQ
USPSTF
SAMHSA
FDA
VHA
CMS MedCAC
CDC
Activity
• Produces reviews
• Sponsors or purchases reviews
Principal use
• Development of practice guidelines and recommendations
• Decisions regarding health coverage
• Regulatory approval
NOTE: BCBSA TEC = Blue Cross and Blue Shield Association Technology Evaluation Center.
health interventions and to improve the underlying research methods (CDC, 2007).
The Blue Cross and Blue Shield Association (BCBSA) Technology Evaluation Center (TEC) produces systematic reviews that assess medical technologies for decision makers in its member plans but also provides the results of these reviews to the public for free.2 Many other health plans look to private research organizations, such as the ECRI Institute and Hayes, Inc., that produce systematic evidence assessments available by subscription or for purchase (ECRI, 2006a,b; Hayes, Inc., 2007). Because the reviews are proprietary, they are not free to the public and the subscription fees are considerable. At Hayes, Inc., for example, subscriptions range from $10,000 to $300,000, depending on the size of the subscribing organizations and the types of products licensed.3
The Cochrane Collaboration is an international effort that produces systematic reviews of health interventions; 11 percent (nearly 1,700 individuals) of its active contributors are in the United States (Allen and Clarke, 2007). Cochrane reviews are available by subscription to The Cochrane Library, and abstracts are available for free through PubMed or www.cochrane.org.
2
See http://www.bcbs.com/betterknowledge/tec/.
3
Personal communication, W. S. Hayes, Hayes, Inc., August 29, 2007.
OCR for page 85
Knowing what Works in Health Care: A Roadmap for the Nation
Private Research Firms
Other Entities
ECRI Institute
BCBSA TEC
Hayes, Inc.
Cochrane Collaboration
Health Plans
Specialty Societies
Manufacturers
Professional medical societies often sponsor or conduct evidence reviews as the first step in developing a practice guideline. These include, for example, the American College of Physicians, several cardiology groups (the American College of Cardiology, the American College of Chest Physicians, and the American Heart Association), the American Academy of Neurology, and the American Society of Clinical Oncology.
Origins of Systematic Review Methods
The term “meta-analysis” was first used by social scientists in the 1970s to describe the process of identifying a representative set of studies of a given topic and summarizing their results quantitatively. In a groundbreaking 1976 assessment of treatment for depression, Glass (1976) first used the term “meta-analysis” to describe what is now referred to as systematic review. Textbooks describing the concept and methods of systematic reviews (Cooper and Rosenthal, 1980; Glass et al., 1981; Hedges and Olkin, 1985; Light and Pillemer, 1984; Rosenthal, 1978; Sutton et al., 2000), and research articles exploring issues such as publication bias followed during that and the subsequent decade.
Subsequently, as quantitative syntheses started to include qualitative summaries and medical scientists adopted the methods, a new terminology emerged. Richard Peto and colleagues used the term “overview” for
OCR for page 86
Knowing what Works in Health Care: A Roadmap for the Nation
the new combined approach (Early Breast Cancer Trialists’ Collaborative Group, 1988). Chalmers and Altman (1995) appear to have introduced the term “systematic review” in their book Systematic Reviews. They also suggested that the term “meta-analysis” be restricted to the statistical summary of the results of studies identified as a product of the review process (Chalmers and Altman, 1995). Confusion over terminology persists today, perhaps because the methods grew up in the social sciences and only later were embraced by the medical sciences.
The statistical methods underlying the quantitative aspects of systematic review—i.e., meta-analysis—date to the early 20th century, when statisticians started developing methods for combining the findings from separate but similar studies. In 1904, using new statistical methods, Karl Pearson (1904) combined research on the impact of inoculation against enteric fever on mortality in five communities. In a 1907 study on the prevalence of typhoid, Goldberger (1907) again used quantitative synthesis.
Social scientists were the first to use methods to critically synthesize results to allow statistical inference from a sample a population. As early as 1940, Pratt and colleagues (1940) at Duke University published a critical synthesis of more than 60 years of research on extrasensory perception.
Systematic reviews in the health care arena were comparatively slow to catch on, and the growth in their development and use coincided with the general rise of evidence-based medicine (Guyatt, 1991). The early implementers of systematic reviews were those who conducted clinical trials and who saw the need to summarize data from multiple effectiveness trials, many of them with very small sample sizes (Yusuf et al., 1985). In the 1970s, Iain Chalmers organized the first major collaborative effort to develop a clinical trials evidence base, beginning with the Oxford Database of Perinatal Trials (Chalmers et al., 1986). This subsequently led to two major compilations of systematic reviews of clinical trials, one of pregnancy and childbirth (Chalmers et al., 1989) and one of the newborn period (Sinclair and Bracken, 1992). The growth of bioinformatics, specifically, electronic communication, data storage, and improved indexing and retrieval of publications, allowed this collaborative effort in the perinatal field to expand further. In 1993, the Cochrane Collaboration was formed (Dickersin and Manheimer, 1998) with the aim of synthesizing information from studies of interventions on all health topics.
Up to this time, literature reviews were often used to assess the effectiveness of health care interventions, but empiric research also began to reveal problems in their execution. The methods underlying the reviews were often neither objective nor transparent (Mulrow, 1987; Oxman and Guyatt, 1988); and they did not routinely use scientific methods to identify, assess, and synthesize information. The approach to deciding which literature should be included and which findings should be presented was subjective
OCR for page 87
Knowing what Works in Health Care: A Roadmap for the Nation
and nonsystematic. The reviews may have provided thoughtful, readable discussions of a topic, but the conclusions were generally not credible.
The following sections of the chapter describe the fundamentals of conducting a scientifically rigorous systematic review and then provide the committee’s findings on current efforts.
FUNDAMENTALS OF A SYSTEMATIC REVIEW
Although researchers use a variety of terms to describe the building blocks of a systematic review, the fundamentals are well established (AHRQ EPC Program, 2007; Counsell, 1997; EPC Coordinating Center, 2005; Haynes et al., 2006; Higgins and Green, 2006; Khan and Kleijnen, 2001; Khan et al., 2001a,b; West et al., 2002).4 Five basic steps (listed below) should be followed, and the key decisions that comprise each step of the review should be clearly documented.
Step 1: Formulate the research question.
Step 2: Construct an analytic (or logic) framework.
Step 3: Conduct a comprehensive search for evidence.
Step 4: Critically appraise the evidence.
Step 5: Synthesize the body of evidence.
The following sections briefly describe each of these steps in the process.
Step 1:
Formulate the Research Question
The foundation of a good systematic review is a well-formulated, clearly defined, answerable question. As such, it guides the analytic (or logic) framework for the review, the overall research protocol (i.e., the search for relevant evidence, decisions about which types of evidence should be used, and how best to identify the evidence), and the critical appraisal of the relevant evidence. The objective, in this first step, is to define a precise, unambiguous answerable research question.
Richardson and colleagues (1995) coined the mnemonic PICO (population, intervention, comparison, and outcome of interest) to help ensure that explicit attention is paid to the four key elements of an evidence question.5,6
4
Unless otherwise noted, this section draws from these references.
5
Personal communication, W. S. Richardson, Boonshoft School of Medicine, Wright State University, October 3, 2007.
6
A recent draft version of an AHRQ comparative effectiveness methods manual proposes expanding the PICO format to PICOTS, adding “t” for timing and “s” for settings (AHRQ, 2007a).
OCR for page 88
Knowing what Works in Health Care: A Roadmap for the Nation
Table 4-2 shows examples of how the PICO format can guide the building of a research question.
The characteristics of the study population, such as age, sex, severity of illness, and presence of comorbidities, usually vary among studies and can be important factors in the effect of an intervention. Health care interventions may have numerous outcomes of interest. The research question should be formulated so that it addresses all outcomes—beneficial and adverse—that matter to patients, clinicians, payers, developers of practice guidelines, and others who may be affected (Schünemann et al., 2006). For example, treatments for prostate cancer may affect mortality; but patients are also interested in learning about potential harmful treatment effects, such as urinary incontinence and impotence. Imaging tests for Alzheimer’s disease may lead to the early diagnosis of the condition, but patients and the patients’ caregivers may be particularly interested in whether an early diagnosis improves cognitive outcomes or quality of life.
Many researchers suggest that decision makers be directly involved in formulating the question to ensure that the systematic review is relevant and can inform decision making (Lavis et al., 2005; Schünemann et al., 2006). The questions posed by end users must sometimes be reframed to be answerable by clinical research studies.
TABLE 4-2 PICO Format for Formulating an Evidence Question
PICO Component
Tips for Building Question
Example
Patient population or problem
“How would I describe this group of patients?”
Balance precision with brevity
“In patients with heart failure from dilated cardiomyopathy who are in sinus rhythm …”
Intervention (a cause, prognostic factor, treatment, etc.)
“Which main intervention is of interest?”
Be specific
“… would adding anticoagulation with warfarin to standard heart failure therapy …”
Comparison intervention (if necessary)
“What is the main alternative to be compared with the intervention?”
Be specific
“… when compared with standard therapy alone …”
Outcomes
“What do I hope the intervention will accomplish?” “What could this exposure really affect?”
Be specific
“… lead to lower mortality or morbidity from thromboembolism? Is this enough to be worth the increased risk of bleeding?”
SOURCE: Adapted from the Evidence-based Practice Center Partner’s Guide (EPC Coordinating Center, 2005).
OCR for page 89
Knowing what Works in Health Care: A Roadmap for the Nation
Step 2:
Construct an Analytic Framework
Once the research question is established, it should be articulated in an analytic framework that clearly lays out the chain of logic underlying the case for the health intervention of interest. The complexity of the analysis will vary depending on the number of linkages between the intervention and the outcomes of interest. For preventive services, there may be multiple steps between, for example, screening for a disease and reductions in morbidity and mortality. Figure 4-1 shows the generic analytic framework
FIGURE 4-1 Analytic framework used by the U.S. Preventive Services Task Force.
NOTE: Generic analytic framework for screening topics. Numbers refer to key questions as follow: (1) Is there direct evidence that screening reduces morbidity and/or mortality? (2) What is the prevalence of disease in the target groups? Can a high-risk group be reliably identified? (3) Can the screening test accurately detect the target condition? (a) What are the sensitivity and specificity of the test? (b) Is there significant variation between examiners in how the test is performed? (c) In actual screening programs, how much earlier are patients identified and treated? (4) Does treatment reduce the incidence of the intermediate outcome? (a) Does treatment work under ideal, clinical trial conditions? (b) How do the efficacy and effectiveness of treatments compare in community settings? (5) Does treatment improve health outcomes for people diagnosed clinically? (a) How similar are people diagnosed clinically to those diagnosed by screening? (b) Are there reasons to expect people diagnosed by screening to have even better health outcomes than those diagnosed clinically? (6) Is there intermediate outcome reliability associated with reduced morbidity and/or mortality? (7) Does screening result in adverse effects? (a) Is the test acceptable to patients? (b) What are the potential harms, and how often do they occur? (8) Does treatment result in adverse effects?
SOURCE: Reprinted from the American Journal of Preventive Medicine, 20(3) Harris, R. P., M. Helfand, S. H. Woolf, K. N. Lohr, C. D. Mulrow, S. M. Teutsch, and D. Atkins, Current methods of the US Preventive Services Task Force: A review of the process, 21-35, Copyright 2007, with permission from Elsevier.
OCR for page 90
Knowing what Works in Health Care: A Roadmap for the Nation
that the USPSTF uses to assess screening interventions. It makes explicit the population at risk (left side of the figure), preventive services, diagnostic or therapeutic interventions, and intermediate and health outcomes to be considered (Harris et al., 2001). It also illustrates the chain of logic that the evidence must support to link the service to potential health outcomes: the arrows (linkages), labeled with a service or treatment, represent the questions that the evidence must answer; dotted lines represent associations; and rectangles represent the intermediate outcomes (rounded corners) or the health states (square corners) by which those linkages are measured.
The overarching linkage (Arrow 1) above the primary framework represents evidence that directly links screening to changes in health outcomes. For example, a randomized controlled trial (RCT) of screening for Chlamydia established a direct, causal connection between screening and reductions in the incidence of pelvic inflammatory disease (Meyers et al., 2007; Scholes et al., 1996). That is, a single body of evidence established the connection between the preventive service (screening) and the health outcome (reduced morbidity).
When direct evidence is lacking or is of insufficient quality to be convincing, the USPSTF relies on a chain of linkages to assess the likely effectiveness of a service. These linkages correspond to key questions about the screening test accuracy (Arrow 3), the efficacy of treatment (Arrows 4 and 5 for intermediate and health outcomes, respectively), and the association between intermediate measures and health outcomes (Dotted Line 6). A similar analytic framework can be constructed for questions of drug treatment, devices, behavior change, procedures, health care delivery, or any type of health intervention used in a population or in individuals.
Deciding Which Evidence to Use: Study Selection Criteria
What constitutes evidence that a health care service is highly effective? As noted in Chapter 1, scientists view evidence as knowledge that is explicit, systematic, and replicable. However, patients, clinicians, payers, and other decision makers have different perspectives on what constitutes evidence of effectiveness. For example, some may view the scientific evidence as demonstrating what works under ideal circumstances but not necessarily under a particular set of real world circumstances. A variety of factors can affect the applicability of a particular RCT to individual clinical decisions or circumstances, including patient factors, such as comorbidities, underlying risk, adherence to therapies, disease stage and severity, health insurance coverage, and demographics; intervention factors, such as care setting, level of training, timing and quality of the intervention, and an array of other factors (Atkins, 2007).
OCR for page 91
Knowing what Works in Health Care: A Roadmap for the Nation
The choice of study designs to be included in a systematic review should be based on the type of research question being asked and should have the goal of minimizing bias (Glasziou et al., 2004; Oxman et al., 2006). Table 4-3 provides examples of research questions and the types of evidence that are the most appropriate for addressing them. RCTs can answer questions about the efficacy of screening, preventive, and therapeutic interventions. Although RCTs can best answer questions about the potential harms from interventions, observational study designs, such as cohort studies, case series, or case control studies, may be all that are available or possible for the evaluation of rare or long-term outcomes.7 In fact, because harms from interventions are often rare or occur far in the future, a systematic review of observational research may be the best approach to identifying reliable evidence on potential rare harms (or benefits).
Observational studies are generally the most appropriate for answering questions related to prognosis, diagnostic accuracy, incidence, prevalence, and etiology (Chou and Helfand, 2005; Tatsioni et al., 2005). Cohort studies and case series are useful for examining long-term outcomes because RCTs may not monitor patients beyond the primary outcome of interest or for rare outcomes because they generally have small numbers of participants. Case series are often used, for example, to identify the potential long-term harms of new types of radiotherapy. Similarly, the best evidence on potential harms related to oral contraceptive use (e.g., an increased risk of thromboembolism) may be from nonrandomized cohort studies or case-control studies (Glasziou et al., 2004).
Many systematic reviews use a best evidence approach that allows the use of broader inclusion criteria when higher-quality evidence is lacking (Atkins et al., 2005). In these cases, the systematic reviews consider observational studies because, at a minimum, noting the available evidence helps to delineate what is known and what is not known about the effectiveness of the intervention in question. By highlighting the gaps in knowledge, the review establishes the need for better quality evidence and helps to prioritize research topics.
For intervention effectiveness questions for which RCTs form the highest level of evidence, it is essential to fully document the rationale for including nonrandomized evidence in a review. Current practice does not meet this standard, however. Researchers have found, for example, that 30 of 49 EPC reports that included observational studies did not disclose the rationale for doing so (Norris and Atkins, 2005).
7
See Chapter 1 for the definitions of the types of experimental and observational studies.
OCR for page 110
Knowing what Works in Health Care: A Roadmap for the Nation
BOX 4-5
Unresolved Methodological Issues in Conducting Systematic Reviews
Locating and Selecting Studies
How best to identify all relevant published studies
Whether to include and how best to identify non-English-language studies
Whether to include and how best to identify unpublished studies and studies in the gray literature (e.g., abstracts)
Search strategies for identifying observational studies in MEDLINE, EMBASE, and other databases
Search strategies for identifying studies of diagnostic accuracy in MEDLINE, EMBASE, and other databases
Assessing Study Quality
Understanding the sources of reporting deficiencies in studies being synthesized
Understanding and identifying potential biases and conflicts of interest
Quality thresholds for study inclusion and the management of individual study quality in the context of a review
Collecting Data
Identifying and selecting information to assess treatment harms
Obtaining important unpublished data from relevant studies
Methods used for data abstraction
Analyzing and Presenting Results
Use of qualitative data in systematic reviews
Use of economic data in systematic reviews
Methods for combining results of diagnostic test accuracy
Statistical Methods (e.g., statistical heterogeneity, fixed versus random effects, and meta-regression)
Inclusion of interstudy variability into displays of results
How best to display findings and their reliability for users
Methods and validity of indirect comparisons
Interpreting Results
Understanding why reviews on similar topics may yield different results
Updating systematic reviews
Frequency of updates
SOURCE: Cochrane Collaboration (2007); Higgins and Green (2006).
and provide excitement about the potential to contribute to health research and to health care practice overall. Moreover, the academic community must recognize the scientific scholarship that is required to conduct high-quality systematic reviews.
OCR for page 111
Knowing what Works in Health Care: A Roadmap for the Nation
OTHER PROGRAM CHALLENGES
Keeping Reviews Up-to-Date
Systematic reviews are not only difficult and time consuming, they also must be kept up-to-date to ensure patient safety. Having an organization that exercises oversight on the production of systematic reviews, for example, the Cochrane Collaboration or professional societies that produce clinical practice guidelines, provides an infrastructure and chain of responsibility for the updating of reviews. There has been little research on updating, and the research that does exist indicates that not all organizations have mechanisms for systematically updating their reviews.
In 2001, Shekelle and colleagues (2001) examined how quickly the AHRQ guidelines went out of date. At the time of that study, they classified only 3 of the 17 guidelines in circulation at that time as still valid. About half of the guidelines were out of date in 5.8 years from the time of their release, and at 3.6 years, at least 10 percent were out of date. A more recent report examining a sample of 100 high-quality systematic reviews of interventions found that within 5.5 years, half of the reviews had new evidence that would substantively change the conclusions about the effectiveness of interventions, and within 2 years almost 25 percent had such evidence (Shojania et al., 2007). The frequency of updating was associated with the clinical topic area and the initial heterogeneity of the results.
Thus, it appears that the failure to update systematic reviews and guidelines within a few years could easily result in patient care that is not evidence based and, worse, care that is not as effective as possible or potentially dangerous.
New and Emerging Technologies
Although this chapter has focused on comprehensive, systematic reviews, the committee recognizes that some decision makers have a legitimate need for objective advisories on new and emerging technologies in order to respond to coverage requests when few, if any, high-quality studies or systematic reviews exist. In addition, patients and providers want information on new health care services as soon as the services become known, often because manufacturers are pressing them to adopt a product or because patients have read direct-to-consumer advertising and want answers from their physicians and other health care providers.
Private technology assessment organizations, such as the ECRI Institute and Hayes, Inc., have responded to the market demand for early reviews of new technologies (ECRI, 2006b; Hayes, Inc., 2007). These firms and other private, proprietary organizations offer clients brief reviews based on
OCR for page 112
Knowing what Works in Health Care: A Roadmap for the Nation
readily available sources of information. Two examples are provided in Appendix E (as proprietary products, they are not in the public domain). The reviews aggregate what little is known from searches of electronic databases (e.g., MEDLINE, EMBASE, or the Cochrane Central Register of Controlled Trials) and published conference abstracts. Other easily obtained information, such as reports from FDA advisory committee meetings, may also be included. Typically, the reviews include a brief description of an intervention; its relevance to clinical care; a short, preliminary list of the relevant research citations that have been identified; two- to three-paragraph summaries of selected research abstracts; and details on the methods used to search the literature.
The Program should consider producing brief advisories on new and emerging technologies in addition to full systematic reviews. If so, like the ECRI Institute and Hayes, Inc., products, the advisories produced under the aegis of the Program should clearly emphasize and highlight the limitations of the information. The advisories clearly state their limitations, so that no one will misinterpret them as an adequate substitute for substantive assessments of evidence on effectiveness.
REFERENCES
AHRQ (Agency for Healthcare Research and Quality). 2006. The guide to clinical preventive services 2006: Recommendations of the U.S. Preventive Services Task Force. AHRQ. Pub. No. 06-0588. Rockville, MD: AHRQ.
———. 2007a. Guide for conducting comparative effectiveness reviews (Draft for public comment) http://effectivehealthcare.ahrq.gov/getInvolved/commentFormMethodsGuide.cfm?DocID=1 (accessed October 10, 2007).
———. 2007b. User’s guide to registries evaluating patient outcomes: Summary. AHRQ Pub. No. 07-EHC001-2. Rockville, MD: AHRQ.
AHRQ EPC Program (Evidence-based Practice Center Program). 2007. Template for submissions of topics for AHRQ evidence reports or technology assessments http://www.ahrq.gov/clinic/epcpartner/epcesubtempl.doc (accessed January 17, 2007).
Allen, C., and M. Clarke. 2007 (unpublished). International activity in Cochrane Review Groups with a focus on the USA. Cochrane Collaboration.
Als-Nielsen, B., W. Chen, C. Gluud, and L. L. Kjaergard. 2003. Association of funding and conclusions in randomized drug trials: A reflection of treatment effect or adverse events? JAMA 290(7):921-928.
Annals of Internal Medicine. 2007. Information for authors http://www.annals.org/shared/author_info.html (accessed July 11, 2007).
Antman, E. M., J. Lau, B. Kupelnick, F. Mosteller, and T. C. Chalmers. 1992. A comparison of meta-analyses of randomized control trials and recommendations of clinical experts. Treatments for myocardial infarction. JAMA 268(2):240-248.
Archives of General Psychiatry. 2007. Instructions for authors http://archpsyc.ama-assn.org/misc/ifora.dtl (accessed July 12, 2007).
Atkins, D. 2007. Creating and synthesizing evidence with decision makers in mind: Integrating evidence from clinical trials and other study designs. Medical Care 45(10 Suppl 2): S16-S22.
OCR for page 113
Knowing what Works in Health Care: A Roadmap for the Nation
Atkins, D., K. Fink, and J. Slutsky. 2005. Better information for better health care: The Evidence-based Practice Center program and the Agency for Healthcare Research and Quality. Annals of Internal Medicine 142(12 Part 2):1035-1041.
Barton, M. 2007. Using systematic reviews to develop clinical recommendations (Submitted responses to the IOM HECS committee meeting, January 25, 2007). Washington, DC.
BCBSA (Blue Cross and Blue Shield Association). 2007. Blue Cross and Blue Shield Association proposes payer-funded institute to evaluate what medical treatments work best http://www.bcbs.com/news/bcbsa/blue-cross-and-blue-shield-association-proposes-payer-funded-institute.html (accessed May 2007).
Bekelman, J. E., Y. Li, and C. P. Gross. 2003. Scope and impact of financial conflicts of interest in biomedical research: A systematic review. JAMA 289(4):454-465.
Bhandari, M., F. Morrow, A. V. Kulkarni, and P. Tornetta. 2001. Meta-analyses in orthopaedic surgery: A systematic review of their methodologies. Journal of Bone and Joint Surgery 83A:15-24.
Bossuyt, P. M., J. B. Reitsma, D. E. Bruns, C. A. Gatsonis, P. P. Glasziou, L. M. Irwig, D. Moher, D. Rennie, H. C. de Vet, and J. G. Lijmer. 2003. The STARD statement for reporting studies of diagnostic accuracy: Explanation and elaboration. Clinical Chemistry 49:7-18.
British Medical Journal. 2007. Resources for authors: Article requirements http://resources.bmj.com/bmj/authors/article-submission/article-requirements (accessed July 11, 2007).
Buscemi, N., L. Hartling, B. Vandermeer, L. Tjosvold, and T. P. Klassen. 2006. Single data extraction generated more errors than double data extraction in systematic reviews. Journal of Clinical Epidemiology 59(7):697-703.
CANCER. 2007. CANCER instructions for authors http://www.interscience.wiley.com/cancer/ (accessed July 11, 2007).
CDC (Centers for Disease Control and Prevention). 2007. Community preventive services: Methods: Effectiveness evaluation http://www.thecommunityguide.org/methods/ (accessed October 3, 2007).
Chalmers, I. 2003. Trying to do more good than harm in policy and practice: The role of rigorous, transparent, up-to-date evaluations. Annals of the American Academy of Political and Social Science 589(1):22-40.
———. 2005. Academia’s failure to support systematic reviews. Lancet 365(9458):469.
Chalmers, I., and D. G. Altman. 1995. Systematic reviews. London: BMJ Publications.
Chalmers, I., J. Hetherington, and M. Newdick. 1986. The Oxford database of perinatal trials: Developing a register of published reports of controlled trials. Controlled Clinical Trials 7(4):306-324.
Chalmers, I., M. Enkin, and M. Keirse. 1989. Effective care in pregnancy and childbirth. Oxford, UK: Oxford University Press.
Chan, A. W., A. Hrobjartsson, M. T. Haahr, P. C. Gøtzsche, and D. G. Altman. 2004. Emperical evidence for selective reporting of outcomes in randomized trials: Comparison of protocols to published articles. JAMA 291:2457-2465.
CHEST. 2006. CHEST instructions to authors and statement of CHEST policies http://www.chestjournal.org/misc/PolicyInstruxA.pdf (accessed July 11, 2007).
Chou, R., and M. Helfand. 2005. Challenges in systematic reviews that assess treatment harms. Annals of Internal Medicine 142(12 Part 2):1090-1099.
Circulation. 2007. Instructions for authors http://circ.ahajournals.org/misc/ifora.shtml (accessed July 12, 2007).
Claxton, K., J. T. Cohen, and P. J. Neumann. 2005. When is evidence sufficient? Health Affairs 24(1):93-101.
Cochrane Collaboration. 2007. Methods groups newsletter Vol. 11. Oxford, UK.
OCR for page 114
Knowing what Works in Health Care: A Roadmap for the Nation
Congressional Budget Office. 2007. Research on the comparative effectiveness of medical treatments: Options for an expanded federal role. Testimony by Director Peter R. Orszag before House Ways and Means Subcommittee on Health http://www.cbo.gov/ftpdocs/82xx/doc8209/Comparative_Testimony.pdf (accessed June 12, 2007).
Cook, D. J., C. D. Mulrow, and R. B. Haynes. 1997. Systematic reviews: Synthesis of best evidence for clinical decisions. Annals of Internal Medicine 126(5):376-380.
Cooper, H. M., and R. Rosenthal. 1980. A comparison of statistical and traditional procedures for summarizing research. Psychological Bulletin 87:442-449.
Counsell, C. 1997. Formulating questions and locating primary studies for inclusion in systematic reviews. Annals of Internal Medicine 127(5):380-387.
Delaney, A., S. M. Bagshaw, A. Ferland, B. Manns, and K. B. Laupland. 2005. A systematic evaluation of the quality of meta-analyses in the critical care literature. Critical Care 9: R575-R582.
Diabetes. 2007. Diabetes instructions for authors http://care.diabetesjournals.org/misc/ifora.shtml (accessed July 30, 2007).
Dickersin, K. 2002. Systematic reviews in epidemiology: Why are we so far behind? International Journal of Epidemiology 31(1):6-12.
———. 2005. Publication bias: Recognizing the problem, understanding its origins and scope, and preventing harm. In Publication bias in meta-analysis: Prevention, assessment, and adjustments. Edited by Rothstein, H., A. Sutton, and M. Borenstein. London, UK: John Wiley and Sons, Ltd.
———. 2007 (unpublished). Steps in evidence-based healthcare. PowerPoint Presentation. Baltimore, MD.
Dickersin, K., and Y.-I. Min. 1993. Publication bias: The problem that won’t go away. In Doing more good than harm: The evaluation of health care interventions. Edited by Warren, K. S., and F. Mosteller. New York: New York Academy of Sciences. Pp. 135-148.
Dickersin, K., and E. Manheimer. 1998. The Cochrane Collaboration: Evaluation of health care and services using systematic reviews of the results of randomized controlled trials. Clinical Obstetrics and Gynecology 41(2):315-331.
Dickersin, K., P. Hewitt, and L. Mutch. 1985. Perusing the literature: Comparison of MEDLINE searching with a perinatal trials database. Controlled Clinical Trials 6(4):306-317.
Dickersin, K., R. Scherer, and C. Lefebvre. 1994. Identifying relevant studies for systematic reviews. BMJ 309(6964):1286-1291.
Dickersin, K., E. Manheimer, S. Wieland, K. A. Robinson, C. Lefebvre, S. McDonald, and the CENTRAL Development Group. 2002. Development of the Cochrane Collaboration’s CENTRAL register of controlled clinical trials. Evaluation and the Health Professions 25:38-64.
Early Breast Cancer Trialists’ Collaborative Group. 1988. Effects of adjuvant tamoxifen and of cytotoxic therapy on mortality in early breast cancer: An overview of 61 randomised trials among 28 896 women. New England Journal of Medicine 319:1681-1692.
Easterbrook, P. J., J. A. Berlin, R. Gopalan, and D. R. Matthews. 1991. Publication bias in clinical research. Lancet 337:867-872.
Ebrahim, S., and M. Clarke. 2007. STROBE: New standards for reporting observational epidemiology, a chance to improve. International Journal of Epidemiology 36(5):945-948.
ECRI. 2006a (unpublished). 2006 ECRI price list. ECRI Health Technology Assessment Information Service.
———. 2006b. About ECRI http://www.ecri.org/About_ECRI/About_ECRI.aspx (accessed January 31, 2007).
Eddy, D. M. 2007. Linking electronic medical records to large-scale simulation models: Can we put rapid learning on turbo? Health Affairs 26(2):w125-w136.
OCR for page 115
Knowing what Works in Health Care: A Roadmap for the Nation
Editors. 2005. Reviews: Making sense of an often tangled skein of evidence. Annals of Internal Medicine 142(12 Part 1):1019-1020.
Egger, M., T. Zellweger-Zähner, M. Schneider, C. Junker, C. Lengeler, and G. Antes. 1997. Language bias in randomised controlled trials published in English and German. Lancet 350(9074):326-329.
Egger, M., G. Davie Smith, and K. O’Rourke. 2001. Rationale, potentials, and promise of systematic reviews. In Systematic Reviews in Health Care: Meta-Analysis in Context. Edited by Egger, M. London, UK: BMJ Publishing Group. Pp. 3-19.
Egger, M., P. Juni, C. Bartlett, F. Holenstein, and J. Sterne. 2003. How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Health Technology Assessment 7(1):76.
EPC Coordinating Center. 2005. Evidence-based practice centers partner’s guide http://www.ahrq.gov/clinic/epcpartner/epcpartner.pdf (accessed January 25, 2007).
Francis, J., and J. B. Perlin. 2006. Improving performance through knowledge translation in the Veterans Health Administration. Journal of Continuing Education in the Health Professions 26(1):63-71.
Garfield, E. 2006. The history and meaning of the journal impact factor. JAMA 295(1): 90-93.
Glass, G. V. 1976. Primary, secondary and meta-analysis. Educational Researcher 5(10):3-8.
Glass, G. V., B. McGaw, and M. L. Smith. 1981. Meta-analysis in social research. Newbury Park, CA: Sage Publications.
Glasziou, P., and B. Haynes. 2005. The paths from research to improved health outcomes. ACP Journal Club 142(2):A8-A10.
Glasziou, P., J. Vandenbroucke, and I. Chalmers. 2004. Assessing the quality of research. BMJ 328(7430):39-41.
Glenny, A. M., M. Esposito, P. Coulthard, and H. V. Worthington. 2003. The assessment of systematic reviews in dentistry. European Journal of Oral Sciences 111:85-92.
Gluud, L. L. 2006 Bias in clinical intervention research. American Journal of Epidemiology 163(6):493-501.
Goldberger, J. 1907. Typhoid bacillus carriers. Edited by Rosenau, M. J., L. L. Lumsden, and J. H. Kastle. Report on the origin and prevalence of typhoid fever in the District of Columbia. Hygienic Laboratory Bulletin No. 35 167-174.
Golder, S., H. M. McIntosh, S. Duffy, and J. Glanville. 2006. Developing efficient search strategies to identify reports of adverse effects in Medline and Embase. Health Information and Libraries Journal 23(1):3-12.
Gøtzsche, P. C., A. Hrobjartsson, K. Maric, and B. Tendal. 2007. Data extraction errors in meta-analyses that use standardized mean differences. JAMA 298(4):430-437.
GRADE Working Group. 2004. Grading quality of evidence and strength of recommendations. BMJ 328(7454):1490.
Guyatt, G. H. 1991. Evidence-based medicine. ACP Journal Club 114:A-16.
Harris, R. P., M. Helfand, S. H. Woolf, K. N. Lohr, C. D. Mulrow, S. M. Teutsch, and D. Atkins. 2001. Current methods of the U.S. Preventive Services Task Force: A review of the process. American Journal of Preventive Medicine 20(3 Suppl):21-35.
Hayden, J. A., P. Cote, and C. Bombardier. 2006. Evaluation of the quality of prognosis studies in systematic reviews. Annals of Internal Medicine 144:427-437.
Hayes, Inc. 2007. Welcome to Hayes http://hayesinc.com (accessed May 8, 2007).
Haynes, R. B., D. L. Sackett, G. H. Guyatt, and P. Tugwell. 2006. Clinical epidemiology: How to do clinical practice research. 3rd ed. Philadelphia, PA: Lipincott Williams & Wilkins.
The Health Industry Forum. 2006. Comparative effectiveness forum: Key themes. Washington, DC: The Health Industry Forum.
OCR for page 116
Knowing what Works in Health Care: A Roadmap for the Nation
Hedges, L. V., and I. Olkin. 1985. Statistical methods for meta-analysis. Orlando, FL: Academic Press.
Helfand, M. 2005. Using evidence reports: Progress and challenges in evidence-based decision making. Health Affairs 24(1):123-127.
Heres, S., J. Davis, K. Maino, E. Jetzinger, W. Kissling, and S. Leucht. 2006. Why olanzapine beats risperidone, risperidone beats quetiapine, and quetiapine beats olanzapine: An exploratory analysis of head-to-head comparison studies of second-generation antipsychotics. American Journal of Psychiatry 163(2):185-194.
Higgins, J. T., and S. Green. 2006. Cochrane handbook for systematic reviews of interventions 4.2.6 [updated September 2006], The Cochrane Library, Issue 4, 2006. Chichester, UK: John Wiley & Sons, Ltd.
Hopewell, S., M. Clarke, C. Lefebvre, and R. Scherer. 2007a. Handsearching versus electronic searching to identify reports of randomized trials. Cochrane Database of Systematic Reviews (2).
Hopewell, S., S. McDonald, M. Clarke, and M. Egger. 2007b. Grey literature in meta-analyses of randomized trials of health care interventions. Cochrane Database of Systematic Reviews (2).
Hypertension. 2007. Instructions to authors http://hyper.ahajournals.org/misc/ifora.shtml (accessed July 30, 2007).
ICMJE (International Committee of Medical Journal Editors). 2006. Uniform requirements for manuscripts submitted to biomedical journals http://www.icmje.org (accessed September 5, 2007).
Ioannidis, J. P., and J. Lau. 2004. Systematic review of medical evidence. Journal of Law and Policy 12(2):509-535.
Ioannidis, J. P., J. W. Evans, P. C. Gøtzsche, R. T. O’Neill, D. Altman, K. Schulz, and D. Moher. 2004. Better reporting of harms in randomized trials: An extension of the CONSORT Statement. Annals of Internal Medicine 141:781-788.
IOM (Institute of Medicine). 2007. Learning what works best: The nation’s need for evidence on comparative effectiveness in health care http://www.iom.edu/ebm-effectiveness (accessed April 2007).
Jadad, A. R., and H. J. McQuay. 1996. Meta-analyses to evaluate analgesic interventions: A systematic qualitative review of their methodology. Journal of Clinical Epidemiology 49:235-243.
Jadad, A. R., M. Moher, G. P. Browman, L. Booker, C. Sigouin, M. Fuentes, and R. Stevens. 2000. Systematic reviews and meta-analyses on treatment of asthma: Critical evaluation. BMJ 320:537-540.
JAMA. 2007. Instructions for authors http://jama.ama-assn.org/misc/ifora.dtl (accessed July 12, 2007).
Jones, A. P., T. Remmington, P. R. Williamson, D. Ashby, and R. L. Smyth. 2005. High prevalence but low impact of data extraction and reporting errors were found in Cochrane systematic reviews. Journal of Clinical Epidemiology 58:741-742.
Jorgensen, A. W., J. Hilden, and P. Gøtzsche. 2006. Cochrane reviews compared with industry supported meta-analyses and other meta-analyses of the same drugs: Systematic review. BMJ Online 333:782-786.
Journal of Clinical Oncology. 2007. Information for contributers http://jco.ascopubs.org/misc/ifora.shtml (accessed July 12, 2007).
Journal of the National Cancer Institute. 2007. Instructions to authors http://www.oxfordjournals.org/our_journals/jnci/for_authors/index.html (accessed July 30, 2007).
Khan, K. S., and J. Kleijnen. 2001. Stage II conducting the review: Phase 4 selection of studies. In CRD Report Number 4. Edited by Khan, K. S., G. ter Riet, H. Glanville, A. J. Sowden, and J. Kleijnen. York, UK: NHS Centre for Reviews and Dissemination, University of York.
OCR for page 117
Knowing what Works in Health Care: A Roadmap for the Nation
Khan, K. S., J. Popay, and J. Kleijnen. 2001a. Stage I planning the review: Phase 2 development of a review protocol. In CRD Report Number 4. Edited by Khan, K. S., G. ter Riet, H. Glanville, A. J. Sowden, and J. Kleijnen. York, UK: NHS Centre for Reviews and Dissemination, University of York.
Khan, K. S., G. ter Riet, J. Popay, J. Nixon, and J. Kleijnen. 2001b. Stage II conducting the review: Phase 5 study quality assessment. In CRD Report Number 4. Edited by Khan, K. S., G. ter Riet, H. Glanville, A. J. Sowden, and J. Kleijnen. York, UK: NHS Centre for Reviews and Dissemination, University of York.
Kunz, R., G. Vist, and A. D. Oxman. 2007. Randomisation to protect against selection bias in healthcare trials. Cochrane Database of Systematic Reviews (2).
Lancet. 2007. Information for authors http://www.thelancet.com/authors/lancet/authorinfo/ (accessed July 30, 2007).
Lau, J., E. M. Antman, J. Jimenez-Silva, B. Kupelnick, F. Mosteller, and T. C. Chalmers. 1992. Cumulative meta-analysis of therapeutic trials for myocardial infarction. New England Journal of Medicine 327(4):248-254.
Lavis, J., H. Davies, A. Oxman, J. Denis, K. Golden-Biddle, and E. Ferlie. 2005. Towards systematic reviews that inform health care management and policy-making. Journal of Health Services Research and Policy 10(Suppl 1):35-48.
Lexchin, J., L. A. Bero, B. Djulbegovic, and O. Clark. 2003. Pharmaceutical industry sponsorship and research outcome and quality: Systematic review. BMJ 326:1167-1170.
Light, R. J., and D. B. Pillemer. 1984. Summing up. Cambridge, MA: Harvard University Press.
Mallen, C., G. Peat, and P. Croft. 2006. Quality assessment of observational studies is not commonplace in systematic reviews. Journal of Clinical Epidemiology 59:765-769.
Medicare Payment Advisory Commission. 2007. Chapter 2: Producing comparative effectiveness information. In Report to the Congress: Promoting greater efficiency in Medicare http://www.medpac.gov/documents/Jun07_EntireReport.pdf (accessed June 2007).
Meyers, D., H. Halvorson, and S. Luckhaupt. 2007. Evidence synthesis number 48. Screening for chlamydial infection: A focused evidence update for the U.S. Preventive Services Task Force. Gaithersburg, MD: Agency for Healthcare Research and Quality.
Moher, D., D. J. Cook, S. Eastwood, I. Olkin, D. Rennie, D. F. Stroup, and the QUOROM Group. 1999. Improving the quality of reports of meta-analyses of randomized controlled trials: The QUOROM statement. Lancet 354:1896-1900.
Moher, D., A. Jones, L. Lepage, and the CONSORT Group. 2001a. Use of the CONSORT Statement and quality of reports of randomized trials. JAMA 285(15):1992-1995.
Moher, D., K. F. Schulz, D. Altman, and the CONSORT Group. 2001b. The CONSORT statement: Revised recommendations for improving the quality of reports of parallel-group randomized trials. JAMA 285(15):1987-1991.
Moher, D., J. Tetzlaff, A. C. Tricco, M. Sampson, and D. G. Altman. 2007. Epidemiology and reporting characteristics of systematic reviews. PLoS Medicine 4(3):447-455.
Montori, V., N. Wilczynski, D. Morgan, and R. B. Haynes, for the Hedges Team. 2003. Systematic reviews: A cross-sectional study of location and citation counts. BMC Medicine 1(1):2.
Mulrow, C. 1987. The medical review article: State of the science. Annals of Internal Medicine 106:485-488.
Mulrow, C., and K. Lohr. 2001. Proof and policy from medical research evidence. Journal of Health Politics, Policy and Law 26(2):249-266.
Mulrow, C. D., D. J. Cook, and F. Davidoff. 1997. Systematic reviews: Critical links in the great chain of evidence. Annals of Internal Medicine 126(5):389-391.
Neumann, P. J. 2006. Emerging lessons from the Drug Effectiveness Review Project. Health Affairs 25(4):w262-w271.
OCR for page 118
Knowing what Works in Health Care: A Roadmap for the Nation
Neumann, P. J., N. Divi, M. T. Beinfeld, B. S. Levine, P. S. Keenan, E. F. Halpern, and G. S. Gazelle. 2005. Medicare’s national coverage decisions, 1999-2003: Quality of evidence and review times. Health Affairs 24(1):243-254.
Neumann, P. J., N. Divi, M. T. Beinfeld, and B. S. Levine. 2007 (unpublished). Medicare National Coverage Decision Database. Tufts-New England Medical Center. Sponsored by the Commonwealth Fund.
New England Journal of Medicine. 2007. Instructions for submitting a NEW manuscript http://authors.nejm.org/Misc/NewMS.asp (accessed July 12, 2007).
NGC (National Guideline Clearinghouse). 2007. Search for cardiology http://www.guideline.gov/search/searchresults.aspx?Type=3&txtSearch=cardiology&num=500 (accessed July 11, 2007).
Norris, S. L., and D. Atkins. 2005. Challenges in using nonrandomized studies in systematic reviews of treatment interventions. Annals of Internal Medicine 142(12 Part 2): 1112-1119.
Obstetrics and Gynecology. 2007. Instructions for authors http://www.greenjournal.org/misc/authors.pdf (accessed July 12, 2007).
Oxman, A. D., and G. H. Guyatt. 1988. Guidelines for reading literature reviews. Canadian Medical Association Journal 138:697-703.
Oxman, A. D., H. J. Schünemann, and A. Fretheim. 2006. Improving the use of research evidence in guideline development: 7. Deciding what evidence to include. Health Research Policy and Systems 4(19).
Pearson, K. 1904. Report on certain enteric fever inoculation statistics. BMJ 3:1243-1246.
Pediatrics. 2007. Instructions for authors http://mc.manuscriptcentral.com/societyimages/pediatrics/2004_author_instructions.pdf (accessed July 12, 2007).
Peppercorn, J., E. Blood, E. Winer, and A. Partridge. 2007. Association between pharmaceutical involvement and outcomes in breast cancer clinical trials. Cancer 109(7):1239-1246.
Perlin, J. B., and J. Kupersmith. 2007. Information technology and the inferential gap. Health Affairs 26(2):w192-w194.
Poolman, R., P. Struijs, R. Krips, I. Sierevelt, K. Lutz, and M. Bhandari. 2006. Does a “Level I Evidence” rating imply high quality of reporting in orthopaedic randomised controlled trials? BMC Medical Research Methodology 6(1):44.
Pratt, J. G., J. B. Rhine, B. M. Smith, C. E. Stuart, and J. A. Greenwood. 1940. Extra-sensory perception after sixty years: A critical appraisal of the research in extra-sensory perception. New York: Henry Holt.
Radiology. 2007. Publication information for authors http://www.rsna.org/publications/rad/pdf/pia.pdf (accessed July 11, 2007).
Rayleigh, L. 1884. Address by the Rt. Hon. Lord Rayleigh. In Report of the fifty-fourth meeting of the British Association for the Advancement of Science. Edited by Murray, J. Montreal.
Reviews in Clinical Gerontology. 2007. Instructions for contributors http://assets.cambridge.org/RCG/RCG_ifc.pdf (accessed July 30, 2007).
Richardson, W. S., M. C. Wilson, J. Nishikawa, and R. S. A. Hayward. 1995. The well-built clinical question: A key to evidence-based decisions [editorial]. ACP Journal Club 123: A12-A13.
Rietveld, R. P., H. C. P. M. van Weert, G. ter Riet, and P. J. E. Bindels. 2003. Diagnostic impact of signs and symptoms in acute infectious conjunctivitis: Systematic literature search. BMJ 327(7418):789.
Rosenthal, R. 1978. Combining results of independent studies. Psychological Bulletin 85: 185-193.
Santaguida, P., M. Helfand, and P. Raina. 2005. Challenges in systematic reviews that evaluate drug efficacy or effectiveness. Annals of Internal Medicine 142(12 Part 2):1066-1072.
OCR for page 119
Knowing what Works in Health Care: A Roadmap for the Nation
Scherer, R. W., P. Langenberg, and E. von Elm. 2007. Full publication of results initially presented in abstracts. Cochrane Database of Systematic Reviews (2).
Scholes, D., A. Stergachis, F. E. Heidrich, H. Andrilla, K. K. Holmes, and W. E. Stamm. 1996. Prevention of pelvic inflammatory disease by screening for cervical chlamydial infection. New England Journal of Medicine 334(21):1362-1366.
Schünemann, H., D. Best, G. Vist, and A. D. Oxman. 2003. Letters, numbers, symbols and words: How to communicate grades of evidence and recommendations. Canadian Medical Association Journal 169(7):677-680.
Schünemann, H., A. Oxman, and A. Fretheim. 2006. Improving the use of research evidence in guideline development: 6. Determining which outcomes are important. Health Research Policy and Systems 4(1):18.
Shea, B., D. Moher, I. Graham, B. A. Pham, and P. Tugwell. 2002. A comparison of the quality of Cochrane reviews and systematic reviews published in paper-based journals. Evaluation and the Health Professions 25:116-129.
Shekelle, P. G., E. Ortiz, S. Rhodes, S. C. Morton, M. P. Eccles, J. M. Grimshaw, and S. H. Woolf. 2001. Validity of the Agency for Healthcare Research and Quality Clinical Practice Guidelines: How quickly do guidelines become outdated? JAMA 286:1461-1467.
Shojania, K. G., M. Sampson, M. T. Ansari, J. Ji, S. Doucette, and D. Moher. 2007. How quickly do systematic reviews go out of date? A survival analysis. Annals of Internal Medicine 147:224-233.
Sinclair, J., and M. Bracken. 1992. Effective care of the newborn infant. New York: Oxford University Press.
Slutsky, J. 2007. Approaches to priority setting: Identifying topics and selection. Submitted Responses to the HECS Committee Meeting, January 25, 2007. Washington, DC.
Song, F., A. J. Eastwood, S. Gilbody, L. Duley, and A. J. Sutton. 2000. Publication and related biases. Health Technology Assessment 4(10).
Spine. 2007. Instructions for authors http://edmgr.ovid.com/spine/accounts/ifauth.htm (accessed July 12, 2007).
Steinberg, E. P., and B. R. Luce. 2005. Evidence based? Caveat emptor! Health Affairs 24(1):80-92.
Stewart, W. F., N. R. Shah, M. J. Selna, R. A. Paulus, and J. M. Walker. 2007. Bridging the inferential gap: The electronic health record and clinical evidence. Health Affairs 26(2): w181-w191.
Stroup, D. F., J. A. Berlin, S. C. Morton, I. Olkin, G. D. Williamson, D. Rennie, D. Moher, B. J. Becker, T. A. Sipe, and S. B. Thacker for the Meta-analysis Of Observational Studies in Epidemiology (MOOSE) Group. 2000. Meta-analysis of observational studies in epidemiology: A proposal for reporting. JAMA 283(15):2008-2012.
Sutton, A. J., K. R. Abrams, D. R. Jones, T. A. Sheldon, and F. Song. 2000. Methods for meta-analysis in medical research. London, UK: John Wiley.
Tatsioni, A., D. A. Zarin, N. Aronson, D. J. Samson, C. R. Flamm, C. Schmid, and J. Lau. 2005. Challenges in systematic reviews of diagnostic technologies. Annals of Internal Medicine 142(12 Part 2):1048-1055.
Treadwell, J. R., S. J. Tregear, J. T. Reston, and C. M. Turkelson. 2006. A system for rating the stability and strength of medical evidence. BMC Medical Research Methodology [electronic resource] 6:52.
Tunis, S. 2006. Improving evidence for health care decisions. Presentation to IOM staff, April 28, 2006. Washington, DC.
USPSTF (U.S. Preventive Services Task Force). 2007. U.S. Preventive Services Task Force ratings http://www.ahrq.gov/clinic/uspstf07/ratingsv2.htm (accessed July 10, 2007).
OCR for page 120
Knowing what Works in Health Care: A Roadmap for the Nation
von Elm, E., D. G. Altman, M. Egger, S. J. Pocock, P. C. Gøtzsche, J. P. Vandenbroucke, and the Strobe Initiative. 2007. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement: Guidelines for reporting observational studies. Annals of Internal Medicine 147(8):573-577.
West, S., V. King, T. Carey, K. Lohr, N. McCoy, S. Sutton, and L. Lux. 2002. Systems to rate the strength of scientific evidence. Evidence Report/Technology Assessment No. 47. (Prepared by the Research Triangle Institute-University of North Carolina Evidence-based Practice Center under Contract No. 290-97-0011.) AHRQ Publication No. 02-E016. Rockville, MD: Agency for Healthcare Research and Quality.
Whiting, P., A. W. Rutjes, J. Dinnes, J. B. Reitsma, P. M. Bossuyt, and J. Kleijnen. 2005. A systematic review finds that diagnostic reviews fail to incorporate quality despite available tools. Journal of Clinical Epidemiology 58:1-12.
Wieland, S., and K. Dickersin. 2005. Selective exposure reporting and Medline indexing limited the search sensitivity for observational studies of the adverse effects of oral contraceptives. Journal of Clinical Epidemiology 58(6):560-567.
Wilczynski, N. L., D. Morgan, R. B. Haynes, and the Hedges Team. 2005. An overview of the design and methods for retrieving high-quality studies for clinical care. BMC Medical Informatics and Decision Making 5(20).
Wilensky, G. R. 2006. Developing a center for comparative effectiveness information. Health Affairs w572.
World Health Organization. 2007. International clinical trials registry platform http://www.who.int/ictrp/en/ (accessed August 9, 2007).
Yusuf, S., R. Peto, J. Lewis, R. Collins, and P. Sleight. 1985. Beta blockade during and after myocardial infarction: An overview of the randomized trials. Progress in Cardiovascular Diseases 27(5):335-371.
Items in cart [1]
