3


The Information Networks Required

INTRODUCTION

The scale of efficiencies that might be gained through developing prioritization and coordinating capacities and improving the methods used for comparative effectiveness research (CER) will be limited by the infrastructure available to support the capture, access, and sharing of relevant data and information. Design and development of robust information networks, and efforts to foster collaboration around common work, will therefore be a critical aspect of creating the infrastructure for expanded CER—necessary for the generation and application of evidence alike, as well as for providing opportunities to support learning from clinical practice. In addition to the federal efforts to increase the adoption and use of electronic health records as described previously, many organizations have developed such capacities, and drawing upon these and other resources through systematic, linked, and coordinated networks would greatly enhance the nation’s fundamental capacity to generate evidence. Papers included in this chapter describe what was known about capacity in 2008, give a rough estimate of the necessary capacity, and offer initial suggestions on policies or activities for progress. These issues are considered in more depth in the Institute of Medicine (IOM) workshop summary publication on The Digital Infrastructure for the Learning Health System: The Foundation for Continuous Improvement in Health and Health Care (IOM, 2011).

Clinical information systems (CISs)—including electronic health records (EHRs)—hold particular promise, given their emerging prominence at the nexus of clinical research, clinical practice, and decision



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 153
3 The Information Networks Required INTRODUCTION The scale of efficiencies that might be gained through developing pri- oritization and coordinating capacities and improving the methods used for comparative effectiveness research (CER) will be limited by the infrastruc- ture available to support the capture, access, and sharing of relevant data and information. Design and development of robust information networks, and efforts to foster collaboration around common work, will therefore be a critical aspect of creating the infrastructure for expanded CER—necessary for the generation and application of evidence alike, as well as for providing opportunities to support learning from clinical practice. In addition to the federal efforts to increase the adoption and use of electronic health records as described previously, many organizations have developed such capacities, and drawing upon these and other resources through systematic, linked, and coordinated networks would greatly enhance the nation’s fundamental capacity to generate evidence. Papers included in this chapter describe what was known about capacity in 2008, give a rough estimate of the necessary capacity, and offer initial suggestions on policies or activities for progress. These issues are considered in more depth in the Institute of Medicine (IOM) workshop summary publication on The Digital Infrastructure for the Learning Health System: The Foundation for Continuous Improvement in Health and Health Care (IOM, 2011). Clinical information systems (CISs)—including electronic health records (EHRs)—hold particular promise, given their emerging promi- nence at the nexus of clinical research, clinical practice, and decision 

OCR for page 153
 LEARNING WHAT WORKS making. Appropriately designed EHRs not only serve as a means for practitioners to access best practices and evidence guidelines, but they also capture a broad array of information important to the diagnosis and treatment of individual patients. To provide policy makers with “order of magnitude” estimates of the spending needed to speed broad adoption of CISs in care delivery organizations throughout the nation, Robert H. Miller from the University of California at San Francisco describes current EHR adoption, future EHR capital and operating expenditure require- ments, and prospects for EHR adoption in the hospital and in physician and clinical services sectors. Work is also needed to develop the technical capacity, methods, stan- dards, and policies for the efficient exchange of information from EHRs and other data sources (e.g., administrative databases, clinical registries) and to disseminate evidence syntheses and other resources to guide practice. Although large databases and clinical registries offer immediate opportuni- ties for learning what works in health care, Carol C. Diamond from the Markle Foundation argues that the greatest promise of health information technology (HIT) lies in its ability to enable quick and efficient learning via a networked and distributed approach to information sharing and evidence development. To maximize this potential, approaches to data and information hubs will need to evolve to address four key challenges: (1) clearly defining the ultimate goal; (2) being open to reset our definitions and assumptions about health data and research approaches; (3) articulat- ing new, broadly accepted working principles based on 21st-century infor- mation paradigms; and (4) developing an information policy framework that broadly addresses public hopes and concerns. Diamond illustrates how these challenges are a jumping-off point for moving to a distributed approach to research—one characterized by connectivity, networks, and feedback loops. Finally, an essential function of any system dedicated to developing a robust evidence base for medical practice is the synthesis of information derived from relevant trials, studies, and insights emerging from clinical practice. As data resources and networks expand, demand will also grow for synthesis work to ensure studies are appropriately reviewed, vetted, and incorporated into the evolving evidence base. Lorne A. Becker from the Cochrane Collaborative provides an overview of current approaches to evi- dence review, synthesis, coordination, and dissemination—internationally and within the United States—and offers some suggestions on key oppor- tunities for expanding capacity to meet the anticipated demand.

OCR for page 153
 THE INFORMATION NETWORKS REQUIRED ELECTRONIC HEALTH RECORDS: NEEDS, STATUS, AND COSTS FOR U.S. HEALTHCARE DELIVERY ORGANIZATIONS Robert H. Miller, Ph.D., Professor of Health Economics in Residence, University of California at San Francisco, Institute for Health & Aging Introduction Implementing ubiquitous evidence-based medicine (EBM) requires robust CISs, especially EHRs. As a result, policy makers want to under- stand what EHR capabilities are needed, to what extent they’ve been imple- mented, likely costs for further adoption and maintenance, and prospects for full implementation of those capabilities.1 Despite keen policy-maker interest, as of mid-2008 few “global” cost estimates for ubiquitous CISs in the healthcare delivery system had been generated (CBO, 2008), in large part because the U.S. healthcare system is so large and diverse, while usable CIS cost and benefit data have been so scarce. In 2006, of the $2.1 trillion in total healthcare expenditures, the $648 billion hospital sector and $447 billion “physician and clinical services” sector were the most intensive users of CISs, incurring the bulk of CIS capi- tal and operating expenses (Catlin et al., 2008). Other, smaller healthcare delivery system sectors that had less intensive CIS use together accounted for another $400 billion or so in expenditures in 2006. Using National Health Expenditure Accounts terminology, these sectors included dental services, “other” professional and personal healthcare services, nursing home care, and home health care. Healthcare sectors least relevant to this analysis accounted for $600 billion in spending; they included retail outlet sales of medical products, administration and government public health, research, construction, and equipment. Because this report is derived from a presentation given in July 2008, it does not include a description or analysis of recent economic developments or of the 2009 economic stimulus legislation on CIS adoption and cost. It does include a handful of updated references for studies that had been in manuscript form in July 2008 and which were subsequently published. Methods For this overview, two main types of data sources were used: (1) data from peer-reviewed articles and non-peer-reviewed reports, and (2) inter- view data from research conducted on behalf of the 2007–2008 California Governor’s Health Information Technology Financing Advisory Commis- 1 Personal health records constitute a separate set of capabilities that consumers could use to view and act on data from various sources, including from the EHRs that hospitals and physician organizations use.

OCR for page 153
 LEARNING WHAT WORKS sion, for which researchers obtained information on CIS adoption, CIS business case and value proposition, and ability to finance CISs from large health systems, rural hospitals, public hospitals, medical groups, indepen- dent practice associations, and community health centers—albeit only for one state, California. For the July 2008 IOM workshop, the author generated rough order- of-magnitude estimates of new or additional spending on CISs needed to implement EBM, over and above current spending on CIS capital projects and operating costs. Such ballpark cost estimates could be useful to policy makers that want to know whether the likely new spending on CISs is closer to (say) $50 billion than it is to $500 billion dollars, whether most healthcare delivery system organizations can afford the new CIS spending, and what public policies are needed to achieve ubiquitous CIS adoption. Given that any CIS cost estimates would be rough, we aimed to create cost estimates that were more likely to err on the high than on the low side—if conservative (worst-case) estimates of CIS costs were “manageable” for delivery system organizations, then, likely CIS costs would be even more manageable. Hospital Sector Clinical Information Systems What’s Needed and What’s Been Adopted Table 3-1 contains a brief description of hospital CIS (EHR) capabilities and adoption, using a stages-of-CIS adoption schema used by the Health Information Management Systems Society (HIMSS) and data that HIMSS obtained for early 2008 (HIMMS Analytics, 2008). The schema shows a hierarchy of CIS adoption, with organizations at a higher stage of adoption typically having capabilities found at lower stages. Most hospitals had new or old (i.e., “legacy”) stage 1 ancillary systems that manage basic informa- tion on radiology orders, laboratory orders, and pharmacy prescriptions. The most basic systems depend on orders that providers first write out by hand, and that are made electronic at some point prior to test/prescription processing. Stage 2 CIS capabilities can pull patient data together from many (often isolated and disparate) information systems into a central data repository that enables managers to generate reports and providers and staff to more easily view more demographic, test result, prescription, and other data. Stage 3 capabilities enable improved data presentation and capture, some checking for errors in prescription and test ordering, as well as digital imaging. As of 2007, CIS capabilities at stage 4 and beyond were still relatively rare—according to HIMSS data, less than 5 percent of hospitals had com-

OCR for page 153
 THE INFORMATION NETWORKS REQUIRED TABLE 3-1 Hospital Electronic Health Record Capabilities and Adoption Estimates Stage Description 2008 Stage 7 Medical record fully electronic; healthcare organization 0.1% able to contribute continuity of care document as by-product of electronic medical record; data warehousing/mining Stage 6 Physician documentation (structured templates), 1.0% full clinical decision support, full Radiology Picture Archiving and Communication System (PACS) Stage 5 Closed loop medication administration 1.3% Stage 4 Computerized physician order entry, clinical decision 1.9% support (clinical protocols) Stage 3 Clinical documentation (flow sheets), clinical decision 32.9% support system (error checking), PACS available outside radiology Stage 2 Clinical data repository, controlled medical vocabulary, 33.2% clinical decision support system inference engine, may have document imaging Stage 1 Ancillaries—lab, radiology, pharmacy 12.5% Stage 0 All three ancillaries not installed 17.1% SOURCE: HIMSS Analytics, 2008. For more information see www.connectingforhealth.org/ resources/CCEndorser.pdf (accessed September 8, 2010). puterized physician order entry (CPOE), where the ordering physician (rather than support staff) did the data entry. CPOE systems are considered more likely to affect decision making than simpler ordering systems, as they can generate patient safety/quality reminders and alerts when physicians enter data at the point of care, rather than when staff enter order data later in the ordering process. At the far end of the spectrum are the least implemented capabilities, such as “closed loop” medication administra- tion (with bar coding), physician documentation, and robust capability for health information exchange. American Hospital Association (AHA) data from 2006 also provided information on the CIS adoption in U.S. hospitals, although they likely overstated CIS adoption because executives and staff in hospitals with advanced CISs were more likely to respond to a survey on CIS adoption than were respondents in hospitals without advanced CISs (AHA, 2007a). The AHA survey findings indicated that 66 percent of hospitals had results

OCR for page 153
 LEARNING WHAT WORKS viewing for lab and radiology, and between 46 percent and 66 percent of hospitals had lab/radiology/pharmacy order entry by staff; however, in only 10 percent of hospitals did more than half of physicians routinely use CPOE capabilities for medications ordering. Similarly, while the survey results also showed some progress in implementing CIS capabilities with built-in EBM rules. For example, 31 percent of hospitals provided real-time drug alerts at the point of order data entry (by staff or providers), 37 percent provided “back-end” (not real-time) drug alerts, but only about 10 percent of hospitals offered providers with suggested clinical guidelines and path- ways for patient care. Order-of-Magnitude Cost Estimates Generating even rough estimates of future CIS costs throughout the U.S. healthcare delivery system is a perilous endeavor due to a lack of high- quality evidence about CIS adoption and cost. Nevertheless, it is possible to show how rough CIS capital and operating costs estimates could be gener- ated and describe the pitfalls of any one estimate. A crude estimate of hospital sector CIS capital costs can be created by multiplying the number of staffed U.S. community hospital beds by the estimated cost per hospital bed of implementing robust CIS capabili- ties, and then subtracting the proportion of the CIS capital cost already incurred. If one was to take a CIS cost per bed estimate of $57,000 found in a 2005 RAND report (which the RAND researchers believed was very rough) and increase it to $100,000 in order to account for inflation and to create a distinctly conservative (high) bias to the cost estimate (Girosi et al., 2005), robust CISs in all U.S. hospitals would cost $90 billion in capital costs, given about 800,000 community hospital beds, and perhaps another estimated 100,000 hospital beds with similar characteristics in federal and state hospitals (AHA, 2007b). Hospitals already have spent some portion of this hypothetical $90 bil- lion in hospital sector CIS capital cost; however, how much they’ve spent is unclear due to limitations in evidence on the CIS adoption, cost, and spending. Suppose that hospital-sector organizations have already incurred 25 percent of the capital cost of robust CISs on average, and so need to incur nearly $70 billion in additional capital expenditures (75 percent of the hypothetical $90 billion). We assume an 8-year time horizon for achieving robust CISs for nearly all hospitals, since implementing CISs can take years, even in a large health system with substantial information systems staffing, while U.S. hospitals and health systems are at varying stages of implement-

OCR for page 153
9 THE INFORMATION NETWORKS REQUIRED ing CISs. In that case, CIS capital spending per year would amount to $8.5 billion on average.2 Hospitals already are spending some portion of the hypothetical $8.5 billion per year on new CIS capabilities—but again, we don’t know how much. For example, while one 2008 survey projected that U.S. hospitals would spend about $10 billion per year on all HIT-related capital projects, it provided no estimates of spending only on CISs or on only new CIS capabilities (rather than on replacements for old capabilities) (HIMMS Analytics, 2008). To show how calculating “new” CIS expenditures might work, sup- pose for example that CISs accounted for half of the $10 billion per year in hospital HIT-related capital projects, and that new CIS capabilities (not just replacement of old capabilities) accounted for 60 percent of that $5 billion per year. In that case, hospitals would already be spending about $3 billion of the $8.5 per year billion for needed CISs, leaving about $5.5 billion per year in additional “new” CIS capital expenditures per year, or about $42 billion over 8 years.3 In one of many simple simulations, and assuming that each dollar of new capital spending creates $0.25 in new operating expenses, hospitals would incur about $48 billion for additional operating costs over the 8 year period, for a total of $90 billion in new CIS expenditures—about $11 billion total in CIS spending per year. Given the generally poor quality of the evidence behind the assump- tions, a much lower (e.g. $50 billion) or much higher (e.g., $130 billion) amount over 8 years is equally plausible for the hospital sector. The impor- tant point is that these amounts can be seen as a rough, order-of-magnitude range for additional hospital CIS spending. Prospects for New Clinical Information System Deployment Is the needed CIS spending over 8 years “feasible” for the U.S. hospital sector? In 2006, $90 billion for new CIS spending would translate into an average 1.7 percent increase in hospital spending per year. Assuming no financial benefit from CIS investment, and given median hospital net margins of around 5 percent in 2007 (MedPAC, 2008), such CIS spend- ing might be feasible for many hospitals only if some major construction projects were delayed. Obviously, such spending would not be feasible for 2 To keep the exposition simple, we do not include spending to replace some of the ad- ditional CIS hardware (a shrinking part of total CIS expenditure) and ignore interest and discount rates for future costs and benefits. 3 In fact, the total “new” CIS costs would be somewhat higher, since any new CIS capital expenditures create new capital replacement (depreciation) costs and new operating costs—for software maintenance, additional information systems, clinical staffing, and so on.

OCR for page 153
0 LEARNING WHAT WORKS financially weaker hospitals or if hospital margins deteriorated due to a difficult economy. Stakeholders care about “net” CIS costs—that is, total CIS costs less cost savings and new revenues. From the hospital’s perspective, the good news is that some CIS-related financial return to the hospital is likely, since hospitals will use EHRs to create some efficiency savings from operations, and generate some revenues from higher reimbursement coding and from new services—benefits that potentially could make a substantial contribu- tion towards paying for new CIS costs. Any government subsidies would contribute an additional amount. Moreover, despite a likely unfavorable measurable financial return on investment (as of mid-2008), many execu- tives and boards—especially in larger hospital systems and larger nonsystem hospitals—appeared to view advanced CISs as a cost of doing business. That is, while a CIS investment might not be justified based on a measur- able return on investment analysis, many health system leaders see it as necessary expense in order to compete successfully in a market place that increasingly will compare organizations by the quality of care provided (Miller et al., 2009a). Physician Practice Clinical Information Systems What’s Needed and What’s Been Adopted Physician offices typically can use two different types of information systems. The best chronic disease management systems (CDMSs) for chronic and preventive care (e.g., for diabetics, asthmatics, women needing cervical cancer screening) use electronic data from billing, scheduling, registration, and lab systems, plus manually entered data, to create paper patient data summaries and reminders for visits, along with lists of patients needing ser- vices and provider performance reporting. In this setting, CDMS software coexists with the paper medical record. Simpler CDMS software imports no electronic data. While such systems are useful, policy attention has focused on ambulatory care EHR software that typically includes a suite of capabili- ties that physicians can use in day-to-day care, including electronic viewing, documenting, prescribing, lab order entry, care reminders, and messaging, as well as the capabilities found in CDMS software (see Table 3-2). Here we focus only on EHRs. Order-of-Magnitude Cost Estimates A rough estimate of overall “new” CIS capital spending on physician EHRs can be created by multiplying the number of active office-based phy-

OCR for page 153
 THE INFORMATION NETWORKS REQUIRED TABLE 3-2 System and Capabilities in Chronic Disease Management Systems and Electronic Health Records System and Capabilities Explanation and/or Benefits Chronic Disease Management Systems (for Best products use electronic data from billing, chronic/preventive care patients) scheduling, registration, and lab systems, plus some manually inputted data; keeps paper chart Patient data summaries (paper) Provides relevant data at the point of care Reminders (paper) On the paper data summaries Lists of patient needing services Permits outreach to patients overdue for tests or visits Provider performance reporting Enables managers and providers to understand Quality Improvement performance Electronic Health Records Replaces paper chart; best ones also replace chronic disease management systems Prescribing Permits drug–drug/allergy interaction alerts; reduces input errors Lab ordering Reduces input errors Documenting Best products have templates for types of patients Messaging with providers Improves provider communication Messaging with patients Improves patient–provider communication; best products enable patients to view data, order prescriptions, make appointments AND Patient data summaries Provides relevant data during visit; enables customizable views Reminders Typically built into documenting and ordering Lists of patient needing services Permits outreach to patients overdue for tests or visits Provider performance reporting Enables managers and providers to understand Quality Improvement performance SOURCE: Derived from Miller et al., 2009b. sicians times the EHR capital cost per physician, less EHR costs already incurred (based on an estimate of physician EHR adoption), and adding new operating and depreciation costs over time. In 2005–2006, between 310,000 and 500,000 physicians practiced in office settings (depending on the data source), while the EHR capital cost per physician was around $40,000.4 If only 10 percent of the higher estimate of office-based physicians had robust EHRs, the total EHR capital cost over 8 years would be around 4 The lower estimate physician cost is based on data from E. Hing and C. Burt, Charac- teristics of Office-Based Physicians and Their Medical Practices: United States, 200-200, (Hyattsville, MD: National Center for Health Statistics, 2008), while the higher estimate is

OCR for page 153
2 LEARNING WHAT WORKS $20 billion, or $2.5 billion per year, plus some hardware replacement cost. Since physicians have already spent funds on EHR capital projects, and are already spending funds on EHR capital expenses, “new” spending on EHRs might amount to about $15 billion over an 8-year period. Using the same approach for office-based physician offices as for hospi- tals, and taking the higher office physician estimate, an order-of-magnitude estimate for new physician EHR spending would amount to roughly $40 billion to 50 billion over 8 years, including new operating expenses—or about a 1 percent to 1.25 percent average increase in physicians-services sector expenditures per year averaged over each of the 8 years. Here, too, CIS cost over 8 years is a feasible expenditure for most physician practices, even in a worst-case scenario of no offsetting savings or increased revenues or subsidies. In fact, some evidence suggests that the financial return to office-based physicians could be substantial (even with- out subsidies), which could greatly reduce the net CIS expenditure figure (Miller et al., 2009a). Prospects for New Clinical Information System Deployment Large medical groups have been implementing EHRs at a good pace (DesRoches et al., 2008), because some groups face a favorable CIS busi- ness case, and because some large groups consider CISs a cost of doing business for reasons similar to the hospital sector. Solo/small groups (i.e., 10 physicians or fewer) were adopting EHRs more slowly, because the EHR business case was not perceived as favorable enough to physician practice owners, EHRs were disruptive and stressful to implement, and adequate technical support was typically hard to find. The pace of CIS implemen- tation for all types of practices should increase to some extent, due to anticipated or actual patient pressure and greater reimbursement rewards for EHR-enabled performance, and (possibly) some subsidies from hospi- tals seeking to bind physicians to their organizations. Obviously, any new government subsidies or new support services could substantially increase EHR adoption. In the physician sector, absent any special CIS subsidies, financially weaker organizations would fall behind in CIS adoption, a spe- cial concern when some of those organizations also serve the disadvantaged and underserved. based on D. Smart, Physician Characteristics and Distribution in the U.S.: 200 Edition, (Chicago, IL: American Medical Association, 2006).

OCR for page 153
 THE INFORMATION NETWORKS REQUIRED Is the Clinical Information System Expense Worthwhile? Again, these estimates of new CIS costs for hospitals and physicians are very rough. They are intended as order-of-magnitude estimates to put the overall potential cost in the perspective of the overall spending in the hospital and physician healthcare sectors. Will benefits justify the substantial cost for hospital and physician EHRs? We know that simply implementing EHR capabilities doesn’t mean they will be used for EBM. Much out-of-the-box EHR software lacks easy-to-use and useful evidence-based templates with reminders and alerts, reports on patients needing services, and reports on provider performance; meanwhile, the information systems staff expertise that hospitals and large groups can tap to compensate for software limitations has been unavail- able to the majority of physicians in solo and small groups. Moreover, without special performance incentives, many healthcare providers won’t use even easy-to-use and useful software since practicing EBM requires dif- ficult changes in workflow and sometimes additional staff. Even if provid- ers did use the software, it could take years for comparative effectiveness researchers to obtain truly comparable data from many different organiza- tions’ practice settings. Obtaining such data requires promulgating precise definitions of measures and methods of obtaining data, but enforcing such standards would be especially difficult to achieve given the wide variation in EHR and billing software, physician documentation and data validation practices, and data from health information exchange—all of which could affect the quality of CER measures. Increasing EHR use and especially EHR use for quality improvement will depend on a series of substantial changes in out-of-the-box EHR soft- ware, government and payer financial incentives, public performance report- ing, EHR support services, and improved health information exchange. While each can contribute to increasing EHR adoption and especially use for quality improvement, appropriate financial (dis-)incentives and public reporting are the most important policy carrots and sticks that can encour- age providers to practice EBM. DATA AND INFORMATION HUB REQUIREMENTS Carol C. Diamond. M.D., Ph.D., Managing Director, Healthcare Program, Markle Foundation Overview The vision set forth by the IOM’s The Learning Healthcare System is compelling, and it has been clearly articulated in that workshop summary (IOM, 2007). In a learning health system,

OCR for page 153
0 LEARNING WHAT WORKS received mentorship. Three reviews have been published in the Cochrane Library, 12 are in progress, and the initiative has now been extended to include authors from South Asia. The primary product of the collaboration is the Cochrane Database of Systematic Reviews (CDSR).10 From a small beginning in 1995 when it contained only 36 full systematic reviews, this database has grown with each issue and now contains almost 4,000 systematic reviews covering the whole range of healthcare interventions. Initially, Cochrane reviews addressed only questions of effectiveness of interventions, using primarily evidence from controlled clinical trials. While this is still largely true, many Cochrane reviews now also include results of studies using other designs, such as controlled before/after and interrupted time series designs, for questions that have not been studied using randomized trials. Because of the international scope of the Cochrane Collaboration, the reviews cover a broad range of topical areas, with applicability for both developing and developed countries. Beginning in 2008, the database was expanded beyond its initial coverage of only systematic reviews of interventions and now includes reviews of diagnostic test accuracy (Leeflang et al., 2008) and reviews that synthesize research on issues relevant to systematic review methodology. The Cochrane approach of producing a coordinated database of focused systematic reviews using an international collaborative process has a number of advantages and has been an effective way to build capacity for evidence synthesis in the countries participating in this effort. In addition to the mechanisms for author development noted above, the Cochrane process has contributed to capacity building by advancing systematic review meth- ods, developing tools to help in the evidence synthesis process, and forging important partnerships with universities and other academic institutions. Working in a Cochrane methods group provides opportunities for methodologists to further develop the methods used in systematic reviews and also provides them with a large set of reviews and protocols to serve as a substrate for their research. Detailed guidance on systematic review production from these groups has been incorporated into two Cochrane Handbooks (Diagnostics Test Accuracy Working Group, 2009; Higgins and Green, 2008). The groups have also developed the Cochrane Methodology Register11 which is continuously updated and now includes more than 11,000 citations to journal articles, book chapters, conference proceedings, conference abstracts, and reports of ongoing methodological research. The aim of the register is to include all published reports of empirical method- 10 Available from http://www.thecochranelibrary.com (accessed June 15, 2009). 11 Available at http://www3.cochrane.org/access_data/cmr/accessDB_cmr.asp (accessed June 16, 2009).

OCR for page 153
 THE INFORMATION NETWORKS REQUIRED ological studies that could be relevant for inclusion in a Cochrane method- ology review, along with comparative and descriptive studies relevant to the conduct of systematic reviews of healthcare interventions. In order to facilitate the production of high-quality systematic reviews by a widely dispersed international group of authors, the collaboration has developed a number of tools to help authors identify studies for inclusion in their reviews, use appropriate and standardized methods in conducting their reviews, and produce their reviews in the format specified for CDSR. The literature identification tool is the Cochrane Central Register of Controlled Trials (CENTRAL). Each Cochrane review group maintains a database of relevant studies that includes references to both published and unpublished reports. These individual study registers are assembled quarterly into CENTRAL, which is then published as part of the Cochrane Library to make it available for broader public use by health researchers and others wishing to perform evidence syntheses. Approximately two-thirds of the references in CENTRAL are derived from specially designed searches of MEDLINE and Excerpta Medica Database (EMBASE). The remainder con- . con- sist of references uncovered by authors or by organized efforts of Cochrane review groups, centers, or fields to find additional studies through activities, such as handsearching of journals or conference proceedings, follow-up of references from other studies, or contact with trialists or others who may have knowledge of additional studies not included in MEDLINE or EMBASE. A recent assessment showed that searching beyond CENTRAL found only a very small number of trials (Royle and Milne, 2003). The collaboration’s authoring tool is a complex piece of software known as RevMan. The software has been continually refined over many years. It is structured to guide authors through the appropriate steps in con- ducting and writing up their review, and links are provided to the relevant section of the Cochrane handbooks at each step. RevMan also incorporates a number of statistical tools, developed in conjunction with the Cochrane methods groups, that allow authors to perform meta-analyses of their data in a standardized way. In addition to serving as essential infrastructure components for the collaboration, these resources have been made freely available and are now in widespread use in the production of evidence syntheses by others. For example, the majority of systematic reviews include a search of CENTRAL as one method of identifying studies, many reference the methods outlined in one of the Cochrane handbooks, and a large number are prepared using RevMan. Many of the aims and activities of the Cochrane Collaboration fit well with the missions of universities and academic institutions. This has led to a number of fruitful collaborations. Most Cochrane editorial groups and many centers and fields are located in university settings or have close ties

OCR for page 153
2 LEARNING WHAT WORKS with specific university departments or other units. Both parties benefit from this sort of collaboration—the Cochrane entity is supported and finds colleagues and collaborators within the university, while the university faculty not directly employed in the Cochrane entity become linked with an international network of individuals with unique methods and content skills and knowledge. The ability of the Cochrane Collaboration to simultaneously increase the available number of systematic reviews and to improve and build the infrastructure required to support production of evidence syntheses of all types has led several countries to build support for the collaboration into their budgets. The majority of this support has been directed toward fund- ing of Cochrane infrastructure (for example, support for Cochrane Review Groups) and has not been tied to the production of systematic reviews on specific topics. The United Kingdom has been the leader in this regard. Cochrane activities in the United Kingdom have been funded continuously since 1992, and the funders continue to feel that this approach of funding the infrastructure to support the production of methodologically sound systematic reviews on topics chosen by review authors is an important component of their approach to supporting evidence synthesis. The United States has provided support to some Cochrane groups as well, using a variety of funding mechanisms. One of the first Cochrane edi- torial groups to be formed, the Neonatal Review Group has had funding from the National Institute of Child Health and Human Development (USA) for the support of its infrastructure since its inception. This has allowed the preparation and continuous updating of a classified bibliography of virtu- ally all reports of randomized trials in the field of neonatology and of sys- tematic reviews (incorporating meta-analysis) of the results of this body of research.12 This group currently has 253 completed reviews with 65 reviews in progress. The HIV/AIDs review group has had CDC/Global AIDS Pro- gram (2008–present) and National Institute of Mental Health (2007–2008) support and has produced 52 reviews, with 51 reviews in progress. The Prostatic Diseases and Urologic Cancers has had Veterans’ Administration (1998–2003) and National Institute of Diabetes and Digestive and Kidney Diseases (2005–present) support and has produced 29 reviews, with 23 in progress. The Cochrane Eyes and Vision Group (CEVG), started in 1997, has 73 completed reviews and 54 reviews in progress. The CEVG U.S. Satel- lite, with support from the National Eye Institute from 2002 to 2009, has produced 18 completed reviews and has 36 additional reviews in progress, to date. One of the most active of the collaboration’s Fields/Networks is 12 The Cochrane Neonatal Group. Available at http://neonatal.cochrane.org (accessed De- cember 14, 2008).

OCR for page 153
 THE INFORMATION NETWORKS REQUIRED the U.S.-based Complementary Medicine Field,13 which is supported by the National Center for Complementary and Alternative Medicine. Unlike Cochrane Review Groups, Fields do not have a direct editorial role in the production of Cochrane reviews but identify health issues of importance to specific populations and/or intervention types and support CRGs in their production of relevant reviews in a variety of ways. One important func- tion of Fields is their contribution of trials to CENTRAL, and the Comple- mentary Medicine Field has been a major contributor in this regard, with a database that includes over 7,000 reports of clinical trials. The field has had a very active role in the identification, translation, and critical appraisal of reports of complementary medicine interventions published in Chinese journals, and also has assembled an organized list of all of the Cochrane reviews addressing complementary medicine interventions. Advantages of a Collaborative International Approach to Evidence Synthesis The example of the Cochrane Collaboration demonstrates the many advantages of an organized international approach to the production of evidence syntheses, and the benefits it brings in terms of prioritization, methods development, and capacity development. This section explores these issues in more detail and also notes other international collaborative approaches, some of which involve more complex evidence syntheses. Prioritization Because of the many disparate stakeholders involved and the large variations in morbidity, resources, and other factors from country to coun- try, it is difficult to conceptualize an international process that would adequately represent all of the relevant perspectives in prioritizing ques- tions in order to decide on a relatively small number of questions to be addressed by complex evidence syntheses. However, as noted above, in the case of focused systematic reviews, prioritization is more about coverage of the entire evidence terrain and coordination of efforts to avoid duplication. Thus a model in which local actors produce focused evidence syntheses, in accord with their own local priorities but also in a coordinated collabora- tive fashion, can result in a comprehensive set of syntheses that addresses multiple disparate priorities and that can serve as building blocks for more specific analyses performed at a local level. 13 Available at http://medschool.umaryland.edu/integrative/cochrane_about.asp (accessed September 8, 2010).

OCR for page 153
 LEARNING WHAT WORKS Methods Development An international approach can be helpful in developing the rigorous methods needed for evidence syntheses of all sorts. The required meth- odological expertise is frequently not available within a single country or region, and many of the issues are sufficiently complex that only a handful of individuals around the globe have a good grasp of them and are at the leading edge in their development. In addition to the work done within the Cochrane Collaboration, international collaborative groups have advanced the science of evidence synthesis in a number of ways. One example is the family of standards for reporting of research stud- ies and of evidence syntheses. The complete set has been assembled by the Enhancing the Quality and Transparency of Health Research Network— an international initiative that seeks to enhance the reliability of medi- cal research literature by promoting transparent and accurate reporting of research studies.14 The Quality of Reporting Meta-analyses (recently renamed Preferred Reporting Items for Systematic Reviews and Meta- Analyses) guidelines address reporting of systematic reviews of random- ized trials (Moher et al., 2000), while the Meta-Analyses and Systematic Reviews of Observational Studies guidelines address reporting of systematic reviews of observational studies (Stroup et al., 2000). A second international collaborative activity has addressed the methods to be used to assess the quality of evidence and the strength of evidence- based recommendations in a standardized way. Such methods have been developed by groups such as the Grading of Recommendations Assess- ment, Development, and Evaluation Working Group alliance (Guyatt et al., 2008). Additional Examples of International Collaboration in Evidence Synthesis A number of groups other than the Cochrane Collaboration have now begun to organize evidence syntheses using an international collabora- tive model. The Joanna Briggs Institute publishes a library of systematic reviews relevant to nursing.15 The Campbell Collaboration, an interna- tional research network modelled after the Cochrane Collaboration, pro- duces systematic reviews of the effects of social interventions involving areas such as education, crime and justice, and social welfare (Campbell Collaboration, n.d.). Some groups have had a more narrow content focus, such as the consortium of guideline development organizations and profes- 14 See http://www.equator-network.org/about-equator (accessed June 12, 2009). 15 See http://www.joannabriggs.edu.au/pdf/JBI_LibSR_info.pdf (accessed June 12, 2009) . accessed

OCR for page 153
 THE INFORMATION NETWORKS REQUIRED sional societies formed to produce joint guidelines for the management of COPD (Schünemann et al., 2009). In other cases the international efforts have been directed at organizing a standard format for specific components of complex evidence syntheses so as to allow portions to be shared and used in other countries or set- tings. Developers of health technology assessments in the European Union have taken this approach. Through a consortium organization (European Network for Health Technology Assessment) with 63 partners from 32 countries and that they are developing a “core model” that defines 9 dif- ferent domains of a health technology assessment and that defines stan- dard elements for each domain.16 The model currently under development addresses only medical and surgical interventions, but a similar effort directed at diagnostic technologies is planned for the future. Similar work on standardization is being undertaken by the Guidelines International Net- work (GIN)—an international not-for-profit association of organizations and individuals involved in the development of clinical practice guidelines. GIN has defined a minimum data set that should be included in all evidence tables summarizing interventions (GIN, 2009) and is in the process of formulating a second to address evidence tables relating to diagnostic test accuracy. The aim is to have data in these tables presented in a consistent format that would allow guideline developers to use the efforts of others in developing their own evidence tables. Future Directions While the examples just listed show some beginning steps toward international integrated vehicles for evidence synthesis, there is much still to be done. Some of the additional needs include continued advancement in methods, particularly for complex evidence synthesis or syntheses involving designs other than RCTs; continued improvements in the quality of evi- dence syntheses; and improved coordination so as to decrease unnecessary duplication of effort. Methods for focused systematic reviews that combine data from con- trolled clinical trials are well advanced. There is much less agreement, however, on the most appropriate methods for combining results from stud- ies with other designs—primarily because of the difficulty in assessing the probability and magnitude of bias in these studies. Methods development for complex reviews and guidelines is even less advanced. As noted, inter- national collaborative efforts have already made some beginnings in this area. On a national level, both the Centre for Reviews & Dissemination in 16 See http://www.eunethta.net/upload/Founding%20Partners/EUnetHTA%20Collaboration %20Work%20Plan%202009_June292009_FINAL.pdf (accessed July 20, 2010).

OCR for page 153
 LEARNING WHAT WORKS the United Kingdom and AHRQ through its EPCs in the United States have made major contributions. Both have produced publications that address some of the methodological issues involved in complex syntheses (AHRQ, 2008; CRD, 2009) Methods development for guidelines is also needed. In an international survey of 18 clinical guideline programs, Burgers et al. (2003) found lit- tle consistency in methods, although they did note that all respondents intended to develop their guidelines using rigorous methods. They also reported a trend toward increasing use of evidence-based methods, such as the use of electronic database searches and systematic reviews. Most guideline processes also incorporated consensus procedures of some sort. A recent literature review commissioned by the WHO (Schünemann et al., Schünemann et 2006) to inform its guideline production process found no experimental research or studies that compared components of guideline methods advice. They did note, however that many organizations that produce guidelines have a “guidelines for guidelines” document to guide their processes, and they found empirical evidence that organizations that publish their guide- lines for guidelines produce more methodologically sound guidelines. The authors of the review were able to recommend a set of 19 principles for use by the WHO in guideline development. Given the current state of methods development for evidence syn- theses, it is clear that at least some of the funding for comparative effec- tiveness studies in the United States should be directed to promotion of further advances in methodology. While this funding could take the form of increased support for existing organizations within the United States, there would be clear advantages to align these efforts with international groups that are performing similar work. Avoiding Duplication of Effort Currently many different groups perform evidence synthesis of vari- ous sorts, and do so in a relatively uncoordinated way—leading to much needless duplication of effort. Greater organization of these efforts on an international scale would be helpful. One useful first step would be the formation of a registry of systematic reviews, analogous to the registries of clinical trials currently being set up. Prospective registration would allow any individual contemplating the performance of a systematic review to determine if a relevant review were already available or in progress, and it would also simplify the process of searching for systematic reviews. A registry could also be effective in improving the quality of systematic reviews—particularly if it included a mechanism for ensuring that review protocols are always produced. Prospective registration and publication of protocols, as done by the Cochrane Collaboration and the Joanna

OCR for page 153
 THE INFORMATION NETWORKS REQUIRED Briggs Institute, have been suggested as a way to reduce the possibility of selective outcome reporting bias in systematic reviews (Schünemann et al., Schünemann et 2006) and to address the possibility of nonpublication bias. Conclusions In summary, a number of different private and public entities in the United States currently conduct and disseminate evidence syntheses of vari- currently vari- ous types to support clinically effective medical practice, and an expansion of these efforts would be welcome and beneficial. While there is clearly a need for complex evidence syntheses to address the highest-priority topics, the number of these produced is likely to be relatively small (as at pres- ent) and to leave many gaps. These gaps can and will continue to be filled by focused systematic reviews and other evidence syntheses produced in the United States and in other countries. Clinicians, policy makers, and and the public will benefit from these efforts regardless of the degree of direct U.S. involvement. However, increased participation by the United States in international collaborative efforts such as those discussed in this paper would bring a number of benefits in addition to increasing the number of high-quality evidence syntheses produced. These include additional oppor- These oppor- tunities for workforce training in the United States, as well as participation in international efforts to develop the tools, methods, and standards for evidence synthesis. REFERENCES AHA (American Hospital Association). 2007a. Continued progress: Hospital use of informa- tion technology. Chicago, IL: AHA. ———. 2007b. Fast facts on U.S. hospitals. Chicago, IL: AHA. AHRQ (Agency for Healthcare Research and Quality). 2008. Effective healthcare program. Re- search Reviews. http://effectivehealthcare.ahrq.gov/healthInfo.cfm?infotype=rr&ProcessID=60 (accessed July 20, 2010). ). Buetow, K. 2008. Presentation to AHIC. http://archive.healthit.hhs.gov/portal/server.pt/gate- way/PTARGS_0_869092_0_0_18/all_materials.pdf (accessed July 20, 2010). Burgers, J. S., R. Grol, N. S. Klazinga, M. Makela, J. Zaat, and AGREE Collaboration. 2003. Towards evidence-based clinical practice: An international survey of 18 clinical guideline programs. International Journal of Quality Health Care 15(1):31-45. Campbell Collaboration. n.d. http://www.campbellcollaboration.org/about_us/index.php (ac- (ac- cessed July 20, 2010). . Catlin, A., C. Cowan, M. Hartman, S. Heffler, and the National Health Expenditure Accounts Team. 2008. National health spending in 2006: A year of change for prescription drugs. Health Affairs 27(1):14-29. CBO (Congressional Budget Office). 2008. Evidence on the costs and benefits of health information technology. http://www.cbo.gov/ftpdocs/91xx/doc9168/05-20-HealthIT.pdf (accessed July 20, 2010). .

OCR for page 153
 LEARNING WHAT WORKS Clarke, M., S. Hopewell, and I. Chalmers. 2007. Reports of clinical trials should begin and end with up-to-date systematic reviews of other relevant evidence: A status report. Journal of the Royal Society of Medicine 100(4):187-190. Connecting for Health. 2008a. We need a 2st-century privacy approach allowing Americans to protect and share health information to improve quality. http://www.connectingforhealth. com/resources/20080822_policy_brief.pdf (accessed July 20, 2010). ———. 2008b. Americans overwhelmingly believe electronic personal health records could im- prove their health. http://www.connectingforhealth.org/resources/ResearchBrief-200806. pdf (accessed July 20, 2010). ——— n.d.a Connectivity in the 2st century. http://www.connectingforhealth.org/connectivity/ (accessed July 20, 2010). ——— n.d.b Connecting for health decision-making for population health “first principles.” http://www.connectingforhealth.com/resources/first_principles.pdf (accessed July 20, 2010). ———. n.d.c Connecting for health common framework for networked personal health information: Statement of endorsement. http://www.connectingforhealth.org/resources/ CCEndorser.pdf (accessed July 20, 2010). CRD (Centre for Reviews and Dissemination). 2009. Systematic reviews: CRD’s guidance for undertaking reviews in health care. http://www.york.ac.uk/inst/crd/SysRev/!SSL!/ WebHelp/SysRev3.htm (accessed July 20, 2010). DesRoches, C. M., E. G. Campbell, S. R. Rao, K. Donelan, T. G. Ferris, A. Jha, R. Kaushal, D. E. Levy, S. Rosenbaum, A. E. Shields, and D. Blumenthal. 2008. Electronic health records in ambulatory care—A national survey of ohysicians. New England Journal of Medicine 359(1):50-60. Diagnostic Test Accuracy Working Group. 2009. Handbook for DTA reviews. http://srdta. cochrane.org/en/authors.html (accessed June 12, 2009). Eisenberg, J. M. 2002. Globalize the evidence, localize the decision: Evidence-based medicine and international diversity. Health Affairs 21(3):166-168. GIN (Guidelines International Network). 2010. Evidence Tables Working Group. http://www. g-i-n.net/activities/etwg/members-of-the-etwg (accessed July 15, 2010). Girosi, F., R. Meili, and R. Scoville. 2005. Extrapolating evidence of health information tech- nology savings and costs. Santa Monica, CA: RAND Corporation. Guyatt, G. H., A. D. Oxman, G. E. Vist, R. Kunz, Y. Falck-Ytter, P. Alonso-Coello, and H. J. Schünemann. 2008. GRADE: An emerging consensus on rating quality of evidence and strength of recommendations. British Medical Journal 336(7650):924-926. Higgins, J. P. T., and S. Green, eds. 2008. Cochrane handbook for systematic reviews of inter- ventions. Version 5.0.1. http://www.cochrane-handbook.org (accessed June 15, 2009). HIMSS (Health Information Management Systems Society) Analytics. 2008. Hospital IT expenses and budgets related to clinical sophistication. Market findings from HIMSS Analytics. Chicago, IL: HIMSS. IOM (Institute of Medicine). 2007. The learning healthcare system: Workshop summary. Washington, DC: The National Academies Press. ———. 2008. Knowing what works in health care: A roadmap for the nation. Washington, DC: The National Academies Press. ———. 2011. Digital infrastructure for the learning health system: The foundation for con- tinuous improvement in health and health care: Workshop summary. Washington, DC: The National Academies Press. Leeflang, M. M., J. J. Deeks, C. Gatsonis, P. M. Bossuyt, and Cochrane Diagnostic Test Ac- curacy Working Group. 2008. Systematic reviews of diagnostic test accuracy. Annals of Internal Medicine 149(12):889-897.

OCR for page 153
9 THE INFORMATION NETWORKS REQUIRED Lin, K., B. Watkins, T. Johnson, J. A. Rodriguez, and M. B. Barton; U.S. Preventive Services Task Force. 2008. Screening for chronic obstructive pulmonary disease using spirometry: Summary of the evidence for the U.S. Preventive Services Task Force. Annals of Internal Medicine 148:535-543. Mallett, S., and M. Clarke. 2003. How many Cochrane reviews are needed to cover existing evidence on the effects of health care interventions? American College of Physicians Journal Club 139(1):A11. MedPAC (Medicare Payment Advisory Committee). 2008. Medicare payment policy report to the Congress, March 200. Washington, DC: MedPAC. Miller, R. H., K. D’Amato, N. Oliva, C. E. West, and J. W. Adelson. 2009a. California’s digital divide: Clinical information systems for the haves and have-nots. Health Affairs 28(2). ———. 2009b. Barriers to financing clinical information systems in California healthcare delivery system organizations: Report to the Governor’s Health Information Technol- ogy Financing Advisory Commission. Sacramento: California Council on Science and Technology. Moher, D., D. J. Cook, S. Eastwood, I. Olkin, D. Rennie, and D. F. Stroup. 2000. Improving the quality of reports of meta-analyses of randomised controlled trials: The QUOROM statement. British Journal of Surgery 87(11):1448-1454. Moher, D., J. Tetzlaff, A. C. Tricco, M. Sampson, and D. G. Altman. 2007. Epidemiology and reporting characteristics of systematic reviews. PLoS Medicine 4(3):e78. Murthy, V. H., H. M. Krumholz, and C. P. Gross. 2004. Participation in cancer clinical tri- als: Race-, sex-, and age-based disparities. Journal of the American Medical Association 291(22):2720-2726. Nannini, L. J., C. J. Cates, T. J. Lasserson, and P. Poole. 2007. Combined corticosteroid and long-acting beta-agonist in one inhaler versus long-acting beta-agonists for chronic ob- structive pulmonary disease. Cochrane Database System Review 4:CD006829. NCI (National Cancer Institute). 2004. Shared Pathology Informatics Network (SPIN). http:// spin.nci.nih.gov (accessed August 5, 2010). ———. 2008a. NCI Cancer Bulletin. http://www.cancer.gov/ncicancerbulletin/NCI_Cancer_ Bulletin_031808/page4/print (accessed December 15, 2008). ———. 2008b. About caBIG. https://cabig.nci.nih.gov/overview (accessed August 5, 2010). Oliver, J., N. Siegfried, G. Kennedy, and T. Horvath. n.d. Putting Africa first: Supporting novice authors of HIV/AIDS reviews in Africa. http://www.imbi.uni-freiburg.de/OJS/cca/ index.php/cca/article/viewArticle/1334 (accessed June 12, 2009). Oxman, A., H. Schünemann, and A. Fretheim. 2006. Improving the use of research evidence in guideline development: 7. Deciding what evidence to include. Health Research Policy and Systems 4(1):19. PatientsLikeMe. 2008a. http://www.patientslikeme.com/ (accessed December 15, 2008). ———. 2008b. The value of openness. http://blog.patientslikeme.com/ (accessed December 15, 2008). Poole, P. J., E. Chacko, R. W. Wood-Baker, and C. J. Cates. 2006. Influenza vaccine for pa- tients with chronic obstructive pulmonary disease. Cochrane Database System Review 1:CD002733. Qaseem, A., V. Snow, P. Shekelle, K. F. Sherif, T. J. Wilt, S. Weinberger, and D. K. Owens. 2007. Diagnosis and management of stable chronic obstructive pulmonary disease: A clinical practice guideline from the American College of Physicians. Annals of Internal Medicine 147(9):633-638. Royle, P., and R. Milne. 2003. Literature searching for randomized controlled trials used in Cochrane reviews: Rapid versus exhaustive searches. International Journal of Technology Assessment in Health Care 19(4):591-603.

OCR for page 153
90 LEARNING WHAT WORKS Schünemann, H., A. Fretheim, and A. Oxman. 2006. Improving the use of research evidence in guideline development: 1. Guidelines for guidelines. Health Research Policy and Systems 4(1):13. Schünemann, H. J., M. Woodhead, A. Anzueto, S. Buist, W. Macnee, K. F. Rabe, and J. Heffner. 2009. A vision statement on guideline development for respiratory disease: The example of COPD. Lancet 373(9665):774-779. Sheldon, T., and I. Chalmers. 1994. The UK Cochrane Centre and the NHS Centre for re- views and dissemination: Respective roles within the information systems strategy of the NHS R&D programme, coordination and principles underlying collaboration. Health Economics 3(3):201-203. Stroup, D. F., J. A. Berlin, S. C. Morton, I. Olkin, G. D. Williamson, D. Rennie, D. Moher, B. J. Becker, T. A. Sipe, and S. B. Thacker. 2000. Meta-analysis of observational stud- ies in epidemiology: A proposal for reporting. Meta-analysis of Observational Stud- ies in Epidemiology (MOOSE) group. Journal of the American Medical Association 283(15):2008-2012. Whitlock, E. P., J. S. Lin, R. Chou, P. Shekelle, and K. A. Robinson. 2008. Using exist- ing systematic reviews in complex systematic reviews. Annals of Internal Medicine 148(10):776-782. Woolf, S. H. 2008. The meaning of translational research and why it matters. Journal of the American Medical Association 299(2):211-213.