Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 213
Clinical Practice Guidelines We Can Trust Appendix C Clinical Practice Guideline Appraisal Tools
OCR for page 214
Clinical Practice Guidelines We Can Trust TABLE C-1 Examples of Tools for Measuring the Quality of Clinical Practice Guidelines Development Tool/Authors Purpose Tool A Provisional Instrument for Assessing Clinical Practice Guidelines To provide an explicit method for examining the soundness of clinical practice guidelines (CPGs) and to encourage their systematic development. The committee developed this instrument to be used as an education tool, a self-assessment tool, or a means of judging guidelines before their adoption. Each of the tool’s seven attributes subsumes a number of dimensions. Each dimension is assigned a “yes/no” score (the dimension is or is not represented in the guideline). If yes, an associated rating of satisfactory, conditionally satisfactory, or unsatisfactory (degree to which the dimension is satisfied) is given. If the dimension is not represented (i.e., “no” response), then a judgment of whether the omission is unimportant, minor, or major is made. If most responses to the questions are “satisfactory” (or “unimportant omissions”), one might reasonably conclude that such a guideline would be sufficient for most clinical situations. Alternatively, if most responses were unsatisfactory (or major omissions), one would probably argue that the guideline needed to be revised before it could be used effectively. IOM (1992) The seven attributes include Clinical applicability The guideline document describes the patient populations to which the guidelines are meant to apply. The guideline document discusses complex clinical problems for the population(s) covered by the guidelines. The guideline document gives a rationale for excluding patient population(s). Clinical flexibility The guideline document provides specific information about situations in which clinical exceptions might be made in applying the guidelines.
OCR for page 215
Clinical Practice Guidelines We Can Trust The guideline document provides specific information about nonclinical situations in which exceptions might be made in applying the guidelines. The guideline document discusses the role of patient preferences as they relate to healthcare decisions in the particular case that the guidelines cover. The guideline document describes how patient preferences were taken into account during the guideline development process. Reliability/reproducibility The guidelines were subjected to independent review by experts or outside panels. The guideline document explains the lack of independent review. The guidelines were pretested in some manner. Validity: Definition and evaluation questions The guideline document specifically describes the method(s) used to collect (i.e., identify and retrieve) the scientific evidence on which recommendations are based. The guideline document gives adequate references or citations to the sources of information used in developing the guidelines. The guideline document discusses in general terms the strength of the scientific evidence on which recommendations are based. The guideline document explicitly rates the strength of evidence.
OCR for page 216
Clinical Practice Guidelines We Can Trust Tool/Authors Purpose Tool If a formal method of synthesis is used to combine the scientific evidence quantitatively or otherwise to develop summary outcome measures that reflect the strength of the scientific evidence, then the guideline document explicitly describes the method. Given that a formal method of synthesis is used to combine the scientific evidence quantitatively or otherwise to develop summary outcome measures, the guideline document explicitly reports the results of that synthesis. If formal expert or group judgment techniques are used to reach professional consensus, then the guideline document explicitly describes the techniques. Given that expert or group judgment method(s) are used to reach professional consensus, the guideline document explicitly gives information about the strength of professional consensus. The guideline document provides a qualitative description of the health benefits that are expected from a specific health practice. The guideline document provides a qualitative description of the potential harms or risks that may occur as a result of a specific health practice. The guideline document provides quantitative information or estimates about the health benefits to be expected as a result of a specific health practice.
OCR for page 217
Clinical Practice Guidelines We Can Trust The guideline document projects health benefits or outcomes in terms of additional life expectancy or similar measures, such as Quality-Adjusted Life Years (QUALYs). The guideline document provides a quantitative description of the potential harms or risks that may occur as a result of a specific health practice. The guideline document provides a qualitative description of the health costs or expenditures that are expected from a specific health practice. The guideline document provides a quantitative description of the health costs or expenditures that are expected from a specific health practice. If health benefits are projected in terms of additional life expectancy or similar measures, such as QUALYs, then the cost per unit of each identified benefit is estimated. Generally, the estimates of the benefits, harms, and costs are consistent with the strength of the evidence presented in the guideline document. The guideline document explicitly discusses the strength of the scientific evidence on which each major recommendation is based. Each major recommendation is consistent with the estimated benefits, harms, and costs of the service or intervention (and thus with the strength of evidence). The guideline document identifies other sets of guidelines that deal with the same clinical condition, technology, or topic. The guideline document identifies possible conflicts among existing guidelines and the reasons for them.
OCR for page 218
Clinical Practice Guidelines We Can Trust Tool/Authors Purpose Tool Clarity The guidelines describe the health condition to be prevented, detected, or treated in unambiguous terms. The guidelines describe the options for management of the health condition (i.e., the health practice and its alternatives) in unambiguous terms. If the guidelines give major recommendations, each is written in unambiguous terms. Recommendations are comprehensive, insofar as the evidence permits, and recommendations that might be expected are given. Recommendations are consistent (do not conflict with each other). The guideline document uses clear headings, indexes, lists, flow charts, or other devices to identify major topics discussed. The guideline document has a summary or abstract that accurately reflects the methods, content, and recommendations of the entire document. Each user of the guideline document can easily find each major recommendation. Scheduled review The guideline document gives a specific date for the scheduled review, gives other information concerning the procedure by which a scheduled review might be done, or gives a sunset or expiration date.
OCR for page 219
Clinical Practice Guidelines We Can Trust Multidisciplinary process Persons with appropriate clinical and methodological disciplines participated in developing the guideline document—that is, a multidisciplinary approach was followed. The guideline document explicitly notes any potential biases and/or conflicts of interests of the panel members, or states that biases and conflicts of interest were discussed among panel members or otherwise taken into account. Overall, potential biases and/or conflicts of interest appear to be adequately balanced or otherwise accounted for in the guideline development process. The guideline process describes the methods used to solicit views of interested parties not on the guidelines development panel and to present those views to the members of the panel. Appraisal of Guidelines for Research & Evaluation (AGREE) To assess the quality of CPGs: how well a guideline is developed and reported. It does not assess the clinical content or quality of evidence of the recommendations. AGREE consists of 23 key items organized in 6 domains. Each domain is intended to capture a separate dimension of guideline quality. Each item is rated on a 4-point Likert scale ranging from 4 (strongly agree) to 1 (strongly disagree). Domain scores are calculated by summing up all the scores of the individual items in a domain and by standardizing the total as a percentage of the maximum possible score for that domain. The six domain scores are independent and should not be aggregated into a single quality score; there is no threshold score that separates a “good” from a “not good” guideline. However, there is a section at the end for an overall assessment of the guideline, with a series of options (strongly recommend, recommend, would not recommend, and unsure). Cluzeau, F.A., J. Burgers and an international group of researchers from 13 countries. AGREE (2001)
OCR for page 220
Clinical Practice Guidelines We Can Trust Tool/Authors Purpose Tool AGREE II Brouwers et al. (2010) AGREE II represents an update of the original tool to improve its usability and methodological properties, namely its validity and reliability. AGREE II also includes 2 final overall assessment items that require the appraiser to make overall judgments of the practice guideline and consider how they rated the 23 items. AGREE II contains the same six domains as the original AGREE; changes to individual attributes of the domains are highlighted below in italics. Scope and purpose The overall objective(s) of the guideline is (are) specifically described. The clinical question(s) covered by the guideline is (are) specifically described. AGREE II: The health question(s) covered by the guideline is (are) specifically described. The patients to whom the guideline is meant to apply are specifically described. AGREE II: The population (patients, public, etc.) to whom the guideline is meant to apply is specifically described. Stakeholder involvement The guideline development group includes individuals from all the relevant professional groups. The patients’ views and preferences have been sought. AGREE II: The views and preferences of the target population (patients, public, etc.) have been sought. The target users of the guideline are clearly defined. The guideline has been piloted among target users. AGREE II: Deleted item. Incorporated into user guide description of item 19.
OCR for page 221
Clinical Practice Guidelines We Can Trust Rigor of development Systematic methods were used to search for evidence. AGREE II: No change. Renumber to 7. The criteria for selecting the evidence are clearly described. AGREE II: No change. Renumber to 8. AGREE II: New item number 9. The strengths and limitations of the body of evidence are clearly described. The methods used for formulating the recommendations are clearly described. The health benefits, side effects, and risks have been considered in formulating the recommendations. There is an explicit link between recommendations and the supporting evidence. The guideline has been externally reviewed by experts prior to its publication. A procedure for updating the guideline is provided. Clarity and presentation The recommendations are specific and unambiguous. The different options for management of the condition are clearly presented. AGREE II: The different options for management of the condition or health issue are clearly presented. Key recommendations are easily identifiable. The guideline is supported with tools for application. AGREE II: The guideline provides advice and/or tools on how the recommendations can be put into practice. Domain changes from clarity of presentation to applicability, and renumbered to 19.
OCR for page 222
Clinical Practice Guidelines We Can Trust Tool/Authors Purpose Tool Applicability The potential organizational barriers in applying the recommendations have been discussed. AGREE II: The guideline describes facilitators and barriers to its implementation. Change in order from 19 to 18. The potential cost implications of applying the recommendations have been considered. AGREE II: The potential resource implications of applying the recommendations have been considered. The guideline presents key review criteria for monitoring and/or audit purposes. AGREE II: The guideline presents monitoring and/or auditing criteria. Editorial independence The guideline is editorially independent from the funding body. AGREE II: The views of the functioning body have not influenced the content of the guideline. Conflicts of interest of guideline development members have been recorded. AGREE II: Competing interests of guideline development group members have been recorded and addressed. Cluzeau’s “Appraisal Instrument” A critical appraisal instrument that assesses whether guideline developers have minimized biases inherent in creating guidelines and addresses requirements for effective dissemination and implementation. A checklist containing 37 items categorized into 3 conceptual dimensions: rigor of development (validity and reproducibility), context and content (reliability, applicability, flexibility, and clarity), and application (implementation, dissemination, and monitoring strategies). The specific questions are answered yes, no, or unsure, which are then given values of 1, 0, and 0 respectively. Cluzeau et al. (1999)
OCR for page 223
Clinical Practice Guidelines We Can Trust A guideline dimension score is obtained by calculating the mean of the appraiser’s scores, and is then expressed as a percentage of the maximum possible score for that dimension in order to compare scores across the three dimensions. There is also a global qualitative assessment of the guidelines, with the options (strongly recommended, recommended, and not recommended) as a measure for overall quality. Dimension one: Rigor of development process Is the agency responsible for the development of the guidelines clearly identified? Was external funding or other support received for developing the guidelines? If external funding or support was received, is there evidence that potential biases of the funding body(ies) were taken into account? Is there a description of the individuals (e.g., professionals, interest groups—including patients) who were involved in the guideline development group? If so, did the group contain representatives of all key disciplines? Is there a description of the sources of information used to select the evidence on which the recommendations are based? If so, are the sources of information adequate? Is there a description of the method(s) used to interpret and assess the strength of evidence?
OCR for page 224
Clinical Practice Guidelines We Can Trust Tool/Authors Purpose Tool If so, is (are) the method(s) for rating the evidence adequate? Is there a description of the methods used to formulate the recommendations? If so, are the methods satisfactory? Is there an indication of how the views of interested parties not on the panel were taken into account? Is there an explicit link between the major recommendations and the level of supporting evidence? Were the guidelines independently reviewed prior to publication/release? If so, is explicit information given about the methods and how the comments were addressed? Were the guidelines piloted? If so, is explicit information given about the methods used and the results adopted? Is a date mentioned for reviewing or updating the guidelines? Is the body responsible for the reviewing and updating clearly indentified? Overall, have the potential biases of guideline development been adequately addressed? Dimension two: Context and content Are the reasons for developing the guidelines clearly stated? Are the objectives of the guidelines clearly defined?
OCR for page 225
Clinical Practice Guidelines We Can Trust Is there a satisfactory description of the patients to which the guidelines are meant to apply? Is there a description of the circumstances (clinical or nonclinical) in which exceptions might be made in using the guidelines? Is there an explicit statement of how the patient’s preferences should be taken into account in applying the guidelines? Do the guidelines describe the condition to be detected, treated, or prevented in unambiguous terms? Are the different possible options for the management of the condition clearly stated in the guidelines? Are the recommendations clearly presented? Is there an adequate description of the health benefits that are likely to be gained from the recommended management? Is there an adequate description of the potential harms or risks that may occur as a result of the recommended management? Is there an estimate of the costs or expenditures likely to incur from the recommended management? Are the recommendations supported by the estimated benefits, harms, and costs of the intervention? Dimension three: Application of guidelines Does the guideline document suggest possible methods for dissemination and implementation? (National guidelines only) Does the guideline document identify key elements that need to be considered by local guideline groups?
OCR for page 226
Clinical Practice Guidelines We Can Trust Tool/Authors Purpose Tool Does the guideline document specify criteria for monitoring compliance? Does the guideline document identify clear standards or targets? Does the guideline document define measureable outcomes that can be monitored? Hayward et al.’s “Structured Abstracts of CPGs” A structured format for abstracts of articles describing CPGs to help readers assess the applicability, importance, and validity of any guideline. The instrument includes eight topics essential to CPG developmental reporting. Format for structured abstracts of CPGs Objective: The primary objective of the guideline, including health problem and the targeted patients, providers, and settings. Options: The clinical practice options considered in formulating the guideline. Outcomes: Significant health and economic outcomes considered in comparing alternative practices. Evidence: How and when evidence was gathered, selected, and synthesized. Values: Disclosure of how values were assigned to potential outcomes of practice options and who participated in the process. Benefits, harms, and costs: The type and magnitude of benefits, harms, and costs expected for patients from guideline implementation. Hayward et al. (1993)
OCR for page 227
Clinical Practice Guidelines We Can Trust Recommendations: Summary of key recommendations. Validation: Report of any external review, comparison with other guidelines, or clinical testing of guideline use. Sponsors: Disclosure of the persons who developed, funded, or endorsed the guideline. Shaneyfelt’s “Methodological Standards” A tool to assess methodological quality of the development and reporting of clinical practice guidelines in the peer-reviewed medical literature. The 25-item instrument uses a yes or no format to measure adherence to three broadly grouped quality criteria. A guideline’s score reflects the total number of standards it satisfies, that is, “yes” responses (0–25). Shaneyfelt et al. (1999) Standards of guidelines development and format Purpose of the guideline is specified. Rationale and importance of the guideline are explained. The participants in the guideline development process and their areas of expertise are specified. Targeted health problem or technology is clearly defined. Targeted patient population is specified. Intended audience or users of the guideline are specified. The principal preventive, diagnostic, or therapeutic options available to clinicians and patients are specified. The health outcomes are specified. The method by which the guideline underwent external review is specified. An expiration date or date of scheduled review is specified.
OCR for page 228
Clinical Practice Guidelines We Can Trust Tool/Authors Purpose Tool Standards of evidence identification and summary Method of identifying scientific evidence is specified. The time period from which evidence is reviewed is specified. The evidence used is identified by citation and referenced. Method of data extractions is specified. Method for grading or classifying the scientific evidence is specified. Formal methods of combining evidence or expert opinion are used and described. Benefits and harms of specific health practices are specified. Benefits and harms are quantified. The effect on healthcare costs from specific health practices is specified. Costs are quantified. Standards on the formulation of recommendations The role of value judgments used by the guideline developers in making recommendations is discussed. The role of patient preferences is discussed. Recommendations are specific and apply to the stated goals of the guideline. Recommendations are graded according to the strength of the evidence. Flexibility in the recommendations is specified. SOURCES: AGREE (2001); Brouwers et al. (2010); Cluzeau et al. (1999); Hayward et al. (1993); IOM (1992); Shaneyfelt et al. (1999).
OCR for page 229
Clinical Practice Guidelines We Can Trust REFERENCES AGREE. 2001. Appraisal of guidelines for research & evaluation (AGREE) instrument. Brouwers, M. C., M. E. Kho, G. P. Browman, J. S. Burgers, F. Cluzeau, G. Feder, B. Fervers, I. D. Graham, S. E. Hanna, J. Makarski, and for the AGREE Next Steps Consortium. 2010. Development of the AGREE II, part 2: Assessment of validity of items and tools to support application. CMAJ 182(10):E472–E478. Cluzeau, F. A., P. Littlejohns, J. M. Grimshaw, G. Feder, and S. E. Moran. 1999. Development and application of a generic methodology to assess the quality of clinical guidelines. International Journal for Quality in Health Care 11(1):21–28. Hayward, R. S. A., M. C. Wilson, S. R. Tunis, E. B. Bass, H. R. Rubin, and R. B. Haynes. 1993. More informative abstracts of articles describing clinical practice guidelines. Annals of Internal Medicine 118(9):731–737. IOM (Institute of Medicine). 1992. Guidelines for clinical practice: From development to use. Edited by M. J. Field and K. N. Lohr. Washington, DC: National Academy Press. Shaneyfelt, T., M. Mayo-Smith, and J. Rothwangl. 1999. Are guidelines following guidelines? The methodological quality of clinical practice guidelines in the peer-reviewed medical literature. JAMA 281:1900–1905.
OCR for page 230
Clinical Practice Guidelines We Can Trust This page intentionally left blank.