TABLE C-1 Examples of Tools for Measuring the Quality of Clinical Practice Guidelines Development
Tool/Authors |
Purpose |
Tool |
A Provisional Instrument for Assessing Clinical Practice Guidelines |
To provide an explicit method for examining the soundness of clinical practice guidelines (CPGs) and to encourage their systematic development. The committee developed this instrument to be used as an education tool, a self-assessment tool, or a means of judging guidelines before their adoption. |
Each of the tool’s seven attributes subsumes a number of dimensions. Each dimension is assigned a “yes/no” score (the dimension is or is not represented in the guideline). If yes, an associated rating of satisfactory, conditionally satisfactory, or unsatisfactory (degree to which the dimension is satisfied) is given. If the dimension is not represented (i.e., “no” response), then a judgment of whether the omission is unimportant, minor, or major is made. If most responses to the questions are “satisfactory” (or “unimportant omissions”), one might reasonably conclude that such a guideline would be sufficient for most clinical situations. Alternatively, if most responses were unsatisfactory (or major omissions), one would probably argue that the guideline needed to be revised before it could be used effectively. |
IOM (1992) |
||
|
|
The seven attributes include |
|
|
|
|
|
|
Tool/Authors |
Purpose |
Tool |
|
|
|
|
|
|
Tool/Authors |
Purpose |
Tool |
|
|
|
|
|
|
|
|
The guideline process describes the methods used to solicit views of interested parties not on the guidelines development panel and to present those views to the members of the panel. |
Appraisal of Guidelines for Research & Evaluation (AGREE) |
To assess the quality of CPGs: how well a guideline is developed and reported. It does not assess the clinical content or quality of evidence of the recommendations. |
AGREE consists of 23 key items organized in 6 domains. Each domain is intended to capture a separate dimension of guideline quality. Each item is rated on a 4-point Likert scale ranging from 4 (strongly agree) to 1 (strongly disagree). Domain scores are calculated by summing up all the scores of the individual items in a domain and by standardizing the total as a percentage of the maximum possible score for that domain. The six domain scores are independent and should not be aggregated into a single quality score; there is no threshold score that separates a “good” from a “not good” guideline. However, there is a section at the end for an overall assessment of the guideline, with a series of options (strongly recommend, recommend, would not recommend, and unsure). |
Cluzeau, F.A., J. Burgers and an international group of researchers from 13 countries. |
||
AGREE (2001) |
Tool/Authors |
Purpose |
Tool |
AGREE II Brouwers et al. (2010) |
|
AGREE II represents an update of the original tool to improve its usability and methodological properties, namely its validity and reliability. AGREE II also includes 2 final overall assessment items that require the appraiser to make overall judgments of the practice guideline and consider how they rated the 23 items. AGREE II contains the same six domains as the original AGREE; changes to individual attributes of the domains are highlighted below in italics. |
|
|
Scope and purpose
Stakeholder involvement
|
|
|
Rigor of development
Clarity and presentation
|
Tool/Authors |
Purpose |
Tool |
|
|
Applicability
Editorial independence
Conflicts of interest of guideline development members have been recorded. AGREE II: Competing interests of guideline development group members have been recorded and addressed. |
Cluzeau’s “Appraisal Instrument” |
A critical appraisal instrument that assesses whether guideline developers have minimized biases inherent in creating guidelines and addresses requirements for effective dissemination and implementation. |
A checklist containing 37 items categorized into 3 conceptual dimensions: rigor of development (validity and reproducibility), context and content (reliability, applicability, flexibility, and clarity), and application (implementation, dissemination, and monitoring strategies). The specific questions are answered yes, no, or unsure, which are then given values of 1, 0, and 0 respectively. |
Cluzeau et al. (1999) |
|
|
A guideline dimension score is obtained by calculating the mean of the appraiser’s scores, and is then expressed as a percentage of the maximum possible score for that dimension in order to compare scores across the three dimensions. There is also a global qualitative assessment of the guidelines, with the options (strongly recommended, recommended, and not recommended) as a measure for overall quality. |
|
|
Dimension one: Rigor of development process
|
Tool/Authors |
Purpose |
Tool |
|
|
Dimension two: Context and content
|
|
|
Dimension three: Application of guidelines
|
Tool/Authors |
Purpose |
Tool |
|
|
|
Hayward et al.’s “Structured Abstracts of CPGs” |
A structured format for abstracts of articles describing CPGs to help readers assess the applicability, importance, and validity of any guideline. |
The instrument includes eight topics essential to CPG developmental reporting. Format for structured abstracts of CPGs
|
Hayward et al. (1993) |
|
|
Sponsors: Disclosure of the persons who developed, funded, or endorsed the guideline. |
Shaneyfelt’s “Methodological Standards” |
A tool to assess methodological quality of the development and reporting of clinical practice guidelines in the peer-reviewed medical literature. |
The 25-item instrument uses a yes or no format to measure adherence to three broadly grouped quality criteria. A guideline’s score reflects the total number of standards it satisfies, that is, “yes” responses (0–25). |
Shaneyfelt et al. (1999) |
||
|
|
Standards of guidelines development and format
|
Tool/Authors |
Purpose |
Tool |
|
|
Standards of evidence identification and summary
Standards on the formulation of recommendations
Flexibility in the recommendations is specified. |
SOURCES: AGREE (2001); Brouwers et al. (2010); Cluzeau et al. (1999); Hayward et al. (1993); IOM (1992); Shaneyfelt et al. (1999). |
REFERENCES
AGREE. 2001. Appraisal of guidelines for research & evaluation (AGREE) instrument.
Brouwers, M. C., M. E. Kho, G. P. Browman, J. S. Burgers, F. Cluzeau, G. Feder, B. Fervers, I. D. Graham, S. E. Hanna, J. Makarski, and for the AGREE Next Steps Consortium. 2010. Development of the AGREE II, part 2: Assessment of validity of items and tools to support application. CMAJ 182(10):E472–E478.
Cluzeau, F. A., P. Littlejohns, J. M. Grimshaw, G. Feder, and S. E. Moran. 1999. Development and application of a generic methodology to assess the quality of clinical guidelines. International Journal for Quality in Health Care 11(1):21–28.
Hayward, R. S. A., M. C. Wilson, S. R. Tunis, E. B. Bass, H. R. Rubin, and R. B. Haynes. 1993. More informative abstracts of articles describing clinical practice guidelines. Annals of Internal Medicine 118(9):731–737.
IOM (Institute of Medicine). 1992. Guidelines for clinical practice: From development to use. Edited by M. J. Field and K. N. Lohr. Washington, DC: National Academy Press.
Shaneyfelt, T., M. Mayo-Smith, and J. Rothwangl. 1999. Are guidelines following guidelines? The methodological quality of clinical practice guidelines in the peer-reviewed medical literature. JAMA 281:1900–1905.