B A Provisional Instrument for Assessing Clinical Practice Guidelines

KATHLEEN N. LOHR AND MARILYN J. FIELD

Division of Health Care Services

The Institute of Medicine (IOM) of the National Academy of Sciences has been engaged since the beginning of 1990 in two projects relating to the development, implementation, and evaluation of clinical practice guidelines. One IOM committee defined practice guidelines as "systematically developed statements to assist practitioners and patients in choosing appropriate health care for specific clinical conditions." It also delineated several desirable attributes of guidelines that are intended to help users understand the elements of a sound guideline and to recognize good (or not-so-good) guidelines. These aspects of guidelines were discussed in Clinical Practice Guidelines: Directions for a New Program (IOM, 1990) and Guidelines for Clinical Practice: From Development to Use (this report).

The first IOM study committee discovered that no explicit method was available for assessing existing or emerging practice guidelines. At least one instrument was being tested to assess some aspects of guideline development (AMA, 1990), but nothing existed to judge the quality, reliability, and validity of the content of the guideline itself. Therefore, one task the second IOM committee undertook was to develop an "assessment instrument" that could be used by various parties in formal evaluations of guidelines.

The next sections of this document describe, first, the purposes of the "provisional" assessment instrument and, second, its development. The discussion covers several features of the instrument and its application, and notes several cautions and caveats about the present form, all of which warrant further consideration. Finally, the document presents the instru-



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 346
Guidelines for Clinical Practice: From Development to Use B A Provisional Instrument for Assessing Clinical Practice Guidelines KATHLEEN N. LOHR AND MARILYN J. FIELD Division of Health Care Services The Institute of Medicine (IOM) of the National Academy of Sciences has been engaged since the beginning of 1990 in two projects relating to the development, implementation, and evaluation of clinical practice guidelines. One IOM committee defined practice guidelines as "systematically developed statements to assist practitioners and patients in choosing appropriate health care for specific clinical conditions." It also delineated several desirable attributes of guidelines that are intended to help users understand the elements of a sound guideline and to recognize good (or not-so-good) guidelines. These aspects of guidelines were discussed in Clinical Practice Guidelines: Directions for a New Program (IOM, 1990) and Guidelines for Clinical Practice: From Development to Use (this report). The first IOM study committee discovered that no explicit method was available for assessing existing or emerging practice guidelines. At least one instrument was being tested to assess some aspects of guideline development (AMA, 1990), but nothing existed to judge the quality, reliability, and validity of the content of the guideline itself. Therefore, one task the second IOM committee undertook was to develop an "assessment instrument" that could be used by various parties in formal evaluations of guidelines. The next sections of this document describe, first, the purposes of the "provisional" assessment instrument and, second, its development. The discussion covers several features of the instrument and its application, and notes several cautions and caveats about the present form, all of which warrant further consideration. Finally, the document presents the instru-

OCR for page 346
Guidelines for Clinical Practice: From Development to Use ment itself, in three operational parts—a general information sheet, the full instrument, and a summary evaluation sheet. The instrument is termed provisional because the committee firmly believed that more experience needs to be accumulated by testing it on different kinds of guidelines. PURPOSES OF THE ASSESSMENT INSTRUMENT The central purpose of the IOM's instrument for assessing clinical practice guidelines is to provide an explicit method for examining the soundness of such guidelines and to encourage their systematic development. By assessment is meant a prospective judgment of the soundness of both the process used in developing a guideline and the resulting guideline. The intent is to avoid situations in which a guideline that is not consistent with the scientific evidence is nonetheless "rated" as good on procedural criteria alone.1 More concretely, the IOM intended to operationalize its attributes of good guidelines and to provide a standardized approach and structure for the assessment of a guideline document. The resulting form is not simple. Therefore, the IOM does not expect practicing physicians or other clinicians, patients, other nonprofessionals, or policy makers to apply this instrument. Rather it expects individuals (or groups) with three types of expertise to apply it-namely, those with clinical experience with patients who have the conditions or problems covered by the guideline document, those with research experience in the conditions or technologies covered, and those with methodologic skills in developing guidelines. Any final or overall judgments of a guideline document emerging from the application of this instrument would be reported in simpler, summary form in ways that would convey the relative soundness (or lack of it) of a given guideline document to all potential users of the guideline. The IOM committee sees three possible uses of this instrument: as an educational tool, as a self-assessment tool, and as a means of judging guidelines before their adoption. The Agency for Health Care Policy and Research (AHCPR) may want to use this instrument, or one like it, in directing the work of its guidelines-development panels; the agency may also wish to employ it in judging the products of those panels or the guidelines developed by other groups, such as medical specialty societies. Furthermore, other groups may wish to review existing or draft guidelines of their own against this instrument, in an effort to identify guidelines warranting revision or defects in draft guidelines that warrant correction before they are put into final form. Finally, if an organization were to be created for the 1   Evaluation of the eventual impact of guidelines is a separate step. Both IOM reports (1990 and this one) include discussions of evaluation.

OCR for page 346
Guidelines for Clinical Practice: From Development to Use express purpose of certifying or otherwise reporting on the soundness of particular guidelines, it might wish to employ the instrument as part of its review activities. DEVELOPMENT PROCESS The concept of the instrument originated during discussions with members of the AHCPR staff about their responsibilities for practice guidelines under the Omnibus Budget Reconciliation Act of 1989. In seeking to respond to those early ideas, the IOM committee went through four steps. First, with the aid of an outside consultant, staff drafted a set of questions to operationalize the eight conceptual attributes of good practice guidelines identified in the IOM's 1990 report on guidelines. 2 (Discussions of these attributes introduce sections of the assessment instrument.) Second, these questions were combined with background information and instructions for users and subjected to considerable internal and external review, as described below. Members of the IOM committee twice reviewed drafts of the assessment instrument during this time. Third, IOM staff used the critiques and suggestions of reviewers to revise the instrument, and it, together with background material, was subjected to external review according to IOM and National Research Council (NRC) procedures. Fourth, the instrument was revised in response to that external review, resulting in the provisional questionnaire and other forms incorporated into this document. Initial Reviews An interim draft of the instrument was sent to AHCPR in December 1990 and was subsequently forwarded to several professional societies that had volunteered to review and test the instrument against guidelines of their own. By June 1991, the committee had received more than 15 separate responses and commentaries. Responses included a lengthy summary review provided by IMCARE (Internal Medicine Center to Advance Research and Education), which had solicited reviews from the 231 internists then in its Guideline Network. IMCARE also sent the document to 147 physicians who requested more information and received 65 responses from network members who reviewed and, in many cases, applied the instruments to real guidelines. Reactions to the draft instrument were extremely varied. With respect to format, instructions, ease of use, and similar issues, positive comments included the following: 2   Anne-Marie Audet, M.D., of the Health Institute, New England Medical Center, Inc., served as consultant during the initial stage of the project.

OCR for page 346
Guidelines for Clinical Practice: From Development to Use "Well written—concise." "The instrument was easy to apply and the definitions of attributes were helpful in driving the assessment." "The meaning of each attribute, the instructions, and response categories were generally clear." "Overall, the instrument provides a precise algorithm for examining the degree to which a clinical guideline meets defined aspects of seven attributes. In our opinion, the instrument can be used for its intended purpose. In addition, application of an instrument such as this is, in itself, a thought-provoking exercise for individuals active in developing clinical practice guidelines." By contrast, negative comments were of the following kind: The form seems excessively lengthy and not particularly user-friendly." "The instructions are verbose and redundant, and almost legalistic. They are not user-friendly." "This instrument was very confusing, almost 'impossible."' "Instrument of intellectual torture. Beyond Bureaucracy! . . . This makes me feel stupid!" A form that accompanied the instrument asked reviewers to indicate the time (in person-hours) needed to apply the assessment instrument to a guideline of their choice. Among 59 individuals in the IMCARE group who evidently applied the instrument to an actual guideline and completed the form, 13 said that the instrument took under two hours to apply, 38 said two to three hours, and 8 said four hours or more. Several commentators indicated that their learning curve was quite steep and that repeated use of the instrument would make it simpler and less time-consuming to apply. In rating the overall difficulty of understanding or using the assessment instrument, the majority of respondents indicated that it was moderately to very difficult. In addition, the great majority found the instrument good or at least somewhat helpful in helping them reach an overall judgment of the strengths and weaknesses of the guideline they were evaluating. Finally, more than half indicated that the instrument definitely should be revised; most of the rest were uncertain or had no opinion, and only about 1 in 10 advised abandoning the effort. Among the more concrete recommendations for revisions were the following: (1) add more "don't know" or "not applicable" responses to certain questions; (2) simplify and shorten the instructions; (3) include a "prologue'' containing pertinent information from the first IOM (1990) report on guidelines, which defines key terms and similar concepts; (4) consider adding an attribute related to endorsement by appropriate affiliated or outside organizations; and (5) clarify exactly who the intended users are. In addition, many reviewers offered comments on the draft instrument itself, chief-

OCR for page 346
Guidelines for Clinical Practice: From Development to Use ly observations about the wording of questions and the addition of response categories. More general suggestions included employing the instrument to help generate guidelines (rather than rate them) and using the instrument to guide an assessment process (without necessarily requiring that the instrument be completed in full). Final Reviews In accordance with IOM and NRC procedures, the revised instrument was subjected to an external, anonymous review by a panel similar to the full IOM committee. The "provisional" form in this document reflects the reactions of this group of seven experts, and many of their comments have been incorporated into the discussion presented here. IMPORTANT FEATURES OF THE PROVISIONAL ASSESSMENT INSTRUMENT Attributes of Practice Guidelines Types of Attributes Four attributes identified in the first IOM report on practice guidelines concern the substance of the guidelines—clinical applicability or scope, clinical flexibility, reliability/reproducibility, and validity. Four others— clarity, multidisciplinary process, scheduled review, and documentation— have more to do with process. This instrument explicitly incorporates all attributes but documentation; that attribute is captured in questions directed at the other seven. Each attribute is described in the text of the instrument. Implicit Weight Accorded to Different Attributes As discussed below, this instrument has no explicit or quantitative scoring system. The attributes are implicitly weighted, however, according to the number of main questions used to cover them. Of a total of 46 questions, validity has 22 questions; clarity, 8; multidisciplinary process, 4; clinical flexibility, 4; reliability and reproducibility, 4; clinical adaptability, 3; and scheduled review, 1. By this rough metric, validity is accorded by far the major emphasis in the document, reflecting the committee's concern for finding a way to judge the soundness of the guidelines themselves rather than just the acceptability of the development process.

OCR for page 346
Guidelines for Clinical Practice: From Development to Use Questions and Response Categories Questions Most of the seven attributes are dealt with through one or more questions that tap specific "dimensions." For example, validity is divided into five issues: (1) strength of the scientific evidence and professional consensus; (2) qualitative and quantitative statements about health benefits and harms or risks; (3) qualitative and quantitative statements about expected health costs or expenditures; (4) the extent to which specific recommendations are justified by the estimates of benefits, harms, and costs provided in the guidelines and the extent to which those estimates are supported by the evidence amassed in the guidelines document; and (5) potential conflicts among existing guidelines, if any. The instrument has 46 descriptive questions related to the seven attributes. Generally they pertain to the presence of information about the attribute or about a particular dimension of an attribute. Several questions have additional "items" designed to help assessors think about key points implied by the main question and whether a particular dimension of an attribute is satisfactory or not. Responses to Questions Responses to each question are typically "yes" or "no" for questions asking about the presence or absence of certain information, features, or development processes. If the answer is "yes,'' a follow-up question asks about the quality of that information, feature, or development process--essentially whether the information provided is satisfactory or not. If the answer to the main question is "no," the follow-up question probes the significance of the absence of information, a particular feature, or development process and asks whether the omission is important or not. Satisfactory. Assessors can judge information about a particular attribute as satisfactory if all critical elements have been considered and presented. For example, the discussion or description should be thorough and comprehensive; the guideline developers should have based their work on appropriate and correct information (e.g., from the literature review); and they should have used appropriate methods (e.g., for evaluating the strength of the scientific evidence or reaching professional peer consensus). Conditionally satisfactory. The description or discussion of an attribute or dimension is conditionally satisfactory if some, but not all, of the critical elements have been considered and presented. For example, the discussion of a particular aspect of an attribute such as scheduled review may be vague

OCR for page 346
Guidelines for Clinical Practice: From Development to Use or incomplete. Alternatively, guidelines developers may have disregarded important information (about, for instance, likely risks to patients from the use of a technology) in reaching their recommendations, or they may have improperly used certain kinds of methods. These problems with the guideline document may not prevent a clinician from using it or understanding its recommendations, but they may affect its overall usefulness or call certain recommendations into question. Revisions would be presumed to improve the guidelines, but they would not be mandatory. Unsatisfactory. The description or discussion about a specific attribute is unsatisfactory if most of the critical elements have not been considered or presented. For example, well-known pieces of clinical information (or the views of multiple specialists with an interest in the guideline topic) may have been ignored; methods of analysis may have been misapplied; or recommendations may be based on faulty information or poor logic, or both. In such a case, it would be difficult if not impossible to judge the quality of the process of guideline development or the soundness of the resulting guidelines and recommendations (or both); certainly assessors could not mark those attributes as satisfactory. Serious thought must be given to augmenting or revising the guidelines document before it is promoted further. Omissions. Omitting a description or discussion about a specific attribute or dimension may be unimportant if that omission seems likely to have no demonstrable effect either on the ability of a guideline user to apply the guideline effectively in the clinical decision-making process or on the capacity of the assessor to make an independent assessment of the quality of that attribute. Omitting such a description or discussion is of minor importance if it seems likely to affect negatively the ability of a guideline user or of evaluators to apply the guideline effectively or independently to assess its quality. Finally, omitting such a description or discussion is a major omission if the absence of such information essentially prevents guideline users or evaluators from applying the guideline effectively or even making an independent evaluation about the soundness of the guidelines document itself (at least on that particular feature). Special Cases Special cases may arise in which information appropriately is omitted from the guideline because the question or item is not applicable or is inappropriate (given responses to earlier items, for instance). In such an instance, the assessor is asked to mark the response category most appropriate for the given case (e.g., not applicable). In other situations, assessors may find it difficult to arrive at a single answer to the question, especially if the guidelines document being evaluated is very complex or if necessary background information appears to be missing. "Comments" sections are provided throughout for assessors to record additional remarks or qualifying

OCR for page 346
Guidelines for Clinical Practice: From Development to Use statements, to highlight areas not well covered by the instrument, and to note special factors that should either be followed up or taken into account in the overall judgment about the guidelines document. Finally, if assessors conclude that the guidelines document is so complex, clinically esoteric, or methodologically sophisticated that it warrants additional, outside expert review, they are asked to note that at the end of the full instrument and also on the summary evaluation sheet. Alternative Approaches to Responses The main type of response used in this form is categorical (e.g., satisfactory, conditionally satisfactory, and unsatisfactory). Some reviewers noted that this approach is inherently constraining and requires definitions of the three categories, which may not be interpreted consistently. Furthermore, these categories do not allow assessors to distinguish guidelines that more properly should be characterized as excellent or outstanding. To overcome some of these drawbacks, an approach to responses based on a scale might be tested. For example, a five- or even seven-point scale might be adopted, with one end of the range described as excellent (exemplary, highly satisfactory, or a similar superlative) and the opposite end described as poor (inferior, or very unsatisfactory). The equivalent approach might also be tried for the responses concerned with omissions of information. The committee believes this change warrants testing at some point in the future development of this form. Response Scoring After considerable debate and consideration of reviewers' comments, the committee concluded that this instrument should not be "scored" in any quantitative way. Thus, it does not propose any formal weighting or numerical scoring scheme for the main questions, nor does it suggest a particular threshold, cut-point, or floor against which current guidelines might be judged acceptable or unacceptable. If most responses to the questions are "satisfactory" (or "unimportant omissions"), however, one might reasonably conclude that such a guidelines document would be sufficient for most clinical situations. Alternatively, if most responses were unsatisfactory (or major omissions), one would probably argue that the guidelines document needed to be revised before it could be used effectively. The committee was of the view that a defensible scoring system could not be designed a priori in any case, regardless of whether scoring would be purely categorical or more quantitative. Review and testing of the assessment instrument itself—with revisions as necessary—will be required before a sensible scoring system can be proposed. Moreover, different

OCR for page 346
Guidelines for Clinical Practice: From Development to Use users of the assessment instrument may have legitimate reasons to differ on where they would establish such cut-points. Provision of information on the quality of guidelines documents appears to be more in the public interest than is making "one-size-fits-all" judgments on behalf of others. In the same vein, no single question is treated as the signal of a "fatal flaw." That is, for no question will a response of "no," "unsatisfactory," or "major omission'' by itself render the guidelines document unacceptable. Some committee members believed that certain questions, especially those relating to validity, should be so designated. However, the questions that seem to be the most likely candidates for this level of decisiveness3 were added in response to the external review of the draft document; therefore they have not yet been reviewed or tested further. The committee believes designating these (or other) items as potentially "fatal" is premature. Response Aggregation and Display General Comments Information obtained from applying the instrument might eventually be arrayed in one or more qualitative, summary displays or tables, as might be done, for instance, by a Consumer Report article. This might provide a rough indicator of whether the guidelines document could be used effectively in clinical situations. One reviewer, for example, suggested that a report for busy practicing physicians might usefully include "a graphic summary of the degree to which each attribute was successfully achieved, e.g., a bar representing the percent of key items within each attribute that were deemed satisfactory ...; and . . . a brief, narrative summary assessment." Summary Evaluation Sheet The committee did not pursue the design of such displays, chiefly because such an effort was seen as premature for an instrument that itself warrants additional testing and application. As an intermediate step, however, the instrument does include a "summary evaluation sheet," which is actually a set of pages that condense the findings of the assessment from the primary questions in the instrument. It is filled out only after the full instrument has been completed. 3   Two questions that might be candidate "fatal flaw" items, when responses to them were unacceptable (i.e., "no"), are the following from the validity section: * Generally, the estimates of benefits, harms, and costs are consistent with the evidence presented in the guidelines document. (Yes, completely; yes, partially; or no) (Question 28) * Each major recommendation is consistent with the estimated benefits, harms, and costs of the service or intervention (and thus with the strength of the evidence). (Yes, completely; yes, partially; or no) (Question 31).

OCR for page 346
Guidelines for Clinical Practice: From Development to Use In the present version, "better" answers are recorded to the left of the response column, "worse" answers to the right. Thus, a quick scan of this sheet may provide an overall sense of the quality of the guidelines. Put another way, a clinician or other user ought to be able to apply the guideline effectively if all dimensions of the seven attributes (or, at a minimum, all seven attributes) are judged to be "satisfactory" and all omissions of information are considered ''unimportant," as those terms were defined earlier. In this (ideal) situation, all notations on the summary sheet would be on the far left. By contrast, if many or most notations are on the right side of the response column, the user might wish to employ the guidelines only selectively or to request clarifications or revisions. Several reviewers noted that completing the summary evaluation sheet is essentially a clerical task, provided the main part of the assessment instrument has been legibly and fully completed by one or more experts (as discussed earlier). The committee agrees and thus suggests that junior or clerical staff be given this responsibility, and that the instructions on the form indicate that users might wish to do so. Alternatively, the instrument (or at least the recording of responses to its questions) might be computerized. In that case, the summary sheet could be an automatic product of the computer program. The value in pursuing more fully the possibilities of computerization of this form might be considerable. FINAL CAUTIONS AND CAVEATS The Ideal: Enemy of the Good Some commentators noted the IOM's recognition in its 1990 report that most (if not all) guidelines in existence today would "fail" to meet the ideal of this instrument. Reviewers were concerned that "prematurely imposing excessive rigor" would discourage some guideline developers.4 4   Comments from the American Nurses Association were particularly to the point: the criteria used to judge the adequacy of guidelines establishes a high standard that likely would seldom be achieved in reality. Several questions need careful consideration prior to accepting the criteria in this instrument: 1. Do the criteria . . . create false expectations of quality which is not achievable with current fiscal restrictions in health care? 2. What are the potential legal and regulatory ramifications of accepting these criteria as representative of quality practice? 3. How would these criteria eventually influence the costs of care through the pursuit of considerable evidence regarding the "best" method of treatment? 4. Are these guidelines intended to weed out bad practice, or is the intention to demonstrate the "best," often misinterpreted as the "only" way of practicing?" (K. S. O'Connor, Division of Nursing Practice and Economics, American Nurses Association, in a letter to Marilyn Field, IOM study director, dated May 20, 1991.)

OCR for page 346
Guidelines for Clinical Practice: From Development to Use At least one reviewer warned that the assessment process should not be used as a "second level expert panel" and cautioned that designing the assessment instrument process itself would take some care. The IOM committee agrees and, in that light, emphasizes that the educational uses of the instrument are more important than its assessment applications (in the near term at least). Users of the Assessment Instrument Busy practicing physicians or other clinicians are not the intended or anticipated appliers of this assessment instrument. Neither are policy makers, patients, or other nonprofessionals, although all may have some interest in the results. Assessors are expected to have, individually or collectively, expertise in three areas: clinical experience with patient populations covered by the guideline, research experience about the conditions or technologies covered by the guideline, and methodological expertise with techniques and processes of guideline development. Because no one individual is likely to possess all three kinds of expertise, experience with the instrument may suggest that a "group," "panel," or "study section" approach will be needed to apply it satisfactorily. In this way, different individuals would be responsible for different parts of the assessment (particularly to determine validity). Furthermore, turning the full assessment into a review or evaluation that would be understandable to patients, practitioners, or policy makers will be a separate step, as noted earlier. This provisional instrument thus proceeds on several assumptions. First, assessors (individually or collectively) are sufficiently schooled either in the methodologic issues inherent in guidelines development or in the clinical issues related to the main topic of the guidelines document that they are able to complete the bulk of the assessment instrument unaided. Second, questions about clinical topics or methods can be referred to appropriate experts when necessary and without undue delay. This kind of referral is particularly important if the AHCPR or some other entity acquires a specific mandate to certify or ratify guidelines from whatever source. Third, in some cases, having a dual or parallel (i.e., simultaneous) review may be a desirable tactic. Fourth, junior staff may well be used to assemble relevant material, perhaps to do an initial check of the document itself, perhaps to evaluate the document for the attribute of clarity, and to complete the summary evaluation sheet. Finally, the experts assembled or asked to apply the instrument (in its current form or any future, modified form) will be carefully trained in its use.

OCR for page 346
Guidelines for Clinical Practice: From Development to Use VII. MULTI-DISCIPLINARY PROCESS A multi-disciplinary process for practice guidelines means that representatives of a broad range of practitioners, consumers or patients, and other groups likely to be affected by the guidelines have participated in the development process at some stage. These representatives can be individuals who have had direct responsibility for the guideline document or individuals who have reviewed that document or in other ways have contributed to it. This attribute intends that both methodologic and clinical disciplines be involved in the guideline-development process. This document cannot identify in advance all relevant participants, interested parties, or disciplines because each set of guidelines will differ in this respect. This attribute requires that five things be true. First, some combination of individuals directly responsible for guidelines and those who have otherwise contributed to their development collectively represents all the key groups likely to affect or to be affected by the guidelines. Second, the guideline document describes the parties involved (including their credentials and potential biases); "the parties involved" is understood to mean participants in the actual development panel and those in review panels, public hearings, or other review forums. Third, potential biases and conflicts of interests have been discussed or otherwise appropriately taken account of. Fourth, the methods used to solicit panelists' views and arrive at group judgments have been described and are adequate and appropriate to the task of balancing views and potential biases. Fifth, the methods used to solicit outside review comments and present those to panelists have been described and are adequate to the task of making outside views clear to panelists. 43. PERSONS WITH APPROPRIATE CLINICAL AND METHODOLOGIC DISCIPLINES PARTICIPATED IN DEVELOPING THE GUIDELINE DOCUMENT-THAT IS, A MULTI-DISCIPLINARY APPROACH WAS FOLLOWED. _____Yes (Go to Question 43.1) _____No (Go to Question 43.2) _____Don't know or can't tell (Go to Question 43.2) 43.1. ASSESSOR: Respond to Items 43a-43i, below, to determine whether the multi-disciplinary process is satisfactory; then answer Question 43.1 below, using your best judgment as to the overall rating for this element of multi-disciplinary process. Other factors you judge important should be specifically recorded under "Comments or Other Factors."

OCR for page 346
Guidelines for Clinical Practice: From Development to Use 43a. An explanation, discussion, or rationale for selecting the guideline panel chairperson is given. _____Yes _____No 43b. An explanation, discussion, or rationale for selecting the members of the guideline panel is given. _____Yes _____No 43c. An explanation, discussion, or rationale for selecting other individuals directly responsible for the guideline document (such as consultants) is given. _____Yes _____No 43d. The explanation(s), discussion(s), or rationale(s) for selecting the individuals covered in 43a-c is (are): _____Adequate _____Inadequate _____Not applicable 43e. These individuals reflect all appropriate interest groups and disciplines. _____Yes _____No _____Can't tell 43f. One or more outside review panel(s) commented on or reviewed draft guidelines. _____Yes _____No _____Can't tell 43g. One or more public hearing(s) or similar review mechanism(s) were held to allow comment or review on draft guidelines. _____Yes _____No _____Can't tell 43h. Collectively, the review panel(s), public hearing(s), or other review mechanisms reflected all appropriate interest groups and disciplines. _____Yes _____No _____Can't tell 43i. If the answer to either question 43e or question 43h is "No" or "Can't tell," please record what groups or disciplines appear to have been omitted.

OCR for page 346
Guidelines for Clinical Practice: From Development to Use Now answer: 43.1. THE MULTI-DISCIPLINARY APPROACH TO THE GUIDELINES DEVELOPMENT PROCESS IS: _____Satisfactory _____Conditionally _____Unsatisfactory satisfactory (Specify) Comments or Other Factors: 43.2. THE LACK (OR APPARENT LACK) OF A MULTIDISCIPLINARY PROCESS IS: _____Unimportant _____Minor _____Major omission omission Comments: 44. THE GUIDELINE DOCUMENT EXPLICITLY NOTES ANY POTENTIAL BIASES AND/OR CONFLICTS OF INTERESTS OF THE PANEL MEMBERS, OR STATES THAT BIASES AND CONFLICTS OF INTEREST WERE DISCUSSED AMONG PANEL MEMBERS OR OTHERWISE TAKEN INTO ACCOUNT. _____Yes, potential biases and/or conflicts of interest are noted _____Yes, a statement that biases and/or conflicts of interest were discussed is given _____No, no note or statement about biases and/or conflicts of interest is given

OCR for page 346
Guidelines for Clinical Practice: From Development to Use 45. OVERALL, POTENTIAL BIASES AND/OR CONFLICTS OF INTEREST APPEAR TO BE ADEQUATELY BALANCED OR OTHERWISE ACCOUNTED FOR IN THE GUIDELINE DEVELOPMENT PROCESS. _____Yes _____No _____Don't know or can't tell (Specify) Comments: 46. THE GUIDELINE DOCUMENT DESCRIBES THE METHODS USED TO SOLICIT VIEWS OF INTERESTED PARTIES NOT ON THE GUIDELINES DEVELOPMENT PANEL AND TO PRESENT THOSE VIEWS TO THE MEMBERS OF PANEL. _____Yes (Go to Question 46.1) _____No (Go to Question 46.2) 46.1. THE METHODS USED TO SOLICIT VIEWS OF THOSE NOT ON THE PANELS AND PRESENT THOSE VIEWS TO PANELS ARE: _____Satisfactory _____Conditionally _____Unsatisfactory satisfactory (Specify) Comments: 46.2. THE LACK OF A DESCRIPTION OF THE METHODS USED TO SOLICIT VIEWS OF THOSE NOT ON THE PANELS AND TO PRESENT THOSE VIEWS TO PANELS IS: _____Unimportant _____Minor _____Major omission omission Comments:

OCR for page 346
Guidelines for Clinical Practice: From Development to Use PLEASE RECORD ANY SUMMARY JUDGMENTS OR OTHER COMMENTS YOU MAY HAVE AND ANY RECOMMENDATIONS FOR ADDITIONAL REVIEW.

OCR for page 346
Guidelines for Clinical Practice: From Development to Use PART THREE. SUMMARY EVALUATION SHEET Instructions and Key ASSESSOR: Upon completing the entire assessment instrument, please record answers to the main questions (Questions 1-46) below. Circle the relevant answer, according to the following key: KEY Y = Yes; YQ = yes, but response qualified; N = No; NQ = no, but response qualified; S = Satisfactory, CS = Conditionally satisfactory, US = Unsatisfactory; UN = Unimportant, MI = Minor omission, MA = Major omission; NA = Not applicable DK = Don't know, or can't tell I. CLINICAL APPLICABILITY Y S UN N CS MI US MA 1 Description of patient population 1.1. Quality of description 1.2. Omission of description Y S UN N CS MI NA US MA 2 Discussion of complex clinical problems 2.1. Quality of discussion 2.2. Omission of discussion Y S UN N CS MI US MA 3. Rationale for excluding patient populations 3.1. Quality of rationale 3.2. Omission of rationale II. CLINICAL FLEXIBILITY Y S UN N CS MI US MA 4. Information about acceptable clinical exceptions 4.1. Quality of information or statement 4.2. Omission of information or statement Y S UN N CS MI US MA 5. Information about acceptable nonclinical exceptions 5.1. Quality of information or statement 5.2. Omission of information or statement

OCR for page 346
Guidelines for Clinical Practice: From Development to Use Y S UN N CS MI US MA 6. Discussion of patient preferences in the health care decisions 6.1. Quality of discussion  6.2. Omission of discussion Y S UN N CS MI US MA 7. Discussion of patient preferences in guideline development 7.1. Quality of discussion  7.2. Omission of discussion III. RELIABILITY/REPRODUCIBILITY Y S N CS US 8. Independent review by experts or outside panels  8.1. Quality of discussion Y S UN N CS MI US MA 9. Explanation of lack of independent review  9.1. Quality of explanation  9.2. Omission of explanation Y S N CS US 10. Guidelines pretested in some manner  10.1. Quality of discussion Y S UN N CS MI US MA 11. Explanation of lack of pretesting  11.1. Quality of explanation  11.2. Omission of explanation IV. VALIDITY STRENGTH OF THE SCIENTIFIC EVIDENCE AND PROFESSIONAL CONSENSUS Y S UN N CS MI US MA 12. Method of collecting (identifying and retrieving) scientific evidence is specifically described 12.1. Quality of method  12.2. Lack of method Y N   13. Adequate references to sources of scientific evidence Y N   14. General discussion of strength of scientific evidence

OCR for page 346
Guidelines for Clinical Practice: From Development to Use Y S UN N CS MI US MA 15. Explicit rating of the strength of the scientific evidence 15.1. Quality of rating method  15.2. Lack of general discussion of rating method Y S UN N CS MI NQ US MA 16. If a formal method of synthesis is used, explicit description of the method 16.1. Quality of formal method  16.2. Omission of description of formal method Y S UN N CS MI US MA 17. If applicable, the results of a formal synthesis of scientific evidence are explicitly reported 17.1. Quality of results of the synthesis  17.2. Omission of results of the synthesis Y N NQ 18. If applicable, the expert or group judgment techniques used for reaching professional consensus are explicitly described S UN CS MI US MA   18.1. Quality of expert or group judgment techniques  18.2. Omission of description of expert or group judgment techniques Y N   19 If applicable, the strength of professional consensus resulting from use of group judgment techniques is reported S UN CS MI US MA   19.1. Quality of information about strength of professional consensus  19.2. Omission of explicit information about strength of professional consensus HEALTH BENEFITS AND HARMS/RISKS: QUALITATIVE DESCRIPTION Y S UN N CS MI   US MA 20. Qualitative description of health benefits  20.1. Quality of qualitative description 20.2. Omission of qualitative description Y S UN N CS MI US MA 21. Qualitative description of potential harms or risks  21.1. Quality of qualitative description  21.2. Omission of qualitative description

OCR for page 346
Guidelines for Clinical Practice: From Development to Use HEALTH BENEFITS AND HARMS/RISKS: QUANTITATIVE INFORMATION Y S UN N CS MI US MA 22. Quantitative information or estimates of health benefits 22.1. Quality of quantitative information 22.2. Omission of quantitative information Y N NA 23. Health benefits projected in terms of life expectancy or similar measures harms or risks Y S UN N CS MI US MA 24. Quantitative information or estimates of potential 24.1. Quality of quantitative information 24.2. Omission of quantitative information HEALTH COSTS: QUALITATIVE DESCRIPTION Y S UN N CS MI US MA 25. Qualitative description of health costs or expenditures 25.1 Quality of qualitative description 25.2. Omission of qualitative description HEALTH COSTS: QUANTITATIVE DESCRIPTION Y S UN N CS MI US MA 26. Quantitative information or estimates of health costs or expenditures 26.1. Quality of quantitative information 26.2. Omission of quantitative information Y N NA 27. If health benefits projected in terms of life expectancy or similar measures, costs per unit of each identified benefit also estimated Y YQ N   28. Generally, estimates of benefits, harms, and costs are consistent with the strength of provided evidence Y N   29. Major recommendations made in the guideline Y N   30. Discussion of strength of the scientific evidence for each major recommendation

OCR for page 346
Guidelines for Clinical Practice: From Development to Use S UN C S MI US MA   30.1. Quality of discussion 30.2. Omission of discussion Y YQ N 31. Each major recommendation consistent with strength of scientific evidence POTENTIAL CONFLICT AMONG SIMILAR SETS OF GUIDELINES Y N NA 32. Other sets of guidelines identified Y S UN N CS MI US MA 33. Possible conflicts among existing guidelines discussed 33.1. Quality of discussion 33.2. Omission of discussion V. CLARITY LANGUAGE AND TERMS Y N   34. Language describing the health condition is unambiguous Y N   35. Language describing the options for management is unambiguous Y N NA 36. Language for each major recommendation is unambiguous LOGIC Y Y N N   37. Recommendations are comprehensive and present when expected 37.1. Reasons given for lack of expected recommendations Y N NA 38. Recommendations are consistent STRUCTURAL CLARITY Y N   39. Guideline document uses clear headings, indexes, etc. Y N   40. Guideline document has accurate summary or abstract Y N NA 41. Users can find recommendations easily

OCR for page 346
Guidelines for Clinical Practice: From Development to Use VI. SCHEDULED REVIEW: DEFINITION AND EVALUATION QUESTIONS Y S UN N CS MI US MA 42. Scheduled date for review or a procedure for arriving at such a date is provided 42.1. Quality of the scheduled review date or procedure for setting one 42.2. Lack of a scheduled review date or procedure for setting one VII. MULTI-DISCIPLINARY PROCESS Y S UN N CS MI DK US MA 43. Participation of persons in appropriate clinical and methodologic disciplines 43.1. Quality of the multi-disciplinary approach 43.2. Lack of a multi-disciplinary process Y YQ N 44. Guideline document notes potential biases or conflicts of interest or indicates they were taken into account Y N DK 45. Balance of potential biases or conflicts of interest Y S UN N CS MI   US MA 46. Description of the methods used to solicit views of those not on the guidelines development panel and to present those views to the panel 46.1. Quality of methods used 46.2. Lack of a description of methods used SUMMARY JUDGMENT, OTHER COMMENTS, OR NEED FOR ADDITIONAL REVIEW: