D
Decision Conferencing and Multicriteria Decision Analysis
Benefit–risk assessment characterizes information regarding estimates of benefits, estimates of risks, and the severity and comparability among health endpoints associated with benefits and risks. Much has been published about methods for quantitative benefit–risk assessment (Coplan et al., 2011; Guo et al., 2010). Such assessments are widely used in decision-making contexts, particularly with regard to environmental regulation, and the mode of their particular application varies (NRC, 1994, 2009). The committee does not wish to prescribe a particular method for conducting benefit–risk assessment, particularly inasmuch as formal quantitative approaches are likely to be used only in select circumstances when disagreements about potential regulatory actions arise. Instead, the committee highlights in this appendix two decision tools that incorporate key considerations of benefit–risk assessment relevant to regulatory decision-making: decision conferencing, which is a social process intended to engage all the relevant stakeholders to provide scientific judgments at key points in the decision-making process, and multicriteria decision analysis (MCDA), which is a technical model for making decisions that have multiple objectives (Walker et al., 2005). Decision conferencing and MCDA have been used in other contexts and are offered here as examples of potentially useful approaches that integrate analytic and deliberative processes for in-depth evaluation and informed assessments of the benefit–risk balance of approved drugs (Phillips, 2006; Walker et al., 2005). Both processes, when used in a transparent way that documents the inputs into the process, can help to identify the underlying sources of scientific disagreements that are discussed in Chapter 3.
DECISION CONFERENCING
Decision conferencing is a tool that brings key experts and stakeholders together to generate a shared understanding of a challenge, create a sense of common purpose, and gain commitment to a way forward (Phillips, 2006) to aid evaluation and assessment of the benefits and risks associated with a drug (Walker et al., 2005). Decision conferencing has four basic elements: attendance by key stakeholders (for example, regulators; methodologists who have expertise in design, conduct, and analysis of observational studies and clinical trials; decision scientists; physicians who have relevant clinical expertise; patients; and the public); impartial facilitation to guide discussions of estimates of benefit and risk, degree of uncertainty, and values and preferences for health endpoints; on-the-spot modeling with continuous display of the developing model; and an interactive and iterative group process (Phillips, 2006). When quantitative benefit–risk assessments are needed in situations in which disagreements about appropriate regulatory action arise, a neutral facilitator can guide the group through the stages of discussing the issues, developing models for evaluating the issues, and eliciting assumptions about the quality of evidence regarding benefits and risks and about underlying ethical values and preferences for health endpoints without contributing to the content of discussions. Although quantitative estimates resulting from benefit–risk assessment may appear to provide objective information about optimal regulatory decisions, assumptions used in the model are often based on individual judgments about the quality of evidence related to benefits and risks, as discussed in Chapter 3, or based on individual values or preferences for different health endpoints. Using a process like decision conferencing helps to frame the issues and identify the relevant data and evidentiary gaps to guide later data-gathering and thereby improves the efficiency and transparency of the benefit–risk assessment process (Phillips, 2006).
The decision-conferencing process could be integrated into FDA’s current processes and requirements under the Federal Advisory Committee Act,1 and it is similar to initiatives that FDA currently has planned. For example, FDA is working with stakeholders, using faculty of the George Washington University as facilitators, to develop guidance material for approving obesity drugs (McCaughan, 2011). Decision conferencing in benefit–risk assessment has four advantages:
• It facilitates and focuses thinking about a complex challenge.
• It helps to establish common purpose among participants and commitment to move forward.
• It promotes transparency and a shared understanding of how stakeholders define benefits, risks, degree of uncertainty, ethical values regarding outcomes, and potential regulatory actions for managing risk.
1Federal Advisory Committee Act, PL 920-463, 86 Stat. 770 (1972).
• Because decision conferencing can generate both the outcome and a description of the process that leads to the outcome, it can improve stakeholders’ and the public’s understanding of a regulatory decision.
MULTICRITERIA DECISION ANALYSIS
MCDA is a set of methods designed to bring together evaluations of options on different criteria into one overall decision (CHMP, 2008). There are many variants of MCDA, some of which may be adapted to consider the uncertainty of the decision-maker (Linkov and Seager, 2011), and MCDA can be used to provide either a qualitative or a quantitative assessment. The basic methods of MCDA are scoring and weighting (CHMP, 2008). Scoring involves the process of assigning numerical values to options according to particular criteria. Weighting ensures the comparability of the numerical values assigned to all criteria, which allows comparison of different health states with a single metric (Linkov and Seager, 2011). The weights assigned to scores reflect the relative importance of the underlying criteria for the benefit–risk assessment outcome (Walker et al., 2005).
MCDA uses four steps (Linkov and Seager, 2011):
1. Defining the problem and the decision context.
2. Identifying stakeholders, decision-makers, assessment criteria (for example, health outcomes of interest), and the relative importance of different health outcomes.
3. Defining and assessing management alternatives whereby the effects of different regulatory decisions on each criterion, or health outcome, are assessed.
4. Allowing for variability in weighting of different criteria and accounting for the stochastic nature of data through the use of probabilistic sensitivity analysis to provide a rank order of different alternatives for distinct stakeholder groups.
Those four steps help to ensure that all participants understand and are in agreement about the need for a regulatory decision, the criteria by which benefit–risk balance is judged, the evidence and its uncertainties, the values and preferences of different stakeholders, and the consequences of different regulatory decisions. The outputs or information synthesis of the benefit–risk assessment stage of the framework should include model inputs and model outputs (Linkov and Seager, 2011). The model inputs would include estimates of benefits, estimates of risks, the degree of uncertainty, and preferences for health outcomes based on ethical values. The model outputs may be characterized quantitatively, for example, the benefits of a drug outweigh the risks 85 percent of the time; or qualitatively, for example, there is clear and convincing evidence that the benefits of a drug outweigh its risks. The synthesis should also discuss any uncertainty
analyses that were conducted and the process by which the benefit–risk assessment was performed—that is, a description of the “decision conferencing” or other process that FDA used to seek and include stakeholder input as necessary. Useful information includes statements of who provided the inputs and who moderated the process. Both the process and the outcome should be documented as a way to set the stage for understanding the regulatory decision. The outcome of the assessment process, whether quantitative or qualitative, becomes the evidentiary basis of the regulatory decision-making that is at the heart of the next stage—benefit–risk management.
LIMITATIONS OF USING MODELING APPROACHES FOR BENEFIT AND RISK ASSESSMENTS
MCDA is a useful tool for benefit–risk assessment, but it has its limitations, both in its own right and as an aid to regulatory decision-making. Using a quantitative MCDA approach to benefit–risk assessment forces participants to be explicit about how they evaluate existing evidence regarding the effectiveness of a drug and its associated harms and about the degree of uncertainty regarding benefits and risks. Such a quantitative assessment, however, can obscure underlying interpretations of scientific findings. The interpretations should be explicitly described, and decision conferencing can mitigate some concerns by describing differences in underlying assumptions. In addition, quantification of intangible factors, such as a patient’s preferences that might be based on various degrees of dread for different diseases, may be difficult in MCDA models although relevant for the decision-making process.
Preferences regarding the relative importance and severity of health states associated with the disease or the treatment may also vary widely among stakeholders and could potentially be obscured by using modeling approaches. However, decision conferencing can serve as a useful tool for conducting MCDA by explicitly describing stakeholders’ values and preferences and how they may affect regulatory decisions.
Even with the aid of social and technical decision tools, benefit–risk assessment is unavoidably limited by the quality and quantity of available evidence. Uncertainty analysis may identify the key information needed to support a particular regulatory action, but such data may not be available in the time needed to address the public health issue at hand. The available data may lead to multiple interpretations and contribute to decision-making gridlock. That delay could compound limitations of the resources and expertise available in FDA to conduct benefit–risk assessments and result in a backlog. Such risk-assessment backlogs have occurred in the Environmental Protection Agency (NRC, 2009). If tools like decision conferencing and MCDA are used to enhance transparency, the rationale for regulatory decisions can at least be understood by stakeholders even if disagreements about optimal regulatory action remain. The transparency
allows stakeholders to have a shared understanding of differences in scientific assessment of preferences regarding health outcomes, which can help to determine whether additional data are needed and whether those data will meet thresholds for influencing future regulatory action. Decision conferencing and MCDA, however, may not be appropriate for every situation, and benefit–risk assessments should be scalable to the severity and scope of the particular public health concern at issue, the level of controversy surrounding it, and FDA resource constraints.
REFERENCES
CHMP (Committee for Medical Products for Human Use). 2008. Reflection paper on benefit-risk assessment methods in the context of the evaluation of marketing authorisation applications of medicinal products for human use. London, UK: European Medicines Agency.
Coplan, P. M., R. A. Noel, B. S. Levitan, J. Ferguson, and F. Mussen. 2011. Development of a framework for enhancing the transparency, reproducibility and communication of the benefit-risk balance of medicines. Clinical Pharmacology & Therapeutics 89(2):312-315.
Guo, J. J., S. Pandey, J. Doyle, B. Bian, Y. Lis, and D. W. Raisch. 2010. A review of quantitative risk-benefit methodologies for assessing drug safety and efficacy-report of the ISPOR risk-benefit management working group. Value Health 13(5):657-666.
Linkov, I., and T. P. Seager. 2011. Coupling multi-criteria decision analysis, life-cycle assessment, and risk assessment for emerging threats. Environmental Science and Technology 45(12):5068-5074.
McCaughan, M. 2011. FDA rethinking weight loss standards: A test case for patient-focused drug development. “The Pink Sheet” Daily, June 6, 2011.
NRC (National Research Council). 1994. Science and judgment in risk assessment. Washington, DC: National Academy Press.
NRC. 2009. Science and decisions: Advancing risk assessment. Washington, DC: The National Academies Press.
Phillips, L. D. 2006. Decision conferencing. Operational research working papers, LSEOR 06.85. London, UK: Operational Research Group, Department of Management, London School of Economics and Political Science.
Walker, S., L. Phillips, and M. Cone. 2005 (unpublished). Benefit-risk assessment model for medicines: Developing a structured approach to decision making.
This page intentionally left blank.