2

Modeling Strategy: From Single
Attribute to Multiple Attributes

The vaccine prioritization techniques of the earlier Institute of Medicine (IOM) studies published in 1985–1986 and 2000 relied on two criteria: (1) reduction of health burden (IOM, 1985, 1986) and (2) incremental cost or savings (IOM, 2000) due to use of the vaccine in a defined population. More specifically, the 1985–1986 work used only a single attribute—infant deaths averted—for ranking vaccine candidates; it did not consider cost attributes. The 2000 report used an approach based on cost-effectiveness to prioritize vaccines.

Those studies saw the central “modeling task” as numerical estimation of the expected costs and benefits of the vaccines. The principles underlying this approach derive from the economic theory of social welfare as implemented in the classic utility frameworks (Garber and Phelps, 1997). The computational models were the key contributions of the 1985–1986 and 2000 reports. Their work involved many decisions concerning which costs and savings to include and how best to measure health gains.

New Vaccine Development (1985–1986) and
Vaccines for the 21st Century (2000)

The 1985–1986 report measured health benefits using infant mortality equivalents (IMEs), which involved subjective judgments relating to morbidity and mortality reductions compared to an equivalent number of infant deaths averted. Since the time that report was published, analytical techniques have advanced. Standardized measures of health-related qual-



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 27
2 Modeling Strategy: From Single Attribute to Multiple Attributes The vaccine prioritization techniques of the earlier Institute of Medicine (IOM) studies published in 1985–1986 and 2000 relied on two criteria: (1) reduction of health burden (IOM, 1985, 1986) and (2) incremental cost or savings (IOM, 2000) due to use of the vaccine in a defined population. More specifically, the 1985–1986 work used only a single attribute—infant deaths averted—for ranking vaccine candidates; it did not consider cost attributes. The 2000 report used an approach based on cost-effectiveness to prioritize vaccines. Those studies saw the central “modeling task” as numerical esti- mation of the expected costs and benefits of the vaccines. The principles underlying this approach derive from the economic theory of social wel- fare as implemented in the classic utility frameworks (Garber and Phelps, 1997). The computational models were the key contributions of the 1985– 1986 and 2000 reports. Their work involved many decisions concerning which costs and savings to include and how best to measure health gains. New Vaccine Development (1985–1986) and Vaccines for the 21st Century (2000) The 1985–1986 report measured health benefits using infant mortal- ity equivalents (IMEs), which involved subjective judgments relating to morbidity and mortality reductions compared to an equivalent number of infant deaths averted. Since the time that report was published, analytical techniques have advanced. Standardized measures of health-related qual- 27

OCR for page 27
28 RANKING VACCINES: A Prioritization Framework ity of life (HRQOL) such as the Health Utilities Index Mark 2, or HUI2—a tool to measure morbidity reduction—have been developed using methods of multi-attribute utility theory (Feeny et al., 1996). HUI2 has been com- bined with actuarial measures of life expectancy changes in order to com- pute quality-adjusted life years (QALYs) as one of the main health valuation measures. To derive its vaccine priorities, the 2000 report relied on incremen- tal dollar costs per incremental QALY gained ($/QALY) for both preven- tive and therapeutic vaccines that are of importance to the United States. In the nearly three decades since the 1985–1986 report was published, the theoretical basis for its calculations has not changed. By contrast, in the years since the 2000 report, the methods of cost-effectiveness analysis have become somewhat more sophisticated when it comes to assessing the effectiveness of $/QALY values for health care technologies. The self-reported health status data needed for population-based measures such as HUI2 are not available in much of the world. Instead, researchers at the World Health Organization in collaboration with researchers at other institutions developed a similar tool: disability- adjusted life years (DALYs). In calculating DALYs, disability weights are assigned to typical manifestations of a wide variety of diseases; such mea- sures have been used for many countries around the world (Fox-Rushby and Hanson, 2001; Gold et al., 2002; Murray and Lopez, 2000). Methods to incorporate uncertainties in decision models were undergoing rapid development at the time of the 2000 report. They have since progressed and become more generally applicable (Fenwick et al., 2001; Meckley et al., 2010). There have also been advances in population- based data collection supporting HUI2 and similar indexes of generic health-related quality of life that the 2000 report incorporated (Fryback et al., 2007, 2010; Luo et al., 2005, 2009). In recent years, advances in complex systems modeling have helped characterize the nature and spread of infections in populations. These dynamical techniques can now be used for estimating the impact of a new vaccine for a specific population (e.g., Epstein et al., 2008). But the under- lying decision framework and conceptual approaches to estimating costs and health benefits have essentially remained unchanged. The previous reports developed a computational model based on two important (but distinctly different) attributes for prioritizing vaccines, although more sophisticated methods could have been used. The main crit- icism of the 2000 report was related to the basic framework itself: the sys- tem was too limited and considered only costs and aggregated health benefits (e.g., see Plotkin et al., 2000).

OCR for page 27
29 Modeling Strategy: From Single Attribute to Multiple Attributes Modeling beyond cost-effectiveness The committee revisited the assumptions and limitations of the 1985– 1986 and 2000 approaches. Instead of taking the path of developing a de novo computational model, the committee chose to significantly expand the previous IOM works by using a multi-attribute utility framework and develop a novel software application. In this work, therefore, some aggre- gate measure of health benefits (such as infant deaths averted) or an effi- ciency criterion (such as cost-effectiveness) has simply become one among the many criteria—rather than the only criterion—that influence vaccine prioritization. The committee took on the task of expanding the list of attributes characterizing vaccine candidates and developing a prototype software— SMART Vaccines Beta—to weigh not only economic and health attributes but also demographic, scientific, business, programmatic (field-level logis- tics), social, and policy aspects relating to new vaccine development. The short-listing of 29 attributes used in SMART Vaccines Beta was informed by stakeholder and concept evaluator feedback, committee discussions, and literature review (Burchett et al., 2011). Values and objectives in priority setting Priority setting means assigning values and objectives. If the main objec- tive of a new preventive vaccine is to minimize the disease burden in the target population, then assuming that all else is equal, the highest priority typically would be given to the vaccine candidate expected to produce the largest health benefit compared to other candidates, and a set of vaccine candidates would be prioritized according to their expected health ben- efits, going from most expected benefits to least. But all else is not equal. Priorities must also reflect such consider- ations as the fact that resources are constrained. Such a limited-resources constraint points to a different objective: to minimize the costs associated with bringing a vaccine to licensure and then administering it in the tar- get population. If minimizing costs is the main objective, then the program with the lowest development and implementation costs would be favored, and priorities would simply be ordered according to the increasing costs of the different programs. These two objectives—maximizing health benefits and minimizing costs—are often in conflict. One vaccine candidate may potentially have a very large aggregate impact on health burden but also have greater expected costs than a vaccine addressing a different disease where the effect on health burden may be smaller.

OCR for page 27
30 RANKING VACCINES: A Prioritization Framework When objectives are in conflict, decision makers often deal with trade-offs. In this case, each vaccine candidate is associated with expected health benefits and costs. Expressing a priority order among candidates requires us to weigh the extent to which each vaccine candidate achieves the two objectives jointly, perhaps preferring one objective over the other. In this case, cost-effectiveness analysis is appropriate and may be used to prioritize vaccine candidates when there are trade-offs between these two important attributes. But several other objectives could also influence the ranking of vac- cine candidates under consideration. These objectives depend on whose priorities are being expressed toward maximizing the overall value of the vaccines. For example, decision makers may want to represent a public desire to minimize the burden of disease in specific target populations such as women, infants, and children; the socioeconomically disadvantaged; or military personnel. There may be certain diseases that raise special con- cerns or fear in the public mind—for example, a rare but particularly grue- some condition, an unrelenting infection, or a terribly disfiguring disease. Extra priority may be given to a vaccine that prevents such a disease, esca- lating its priority despite high costs or a relatively small aggregate health burden imposed by the disease in the population compared to a vaccine preventing a condition that is more common but that has a relatively minor health burden. Other objectives are also possible. One might wish, for instance, to maximize the benefit to future generations by investing in a vaccine that could eliminate a particular disease altogether or mitigate its epidemic potential. Similarly, one might wish to prioritize a vaccine that has the potential to significantly advance the scientific base, including new pro- duction, preservation, and delivery methods. A prioritization exercise starts with a set of vaccine candidates, each of which is expected to meet, to a greater or lesser degree, a number of desired objectives. The basic purpose of prioritization is to place these can- didates in order from “most preferred” to “least preferred” in accordance with values held by or represented in proxy by the decision maker. The methods used to accomplish this task in a rigorous fashion fall generally under the rubric “multi-criteria decision making.” Multi-criteria decision-making methods From the family of multi-criteria decision-making models, the commit- tee chose to use a version of multi-attribute utility theory. As a starting point, the committee limited the models under consideration to those

OCR for page 27
31 Modeling Strategy: From Single Attribute to Multiple Attributes that included multiple attributes. The committee heard from a number of stakeholders that the narrow range of attributes used to rank vaccine pri- orities in previous IOM studies significantly limited their value and appli- cations. Thus, the committee reviewed three multiple-attribute modeling approaches (listed in the order of historical development): (1) mathemati- cal programming (or optimization), (2) multi-attribute utility theory, and (3) analytical hierarchy process. The approaches were evaluated against four criteria: axiomatic foundation; priority scaling; sensitivity analysis; and transparency. Axiomatic foundation Multi-attribute utility theory and mathematical programming are based on axiomatic theory—the former being derived from principles of utility maxi- mization (Krantz et al., 1971), and the latter being based on mathematical optimization. The analytical hierarchy process has an axiomatic base that the committee considered incomplete. To elaborate, the issue of indepen- dence from irrelevant alternatives (IIA) was of particular importance to the committee’s considerations. IIA means the following: Given a particu- lar set of options (candidate vaccines) in which candidate A is preferred to candidate B, if an additional candidate C—unrelated to A and B—is added to the option set, then A continues to be preferred over B. Consider, for example, a comparison of vaccines to prevent tubercu- losis and malaria, ranked with one preferred to the other. Now suppose that the science and technology evolves to allow a new vaccine against dengue fever. IIA would mean that the ranking of vaccine candidates for tuber- culosis and malaria remains unchanged when the dengue fever vaccine is added to the mix for consideration. The new dengue fever vaccine may be more or less preferred than either tuberculosis or malaria or both vaccines, but the rankings of tuberculosis and malaria vaccines with respect to each other must remain unchanged. Since the appearance of new candidate vac- cines can be anticipated over time, the committee concluded that IIA was particularly important to consider. Priority scaling The 1985–1986 and 2000 IOM reports relating to vaccine prioritization and the international stakeholder testimonies made it very clear that this com- mittee’s work would need to offer greater value in terms of allowing differ- ent users to apply their individual preferences in a prioritization model. The committee defines the term “prioritize” consistently with the stan-

OCR for page 27
32 RANKING VACCINES: A Prioritization Framework dard dictionary definition “to arrange in the order of relative importance.” Thus, prioritization at a minimum requires an ordinal ranking and nothing more—simply stating an order of preference. The three modeling methods considered by the committee all provide additional information beyond an ordinal scale—either interval or ratio scale numbers assigned to vaccine candidates to represent relative priority. To use an analogy relating to temperature measurement, with inter- val scales the difference between two values has the same meaning at dif- ferent points along the scale. For example, the difference between 20°C and 40°C has the same meaning as the difference between 30°C and 50°C. But 40°C is not twice as hot as 20°C. Ratio scales also provide informa- tion about relative values, thus requiring identification of true “zero” on the scale. Kelvin temperature allows for this: 300K is twice as hot as 150K, whereas statements about ratios of temperatures are incorrect in either °C or °F scales—but ratios of differences in temperatures are the same on K, °C, and °F scales. Since only ordinal ranking is required in prioritization, any modeling approach providing interval or ratio scaling is sufficient. Sensitivity analysis The committee also wanted to allow users to conduct sensitivity analysis on their results. This sensitivity analysis has several purposes, including (a) enhancing understanding of the inputs to which the results were most sensitive, (b) pointing toward areas where improved data have the greatest value, and hence potentially (c) spurring efforts and investments in data generation. All three modeling approaches had the capability for ably sup- porting sensitivity analyses. Transparency Another important criteria for the committee was transparency. In the committee’s view, the multi-attribute utility approach was more transpar- ent than other possible approaches. In mathematical programming, for example, one could subtly alter the constraint set (in ways very difficult for others to see) so as to eliminate some candidates from the solution set in favor of others, or else modify the way the objective function was specified. In analytic hierarchy process, the value weights emerge only after a long series of pair-wise comparisons have been recorded and modified through normalization processes involving complex matrix manipulations. By con- trast, in multi-attribute utility theory the weights and data are available for everybody to see and use. In that regard, multi-attribute utility theory was

OCR for page 27
33 Modeling Strategy: From Single Attribute to Multiple Attributes found to be the best fit for satisfying the transparency requirement. Indeed, the committee saw this as a strength of the SMART Vaccines, highlighting its potential in promoting cross-comparison of different users’ rationale and conclusions and leading to more informed discussions about priorities among different stakeholders. Each modeling alternative is summarized in the following sections. Mathematical programming or optimization Mathematical programming (linear programming, nonlinear program- ming, stochastic programming, and more complex optimization algo- rithms) has been widely and successfully employed in many areas to tackle complex challenges. In concept, mathematical programming is an appro- priate method for vaccine prioritization. Its optimization characteristics are well understood (Rardin, 1997). In various formulations, it can pro- vide output of at least ordinal nature (ranking) and, in many formulations, interval or ratio scale output, and software to carry out such calculations is widely available in numerous commercial and free-ware environments. It is also amenable to sensitivity analyses. The primary uses of mathematical programming involve optimi- zation of some value function (specified by the user) subject to a set of constraints which are often highly complex and frequently nonlinear. In classical linear and nonlinear programming, the values of relevant compo- nents of the model are known (e.g., cost, consumer preferences, and other factors). Stochastic programming emerged to provide optimization tools when uncertainty exists about certain components of the system under consideration. But, in general, the value of mathematical programming appears when there are many possible solutions (perhaps an infinite num- ber) within the constraint set. Prioritization of vaccines differs considerably from the usual uses of mathematical programming. Typically, only a small number of alterna- tives are considered in the set of potential vaccines (dozens, perhaps, but seldom hundreds, almost never thousands, and certainly not an infinite set of options). Separately, unless a customized stochastic programming method or some equivalent method is developed and used, the dearth of data in regards to new vaccines problem would likely render the optimi- zation capabilities of mathematical programming questionable for the application. Another issue also deterred the full consideration of mathemati- cal programming for vaccine prioritization: Mathematical programming requires a pre-specification of the value function. This is a crucial issue,

OCR for page 27
34 RANKING VACCINES: A Prioritization Framework since many users and stakeholders would not be able to competently spec- ify a value function for such reasons as a lack of a quantitative background. Furthermore, there are no well-developed and tested methods for value elicitation associated with mathematical programming methods. Analytic hierarchy process The analytic hierarchy process has many desirable attributes. It is widely used by people in business and other settings to assist in decision making, often under the tutelage of professional consultants. It provides a ratio- scale value function, which is more than sufficient for the committee’s ranking process. It has a well-developed process for eliciting values from users, based on a large set of pair-wise comparisons of different alterna- tives along the various attribute dimensions. The user must make a sizeable number (typically in the hundreds) of paired comparison assessments. For each pair of candidates (e.g., vaccines) A and B, and for each attribute, xj, the decision maker rates the comparison of xaj versus xbj using a scale of 9, 7, 5, 3, 1, 1/3, 1/5, 1/7, 1/9 to describe how much better A is than B on that attribute, where the numbers are meant to convey a ratio scale of relative performance. Although, in principle, any user can program the calculations necessary for deriving priorities1 from an analytic hierarchy process, most analysts use one of a number of proprietary software packages currently available. These packages lead users through the necessary steps and pro- vide internal consistency checks for many of the comparative assessments. Besides the complexity associated with value elicitation process, two other features make this analytic hierarchy process less friendly for vaccine prioritization. Perhaps most important, the analytic hierarchy process does not maintain IIA, a fact that is widely understood among both proponents and opponents of this method (Dyer, 1990; Saaty, 1987). Proponents of ana- lytic hierarchy process cite this as a beneficial feature, noting that many real world decisions also do not have IIA. But the committee, for reasons stated previously, views IIA as a critical factor in vaccine prioritization. Among the users of analytic hierarchy process the word “priority” has a specific techni- 1 cal meaning (relating to a normalized eigenvector used in the model) that does not match the standard definition of priority mentioned earlier and used in this report. Thus, one should not confuse the specific analytic hierarchy process definition of priority with the one used by the committee.

OCR for page 27
35 Modeling Strategy: From Single Attribute to Multiple Attributes Multi-attribute utility theory A multi-attribute utility-based prioritization exercise consists of several steps. First, the set of vaccine candidates to be considered must be iden- tified. Next, a set of objectives that underpin the valuation of candidates must be listed. For each objective there must be a specific measure—called an “attribute”—developed. The attributes may be natural scales (such as expected net present value of annualized dollar costs or savings), well- established indexes (such as net annualized increase in QALYs due to the vaccine), or customized categorical scales. If we denote each candidate vaccine by xi, then the outcome attri- butes characterizing that vaccine may be viewed as a vector, ci = (xi1, xi2, . . . , xin), where n is the number of attributes being considered when setting priorities, and xij is the value of the scale for the jth attribute for the ith vaccine candidate. Multi-attribute utility models can combine attributes of each type, whether continuous or categorical. Keeney and Raiffa (1976) as well as a number of others (Barron and Barrett, 1996; Edwards and Barron, 1994; Edwards and Newman, 1982; von Winterfeldt and Edwards, 1986) have described methods to specify n single-attribute functions, 0 ≤ uj(xj) ≤ 1, and a global utility function, U(ci) = f(u1(xi1), u2(xi2), . . . , un(xin)), such that 0 ≤ U(ci) ≤ 1. The function U is con- structed so that ca is preferred to cb if and only if U(ca) > U(cb). Often the function f is additive, U(ci) = w1 u1(xi1) + w2 u2(xi2) + . . . + wn un(xin), where the wjs are constants that sum to 1. The ratios wj/wk reflect the change in value achieved by changing the jth attribute from its min- imum to maximum level in the set of vaccine candidates versus making the corresponding change in the kth attribute. Although there are strong arguments for using an additive function as a first approximation (Edwards and Barron, 1994; Keeney and von Winterfeldt, 2007; von Winterfeldt and Edwards, 1986), in some cases a multiplicative function or multi-linear function might be more appropriate in order to account for interactions among the attributes based on user preferences (Keeney and Raiffa, 1976). Additive functions are often satisfactory for broad policy purposes. The committee employs an additive version of multi-attribute utility method in SMART Vaccines Beta. Determining what weights (w1, w2, . . . , wn) to use is a separate prob- lem from that of choosing the functional form (e.g., additive or multipli- cative). Edwards and Barron (1994) proposed a method to approximate the wjs using the decision maker’s rank order of the relative importance of the attributes. In particular, they proposed using the rank order centroid method to derive weights for a set of attributes, a method that was later extensively evaluated by Barron and Barrett (1996).

OCR for page 27
36 RANKING VACCINES: A Prioritization Framework The rank order centroid approximation The decision maker’s major input is to produce a rank order of the relative importance of the attributes in order to differentiate the priority of the vac- cine candidates. This induces a rank order on the weights in the additive model. Suppose that the rank order is w1 ≥ w2 ≥ … ≥ wn for n attributes. The rank order centroid approximation for the constants in an additive model would then be as follows:  11 1  1 + 2 + 3 + ... + n  w1 =   n      1 11  0 + 2 + 3 + ... + n  w2 =   n      1 1 0 + 0 + + ... +   3 n w3 =   n      1  0 + 0 + 0 + ... + n  wn =   n     More compactly the weights can be expressed by 1 n wi = ∑ i = 1n j =i j Barron and Barrett showed this rank order centroid approximation for weights to be superior to other often-proposed methods, such as the normalized sum of ranks. It is important to realize that rank order centroid weights are not essential to the multi-attribute utility models; rather they are an approximation used to reduce the workload of the potential user. In SMART Vaccines Beta, the rank order centroid-based weighting approach was employed in order to speed up development of other parts of the model. In many policy settings using multi-attribute utility theory, these weights are developed with experts guiding the process of decision makers elucidating their preferred weights (Keeney and Raiffa, 1976; von Winterfeld and Edwards, 1986). The multi-attribute decision techniques (or related proprietary soft- ware packages) have been used in practical applications in a number of

OCR for page 27
37 Modeling Strategy: From Single Attribute to Multiple Attributes public policy settings, including to evaluate alternative plans to desegregate schools (Edwards, 1979), to plan wastewater treatment facilities (Keeney et al., 1996), to evaluate accounting regulations for control of nuclear materi- als (Keeney and Smith, 1982), and to evaluate homeland security decisions (Keeney and von Winterfeld, 2011). Additional applications have been reviewed by Keefer and colleagues (2004). Data demands The multi-attribute utility approach places considerable data demands on users. The committee continually sought to balance the model’s capabili- ties and complexity with the data demands it would place on users. The challenge, however, spans every approach considered by the committee. It is intrinsic not to the multi-attribute utility approach itself, but rather to the underlying complexity of prioritization and how to model it. Had mathematical programming or analytic hierarchy process been adopted, a level of data demands similar to those in the multi-attribute utility theory would have been required. The only way to reduce data demands is to have limited capabilities in SMART Vaccines. A parallel issue relates to how the necessary data must be structured. In the committee’s view, the data inputs necessary for the multi-attribute approach are at least as simple—and often simpler—for users to understand than would be the case in alternative models. For example, many formula- tions of mathematical programming have inequality constraints, a concept that could seem alien to many potential users of our software. The modeling framework for SMART Vaccines Beta Multi-attribute utility theory provides the analytical framework that underpins the committee’s work, and the specific model within this frame- work is an additive multi-attribute utility model. A schematic diagram of the model’s organization is presented in Figure 2-1. Within the multi-attribute utility framework, a vaccine candidate is viewed as a means to achieve an end in a specified population. The various objectives that the development and delivery of a new vaccine may address include • enhancing public health by reducing the burden due to a particular disease or condition; • minimizing the societal costs of the disease, and its prevention and treatment;

OCR for page 27
56 RANKING VACCINES: A Prioritization Framework In SMART Vaccines Beta, the weights are computed from a strict rank order of attributes supplied by the user. In future versions, the com- mittee expects that this approximation will be replaced by a more elabo- rate elicitation of weights, perhaps using a hierarchical clustering of the selected attributes, and based at least in part on direct ratio estimates of the importance of the ranges of value described by each attribute. This elabo- ration will require considerable attention to the user interface design and was beyond the current demonstration of concept exercise. Appendix B lists the computations described in this chapter. User entries and prioritization categories It is important that all of the vaccine candidates to be prioritized are assessed using the same criteria and measures. At the very outset the user must make two choices that must apply to all vaccine candidates in the set of candidates to be prioritized. The first choice is which metric will be used to measure health benefits—QALYs gained or DALYs averted. The second choice is the selection of attributes by which the value of the vaccines to be compared and prioritized will be measured in the SMART Vaccines Beta. The reason that these choices, once made, are fixed across all vaccine candidates is that the priorities must be determined using the same criteria and measures for each alternative vaccine. The value scores computed for the alternative vaccines are only meaningful relative to one another. These scores have no intrinsic meaning per se, and they gain validity for compari- sons only through the fact that exactly the same basis for evaluation is used for all the alternatives being considered. Demographic inputs The computational submodel requires knowledge of the target population for the vaccine. If vaccines are being prioritized for one country, then that country’s population is the one for which data are needed. If vaccines are being prioritized across a region with more than one country (say, a “super- nation” entity such as the Pan American Health Organization, which has dozens of member countries), the combined population of the region is the target. SMART Vaccines cannot at this time aggregate data across coun- tries, although the current model can deal with multiple populations that have been aggregated a priori and then entered into SMART Vaccines as a new “region.” The population is segmented by age groups—infants, children aged 1 to 4, and then 5-year age bins up to age 99—and also divided into males and

OCR for page 27
57 Modeling Strategy: From Single Attribute to Multiple Attributes females. The average population is represented by the most recent avail- able census data, with the number living in each age range and a standard life table. Age-specific average health-related quality-of-life (HUI2) weights and average hourly wage rates (parental wage rates for persons aged less than 15) are also used in the software. In the United States, the life table data are available from the National Center for Health Statistics and the U.S. Census Bureau; the HUI2 data are available from population surveys (see, for example, Fryback et al., 2007; Luo et al., 2009); and the wage data are available from the Bureau of Labor Statistics. International population data, which are available through the World Health Organization, have been used to pre-populate the data fields and are selectable by country. In the current version of the software, data for hypo- thetical vaccines for three conditions in South Africa and the United States have been entered. Vaccine selection criteria are discussed in Chapter 3. HUI2 data are not generally available outside of the United States and Can- ada unless special surveys have been completed, and DALY weights may be used instead. Wage data outside of developed countries where these statis- tics are usually maintained will have to be estimated subjectively. SMART Vaccines Beta allows assumptions to be tailored for sub- groups of special interest or priority. For example, among persons with tuberculosis the subgroup with HIV infections is of special interest both because immunization may not be effective and because tuberculosis inci- dence is higher in this subgroup. Infants and children or military personnel might also be the special targets of particular vaccination programs. The impact of the immuniza- tion program in a special population is controlled by different input con- stants than those used for the “usual” male and female populations. If a special population is specified, it must be subtracted from the general male and female populations so that the total population is the sum of the three parts; in SMART Vaccines Beta, this subtraction must be done outside the program before inputting data. Disease epidemiology and clinical inputs The computational submodel requires information on the incidence of the disease by sex and by age range as well as case fatality proportions. The time course of the disease is modeled by inputting time-limited states of illness without outpatient visits, of illness with outpatient visits, and of ill- ness with hospitalization; the fraction of cases experiencing each of these; and the time that a typical person experiencing these states would spend in

OCR for page 27
58 RANKING VACCINES: A Prioritization Framework the state. Permanent disability is modeled as a separate outcome, and the percent of cases experiencing permanent disability is entered. Economic inputs The aggregate incremental costs of vaccination versus treatment of the disease are computed in the computational submodel shown in Figure 2-1. This submodel estimates the net incremental costs (or savings) of having a vaccine program versus not having one. The estimation is done by simulat- ing the incidence of disease cases and then simulating the utilization of the units of care, such as visits to a physician’s office, a day of hospitalization, medications, and so forth. To compute the costs of treatment for the target disease, common events in the care of patients, such as over-the-counter medications, a visit to a physician’s office, emergency department visits, and days of hospital- ization, are needed as inputs. To compute the costs of vaccination, it is nec- essary to input the number of doses needed, the cost per dose for vaccine, and the cost per dose to administer the vaccine. Estimates for one-time costs are also entered: research costs for development of the vaccine, costs of the trials and data needed for licensure, and any one-time start-up costs for the initiation of a vaccination program. The committee recognizes that the modeling of costs is at best a broad-brush approximation. But it is simply not possible—especially for hypothetical vaccines—to carry out a microscopic costing of all possible inputs, modeling the various intricacies of the vaccine delivery process. Accordingly, this model allows users to specify the main components of cost in a summary form common to all vaccines. It will require users to roll many aspects of costs into a few generic slots. For example, cost per dose will need to account for manufacturing, storage, transportation, and suit- able profits for all private entities involved in these steps, all in one input number. Sophisticated cost-effectiveness models used to evaluate existing vaccines may break this one input into many subparts in the future, but for now SMART Vaccines Beta uses rough estimates for hypothetical new vaccines. Vaccine inputs The health impacts of vaccination are modeled using estimated duration of immunity conferred, incidence of the disease, and vaccine-associated complications that may be experienced. The effectiveness of the vaccine is modeled by inputs quantifying anticipated uptake or coverage in the vari-

OCR for page 27
59 Modeling Strategy: From Single Attribute to Multiple Attributes ous age groups targeted for vaccination. These estimates should take into account public perceptions of the disease; anticipated vaccine-induced complications, including potential deaths resulting due to the vaccine; and how well the vaccination schedule and doses required fit existing sched- ules in the health system. The herd immunity threshold is set at 100 per- cent in SMART Vaccines. Disease burden summary measures A number of measures of the health burden of disease are incorporated in the model. Some users may prefer to use premature deaths averted or cases prevented. Others may prefer measures such as DALYs averted or QALYs gained—measures that combine the effects of both mortality and morbidity into one number. To compute QALYs, the model must know about the age-specific average health-related quality-of-life (HRQOL) as measured in the popula- tion. The impacts of the disease that could be prevented by vaccination are modeled by assessing a decrement, or “toll,” from the age-specific average for the various health states that an affected person might experience. The reduced HRQOL is then weighted by the length of time that the person is affected in order to get QALYs lost to the disease. SMART Vaccines Beta uses the Health Utilities Index Mark 2 (HUI2) to measure HRQOL, as did the 2000 IOM report. The age- and sex-specific average population baseline HUI2 weights are input as population characteristics. For example, an average observed HUI2 weight of 0.81 is reported for women aged 60 to 64 years in the United States. The HRQOL tolls for the health states associated with the disease must be estimated. For example, using data from the U.S. National Health Measurement Study (Fryback, 2009), the estimated average decre- ment in HUI2 weight for adults who report “cough” versus those who do not report “cough,” age-adjusted, is –0.09. This is used as the daily decre- ment, or toll, from the population average for each day with influenza not requiring an outpatient health care visit. The decrement is –0.13 for those reporting fever, which is used as the daily toll in HUI2 weight for persons requiring an outpatient visit for influenza. Cough and fever are not adjusted here for co-occurrence of other symptoms but rather are used as markers for health states that could be equivalent, on average, to the corresponding influenza health states. A day of hospitalization incurs a toll of –0.2 based on cost–utility analyses from the literature that involve acute illness hospitalization. The 2000 report from the IOM study used subjective role playing by committee members

OCR for page 27
60 RANKING VACCINES: A Prioritization Framework using the HUI2 scales to record their level of functioning and symptoms for health states they were imagining. In the decade since that report more data sources have appeared, such as the National Health Measurement Study and published analyses, from which to estimate HUI2 tolls to model the time course of diseases. Similarly, HUI2 weights must be estimated for permanent disabili- ties resulting from disease- and vaccine-related complications. Estimating the quantities needed for the computational submodel can be vexing, as the needed data are rarely available or reported in the literature. This is further discussed in Chapter 3. If the user elects to compute using DALYs, then similar average health and disability weights must be estimated for disease states. Other attributes If the user selects any other attributes listed in Table 2-1, then appropriate levels of the attributes for each vaccine candidate should be entered by the user. All of these attributes are categorical in nature, with some requiring a simple “yes” or “no” entry. The users will need to make subjective assess- ments where necessary to make the appropriate categorizations. Attribute selection and ranking is accomplished by a drag-and-drop interface (which can be seen in the screenshots of the SMART Vaccines Beta found in Chapter 3). Attributes are selected one at a time and dropped into the ranking box. The selection and ranking of attributes is done once, and all the vaccine candidates are evaluated using the same criteria with fixed weights. This does not prohibit the user from entertaining “What if?” scenarios by changing the attribute selection or the rankings—or both—to see how the value score is affected. But any one set of priorities for vaccine candidates should be based on only one set of attributes and weights. Ranking method In SMART Vaccines Beta the user-selected attributes are not all equally important in establishing priorities. As discussed earlier in this chapter, the rank order of attributes is used to create a set of weights for the factors— the wjs in the equation that is used to compute priority value scores for the vaccines using the Edwards and Barron additive multi-attribute approach (Edwards and Barron, 1994). This method for using the number and rank order of attributes to determine the weights gives most of the weight to the first few attributes in the rank order. The weights are assigned by an approximation algorithm—

OCR for page 27
61 Modeling Strategy: From Single Attribute to Multiple Attributes rank order centroids—as discussed earlier in this chapter. Weights in the additive multi-attribute model are each bounded by 0 and 1, and they col- lectively sum to 1. A vector of n weights, each a number between 0 and 1, may be viewed as a point in the n-dimension cube. Suppose a proper rank order of the set of weights is specified. Consider the subspace of the n-dimension cube formed by the set of all weight vectors that are consistent with the specified rank order and that sum to 1. The vertices (extreme points) of this subset form a simplex, and averaging the coordinates of the n vertices gives the centroid of the simplex. The rank order centroid algorithm takes this cen- troid as the set of weights to be used in the additive multi-attribute model for computing priority scores. It can be thought of as the average of all sets of weights consistent with the user-specified rank order; it is also the mean of a uniform probability distribution over the simplex bounded by the n vertices. Given no other information than the rank order of weights, the rank order centroid is the best statistical estimator for the vector of weights for the additive multi-attribute utility model. Further development of SMART Vaccines can allow users to mod- ify this set of weights by increasing or decreasing single attribute weights while maintaining the rank order. This would give selected attributes slightly more or less importance in the priority calculations. The rank order centroid can be easily extended to include ties in the rank order. But, as discussed by Barron and Barrett (1996), the key information is contained in the rank ordering, and refinements to the weights consistent with the rank order provide, at most, second-order changes. There is one important factor that influences the weights: the num- ber of attributes selected. Table 2-3 displays examples of rank order cen- troid weights for various numbers of attributes in the model. The first few attributes receive most of the total weight using this method, and adding more factors to the prioritization problem has a decreasing effect on the final priority ordering of vaccine candidates. The committee considered limiting the number of weights to some arbitrary number (e.g., seven attributes). After considering this, the com- mittee concluded that introducing a limit on the number of allowed weights would not affect the model much one way or the other, but proceeding without a limit would satisfy those users who really did wish to add a large number of attributes to the model. Users should be aware (see Table 2-3) that once 10 attributes are included, the weight on each subsequent weight is smaller than 0.01 (1 percent) and is extremely unlikely to affect rankings meaningfully. Indeed, even with just five attributes ranked, the weight on the fifth is only 0.04, and with seven attributes, the weight on the seventh is

OCR for page 27
62 RANKING VACCINES: A Prioritization Framework only 0.02. In both cases, the final attribute has little effect on final rankings unless candidate vaccines diverge dramatically on the ranked dimension. Moreover, the current model does not allow for ties in attributes because of the programming complexity in allowing ties in the rank order centroid process. Subsequent modifications to the software will allow users to establish their own rankings independent of this process, includ- ing the possibility of beginning with the rank order centroid weights and then altering pairs of them to allow for ties. For example, if users had five items ranked and wished to establish the top two as having equal weights, then the weights (say, 0.457 and 0.257) created by the rank order centroid method could be averaged as 0.357. The meaning and interpretation of weights This chapter would not be complete without a discussion of the meaning of the weights in the additive multi-attribute utility model. It is tempting to say, as indicated above, that these represent the importance of the different attributes in the prioritization problem. This is a common, but not techni- cally accurate understanding. The mathematical use of the weights is to change the natural attri- bute scales into a common unit of value. For each attribute in Table 2-1, there is a “most preferred” and “least preferred” level or category of the attribute, where “most preferred” means leading to the highest contribu- tion to the priority value of the vaccine candidate. These most and least preferred categories define the range of value through which that attribute can change. If the user defines a single-attribute value function, then each attri- bute will be equal to 1.0 for the most preferred category and to 0.0 for the least preferred one. Other categories between these are scaled linearly between these end values of the scale. A two-level attribute is simply scaled 1 or 0. A three-level attribute is scaled 1.0, 0.5, and 0.0. A four-level attribute will be scored as 1.0 if the attribute is in the highest category, 0.67 if in the second most preferred, 0.33 in the next lower category, and 0.0 in the low- est category. And, a five-level attribute is scaled 1.0, 0.75, 0.50, 0.25, and 0.0 from the most to the least preferred category. Again, these linear scales are an approximation used to simplify the model for users. The next step in modeling is to allow users to specify the spacing between the categories on these single-attribute scales. Consider two attributes in Table 2-1, say Cost-Effectiveness (CE) and Serious Pandemic Potential (SPP). The least-preferred level of CE is Level 4, and the least-preferred level of SPP is Level 2. Suppose there is a vaccine

OCR for page 27
63 Modeling Strategy: From Single Attribute to Multiple Attributes candidate that has CE of Level 4 and SPP of Level 2. To determine which should have the higher weight—CE or SPP—the question arises: Which change improves the overall priority of the vaccine by the larger amount, changing CE to its Level 1 or changing SPP to its Level 1? If the answer is to change CE, then the weight assigned to CE should be larger than the weight assigned to SPP, and vice versa. In the attribute selection and weighting phase of this model, the user is asked to pick those attributes from Table 2-1 that should serve as the basis of comparison for all vaccine candidates. The most- and least- preferred levels of each attribute are displayed to aid this choice. The user is instructed to pick the subset of attributes for which a change from lowest to highest level marks a significant change in priority of a vaccine. This sub- set is then to be rank ordered using exactly the same question as above— sorting the attributes pair-wise according to how much change in priority is implied by changing attributes from least to most preferred levels. The attribute at the top of the user’s rank order should have the largest implied change in overall priority when it changes from least to most preferred, and the attribute at the bottom of the user’s rank order will result in the least change in priority when it changes from the least- to the most-preferred level. Selecting more than seven or eight attributes results in diminished or negligible weights for attributes ranked below 8 (see Table 2-3). This is not to say that the user is restricted from selecting all of the 29 attributes in Table 2-1. But it is true that a handful of attributes generally contain the most weight in establishing priorities. Adding additional attributes beyond seven or eight is unlikely to lead to a decisive change in the priority order of the vaccine candidates. However, Edwards, in an extended case study using multi-attribute utility theory to rank different desegregation plans for the Los Angeles school district (Edwards, 1979), observed that if a number of groups holding strong opinions are attempting to negotiate differences and agree on a ranking of decision alternatives, then one can end up including many attributes to make sure that each group sees all attributes of impor- tance to its viewpoint in the final model. Edwards ended up using more than 100 attributes to tackle this challenge. But such efforts are very rare. Five to fifteen attributes in the final model is much more common (von Winterfeldt and Edwards, 1986). The risk of double counting The committee understands that the model (as presented) carries some risk of double counting some attributes. Double counting in multi-attribute

OCR for page 27
64 RANKING VACCINES: A Prioritization Framework utility theory means putting weight on a pair (or a larger set) of attributes that are highly correlated. The higher the correlation between the attri- butes, the higher is the chance for double counting. The consequence of double counting is that users who include highly correlated attributes will ( perhaps inadvertently) put more weight on the “concept” measured by these attributes than intended. The effect on final value scores will depend in part on how many attributes are included by the user and how high the correlated attributes are placed in the user’s ranking. If the user includes 10 to 15 attributes and places the highly correlated ones near the bottom of the list, the rankings will not change much, since they will receive little weight anyway. However, placing two highly correlated attributes at the top of a short list in the value function can lead to greater emphasis on that underlying concept than perhaps intended. To avoid double counting, the committee selected for inclusion only those value attributes that are intended to capture something desirable about a vaccine that (because of limitations to the sub-sectioning of popula- tion variables) could not be captured directly in computed attributes. Thus the committee sought to exclude from consideration qualitative attributes that were otherwise used in the computation of quantitative attributes. For example, the rate of uptake of a vaccine and a vaccine’s efficacy rate are used in the calculation of the number of persons effectively vac- cinated (and hence in the calculation of reduced disease burden). Thus to include the rate of uptake or the efficacy rate as separate qualitative attri- butes would create the risk of double counting, and hence they are omitted. The most obvious of these double-counting risks would involve the use of DALY or QALY measures of health gains (or losses). While they are not exact mirror images of one another, the DALY and QALY measures are sufficiently similar that the SMART Vaccines software blocks the simulta- neous use of both as indicated attributes. If users select an efficiency mea- sure, they can use $/DALY or $/QALY, but not both. There still remains some potential for double counting. For example, deaths averted contains some of the same information as DALYs averted (or QALYs gained), but these measures do contain independent information about the disease burden. Deaths averted would implicitly count each life saved as the same, no matter what the age of the individual. Life years saved or the more sophisticated QALYs saved contain the additional dimension of duration of the saved life. The 1985–1986 IOM study used infant mortality equivalents prevented for a similar reason—to account for the longevity of the surviving persons. Similarly, combinations of one or more computed quantitative vari- ables can closely correlate with other computed variables. For example,

OCR for page 27
65 Modeling Strategy: From Single Attribute to Multiple Attributes premature deaths averted per year can be closely approximated by the combination of incident cases prevented per year and fatality proportions. Thus including all three of these variables would lead to double counting. These are not identical in the case where a disease has lingering side effects that cause mortality in later years. An example would be an infection that created a chronic condition with some later-year mortality risk. Presently, the software does not take into account such nuances of double counting. However, the committee’s approach to dealing with these double-counting risks will necessarily involve more sophisticated programming in future versions of SMART Vaccines. Discounting and inflation Discounting involves making events that occur in the future commensurate with those that occur in the present. Future events are brought to a “pres- ent value” by discounting them at a pre-selected annual rate. The default value in SMART Vaccines Beta is set at a discount rate of 3 percent, which is presently the standard rate in the U.S. cost-effectiveness literature for health and medicine (see Gold et al., 1996; Ramsey et al., 2005), but the user can alter this at any point. With discounting at 3 percent, an event that occurs 1 year into the future—cost or benefit—carries only 97 percent as much value as one occurring in the present year. An event occurring 2 years into the future as a present value is weighted at 0.972, or 0.9409. One occur- ring 3 years later would have a present value weight of 0.973, or 0.9127. The greater the discount rate—for example, 5 percent instead of 3 percent—the faster these present-value weights diminish over time. Perhaps most important, SMART Vaccines Beta discounts both costs and benefits at the same rate. The logic for this comes from an extended discussion in the literature of cost-effectiveness analysis that generally concludes that discounting benefits and costs at the same rate is the only appropriate strategy (Keeler and Cretin, 1983). In its current version, SMART Vaccines Beta does not allow for different discount rates for costs and benefits. The user must use the same discount rate for all candidate vaccines that are being compared. Using different discount rates could seriously dis- tort the comparisons between vaccines. However, a feature that allows one to specify different discount rates for different vaccines has been included in SMART Vaccines Beta. A separate issue remains regarding the handling of anticipated infla- tion rates within the economy in which the vaccine comparisons are being made. This is a challenge that is distinct from the question of discounting.

OCR for page 27
66 RANKING VACCINES: A Prioritization Framework Many cost–benefit analyses presume some background rate of inflation in the economy—for example, 2 percent per year—so that $1 million in costs this year becomes $1.02 million in costs the following year. Adjusting for inflation before discounting is equivalent to simply computing all future economic costs and benefits in today’s dollars at today’s prices and not worrying about what inflation might be in the future. This is the approach taken by the current model. This is done to avoid questions concerning what inflation rate is appropriate—consumer price inflation, monetary inflation, or sectoral inflation confined to health care. For example the inflationary growth in wages, used to measure worker productivity losses and gains, is quite different from inflation in the costs of health care, which itself is a market basket of services and durable goods with different rates of inflation. Time horizon and uncertainty This model always operates within a fixed 100-year time horizon. This has been done to simplify the software programming and to reduce the poten- tial for coding errors. SMART Vaccines Beta does not include the ability to set distributions on the input parameters to reflect uncertainties relating to the disease or vaccine data. Therefore, in its current version the multi- attribute output values do not have standard errors. A dynamic sensitiv- ity analysis may be required to detect changes in the priority score with changes in key values. These possibilities, along with others, are discussed in Chapter 3. The committee’s prototyping and testing efforts are described in Chapter 3, which also provides representative screenshots of SMART Vac- cines Beta.