Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 27
2
Modeling Strategy: From Single
Attribute to Multiple Attributes
The vaccine prioritization techniques of the earlier Institute of Medicine
(IOM) studies published in 1985–1986 and 2000 relied on two criteria: (1)
reduction of health burden (IOM, 1985, 1986) and (2) incremental cost or
savings (IOM, 2000) due to use of the vaccine in a defined population. More
specifically, the 1985–1986 work used only a single attribute—infant deaths
averted—for ranking vaccine candidates; it did not consider cost attributes.
The 2000 report used an approach based on cost-effectiveness to prioritize
vaccines.
Those studies saw the central “modeling task” as numerical esti-
mation of the expected costs and benefits of the vaccines. The principles
underlying this approach derive from the economic theory of social wel-
fare as implemented in the classic utility frameworks (Garber and Phelps,
1997). The computational models were the key contributions of the 1985–
1986 and 2000 reports. Their work involved many decisions concerning
which costs and savings to include and how best to measure health gains.
New Vaccine Development (1985–1986) and
Vaccines for the 21st Century (2000)
The 1985–1986 report measured health benefits using infant mortal-
ity equivalents (IMEs), which involved subjective judgments relating to
morbidity and mortality reductions compared to an equivalent number of
infant deaths averted. Since the time that report was published, analytical
techniques have advanced. Standardized measures of health-related qual-
27
OCR for page 28
28 RANKING VACCINES: A Prioritization Framework
ity of life (HRQOL) such as the Health Utilities Index Mark 2, or HUI2—a
tool to measure morbidity reduction—have been developed using methods
of multi-attribute utility theory (Feeny et al., 1996). HUI2 has been com-
bined with actuarial measures of life expectancy changes in order to com-
pute quality-adjusted life years (QALYs) as one of the main health valuation
measures.
To derive its vaccine priorities, the 2000 report relied on incremen-
tal dollar costs per incremental QALY gained ($/QALY) for both preven-
tive and therapeutic vaccines that are of importance to the United States.
In the nearly three decades since the 1985–1986 report was published, the
theoretical basis for its calculations has not changed. By contrast, in the
years since the 2000 report, the methods of cost-effectiveness analysis
have become somewhat more sophisticated when it comes to assessing the
effectiveness of $/QALY values for health care technologies.
The self-reported health status data needed for population-based
measures such as HUI2 are not available in much of the world. Instead,
researchers at the World Health Organization in collaboration with
researchers at other institutions developed a similar tool: disability-
adjusted life years (DALYs). In calculating DALYs, disability weights are
assigned to typical manifestations of a wide variety of diseases; such mea-
sures have been used for many countries around the world (Fox-Rushby
and Hanson, 2001; Gold et al., 2002; Murray and Lopez, 2000).
Methods to incorporate uncertainties in decision models were
undergoing rapid development at the time of the 2000 report. They have
since progressed and become more generally applicable (Fenwick et al.,
2001; Meckley et al., 2010). There have also been advances in population-
based data collection supporting HUI2 and similar indexes of generic
health-related quality of life that the 2000 report incorporated (Fryback
et al., 2007, 2010; Luo et al., 2005, 2009).
In recent years, advances in complex systems modeling have helped
characterize the nature and spread of infections in populations. These
dynamical techniques can now be used for estimating the impact of a new
vaccine for a specific population (e.g., Epstein et al., 2008). But the under-
lying decision framework and conceptual approaches to estimating costs
and health benefits have essentially remained unchanged.
The previous reports developed a computational model based on
two important (but distinctly different) attributes for prioritizing vaccines,
although more sophisticated methods could have been used. The main crit-
icism of the 2000 report was related to the basic framework itself: the sys-
tem was too limited and considered only costs and aggregated health benefits
(e.g., see Plotkin et al., 2000).
OCR for page 29
29
Modeling Strategy: From Single Attribute to Multiple Attributes
Modeling beyond cost-effectiveness
The committee revisited the assumptions and limitations of the 1985–
1986 and 2000 approaches. Instead of taking the path of developing a de
novo computational model, the committee chose to significantly expand
the previous IOM works by using a multi-attribute utility framework and
develop a novel software application. In this work, therefore, some aggre-
gate measure of health benefits (such as infant deaths averted) or an effi-
ciency criterion (such as cost-effectiveness) has simply become one among
the many criteria—rather than the only criterion—that influence vaccine
prioritization.
The committee took on the task of expanding the list of attributes
characterizing vaccine candidates and developing a prototype software—
SMART Vaccines Beta—to weigh not only economic and health attributes
but also demographic, scientific, business, programmatic (field-level logis-
tics), social, and policy aspects relating to new vaccine development. The
short-listing of 29 attributes used in SMART Vaccines Beta was informed
by stakeholder and concept evaluator feedback, committee discussions,
and literature review (Burchett et al., 2011).
Values and objectives in priority setting
Priority setting means assigning values and objectives. If the main objec-
tive of a new preventive vaccine is to minimize the disease burden in the
target population, then assuming that all else is equal, the highest priority
typically would be given to the vaccine candidate expected to produce the
largest health benefit compared to other candidates, and a set of vaccine
candidates would be prioritized according to their expected health ben-
efits, going from most expected benefits to least.
But all else is not equal. Priorities must also reflect such consider-
ations as the fact that resources are constrained. Such a limited-resources
constraint points to a different objective: to minimize the costs associated
with bringing a vaccine to licensure and then administering it in the tar-
get population. If minimizing costs is the main objective, then the program
with the lowest development and implementation costs would be favored,
and priorities would simply be ordered according to the increasing costs of
the different programs. These two objectives—maximizing health benefits
and minimizing costs—are often in conflict. One vaccine candidate may
potentially have a very large aggregate impact on health burden but also
have greater expected costs than a vaccine addressing a different disease
where the effect on health burden may be smaller.
OCR for page 30
30 RANKING VACCINES: A Prioritization Framework
When objectives are in conflict, decision makers often deal with
trade-offs. In this case, each vaccine candidate is associated with expected
health benefits and costs. Expressing a priority order among candidates
requires us to weigh the extent to which each vaccine candidate achieves
the two objectives jointly, perhaps preferring one objective over the other.
In this case, cost-effectiveness analysis is appropriate and may be used to
prioritize vaccine candidates when there are trade-offs between these two
important attributes.
But several other objectives could also influence the ranking of vac-
cine candidates under consideration. These objectives depend on whose
priorities are being expressed toward maximizing the overall value of the
vaccines. For example, decision makers may want to represent a public
desire to minimize the burden of disease in specific target populations such
as women, infants, and children; the socioeconomically disadvantaged; or
military personnel. There may be certain diseases that raise special con-
cerns or fear in the public mind—for example, a rare but particularly grue-
some condition, an unrelenting infection, or a terribly disfiguring disease.
Extra priority may be given to a vaccine that prevents such a disease, esca-
lating its priority despite high costs or a relatively small aggregate health
burden imposed by the disease in the population compared to a vaccine
preventing a condition that is more common but that has a relatively minor
health burden.
Other objectives are also possible. One might wish, for instance, to
maximize the benefit to future generations by investing in a vaccine that
could eliminate a particular disease altogether or mitigate its epidemic
potential. Similarly, one might wish to prioritize a vaccine that has the
potential to significantly advance the scientific base, including new pro-
duction, preservation, and delivery methods.
A prioritization exercise starts with a set of vaccine candidates, each
of which is expected to meet, to a greater or lesser degree, a number of
desired objectives. The basic purpose of prioritization is to place these can-
didates in order from “most preferred” to “least preferred” in accordance
with values held by or represented in proxy by the decision maker. The
methods used to accomplish this task in a rigorous fashion fall generally
under the rubric “multi-criteria decision making.”
Multi-criteria decision-making methods
From the family of multi-criteria decision-making models, the commit-
tee chose to use a version of multi-attribute utility theory. As a starting
point, the committee limited the models under consideration to those
OCR for page 31
31
Modeling Strategy: From Single Attribute to Multiple Attributes
that included multiple attributes. The committee heard from a number of
stakeholders that the narrow range of attributes used to rank vaccine pri-
orities in previous IOM studies significantly limited their value and appli-
cations. Thus, the committee reviewed three multiple-attribute modeling
approaches (listed in the order of historical development): (1) mathemati-
cal programming (or optimization), (2) multi-attribute utility theory, and
(3) analytical hierarchy process. The approaches were evaluated against
four criteria: axiomatic foundation; priority scaling; sensitivity analysis;
and transparency.
Axiomatic foundation
Multi-attribute utility theory and mathematical programming are based on
axiomatic theory—the former being derived from principles of utility maxi-
mization (Krantz et al., 1971), and the latter being based on mathematical
optimization. The analytical hierarchy process has an axiomatic base that
the committee considered incomplete. To elaborate, the issue of indepen-
dence from irrelevant alternatives (IIA) was of particular importance to
the committee’s considerations. IIA means the following: Given a particu-
lar set of options (candidate vaccines) in which candidate A is preferred to
candidate B, if an additional candidate C—unrelated to A and B—is added
to the option set, then A continues to be preferred over B.
Consider, for example, a comparison of vaccines to prevent tubercu-
losis and malaria, ranked with one preferred to the other. Now suppose that
the science and technology evolves to allow a new vaccine against dengue
fever. IIA would mean that the ranking of vaccine candidates for tuber-
culosis and malaria remains unchanged when the dengue fever vaccine is
added to the mix for consideration. The new dengue fever vaccine may be
more or less preferred than either tuberculosis or malaria or both vaccines,
but the rankings of tuberculosis and malaria vaccines with respect to each
other must remain unchanged. Since the appearance of new candidate vac-
cines can be anticipated over time, the committee concluded that IIA was
particularly important to consider.
Priority scaling
The 1985–1986 and 2000 IOM reports relating to vaccine prioritization and
the international stakeholder testimonies made it very clear that this com-
mittee’s work would need to offer greater value in terms of allowing differ-
ent users to apply their individual preferences in a prioritization model.
The committee defines the term “prioritize” consistently with the stan-
OCR for page 32
32 RANKING VACCINES: A Prioritization Framework
dard dictionary definition “to arrange in the order of relative importance.”
Thus, prioritization at a minimum requires an ordinal ranking and nothing
more—simply stating an order of preference. The three modeling methods
considered by the committee all provide additional information beyond an
ordinal scale—either interval or ratio scale numbers assigned to vaccine
candidates to represent relative priority.
To use an analogy relating to temperature measurement, with inter-
val scales the difference between two values has the same meaning at dif-
ferent points along the scale. For example, the difference between 20°C
and 40°C has the same meaning as the difference between 30°C and 50°C.
But 40°C is not twice as hot as 20°C. Ratio scales also provide informa-
tion about relative values, thus requiring identification of true “zero” on
the scale. Kelvin temperature allows for this: 300K is twice as hot as 150K,
whereas statements about ratios of temperatures are incorrect in either °C
or °F scales—but ratios of differences in temperatures are the same on K, °C,
and °F scales. Since only ordinal ranking is required in prioritization, any
modeling approach providing interval or ratio scaling is sufficient.
Sensitivity analysis
The committee also wanted to allow users to conduct sensitivity analysis
on their results. This sensitivity analysis has several purposes, including
(a) enhancing understanding of the inputs to which the results were most
sensitive, (b) pointing toward areas where improved data have the greatest
value, and hence potentially (c) spurring efforts and investments in data
generation. All three modeling approaches had the capability for ably sup-
porting sensitivity analyses.
Transparency
Another important criteria for the committee was transparency. In the
committee’s view, the multi-attribute utility approach was more transpar-
ent than other possible approaches. In mathematical programming, for
example, one could subtly alter the constraint set (in ways very difficult for
others to see) so as to eliminate some candidates from the solution set in
favor of others, or else modify the way the objective function was specified.
In analytic hierarchy process, the value weights emerge only after a long
series of pair-wise comparisons have been recorded and modified through
normalization processes involving complex matrix manipulations. By con-
trast, in multi-attribute utility theory the weights and data are available for
everybody to see and use. In that regard, multi-attribute utility theory was
OCR for page 33
33
Modeling Strategy: From Single Attribute to Multiple Attributes
found to be the best fit for satisfying the transparency requirement. Indeed,
the committee saw this as a strength of the SMART Vaccines, highlighting
its potential in promoting cross-comparison of different users’ rationale
and conclusions and leading to more informed discussions about priorities
among different stakeholders. Each modeling alternative is summarized in
the following sections.
Mathematical programming or optimization
Mathematical programming (linear programming, nonlinear program-
ming, stochastic programming, and more complex optimization algo-
rithms) has been widely and successfully employed in many areas to tackle
complex challenges. In concept, mathematical programming is an appro-
priate method for vaccine prioritization. Its optimization characteristics
are well understood (Rardin, 1997). In various formulations, it can pro-
vide output of at least ordinal nature (ranking) and, in many formulations,
interval or ratio scale output, and software to carry out such calculations is
widely available in numerous commercial and free-ware environments. It
is also amenable to sensitivity analyses.
The primary uses of mathematical programming involve optimi-
zation of some value function (specified by the user) subject to a set of
constraints which are often highly complex and frequently nonlinear. In
classical linear and nonlinear programming, the values of relevant compo-
nents of the model are known (e.g., cost, consumer preferences, and other
factors). Stochastic programming emerged to provide optimization tools
when uncertainty exists about certain components of the system under
consideration. But, in general, the value of mathematical programming
appears when there are many possible solutions (perhaps an infinite num-
ber) within the constraint set.
Prioritization of vaccines differs considerably from the usual uses
of mathematical programming. Typically, only a small number of alterna-
tives are considered in the set of potential vaccines (dozens, perhaps, but
seldom hundreds, almost never thousands, and certainly not an infinite
set of options). Separately, unless a customized stochastic programming
method or some equivalent method is developed and used, the dearth of
data in regards to new vaccines problem would likely render the optimi-
zation capabilities of mathematical programming questionable for the
application.
Another issue also deterred the full consideration of mathemati-
cal programming for vaccine prioritization: Mathematical programming
requires a pre-specification of the value function. This is a crucial issue,
OCR for page 34
34 RANKING VACCINES: A Prioritization Framework
since many users and stakeholders would not be able to competently spec-
ify a value function for such reasons as a lack of a quantitative background.
Furthermore, there are no well-developed and tested methods for value
elicitation associated with mathematical programming methods.
Analytic hierarchy process
The analytic hierarchy process has many desirable attributes. It is widely
used by people in business and other settings to assist in decision making,
often under the tutelage of professional consultants. It provides a ratio-
scale value function, which is more than sufficient for the committee’s
ranking process. It has a well-developed process for eliciting values from
users, based on a large set of pair-wise comparisons of different alterna-
tives along the various attribute dimensions. The user must make a sizeable
number (typically in the hundreds) of paired comparison assessments. For
each pair of candidates (e.g., vaccines) A and B, and for each attribute, xj,
the decision maker rates the comparison of xaj versus xbj using a scale of 9,
7, 5, 3, 1, 1/3, 1/5, 1/7, 1/9 to describe how much better A is than B on that
attribute, where the numbers are meant to convey a ratio scale of relative
performance. Although, in principle, any user can program the calculations
necessary for deriving priorities1 from an analytic hierarchy process, most
analysts use one of a number of proprietary software packages currently
available. These packages lead users through the necessary steps and pro-
vide internal consistency checks for many of the comparative assessments.
Besides the complexity associated with value elicitation process, two
other features make this analytic hierarchy process less friendly for vaccine
prioritization. Perhaps most important, the analytic hierarchy process does
not maintain IIA, a fact that is widely understood among both proponents
and opponents of this method (Dyer, 1990; Saaty, 1987). Proponents of ana-
lytic hierarchy process cite this as a beneficial feature, noting that many
real world decisions also do not have IIA. But the committee, for reasons
stated previously, views IIA as a critical factor in vaccine prioritization.
Among the users of analytic hierarchy process the word “priority” has a specific techni-
1
cal meaning (relating to a normalized eigenvector used in the model) that does not match
the standard definition of priority mentioned earlier and used in this report. Thus, one
should not confuse the specific analytic hierarchy process definition of priority with the
one used by the committee.
OCR for page 35
35
Modeling Strategy: From Single Attribute to Multiple Attributes
Multi-attribute utility theory
A multi-attribute utility-based prioritization exercise consists of several
steps. First, the set of vaccine candidates to be considered must be iden-
tified. Next, a set of objectives that underpin the valuation of candidates
must be listed. For each objective there must be a specific measure—called
an “attribute”—developed. The attributes may be natural scales (such as
expected net present value of annualized dollar costs or savings), well-
established indexes (such as net annualized increase in QALYs due to the
vaccine), or customized categorical scales.
If we denote each candidate vaccine by xi, then the outcome attri-
butes characterizing that vaccine may be viewed as a vector, ci = (xi1, xi2,
. . . , xin), where n is the number of attributes being considered when setting
priorities, and xij is the value of the scale for the jth attribute for the ith
vaccine candidate. Multi-attribute utility models can combine attributes of
each type, whether continuous or categorical.
Keeney and Raiffa (1976) as well as a number of others (Barron and
Barrett, 1996; Edwards and Barron, 1994; Edwards and Newman, 1982;
von Winterfeldt and Edwards, 1986) have described methods to specify n
single-attribute functions, 0 ≤ uj(xj) ≤ 1, and a global utility function, U(ci)
= f(u1(xi1), u2(xi2), . . . , un(xin)), such that 0 ≤ U(ci) ≤ 1. The function U is con-
structed so that ca is preferred to cb if and only if U(ca) > U(cb).
Often the function f is additive, U(ci) = w1 u1(xi1) + w2 u2(xi2) + . . . + wn
un(xin), where the wjs are constants that sum to 1. The ratios wj/wk reflect
the change in value achieved by changing the jth attribute from its min-
imum to maximum level in the set of vaccine candidates versus making
the corresponding change in the kth attribute. Although there are strong
arguments for using an additive function as a first approximation (Edwards
and Barron, 1994; Keeney and von Winterfeldt, 2007; von Winterfeldt and
Edwards, 1986), in some cases a multiplicative function or multi-linear
function might be more appropriate in order to account for interactions
among the attributes based on user preferences (Keeney and Raiffa, 1976).
Additive functions are often satisfactory for broad policy purposes. The
committee employs an additive version of multi-attribute utility method in
SMART Vaccines Beta.
Determining what weights (w1, w2, . . . , wn) to use is a separate prob-
lem from that of choosing the functional form (e.g., additive or multipli-
cative). Edwards and Barron (1994) proposed a method to approximate
the wjs using the decision maker’s rank order of the relative importance of
the attributes. In particular, they proposed using the rank order centroid
method to derive weights for a set of attributes, a method that was later
extensively evaluated by Barron and Barrett (1996).
OCR for page 36
36 RANKING VACCINES: A Prioritization Framework
The rank order centroid approximation
The decision maker’s major input is to produce a rank order of the relative
importance of the attributes in order to differentiate the priority of the vac-
cine candidates. This induces a rank order on the weights in the additive
model. Suppose that the rank order is w1 ≥ w2 ≥ … ≥ wn for n attributes. The
rank order centroid approximation for the constants in an additive model
would then be as follows:
11 1
1 + 2 + 3 + ... + n
w1 =
n
1
11
0 + 2 + 3 + ... + n
w2 =
n
1
1
0 + 0 + + ... +
3 n
w3 =
n
1
0 + 0 + 0 + ... + n
wn =
n
More compactly the weights can be expressed by
1
n
wi = ∑ i = 1n
j =i j
Barron and Barrett showed this rank order centroid approximation
for weights to be superior to other often-proposed methods, such as the
normalized sum of ranks. It is important to realize that rank order centroid
weights are not essential to the multi-attribute utility models; rather they
are an approximation used to reduce the workload of the potential user.
In SMART Vaccines Beta, the rank order centroid-based weighting
approach was employed in order to speed up development of other parts
of the model. In many policy settings using multi-attribute utility theory,
these weights are developed with experts guiding the process of decision
makers elucidating their preferred weights (Keeney and Raiffa, 1976; von
Winterfeld and Edwards, 1986).
The multi-attribute decision techniques (or related proprietary soft-
ware packages) have been used in practical applications in a number of
OCR for page 37
37
Modeling Strategy: From Single Attribute to Multiple Attributes
public policy settings, including to evaluate alternative plans to desegregate
schools (Edwards, 1979), to plan wastewater treatment facilities (Keeney et
al., 1996), to evaluate accounting regulations for control of nuclear materi-
als (Keeney and Smith, 1982), and to evaluate homeland security decisions
(Keeney and von Winterfeld, 2011). Additional applications have been
reviewed by Keefer and colleagues (2004).
Data demands
The multi-attribute utility approach places considerable data demands on
users. The committee continually sought to balance the model’s capabili-
ties and complexity with the data demands it would place on users. The
challenge, however, spans every approach considered by the committee.
It is intrinsic not to the multi-attribute utility approach itself, but rather
to the underlying complexity of prioritization and how to model it. Had
mathematical programming or analytic hierarchy process been adopted, a
level of data demands similar to those in the multi-attribute utility theory
would have been required. The only way to reduce data demands is to have
limited capabilities in SMART Vaccines.
A parallel issue relates to how the necessary data must be structured.
In the committee’s view, the data inputs necessary for the multi-attribute
approach are at least as simple—and often simpler—for users to understand
than would be the case in alternative models. For example, many formula-
tions of mathematical programming have inequality constraints, a concept
that could seem alien to many potential users of our software.
The modeling framework for SMART Vaccines Beta
Multi-attribute utility theory provides the analytical framework that
underpins the committee’s work, and the specific model within this frame-
work is an additive multi-attribute utility model. A schematic diagram of
the model’s organization is presented in Figure 2-1.
Within the multi-attribute utility framework, a vaccine candidate is
viewed as a means to achieve an end in a specified population. The various
objectives that the development and delivery of a new vaccine may address
include
• enhancing public health by reducing the burden due to a particular
disease or condition;
• minimizing the societal costs of the disease, and its prevention and
treatment;
OCR for page 56
56 RANKING VACCINES: A Prioritization Framework
In SMART Vaccines Beta, the weights are computed from a strict
rank order of attributes supplied by the user. In future versions, the com-
mittee expects that this approximation will be replaced by a more elabo-
rate elicitation of weights, perhaps using a hierarchical clustering of the
selected attributes, and based at least in part on direct ratio estimates of the
importance of the ranges of value described by each attribute. This elabo-
ration will require considerable attention to the user interface design and
was beyond the current demonstration of concept exercise.
Appendix B lists the computations described in this chapter.
User entries and prioritization categories
It is important that all of the vaccine candidates to be prioritized are
assessed using the same criteria and measures. At the very outset the user
must make two choices that must apply to all vaccine candidates in the set
of candidates to be prioritized. The first choice is which metric will be used
to measure health benefits—QALYs gained or DALYs averted. The second
choice is the selection of attributes by which the value of the vaccines to be
compared and prioritized will be measured in the SMART Vaccines Beta.
The reason that these choices, once made, are fixed across all vaccine
candidates is that the priorities must be determined using the same criteria
and measures for each alternative vaccine. The value scores computed for
the alternative vaccines are only meaningful relative to one another. These
scores have no intrinsic meaning per se, and they gain validity for compari-
sons only through the fact that exactly the same basis for evaluation is used
for all the alternatives being considered.
Demographic inputs
The computational submodel requires knowledge of the target population
for the vaccine. If vaccines are being prioritized for one country, then that
country’s population is the one for which data are needed. If vaccines are
being prioritized across a region with more than one country (say, a “super-
nation” entity such as the Pan American Health Organization, which has
dozens of member countries), the combined population of the region is the
target. SMART Vaccines cannot at this time aggregate data across coun-
tries, although the current model can deal with multiple populations that
have been aggregated a priori and then entered into SMART Vaccines as a
new “region.”
The population is segmented by age groups—infants, children aged 1
to 4, and then 5-year age bins up to age 99—and also divided into males and
OCR for page 57
57
Modeling Strategy: From Single Attribute to Multiple Attributes
females. The average population is represented by the most recent avail-
able census data, with the number living in each age range and a standard
life table.
Age-specific average health-related quality-of-life (HUI2) weights
and average hourly wage rates (parental wage rates for persons aged less
than 15) are also used in the software. In the United States, the life table
data are available from the National Center for Health Statistics and the
U.S. Census Bureau; the HUI2 data are available from population surveys
(see, for example, Fryback et al., 2007; Luo et al., 2009); and the wage data
are available from the Bureau of Labor Statistics.
International population data, which are available through the World
Health Organization, have been used to pre-populate the data fields and are
selectable by country. In the current version of the software, data for hypo-
thetical vaccines for three conditions in South Africa and the United States
have been entered. Vaccine selection criteria are discussed in Chapter 3.
HUI2 data are not generally available outside of the United States and Can-
ada unless special surveys have been completed, and DALY weights may be
used instead. Wage data outside of developed countries where these statis-
tics are usually maintained will have to be estimated subjectively.
SMART Vaccines Beta allows assumptions to be tailored for sub-
groups of special interest or priority. For example, among persons with
tuberculosis the subgroup with HIV infections is of special interest both
because immunization may not be effective and because tuberculosis inci-
dence is higher in this subgroup.
Infants and children or military personnel might also be the special
targets of particular vaccination programs. The impact of the immuniza-
tion program in a special population is controlled by different input con-
stants than those used for the “usual” male and female populations. If a
special population is specified, it must be subtracted from the general male
and female populations so that the total population is the sum of the three
parts; in SMART Vaccines Beta, this subtraction must be done outside the
program before inputting data.
Disease epidemiology and clinical inputs
The computational submodel requires information on the incidence of the
disease by sex and by age range as well as case fatality proportions. The
time course of the disease is modeled by inputting time-limited states of
illness without outpatient visits, of illness with outpatient visits, and of ill-
ness with hospitalization; the fraction of cases experiencing each of these;
and the time that a typical person experiencing these states would spend in
OCR for page 58
58 RANKING VACCINES: A Prioritization Framework
the state. Permanent disability is modeled as a separate outcome, and the
percent of cases experiencing permanent disability is entered.
Economic inputs
The aggregate incremental costs of vaccination versus treatment of the
disease are computed in the computational submodel shown in Figure 2-1.
This submodel estimates the net incremental costs (or savings) of having a
vaccine program versus not having one. The estimation is done by simulat-
ing the incidence of disease cases and then simulating the utilization of the
units of care, such as visits to a physician’s office, a day of hospitalization,
medications, and so forth.
To compute the costs of treatment for the target disease, common
events in the care of patients, such as over-the-counter medications, a visit
to a physician’s office, emergency department visits, and days of hospital-
ization, are needed as inputs. To compute the costs of vaccination, it is nec-
essary to input the number of doses needed, the cost per dose for vaccine,
and the cost per dose to administer the vaccine. Estimates for one-time
costs are also entered: research costs for development of the vaccine, costs
of the trials and data needed for licensure, and any one-time start-up costs
for the initiation of a vaccination program.
The committee recognizes that the modeling of costs is at best a
broad-brush approximation. But it is simply not possible—especially for
hypothetical vaccines—to carry out a microscopic costing of all possible
inputs, modeling the various intricacies of the vaccine delivery process.
Accordingly, this model allows users to specify the main components of
cost in a summary form common to all vaccines. It will require users to roll
many aspects of costs into a few generic slots. For example, cost per dose
will need to account for manufacturing, storage, transportation, and suit-
able profits for all private entities involved in these steps, all in one input
number. Sophisticated cost-effectiveness models used to evaluate existing
vaccines may break this one input into many subparts in the future, but
for now SMART Vaccines Beta uses rough estimates for hypothetical new
vaccines.
Vaccine inputs
The health impacts of vaccination are modeled using estimated duration
of immunity conferred, incidence of the disease, and vaccine-associated
complications that may be experienced. The effectiveness of the vaccine is
modeled by inputs quantifying anticipated uptake or coverage in the vari-
OCR for page 59
59
Modeling Strategy: From Single Attribute to Multiple Attributes
ous age groups targeted for vaccination. These estimates should take into
account public perceptions of the disease; anticipated vaccine-induced
complications, including potential deaths resulting due to the vaccine; and
how well the vaccination schedule and doses required fit existing sched-
ules in the health system. The herd immunity threshold is set at 100 per-
cent in SMART Vaccines.
Disease burden summary measures
A number of measures of the health burden of disease are incorporated in
the model. Some users may prefer to use premature deaths averted or cases
prevented. Others may prefer measures such as DALYs averted or QALYs
gained—measures that combine the effects of both mortality and morbidity
into one number.
To compute QALYs, the model must know about the age-specific
average health-related quality-of-life (HRQOL) as measured in the popula-
tion. The impacts of the disease that could be prevented by vaccination are
modeled by assessing a decrement, or “toll,” from the age-specific average
for the various health states that an affected person might experience. The
reduced HRQOL is then weighted by the length of time that the person is
affected in order to get QALYs lost to the disease. SMART Vaccines Beta
uses the Health Utilities Index Mark 2 (HUI2) to measure HRQOL, as did
the 2000 IOM report.
The age- and sex-specific average population baseline HUI2 weights
are input as population characteristics. For example, an average observed
HUI2 weight of 0.81 is reported for women aged 60 to 64 years in the
United States. The HRQOL tolls for the health states associated with the
disease must be estimated. For example, using data from the U.S. National
Health Measurement Study (Fryback, 2009), the estimated average decre-
ment in HUI2 weight for adults who report “cough” versus those who do
not report “cough,” age-adjusted, is –0.09. This is used as the daily decre-
ment, or toll, from the population average for each day with influenza not
requiring an outpatient health care visit. The decrement is –0.13 for those
reporting fever, which is used as the daily toll in HUI2 weight for persons
requiring an outpatient visit for influenza.
Cough and fever are not adjusted here for co-occurrence of other
symptoms but rather are used as markers for health states that could be
equivalent, on average, to the corresponding influenza health states. A day
of hospitalization incurs a toll of –0.2 based on cost–utility analyses from
the literature that involve acute illness hospitalization. The 2000 report
from the IOM study used subjective role playing by committee members
OCR for page 60
60 RANKING VACCINES: A Prioritization Framework
using the HUI2 scales to record their level of functioning and symptoms
for health states they were imagining. In the decade since that report more
data sources have appeared, such as the National Health Measurement
Study and published analyses, from which to estimate HUI2 tolls to model
the time course of diseases.
Similarly, HUI2 weights must be estimated for permanent disabili-
ties resulting from disease- and vaccine-related complications. Estimating
the quantities needed for the computational submodel can be vexing, as the
needed data are rarely available or reported in the literature. This is further
discussed in Chapter 3. If the user elects to compute using DALYs, then
similar average health and disability weights must be estimated for disease
states.
Other attributes
If the user selects any other attributes listed in Table 2-1, then appropriate
levels of the attributes for each vaccine candidate should be entered by the
user. All of these attributes are categorical in nature, with some requiring a
simple “yes” or “no” entry. The users will need to make subjective assess-
ments where necessary to make the appropriate categorizations.
Attribute selection and ranking is accomplished by a drag-and-drop
interface (which can be seen in the screenshots of the SMART Vaccines
Beta found in Chapter 3). Attributes are selected one at a time and dropped
into the ranking box. The selection and ranking of attributes is done once,
and all the vaccine candidates are evaluated using the same criteria with
fixed weights. This does not prohibit the user from entertaining “What if?”
scenarios by changing the attribute selection or the rankings—or both—to
see how the value score is affected. But any one set of priorities for vaccine
candidates should be based on only one set of attributes and weights.
Ranking method
In SMART Vaccines Beta the user-selected attributes are not all equally
important in establishing priorities. As discussed earlier in this chapter, the
rank order of attributes is used to create a set of weights for the factors—
the wjs in the equation that is used to compute priority value scores for the
vaccines using the Edwards and Barron additive multi-attribute approach
(Edwards and Barron, 1994).
This method for using the number and rank order of attributes to
determine the weights gives most of the weight to the first few attributes in
the rank order. The weights are assigned by an approximation algorithm—
OCR for page 61
61
Modeling Strategy: From Single Attribute to Multiple Attributes
rank order centroids—as discussed earlier in this chapter. Weights in the
additive multi-attribute model are each bounded by 0 and 1, and they col-
lectively sum to 1.
A vector of n weights, each a number between 0 and 1, may be viewed
as a point in the n-dimension cube. Suppose a proper rank order of the
set of weights is specified. Consider the subspace of the n-dimension cube
formed by the set of all weight vectors that are consistent with the specified
rank order and that sum to 1. The vertices (extreme points) of this subset
form a simplex, and averaging the coordinates of the n vertices gives the
centroid of the simplex. The rank order centroid algorithm takes this cen-
troid as the set of weights to be used in the additive multi-attribute model
for computing priority scores. It can be thought of as the average of all sets
of weights consistent with the user-specified rank order; it is also the mean
of a uniform probability distribution over the simplex bounded by the n
vertices. Given no other information than the rank order of weights, the
rank order centroid is the best statistical estimator for the vector of weights
for the additive multi-attribute utility model.
Further development of SMART Vaccines can allow users to mod-
ify this set of weights by increasing or decreasing single attribute weights
while maintaining the rank order. This would give selected attributes
slightly more or less importance in the priority calculations. The rank
order centroid can be easily extended to include ties in the rank order. But,
as discussed by Barron and Barrett (1996), the key information is contained
in the rank ordering, and refinements to the weights consistent with the
rank order provide, at most, second-order changes.
There is one important factor that influences the weights: the num-
ber of attributes selected. Table 2-3 displays examples of rank order cen-
troid weights for various numbers of attributes in the model. The first few
attributes receive most of the total weight using this method, and adding
more factors to the prioritization problem has a decreasing effect on the
final priority ordering of vaccine candidates.
The committee considered limiting the number of weights to some
arbitrary number (e.g., seven attributes). After considering this, the com-
mittee concluded that introducing a limit on the number of allowed weights
would not affect the model much one way or the other, but proceeding
without a limit would satisfy those users who really did wish to add a large
number of attributes to the model. Users should be aware (see Table 2-3)
that once 10 attributes are included, the weight on each subsequent weight
is smaller than 0.01 (1 percent) and is extremely unlikely to affect rankings
meaningfully. Indeed, even with just five attributes ranked, the weight on
the fifth is only 0.04, and with seven attributes, the weight on the seventh is
OCR for page 62
62 RANKING VACCINES: A Prioritization Framework
only 0.02. In both cases, the final attribute has little effect on final rankings
unless candidate vaccines diverge dramatically on the ranked dimension.
Moreover, the current model does not allow for ties in attributes
because of the programming complexity in allowing ties in the rank order
centroid process. Subsequent modifications to the software will allow
users to establish their own rankings independent of this process, includ-
ing the possibility of beginning with the rank order centroid weights and
then altering pairs of them to allow for ties. For example, if users had five
items ranked and wished to establish the top two as having equal weights,
then the weights (say, 0.457 and 0.257) created by the rank order centroid
method could be averaged as 0.357.
The meaning and interpretation of weights
This chapter would not be complete without a discussion of the meaning
of the weights in the additive multi-attribute utility model. It is tempting to
say, as indicated above, that these represent the importance of the different
attributes in the prioritization problem. This is a common, but not techni-
cally accurate understanding.
The mathematical use of the weights is to change the natural attri-
bute scales into a common unit of value. For each attribute in Table 2-1,
there is a “most preferred” and “least preferred” level or category of the
attribute, where “most preferred” means leading to the highest contribu-
tion to the priority value of the vaccine candidate. These most and least
preferred categories define the range of value through which that attribute
can change.
If the user defines a single-attribute value function, then each attri-
bute will be equal to 1.0 for the most preferred category and to 0.0 for the
least preferred one. Other categories between these are scaled linearly
between these end values of the scale. A two-level attribute is simply scaled
1 or 0. A three-level attribute is scaled 1.0, 0.5, and 0.0. A four-level attribute
will be scored as 1.0 if the attribute is in the highest category, 0.67 if in the
second most preferred, 0.33 in the next lower category, and 0.0 in the low-
est category. And, a five-level attribute is scaled 1.0, 0.75, 0.50, 0.25, and 0.0
from the most to the least preferred category.
Again, these linear scales are an approximation used to simplify the
model for users. The next step in modeling is to allow users to specify the
spacing between the categories on these single-attribute scales.
Consider two attributes in Table 2-1, say Cost-Effectiveness (CE) and
Serious Pandemic Potential (SPP). The least-preferred level of CE is Level
4, and the least-preferred level of SPP is Level 2. Suppose there is a vaccine
OCR for page 63
63
Modeling Strategy: From Single Attribute to Multiple Attributes
candidate that has CE of Level 4 and SPP of Level 2. To determine which
should have the higher weight—CE or SPP—the question arises: Which
change improves the overall priority of the vaccine by the larger amount,
changing CE to its Level 1 or changing SPP to its Level 1? If the answer is
to change CE, then the weight assigned to CE should be larger than the
weight assigned to SPP, and vice versa.
In the attribute selection and weighting phase of this model, the
user is asked to pick those attributes from Table 2-1 that should serve as
the basis of comparison for all vaccine candidates. The most- and least-
preferred levels of each attribute are displayed to aid this choice. The user
is instructed to pick the subset of attributes for which a change from lowest
to highest level marks a significant change in priority of a vaccine. This sub-
set is then to be rank ordered using exactly the same question as above—
sorting the attributes pair-wise according to how much change in priority
is implied by changing attributes from least to most preferred levels. The
attribute at the top of the user’s rank order should have the largest implied
change in overall priority when it changes from least to most preferred, and
the attribute at the bottom of the user’s rank order will result in the least
change in priority when it changes from the least- to the most-preferred
level.
Selecting more than seven or eight attributes results in diminished
or negligible weights for attributes ranked below 8 (see Table 2-3). This is
not to say that the user is restricted from selecting all of the 29 attributes
in Table 2-1. But it is true that a handful of attributes generally contain the
most weight in establishing priorities. Adding additional attributes beyond
seven or eight is unlikely to lead to a decisive change in the priority order of
the vaccine candidates. However, Edwards, in an extended case study using
multi-attribute utility theory to rank different desegregation plans for the
Los Angeles school district (Edwards, 1979), observed that if a number of
groups holding strong opinions are attempting to negotiate differences and
agree on a ranking of decision alternatives, then one can end up including
many attributes to make sure that each group sees all attributes of impor-
tance to its viewpoint in the final model. Edwards ended up using more
than 100 attributes to tackle this challenge. But such efforts are very rare.
Five to fifteen attributes in the final model is much more common (von
Winterfeldt and Edwards, 1986).
The risk of double counting
The committee understands that the model (as presented) carries some
risk of double counting some attributes. Double counting in multi-attribute
OCR for page 64
64 RANKING VACCINES: A Prioritization Framework
utility theory means putting weight on a pair (or a larger set) of attributes
that are highly correlated. The higher the correlation between the attri-
butes, the higher is the chance for double counting. The consequence of
double counting is that users who include highly correlated attributes will
( perhaps inadvertently) put more weight on the “concept” measured by
these attributes than intended. The effect on final value scores will depend
in part on how many attributes are included by the user and how high the
correlated attributes are placed in the user’s ranking. If the user includes
10 to 15 attributes and places the highly correlated ones near the bottom
of the list, the rankings will not change much, since they will receive little
weight anyway. However, placing two highly correlated attributes at the
top of a short list in the value function can lead to greater emphasis on that
underlying concept than perhaps intended.
To avoid double counting, the committee selected for inclusion only
those value attributes that are intended to capture something desirable
about a vaccine that (because of limitations to the sub-sectioning of popula-
tion variables) could not be captured directly in computed attributes. Thus
the committee sought to exclude from consideration qualitative attributes
that were otherwise used in the computation of quantitative attributes.
For example, the rate of uptake of a vaccine and a vaccine’s efficacy
rate are used in the calculation of the number of persons effectively vac-
cinated (and hence in the calculation of reduced disease burden). Thus to
include the rate of uptake or the efficacy rate as separate qualitative attri-
butes would create the risk of double counting, and hence they are omitted.
The most obvious of these double-counting risks would involve the
use of DALY or QALY measures of health gains (or losses). While they are
not exact mirror images of one another, the DALY and QALY measures are
sufficiently similar that the SMART Vaccines software blocks the simulta-
neous use of both as indicated attributes. If users select an efficiency mea-
sure, they can use $/DALY or $/QALY, but not both.
There still remains some potential for double counting. For example,
deaths averted contains some of the same information as DALYs averted (or
QALYs gained), but these measures do contain independent information
about the disease burden. Deaths averted would implicitly count each life
saved as the same, no matter what the age of the individual. Life years saved
or the more sophisticated QALYs saved contain the additional dimension of
duration of the saved life. The 1985–1986 IOM study used infant mortality
equivalents prevented for a similar reason—to account for the longevity of
the surviving persons.
Similarly, combinations of one or more computed quantitative vari-
ables can closely correlate with other computed variables. For example,
OCR for page 65
65
Modeling Strategy: From Single Attribute to Multiple Attributes
premature deaths averted per year can be closely approximated by the
combination of incident cases prevented per year and fatality proportions.
Thus including all three of these variables would lead to double counting.
These are not identical in the case where a disease has lingering side effects
that cause mortality in later years. An example would be an infection that
created a chronic condition with some later-year mortality risk. Presently,
the software does not take into account such nuances of double counting.
However, the committee’s approach to dealing with these double-counting
risks will necessarily involve more sophisticated programming in future
versions of SMART Vaccines.
Discounting and inflation
Discounting involves making events that occur in the future commensurate
with those that occur in the present. Future events are brought to a “pres-
ent value” by discounting them at a pre-selected annual rate. The default
value in SMART Vaccines Beta is set at a discount rate of 3 percent, which
is presently the standard rate in the U.S. cost-effectiveness literature for
health and medicine (see Gold et al., 1996; Ramsey et al., 2005), but the
user can alter this at any point. With discounting at 3 percent, an event that
occurs 1 year into the future—cost or benefit—carries only 97 percent as
much value as one occurring in the present year. An event occurring 2 years
into the future as a present value is weighted at 0.972, or 0.9409. One occur-
ring 3 years later would have a present value weight of 0.973, or 0.9127. The
greater the discount rate—for example, 5 percent instead of 3 percent—the
faster these present-value weights diminish over time.
Perhaps most important, SMART Vaccines Beta discounts both costs
and benefits at the same rate. The logic for this comes from an extended
discussion in the literature of cost-effectiveness analysis that generally
concludes that discounting benefits and costs at the same rate is the only
appropriate strategy (Keeler and Cretin, 1983). In its current version,
SMART Vaccines Beta does not allow for different discount rates for costs
and benefits.
The user must use the same discount rate for all candidate vaccines
that are being compared. Using different discount rates could seriously dis-
tort the comparisons between vaccines. However, a feature that allows one
to specify different discount rates for different vaccines has been included
in SMART Vaccines Beta.
A separate issue remains regarding the handling of anticipated infla-
tion rates within the economy in which the vaccine comparisons are being
made. This is a challenge that is distinct from the question of discounting.
OCR for page 66
66 RANKING VACCINES: A Prioritization Framework
Many cost–benefit analyses presume some background rate of inflation in
the economy—for example, 2 percent per year—so that $1 million in costs
this year becomes $1.02 million in costs the following year. Adjusting for
inflation before discounting is equivalent to simply computing all future
economic costs and benefits in today’s dollars at today’s prices and not
worrying about what inflation might be in the future. This is the approach
taken by the current model. This is done to avoid questions concerning
what inflation rate is appropriate—consumer price inflation, monetary
inflation, or sectoral inflation confined to health care. For example the
inflationary growth in wages, used to measure worker productivity losses
and gains, is quite different from inflation in the costs of health care, which
itself is a market basket of services and durable goods with different rates
of inflation.
Time horizon and uncertainty
This model always operates within a fixed 100-year time horizon. This has
been done to simplify the software programming and to reduce the poten-
tial for coding errors. SMART Vaccines Beta does not include the ability to
set distributions on the input parameters to reflect uncertainties relating
to the disease or vaccine data. Therefore, in its current version the multi-
attribute output values do not have standard errors. A dynamic sensitiv-
ity analysis may be required to detect changes in the priority score with
changes in key values. These possibilities, along with others, are discussed
in Chapter 3.
The committee’s prototyping and testing efforts are described in
Chapter 3, which also provides representative screenshots of SMART Vac-
cines Beta.
