Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 299
6
Efficient Analysis of HIV Care
Indicators and Dissemination of
Data by Federal Agencies
In this chapter the committee describes how federal agencies can ef-
ficiently analyze indicators and disseminate data to improve the quality
of HIV care (statement of task question 5). The chapter begins with an
overview of the challenges to the analysis of indicators, including those re-
lated to combining data drawn from multiple sources, and how to address
those challenges. The committee then describes how federal agencies can
efficiently disseminate HIV care data to improve care quality. The chapter
ends with the committee’s conclusions and recommendations.
EFFICIENT ANALYSIS OF HIV CARE
INDICATORS BY FEDERAL AGENCIES
As discussed in Chapter 3, no single data system can be used to gauge
the impact of the National HIV/AIDS Strategy (NHAS) and the Patient
Protection and Affordable Care Act (ACA) on improvements in HIV care.
Rather, estimates of the indicators of clinical HIV care and mental health,
substance use, and supportive services recommended by the committee
often will require the use of data elements from two or more data systems.
Combining data from multiple systems may also be necessary to compen-
sate for the weaknesses of any individual data system, such as a lack of rep-
resentativeness of the population of people living with HIV/AIDS (PLWHA)
or incompleteness of data (e.g., due to a low response rate).
The committee was asked to describe how federal agencies can ef-
ficiently analyze indicators. The data systems described in Chapter 3 that
are maintained by federal entities represent a mix of surveillance (e.g., the
299
OCR for page 300
300 MONITORING HIV CARE IN THE UNITED STATES
National HIV Surveillance System [NHSS]), claims (e.g., Medicaid Statisti-
cal Information System), and programmatic (e.g., Ryan White HIV/AIDS
Program) data sources, as well as epidemiologic studies of PLWHA (e.g, the
Medical Monitoring Project). Efficient analysis of the indicators will require
overcoming challenges to combining data across these disparate systems.
One analytic challenge to the efficient analysis of indicators relates to
differences in the way that data systems operationalize data elements or
define concepts to allow them to be measured. An area in which this may
be relevant is in the calculation of indicators for subgroups of PLWHA,
because data systems may vary in how they define certain demographic
data such as income, geographic marker of residence, race or ethnicity, and
sex or gender. Another challenge is differences across data systems in the
periodicity for particular data elements. Although claims systems will have
continuous data on dispensing of antiretroviral drugs, the Ryan White HIV/
AIDS Program collects information on whether antiretroviral drugs were
prescribed within a 12-month reporting period. This presents an obstacle
to combining data from these systems for purposes of estimating the pro-
portion of PLWHA who were or were not on antiretroviral therapy (ART)
during a given period. Although technically difficult, there are approaches
to deal with the analytic challenges of combining data, as discussed below.
Additional impediments to the efficient analysis of the indicators by
federal agencies that relate to combining data from multiple systems in-
clude the current lack of an infrastructure to support the secure exchange
of health information across health information technology systems (e.g.,
electronic health records) and organizations, and other barriers to data
sharing. These issues are discussed in Chapters 4 and 5 of this report.
An Example of Challenges to the Efficient Analysis
of an Indicator for Clinical HIV Care
One of the core indicators for clinical HIV care recommended by the
committee (see Recommendation 2-1 in Chapter 2) is the proportion of
people with diagnosed HIV infection and a CD4+ cell count <500 cells/
mm3 who are not on ART among all patients who receive such counts. To
define this indicator more precisely, one must take timing into account.
For example, one might ask: What proportion of individuals who received
a CD4+ cell count measurement of <500 cells/mm3 in 2011 also received
ART at any point in 2011. Although this definition is clear, it suffers from
the problem that those who received such a count late in 2011 had less op-
portunity to receive ART in that year. Therefore, it may make more sense
to rephrase the question: How many individuals who received a CD4+ cell
count <500 cells/mm3 in 2011 received ART treatment within a fixed win-
dow of time (e.g., 6 months) of receipt of that measurement. In addition
OCR for page 301
301
EFFICENT ANALYSIS OF HIV CARE INDCATORS
to estimating a population average, there is also interest in estimating the
effects of demographic factors or insurance status on this indicator.
To estimate such an indicator requires information on date of measure-
ment and level of CD4 count as well as date of ART prescriptions given
or filled. Some data sources, such as health maintenance organizations
(HMOs; e.g., Kaiser Permanente), the Department of Veterans Affairs (VA),
and federal prisons provide all of the relevant information needed, permit-
ting a relatively straightforward estimation for subsets of the population.
However, analytic issues arise from the fact that patients may leave these
systems at any point—possibly after a CD4+ cell count <500 cells/mm3 is
measured but before the prescription is provided or 6 months have elapsed.
Furthermore, delays in reporting (e.g., of HIV/AIDS cases, CD4 counts)
must be taken into account, particularly if the goal is to investigate trends
over time. In addition, patients may die within 6 months of receiving a
CD4 count—a situation that makes it impossible to obtain the indicator.
For patients who leave a system before their contribution to the indica-
tor can be assessed, it is important to make use of the available partial
follow-up information in an attempt to avoid, or at least reduce, bias. This
is fairly straightforward using methods for failure-time data if the loss to
follow-up is not informative (i.e., unassociated with greater or lower risk
of starting treatment). If it is informative, appropriate methods must be
used to minimize bias; however, unbiased estimation is possible only if all
potentially confounding variables are available (a very unlikely situation).
To investigate the effect of demographic and other factors on the risk of
not receiving appropriate ART, regression methods can be used. Limitations
arise from losses to follow-up, as described above, as well as from the fact
that with the exception of the NHSS, which captures data on the vast ma-
jority of people identified with HIV/AIDS in the United States, none of the
data sources is representative of either the American population as a whole
or any particular demographic group.
The limitation of representativeness can be addressed by making use
of other sources of data that have broader coverage. To do so, however,
one must make use of data systems that provide only part of the necessary
information by combining them in some way. For example, the NHSS pro-
vides dates of measurements and CD4 counts but not (reliably) the time
of receiving ART. By contrast, Medicare and Medicaid databases provide
information about dates of ART prescriptions filled but not CD4 counts.
In the absence of unique identifiers, no direct linkage between databases
can be made. However, combining across sources is still feasible through
linkage by demographic factors. For example, suppose one knew that for
one demographic group in a given state, 400 people had CD4+ cell counts
<500 cells/mm3 at some point in 2011 among 600 people who had CD4+
cell counts drawn. Suppose one also knew for this group that 300 people
OCR for page 302
302 MONITORING HIV CARE IN THE UNITED STATES
received or filled prescriptions for ART. One then would know that at a
minimum, there had to be 100 patients who should have been on ART but
were not. In fact, however, the number might be considerably greater be-
cause some of the ART use may have been among patients who had CD4+
cell counts of 500 cells/mm3 or greater. However, if one could estimate this
number from other sources (for example, from people within HMO-type
systems who are similar in demographic category to those under study), the
estimate could be refined further. Suppose that of the 200 people who never
had a CD4+ cell count <500 cells/mm3 during the year, one estimates, from
some other data source, that about 100 of them were on ART. Then one
could estimate that about 200 of the 400 patients who should have been
receiving ART were not.
The above discussion illustrates the underlying logic for making infer-
ence but does not address the question of uncertainty in estimation. Of
course there would be errors associated with all of these estimates. How
to calculate the variability in estimates obtained by combining data from
different sources is an area of active research. Bayesian methods have
been used in a variety of settings to characterize the uncertainty associated
with such estimates, reflecting the limitations of the data and the need to
combine across sources. Similarly, Bayesian methods can also be used to
conduct regression analyses that would allow for estimation of the effect of
demographic factors on risk of receiving inadequate treatment.
Issues in Combining Information
Many problems can bedevil analyses of data sets that are derived from
clinical program or public health systems and from which treatment or
intervention effects are being estimated; these include missing data, un-
known population sizes and denominators, and sampling bias. Analysis of
randomized studies generally also suffers from these challenges, since they
are subject to some level of participant attrition, unplanned crossovers,
and inadvertent unblinding. Combining sources of information can help
to overcome shortcomings in each source but creates new challenges for
the analyst, as described in the illustration above. These challenges arise
from the fact that linkage between sources at the individual subject level
may be uncertain or impossible, and even when linkages with high levels of
certainty are possible, all of the relevant information may not be available
on all subjects. Furthermore the level of precision of information may not
be equal across studies and optimal estimation may have to take this factor
into account as well. A large and growing body of work regarding strategies
and methods for combining information is now available.
In 1992, the National Research Council issued an important report
titled Combining Information: Statistical Issues and Opportunities for
OCR for page 303
303
EFFICENT ANALYSIS OF HIV CARE INDCATORS
Research which described many of the principles and methods associated
with combining information (NRC, 2002). Since that time, a number of
developments in methods for combining data from different sources have
occurred that could be applied to HIV research. For example, Bayesian
two-stage hierarchical models have been employed in environmental health
studies that relate air pollution to mortality. The first stage of such studies
estimates the impact on mortality of air pollution for different cities of in-
terest, after controlling for confounding factors. The second stage combines
the estimates across cities using a Bayesian hierarchical model (Lindley and
Smith, 1972; Morris and Normand, 1992) to obtain an overall estimate
and to explore whether some of the geographic variation can be explained
by site-specific explanatory variables (Dominici et al., 2000). Such tech-
niques would also be useful if, for example, there was interest in relating
community-level factors—such as prevalence or incidence of disease, access
to health care, poverty or homelessness rates—to such health outcomes as
HIV morbidity or mortality.
Many of the problems that arise in combining information can be
viewed as related to the issue of missing data. For example, the indicator for
a link between individuals may be seen as missing. Missing data are handled
in a wide variety of ways from the ad hoc (analyze only complete cases) to
sophisticated methods for accommodating incomplete observations.
One approach to dealing with missing data is imputation—replacement
of the missing observation with the best estimate of what it would have
been had it not been missing. Such methods, however, tend to underesti-
mate the uncertainty that arises from the missing data. Multiple imputation
addresses this concern using Bayesian methods (Little and Rubin, 1987).
Likelihood-based methods are also useful; these involve the development
of a likelihood for just the observed data. In either case, one must have
a statistical model for the generation process of the data, including the
probability of its being observed. Given the importance of such models,
considerable effort has been made to expand their flexibility, by allowing
not only fully parametric but also semiparametric models (Tsiatis, 2006).
In some cases it may be possible to make inferences about the sizes of
populations of interest using capture-recapture methods; these are useful
in settings when collection of complete data (i.e., a full enumeration of
the populations) is not feasible or affordable. For example, as described
below, these have been used to estimate the size of an injection drug-using
population.
In addition to the problem of missing data, analyses of observational
data intended to produce causal estimates of the impact of factors, such
as demographics or insurance status, on outcomes must take into account
confounding factors. There is an enormous literature on adjustment for
confounding factors, as well as increased interest in causal modeling for
OCR for page 304
304 MONITORING HIV CARE IN THE UNITED STATES
this purpose. One approach—use of marginal structural models—has re-
ceived increasing attention because of its ability to handle confounding
factors that vary over time (Suarez et al., 2008). All of these techniques
are relevant to the charge to the committee to explore the opportunities
and limits of data sources for HIV program and outcome evaluation in the
United States, but they by no means capture the breadth of methodology
available to cope with data limitations that may bias or confound results
and distort conclusions.
Multiple Imputation for Missing Data
Missing data can arise from settings in which people are asked about
sensitive personal data, when resource constraints limit the completeness
of data collection, or when certain items of information are not routinely
collected. When a given variable is essential for a particular evaluation,
analyses only of complete cases introduces many threats to inference: (1)
bias can be introduced because persons with missing data may be system-
atically different from those with complete data; (2) statistical power can
be reduced when many cases are deleted from analyses due to missing
data; (3) resources can be wasted—for example, when 95 percent of data
are collected on someone, but due to the 5 percent missing data, the entire
data block is left unused; and (4) ethical obligations to research subjects
can be compromised when they have inconvenienced themselves under the
assumption that they were doing this for biomedical or behavioral research,
but the investigator discards their data due to missing variables.
Data on sensitive topics such as sexual risk behaviors or drug use may
be limited by nonresponse bias or biases stemming from socially sensitive
responding. These biases present a special challenge to the collection of
data for surveillance and for epidemiologic research studies where sexual
behaviors or drug use may be relevant (Fenton et al., 2001). Multiple im-
putation helps to circumvent the need to eliminate subjects with partially
observed data imputing (predicting) values for missing variables. Such
imputation requires a statistical model for the complete data (including
the unobserved portions) and for the process that led to the observed
pattern of missing observations. This model is used to predict the missing
observation based on the individuals for whom the data were observed.
The posterior distributions of the unobserved values given the observed
data can then be calculated. Since such calculations may be difficult, Little
and Rubin (1987) propose a resampling-based approach for their calcula-
tion. Like any approach for handling missing data, its validity depends
on the correct specification of a model for the process that generated the
missing data.
OCR for page 305
305
EFFICENT ANALYSIS OF HIV CARE INDCATORS
Capture-Recapture Methods
When the population of interest has not been enumerated and a survey
of the prevalence of a condition or size of a subgroup in a community is
impractical or otherwise unfeasible, the capture-recapture method may be
used. This technique derived from the field of population ecology (Stephen,
1996). For example, one can capture mosquitoes, dust them with harm-
less florescent material, and release them. The proportion of recaptured
mosquitoes in a day or two (allowing sufficient time for remixing but not
allowing time for significant mortality) can be used to estimate the total
number of mosquitoes in the local population, assuming random mix-
ing and equal probability of selecting labeled and unlabeled mosquitoes.
Similarly, small mammals may be trapped, tagged, and released, and then
a second trapping recaptures new and old (tagged) mammals, enabling a
population estimate.
In human biology and epidemiology, the completeness of population
ascertainment can be indirectly estimated using capture-recapture, as with
estimations of persons who need HIV therapy, drug addiction services, or
other social or medical services. Thus, persons must be “captured” and
“marked,” to borrow the ecology model, such that they are available for
recapture after release. Sometimes in epidemiology, this is literal, as with
prisoners who are injection drug users (IDUs), who are arrested but released
after a short time in jail. The proportion who return to jail may be used to
estimate the proportion of drug users at risk of being arrested (presumably
a large proportion of IDUs); combined with population HIV estimates, the
absolute number of HIV-infected drug users can be estimated (Drucker and
Vermund, 1989; Dunn and Ferri, 1999).
One may estimate the size of a population from just two samples or
through multiple samples. Capture histories may be analyzed to estimate
migration, life span, or size in the population of interest. A simple formula
reflects the core principle of the basic capture-recapture approach. This sim-
ple model requires strong assumptions such as full mixing of persons who
have been “captured” and “released” (as with hospitalized patients who
go home) with the general population. The time-to-recapture estimation
must be long enough to permit remixing and short enough for estimation
to be relatively unaffected by deaths, out- and in-migrations, and failure to
identify “marks.” The latter may occur, for example, when rehospitalized
patients use different names when entering an institution. If the assump-
tions are met, the formula is expressed as: N = MC/R (where N = total
population size estimated; M = total number of persons “captured” and
“marked” [i.e., identified] on the first occasion; C = total number of per-
sons “captured” on the second visit; and R = number of identified persons
“marked” from the first occasion that were then reidentified on the second
OCR for page 306
306 MONITORING HIV CARE IN THE UNITED STATES
occasion) (Chao et al., 2001; Hook and Regal, 1995; International Working
Group for Disease Monitoring and Forecasting, 1995a,b; Stephen, 1996).
Marginal Structural Methods and Models
Marginal structural models estimate treatment or intervention effects
in observational studies by statistical strategies of controlling for selection
bias and confounding variables (Robins, 1999). The fundamental concept
behind marginal structural modeling can be explained as follows: Suppose
one wishes to compare exposures A and B, which may vary over time, in
some population and suppose that, at each time, one could create an iden-
tical copy of each study subject. If the actual subject had exposure A at a
given time, we would give the copy exposure B and vice versa. One could
then compare each subject to his or her copy. We refer to the outcomes for
each of the imaginary copies as “counterfactuals” and treat them as miss-
ing data. Inverse-probability weighting (IPW) is an approach to handling
missing data that reweights observations by the inverse of the probability
that they are made. Marginal structural models use IPW to deal with the
unobserved (“missing”) counterfactuals. Using IPW and marginal structural
model procedures reweight data sets so that treatment and covariates are
not confounded.
In “confounding by indication,” an “exposure” is linked to a true
causal exposure (e.g., condom use and commercial sex work) but does not
itself contribute to the outcome. For example, condom use may be statisti-
cally and positively linked to HIV risk, which is counterintuitive (Holmes
et al., 2004), but this association may arise because of confounding by
indication (e.g., disproportionate use of condoms by sex workers in the
population studied). When this occurs in an observational study, the as-
sociation of the putative risk factor cannot be accurately attributed to the
outcome of interest unless one has measured all of the exposures and the
relevant confounding factors.
Estimation of causality must take into account time-dependent con-
founding, and marginal structural models can address selection bias and/
or confounding in such analyses. However, inclusion of such factors as
time-varying covariates in longitudinal models does not correct for this
bias. Such bias occurs most often when “(1) conditional on past treatment
history, a time-dependent variable is a predictor of the subsequent outcome
and [is] also a predictor of subsequent treatment; and (2) [when] past treat-
ment history is an independent predictor of the time-dependent variable”
(Suarez et al., 2008).
Marginal structural models can be used for causal inference even from
nonexperimental designs, comparing treatments or interventions, as long as
information is reasonably accurate, all confounders are measured, and cen-
OCR for page 307
307
EFFICENT ANALYSIS OF HIV CARE INDCATORS
soring either is noninformative or can be modeled accurately as a function
of known covariates. Better control of confounding than available from
simple parametric regression models alone may bring some observational
data closer to values that would be measured in a randomized controlled
clinical trial. One recent example comes from a study showing that hor-
monal contraception is a risk factor for HIV acquisition in African women
(Heffron et al., 2011). Marginal structural model analyses were used to as-
sess the validity of the Cox proportional hazards regression from this large
observational couples study.
Here the committee describes only a few of the challenges that arise in
the use of observational data to make inferences about outcomes or service
coverage (Teresi, 1994) and the approaches to dealing with them. Nonethe-
less, the committee seeks to illustrate a few modern statistical methods to
make surveillance and programmatic data more useful for evaluation pur-
poses and to illustrate the inherent challenges, both to data collection and
to analyses. Correct application of these and other relevant techniques can
improve the chances that inferences drawn from imperfect data are valid.
Analysis of Indicators Involving Small Subgroups of People Living with
HIV/AIDS
Tracking reductions in HIV-related health disparities will require analy-
sis of indicators by race and ethnicity, sexual orientation, and other de-
mographic variables. The NHAS is aimed at improving access to care and
health outcomes for PLWHA and reducing HIV-related health disparities
at the national level. Yet, analysis of indicators may occur at a local level,
such as to disseminate information to local health departments and HIV
care providers on the status of the HIV epidemic in their jurisdictions. In
some communities of the United States, the number of individuals who
comprise a specific demographic group (e.g., racial and ethnic minority men
who have sex with men) may be small. Because the statistical power of an
indicator estimate is linked to the number of observations in a sample, small
subgroups limit the precision of estimates of care indicators and the ability
to compare them with other subpopulations of PLWHA. In epidemiologic
studies, investigators may have little choice but to pool very small subpopu-
lations with the larger study population because there is insufficient power
to extract the effects of defining subgroup characteristics. With respect to
the NHAS, however, this would defeat the purpose of using indicators to
track improvements in HIV-related disparities.
Statistical methods for inference may be used for the analysis of indica-
tors involving small subgroups of PLWHA. In general, Bayesian methods
are useful for combining information about prevalence, incidence, or treat-
ment effects across different population subgroups (Han and Chaloner,
OCR for page 308
308 MONITORING HIV CARE IN THE UNITED STATES
2005). Group-specific Bayesian estimates are “shrunk” toward (moved
closer to) the mean of the quantity of interest over the population included
in the combined data set. Because the amount of shrinkage depends on
the amount of available information, the smaller the size of the subgroup,
the greater will be the reliance on the estimate of the mean. In addition,
group-specific Bayesian estimates are sensitive to assumptions regarding
the distributions of the random effects; the most common approach of as-
suming normal distributions leads to the greatest shrinkage. Using other
types of random effects distributions, such as Student t or mixtures, can
reduce the amount of shrinkage, since they have longer tails and, therefore,
allow for a greater probability of outlying values. As an alternative, one
can base inference on nonparametric approaches, which can achieve the
same goal. Posterior distributions may also tend to be flatter—implying
lower precision in estimates—because the strong normal assumption can
convey a sense that there is more information on which to base inference
than is truly the case if the distributions are nonnormal. While Bayesian
methods provide posterior distributions for any subgroup, no matter its
size, the inference for that group will rest most heavily on the mean and on
underlying assumptions if the subgroup is small. More advanced statistical
methods, such as those that do not require parametric assumptions for the
distributions of the random effects, can provide more reliable and robust
results in this setting.
Growing numbers of studies indicate that social status modifiers such
as race and ethnicity, nativity (place of birth), sexual orientation, geo-
graphic location, and drug use status often have an impact on important
measures of HIV care (e.g., Kempf et al., 2010; Lillie-Blanton et al., 2009;
McGowan et al., 2011). For these subpopulations, among whom social
status contributes to their risk environment (Farley, 2006; Rhodes, 2009)
and treatment access, assumptions of normality of the distribution of ran-
dom effects may be especially problematic, and approaches that allow for
the existence of outliers are particularly needed. One potential consequence
of overshrinkage is underestimation of the impact of indicators of social
status, such as geographic location, economic status, or drug status, on care
experiences. And so although parametric approaches can provide some care
data on individuals in these subpopulation groups, they lie within a more
restrictive set of assumptions that could temper the use of the results for
policy changes.
Epidemiologic studies are an important source of data on care and
supportive services received by PLWHA. Health research in general has
historically been plagued by an inability to recruit and retain large numbers
of racial and ethnic and socioeconomically diverse populations, particularly
of sexual minorities or the homeless (Levkoff and Sanchez, 2003; Moreno-
John et al., 2004; Sengupta et al., 2000), although studies of PLWHA may
OCR for page 309
309
EFFICENT ANALYSIS OF HIV CARE INDCATORS
do better than studies of other populations in terms of representativeness.
Although helpful, statistical methods cannot make up for a lack of sufficient
data to estimate indicators for small populations of PLWHA. The develop-
ment of precise indicator estimates would be facilitated to a greater degree
by inclusion of those groups in greater numbers in HIV-related studies.
In a given community, there may be subpopulations of PLWHA that are
small in number and have complex health care and supportive service needs
for whom the ability to maintain health care regimens depends on access
to supportive services. Improvements in linkages between data systems that
collect information on clinical care and those that collect information on
supportive services (e.g., housing and transportation services) would help
to ensure the availability of the full range of data needed to estimate indica-
tors for these subpopulations. Although data system linkages will not ad-
dress the problem of low statistical power in analyses designed specifically
to provide estimates for small subpopulations of PLWHA, nonparametric
methods can be used to provide some insights into care needs.
Increased support for training of HIV/AIDS researchers in statistics and
methodologies may facilitate the development of expertise in the analysis
of data for subpopulations of PLWHA. Such investment could speed the
provision of effective treatment to all communities and thereby improve
control of HIV transmission.
DISSEMINATION OF DATA TO IMPROVE HIV CARE QUALITY
Analysis of the HIV care and related indicators identified by the com-
mittee will generate data of interest to a number of stakeholders, including
federal and state agencies and policy makers, state and local health depart-
ments, health care systems (e.g., HMOs, VA, prisons), individual provid-
ers, consumers (patients), and academic researchers. Properly presented,
the information provided to each audience has the potential to improve
the quality of HIV care in the United States. Policy makers, agencies, and
health departments may use the information to direct resources and poli-
cies toward areas that are most problematic (e.g., access to health care or
mental health, substance use, or supportive services to improve linkage to
or retention in care). Health care systems and individual providers may use
the information to inform their provision of quality HIV care and to target
patient education efforts. Individual patients, patient groups, and patient
advocates could use the information to direct personal and group advocacy
efforts for access to needed services. Academic researchers could use the
information to support research proposals and projects that might generate
additional information to further improve the quality of HIV care.
The committee’s review of the existing systems that capture data rel-
evant to HIV care shows that many data on various aspects of HIV care
OCR for page 310
310 MONITORING HIV CARE IN THE UNITED STATES
currently exist. However, existing data often are not used to the fullest
extent possible. Although government agencies and some state and local
health departments make some de-identified data available publicly, in
other cases the data reside with the agencies that require the reporting
and are not made accessible to the public (HHS, 2010), including to the
programs and providers who reported the data in the first place. Not only
is broad dissemination of data on HIV care important for improving care
by engaging as many stakeholders as possible; the return of information to
reporting programs and providers increases the collaborative nature of the
relationship, provides them with useful feedback, and may motivate them
to further increase reporting compliance (CDC, 2011, p. 5-32).
Data Dissemination by Federal Agencies
Federal agencies, including the Centers for Disease Control and Preven-
tion (CDC), the Health Resources and Services Administration (HRSA), and
the National Institutes of Health (NIH), have been disseminating health-
related information for decades. Until the advent of the Internet, which
enabled agencies to disseminate large amounts of information, dissemina-
tion had primarily involved making paper copies of documents available to
the public (OMB, 2011). In the context of increasing federal information
dissemination, Congress passed the Information Quality Act (IQA), also
referred to as the Data Quality Act, in December 2000. The IQA required
the Office of Management and Budget (OMB) to issue guidance to federal
agencies to ensure the “quality, objectivity, utility, and integrity” of infor-
mation disseminated to the public. In response, the OMB issued Guidelines
for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity
of Information Disseminated by Federal Agencies, effective October 2001.
These guidelines require that information quality be treated as an integral
step in the information development process. Federal agencies must adopt
a basic standard of quality as a performance goal and take steps to incor-
porate information quality criteria into agency information dissemination.
In addition, agencies are to develop a process for reviewing the quality of
information before it is disseminated. OMB designed the guidelines to apply
to a variety of government dissemination activities and to be generic enough
to fit all media (HHS, 2006a).
The IQA also required that government agencies issue their own infor-
mation quality guidelines and establish mechanisms to allow individuals
to seek correction of information maintained and disseminated by federal
agencies that does not comply with OMB guidance (OMB, 2011). There-
fore, Guidelines for Ensuring the Quality of Information Disseminated to
the Public have been issued for several agencies of the U.S. Department of
OCR for page 311
311
EFFICENT ANALYSIS OF HIV CARE INDCATORS
Health and Human Services (HSS). These guidelines describe the types of
information disseminated by the agency to the public; types of dissemina-
tion methods; agency standards for ensuring the quality of information
disseminated; agency administrative complaint procedures; influential sci-
entific, financial, and statistical information; and any special considerations
for agency dissemination. As one HIV-specific example, the types of infor-
mation disseminated by HRSA listed in its guidelines include HRSA HIV/
AIDS Bureau State Profiles that describe spending and service information
for Ryan White HIV/AIDS Programs, including provider characteristics
(e.g., the number and types of organizations in the state that receive Ryan
White HIV/AIDS Program funding), client demographic information, ser-
vice utilization information (e.g., number of patient visits for core medical
services), and characteristics of AIDS Drug Assistance Program clients
(HHS, 2006b; HRSA, 2012). Under “dissemination methods” the guide-
lines say that the state profiles are available through the HRSA HIV/AIDS
Bureau website and that further requests or feedback can be made by phone
or email (HHS, 2006c). Within CDC, dissemination guidance applies to
HIV/AIDS Surveillance Reports and reports for other infectious and non-
infectious conditions (HHS, 2006b).
Considerations in Data Dissemination
Effective and efficient dissemination of data requires careful attention
to several considerations, including audience, definition and presentation of
the message, data quality and interpretation, and method of dissemination
(CDC, 2009; Marriott et al., 2000; Sofaer and Hibbard, 2010a,b).
In HRSA guidelines for ensuring the quality of information dissemi-
nated to the public, CAREWare, a software package used by Ryan White
HIV/AIDS Program providers to track clients and services, is listed as a
means to ensure the quality of information disseminated to the public
(HHS, 2006c). According to the guidelines, CAREWare helps to ensure the
quality of data because it contains consistency and edit checks on input
data. HIV/AIDS Bureau State Profiles, which present state-level data derived
from these data, provide—for each data element—information on data
limitations, rounding, and restrictions where appropriate (HHS, 2006c).
CDC guidance notes that surveillance information is often obtained from
third parties, such as states and grantees, which places limits on quality
assurance. However, the accuracy, completeness, and timeliness of the in-
formation are subject to sample audits, site visits, and an “evaluation for
completeness and consistency with trends and external controls” (HHS,
2006b).
OCR for page 312
312 MONITORING HIV CARE IN THE UNITED STATES
Audience
Defining and understanding the target audience is one of the first steps
in developing a plan for data dissemination (Marriott et al., 2000; Sofaer
and Hibbard, 2010b). Potential audiences for data derived from the full
set of HIV care indicators identified by the committee already have been
identified (federal and state agencies and policy makers, state and local
health departments, health care systems, individual providers, consumers
[patients], and academic researchers). Selection of the appropriate audience
involves consideration of what the data show, the purpose for which the
data are being disseminated, and the message that is to be conveyed.
Federal, state, or local policy makers and agencies would be the pri-
mary target audience/s if the purpose is to increase or redirect the allocation
of resources or to affect policy changes, including the development of new
programs to address specific areas of need. Such programs could focus on
points in the HIV care continuum that the data might indicate are particu-
larly problematic (e.g., continuity of care) or mediators known to affect
those areas (e.g., access to stable housing). Information about improve-
ments on indicators would be useful as well, by showing which current
policies and programs are working.
Public and private health care systems, as well as individual providers,
might be interested in the data for the purpose of evaluation of, and pos-
sible changes in, the HIV care they provide. Such information could permit
systems and providers to identify their areas of strength, as well as areas
for improvement, in the provision of quality HIV care. Research indicates
that dissemination of clinical practice guidelines alone has a minimal effect
on provider knowledge and performance, while combination strategies,
including those with an education component, are more effective (Marriott
et al., 2000). Results of performance indicators also may be more effective
in changing provider behavior (Marriott et al., 2000).
PLWHA, and advocacy groups for PLWHA, are other potential audi-
ences for the information on indicators. The information could be used
to educate individuals regarding areas in which increased attention and
advocacy could improve HIV care.
Depending on what the data show, the dissemination process might
target any of these general audiences or a more specific audience within a
group, such as policy makers representing a particular region of the United
States, HIV care providers who serve patients in a specific demographic
group, or patients of a particular race or ethnicity. Ultimately, audience
selection should depend on the applicability of the data for that audience
and the purpose the data serve (Sofaer and Hibbard, 2010b). Once the
audience is defined, the message and the remainder of the dissemination
process should be geared to that audience (CDC, 2009).
OCR for page 313
313
EFFICENT ANALYSIS OF HIV CARE INDCATORS
Definition and Presentation of the Message
Another critical consideration in effective data dissemination is the
message to be conveyed. Many audiences are not equipped to understand
and process vast quantities of data (Sofaer and Hibbard, 2010a). Data
provided without the expertise to interpret them can cause more harm
than good. Even the language used to present the information may result
in unanticipated misinterpretation (Hibbard and Sofaer, 2010). It is impor-
tant therefore for an agency to have a clear understanding of the message it
wants to transmit and then relay that message to the target audience clearly
and concisely, along with the data to support it (Hibbard and Sofaer, 2010;
Marriott et al., 2000; Sofaer and Hibbard, 2010a). The details of the mes-
sage may vary depending on the target audience, as will the way in which
the message is presented.
Presentation of the message in the most appropriate way for the target
audience is critical to ensure that the message the agency wants to convey
is the one that is received by the audience (CDC, 2009; Marriott et al.,
2000). Considerations of health literacy and numeracy are important when
preparing information for dissemination (Hibbard and Sofaer, 2010; IOM,
2004). Presentations of data from the HIV care and supportive services
indicators and trends in the quality of HIV care over time will use differ-
ent language depending on the audience (e.g., clinical care professionals,
policy makers, program administrators, members of the public). Clinical
indicators of HIV care that are fully comprehensible to HIV care providers
may be incomprehensible to patients or to policy makers. It is important to
make the information relevant to what the audience understands and the
purpose for which it will use the data. Three papers on “best practices in
public reporting” on health care performance data, prepared for the Agency
for Healthcare Research and Quality, discuss a number of the pitfalls in
and solutions to presenting performance data to health care consumers
(Hibbard and Sofaer, 2010; Sofaer and Hibbard, 2010a,b). Although the
papers focus on a specific type of data and target audience, the concepts
presented may be generalized to other audiences and types of information.
Data Quality and Interpretation
As discussed, the IQA mandates that federal agencies develop quality
assurance guidelines for information releases to the public, and a number of
HHS agencies have issued Guidelines for Ensuring the Quality of Informa-
tion Disseminated to the Public. Although it is important for agencies to
present the message clearly, concisely, and in language that is understood by
and resonates with the target audience, it is also important that they include
information about the quality of the data that support the message and the
methods used to interpret them (Sofaer and Hibbard, 2010a).
OCR for page 314
314 MONITORING HIV CARE IN THE UNITED STATES
Factors affecting data quality include the source of the data, quality
within the system, coverage of the data, confidence range, use of proxies,
and analytic methodology applied. The challenge lies in providing suffi-
cient information to permit independent assessment of the data, while not
overwhelming the target audience with information that it cannot or will
not use (Marriott et al., 2000; Sofaer and Hibbard, 2010a). One approach
is to include with the disseminated information a summary, presented in
language accessible to the target audience, of the data and the data analy-
sis, including discussion of limitations or gaps in the data and any other
relevant information that would enhance the audience’s understanding and
evaluation of the data (HHS, 2006b; Marriott et al., 2000; Sofaer and
Hibbard, 2010a). At the same time, the agency could make available to
interested parties full information on the data set and the methodologies
used to assess it (Sofaer and Hibbard, 2010a). CDC, for example, clearly
documents and makes publicly available the statistical processes and meth-
odologies used to derive published information, which allows independent
statisticians to replicate the results (HHS, 2006b).
Evidence suggests that an audience’s acceptance of data is affected by
its perception of the credibility of the data source and the source reporting
the information (e.g., professional medical journal versus popular press), as
well as proximity of the source to the target audience (Marriott et al., 2000;
Sofaer and Hibbard, 2010a). Information intermediaries can help in this
regard. Engagement with organizations knowledgeable about and trusted
by the target audience may assist in the dissemination of information and
help to support the credibility of the information and its source (Sofaer and
Hibbard, 2010b).
Methods of Dissemination
A final consideration for effective and efficient data dissemination is se-
lection of the most appropriate and cost-effective method of dissemination.
As previously mentioned, federal agencies have a variety of dissemination
methods at their disposal, including traditional print media (e.g., reports,
peer-reviewed articles, fact sheets, newsletters), electronic media (e.g., web-
sites, podcasts), and public forums (e.g., conferences, planned meetings)
(CDC, 2009), and frequently more than one method may be employed.
The target audience and the message and data to be conveyed are fac-
tors in the choice of dissemination method. An agency might choose to
prepare a report or paper for a peer-reviewed professional journal if the
goal is transmit the information to health care systems or providers. Re-
ports, newsletters, and fact sheets might be more effective in reaching policy
makers or other agencies. Websites will reach a larger and broader audi-
ence, including members of the public. The type of information and style of
OCR for page 315
315
EFFICENT ANALYSIS OF HIV CARE INDCATORS
presentation used for a professional journal will differ markedly from that
prepared for dissemination on the agency website. The speed or urgency
with which a message needs to be conveyed is another consideration in the
selection of dissemination method.
CONCLUSIONS AND RECOMMENDATIONS
• Estimation of the committee’s recommended indicators for HIV
care and supportive services will often require combining data from
multiple data systems. Making valid inferences about the indicators
across different populations and over time using data from multiple
data systems presents a range of analytic and logistical challenges.
Such challenges will change over time and will have to be reevalu-
ated periodically.
Recommendation 6-1. At least once every 2 years, the Department
of Health and Human Services should reevaluate mechanisms for
combining data elements to estimate key indicators of HIV care
and access to supportive services, analyze the combined data, and
identify and address barriers to the efficient analysis of such data,
including relevant statistical methodologies. To facilitate this pro-
cess, HHS should engage a center of excellence representing broad
areas of expertise that include information technology, statistical
methodologies for combining data, and data system content.
The center of excellence might also include experts in epidemiology
and surveillance; laws and policies that affect access to HIV-related
data; health services research, including insurance; medical infor-
matics, including integration of public and private data sources
to estimate population-level parameters; clinical HIV care and
relevant social services; and community and patient perspectives.
The center of excellence could address questions such as the extent
to which proxy data elements can be used to estimate indicators;
whether knowledge of an indicator for a subpopulation rather
than the whole cohort of PLWHA might be acceptable for some
indicators; and the level of accuracy to be demanded for any given
indicator (e.g., whether estimates are needed within 1, 5, or 10
percentage points) given the potential costs of data collection and
of obtaining very accurate indicator estimates.
• Information on the indicators recommended by the committee will
be of interest to a variety of stakeholders, including policy makers,
health departments, HIV care providers, patients, and researchers.
OCR for page 316
316 MONITORING HIV CARE IN THE UNITED STATES
The disseminated information can be used in numerous ways—
from informing policy decisions to supporting the development of
research projects—that have the potential to improve the quality
of HIV care.
Recommendation 6-2. The Department of Health and Human
Services should report to the public at least once every 2 years on
indicators of HIV care and access to supportive services to foster
improvements in the quality of HIV care and in monitoring prog-
ress toward meeting the goals of the National HIV/AIDS Strategy.
The reporting interval of at least once every 2 years allows for
regular reporting of the indicator data to monitor the NHAS while
minimizing reporting burden and associated costs. To facilitate
understanding and use of the indicator information by stakehold-
ers, dissemination products and strategies may vary depending
on the target audience and message to be conveyed. Information
about the quality of the indicator data (e.g., confidence ranges for
indicators estimates, use of proxy data elements) might be included
in the dissemination product so that stakeholders are aware of the
limitations of the data.
REFERENCES
CDC (Centers for Disease Control and Prevention). 2009. Basic Concepts for Disseminating
and Communicating Surveillance Data. http://www.cdc.gov/pednss/how_to/disseminate_
data/basic_concepts.htm (accessed January 19, 2012).
CDC. 2011. Principles of Epidemiology in Public Health Practice. Third Edition. Self-Study
Course SS-1000. Atlanta, GA: CDC. http://www.cdc.gov/training/products/ss1000/
ss1000-ol.pdf (accessed January 19, 2012).
Chao, A., P. K. Tsay, S. Lin, W. Shau, and D. Chao. 2001. The applications of capture-
recapture models to epidemiologic data. Statistics in Medicine 20:3123-3157.
Dominici, F., J. M. Samet, and S. L. Zeger. 2000. Combining evidence on air pollution and
daily mortality from the 20 largest U.S. cities: A hierarchical modelling strategy. Journal
of the Royal Statistical Society 163(3):263-302.
Drucker, E., and S. Vermund. 1989. Estimating population prevalence of human immunode-
ficiency virus infection in urban areas with high rates of intravenous drug use: A model
of the Bronx in 1988. American Journal of Epidemiology 130(1):133-142.
Dunn, J., and C. P. Ferri. 1999. Epidemiologic methods for research with drug misusers:
Review of methods for studying prevalence and morbidity. Revista de Saúde Pública
33(2):206-215.
Farley, T. A. 2006. Sexually transmitted diseases in the southeastern United States: Location,
race, and social context. Sexually Transmitted Diseases 33(7):S58-S64.
Fenton, K. A., A. M. Johnson, S. McManus, and B. Erens. 2001. Measuring sexual behavior:
Methodological challenges in survey research. Sexually Transmitted Infections 77:84-92.
OCR for page 317
317
EFFICENT ANALYSIS OF HIV CARE INDCATORS
Han, C., and K. Chaloner. 2005. Design of population studies of HIV dynamics. In Deter-
ministic and Stochastic Models of HIV Epidemics and HIV Infections with Intervention,
edited by W. Tan and H. Wu. Hackensack, NJ: World Scientific Publishing Co.
Heffron, R., D. Donnell, H. Rees, C. Celum, N. Mugo, E. Were, G. de Bruyn, E. Nakku-
Joloba, K. Ngure, J. Kiarie, R.W. Coombs, and J. M. Baeten. 2011. Use of hormonal
contraceptives and risk of HIV-1 transmission: A prospective cohort study. Lancet Infec-
tious Diseases doi:10.1016/S1473-3099(11)70247-X.
HHS (U.S. Department of Health and Human Services). 2006a. HHS Guidelines for Ensur-
ing and Maximizing the Quality, Objectivity, Utility, and Integrity of Information Dis-
seminated to the Public. Part 1: HHS overview. http://aspe.hhs.gov/infoquality/guidelines/
part1.shtml (accessed January 19, 2012).
HHS. 2006b. Guidelines for Ensuring the Quality of Information Disseminated to the Public.
Centers for Disease Control and Prevention and Agency for Toxic Substances and Dis-
ease Registry. http://aspe.hhs.gov/infoquality/guidelines/cdcinfo2.shtml (accessed January
19, 2012).
HHS. 2006c. Guidelines for Ensuring the Quality of Information Disseminated to the Public.
Health Resources and Services Administration. http://aspe.hhs.gov/infoquality/guidelines/
HRSAinfo2.shtml (accessed January 19, 2012).
HHS. 2010. Putting data and innovation to work to help communities and consumers improve
health. News Release. http://www.hhs.gov/news/press/2010pres/06/20100602a.html (ac-
cessed January 19, 2012).
Hibbard, J., and S. Sofaer. 2010. Best Practices in Public Reporting No. 1: How to Effec-
tively Present Health Care Performance Data io Consumers. AHRQ Publication No.10-
0082-EF. Bethesda, MD: Agency for Healthcare Research and Quality.
Holmes, K. K., R. Levine, and M. Weaver. 2004. Effectiveness of condoms in preventing sexu-
ally transmitted infections. Bulletin of the World Health Organization 82(6):454-461.
Hook, E. B., and R. R. Regal. 1995. Capture-recapture methods in epidemiology: Methods
and limitations. American Journal of Epidemiology 17(2):243-264.
HRSA (Health Resources and Services Administration). 2012. Ryan White HIV/AIDS Pro-
gram. 2009 State Profiles. http://hab.hrsa.gov/stateprofiles/index.htm (accessed March
6, 2012).
International Working Group for Disease Monitoring and Forecasting. 1995a. Capture-
recapture and multiple-record systems estimation I: History and theoretical development.
American Journal of Epidemiology 142(10):1047-1058.
International Working Group for Disease Monitoring and Forecasting. 1995b. Capture-
recapture and multiple-record systems estimation II: Applications in human diseases.
American Journal of Epidemiology 142(10):1059-1068.
IOM (Institute of Medicine). 2004. Health Literacy: A Prescription to End Confusion. Wash-
ington, DC: The National Academies Press.
Kempf, M., J. McLeod, A. K. Boehme, M. W. Walcott, L. Wright, P. Seal, W. E. Norton, J. E.
Schumacher, M. Mugavero, and L. Moneyham. 2010. A qualitative study of the barriers
and facilitators to retention-in-care among HIV-positive women in the rural southeast-
ern United States: Implications for targeted interventions. AIDS Patient Care and STDs
24(8):515-520.
Levkoff, S., and H. Sanchez. 2003. Lessons learned about minority recruitment and retention
from the Centers on Minority Aging and Health Promotion. Gerontologist 43:18-26.
Lillie-Blanton, M., V. E. Stone, A. S. Jones, J. Levi, E. T. Golub, M. H. Cohen, N. A. Hessol,
and T. E. Wilson. 2009. Association of race, substance abuse, and health insurance cover-
age with use of highly active antiretroviral therapy among HIV-infected women, 2005.
American Journal of Public Health 100:1493-1499.
OCR for page 318
318 MONITORING HIV CARE IN THE UNITED STATES
Lindley, D. V., and A. F. M. Smith. 1972. Bayes estimates for the linear model. Journal of the
Royal Statistical Society 34(1):1-41.
Little, R. J. A., and D. B. Rubin. 1987. Statistical Analysis with Missing Data. First Edition.
Hoboken, New Jersey: Wiley and Sons, Inc.
Marriott, C., C. Palmer, and P. Lelliott. 2000. Disseminating healthcare information: Getting
the message across. Quality in Health Care 9:58-62.
McGowan, C. C., D. D. Weinstein, C. P. Samenow, S. E. Stinnette, G. Barkinic, P. F. Rubiero,
T. R. Sterling, R. D. Moore, and T. Hulgan. 2011. Drug use and receipt of highly active
antiretroviral therapy among HIV-infected persons in two U.S. clinic cohorts. PLoS One
6(4):e18462.
Moreno-John, G., A. Gachie, C. M. Fleming, A. Napoles-Springer, and E. Mutran. 2004.
Ethnic minority older adults participating in clinical research: Developing trust. Journal
of Aging and Health 16:93S-123S.
Morris, C. N., and S. L. Normand. 1992. Hierarchical models for combining information
and for meta-analysis. In Bayesian Statistics 4, edited by J. M. Bernardo, J. O. Berger,
A. P. Dawid, and A. F. M. Smith. Oxford, UK: Oxford University Press. Pp. 321-344
(with discussion).
NRC (National Research Council). 2002. Combining Information: Statistical Issues and Op-
portunities for Research. Washington, DC: National Academy Press.
OMB (Office of Management and Budget). 2011. Guidelines for Ensuring and Maximizing
the Quality, Objectivity, Utility, and Integrity of Information Disseminated by Federal
Agencies. http://www.whitehouse.gov/omb/fedreg_final_information_quality_guidelines
(accessed January 19, 2012).
Rhodes, T. 2009. Risk environments and drug harms: A social science for harm reduction
approach. International Journal of Drug Policy 20(3):193-201.
Robins, J. M. 1999. Association, causation, and marginal structural models. Synthese 121:
151e79.
Sengupta, S., R. P. Strauss, R. DeVellis, S. C. Quinn, B. DeVellis, and W. B. Ware. 2000.
Factors affecting African-American participation in AIDS research. Journal of Acquired
Immunodeficiency Syndromes 24(3):275-284.
Sofaer, S., and J. Hibbard. 2010a. Best Practices in Public Reporting No. 2: Maximizing
Consumer Understanding of Public Comparative Quality Reports: Effective Use of Ex-
planatory Information. AHRQ Publication No. 10-0082-1-EF. Bethesda, MD: Agency
for Healthcare Research and Quality.
Sofaer, S., and J. Hibbard. 2010b. Best Practices in Public Reporting No. 3: How to Maximize
Public Awareness and Use of Comparative Quality Reports Through Effective Promo-
tion and Dissemination Strategies. AHRQ Publication No. 10-0082-2-EF. Bethesda, MD:
Agency for Healthcare Research and Quality.
Stephen, C. 1996. Capture-recapture methods in epidemiological studies. Infection Control
and Hospital Epidemiology 17(4):262-266.
Suarez, D., J. M. Haro, D. Novick, and S. Ochoa. 2008. Marginal structural models might
overcome confounding when analyzing multiple treatment effects in observational stud-
ies. Journal of Clinical Epidemiology 61:525-530.
Teresi, J. 1994. Overview of methodological issues in the study of chronic care populations.
Alzheimer’s Disease and Associated Disorders 8(Suppl 1):S247-S273.
Tsiatis, A. A. 2006. Semiparametric Theory and Missing Data. New York: Springer Science
and Business Media, LLC.
