Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 49
PART III
Illustrative Evaluation Details for Assessment of
PEPFAR’s Performance and Impact
OCR for page 50
OCR for page 51
The IOM was mandated by Congress in the Lantos–Hyde Act of 2008 to conduct a study
that includes an assessment of the performance of U.S.-assisted global HIV/AIDS programs and
an evaluation of the impact on health of prevention, treatment, and care efforts that are supported
by U.S. funding. Part of the charge to the planning committee in developing a plan for this
evaluation was to be cognizant of the requirements and charges mandated for the evaluation (see
Appendix A). To augment the overview of the evaluation design presented in Part II, this part of
the plan partitions and elaborates the areas of interest laid out in the congressional mandate.
The guiding framework of the program impact pathway is applied to each of these areas,
reflecting the committee’s understanding of the rationale for how PEPFAR’s specific inputs and
activities can be plausibly linked to PEPFAR’s contribution to effects on HIV-specific health
outcomes and impacts. This part of the report illustrates the types of questions that will guide the
evaluation of PEPFAR’s activities in prevention, adult and pediatric treatment, care and support,
child and adolescent wellbeing (including orphans and vulnerable children), and gender-related
vulnerability and risk activities. The evaluation will also consider other fundamental activities in
the areas of knowledge management and funding flows; these are considered first in this part of
the report because they underlie the success of all other programmatic areas. This part of the
report culminates with a discussion of cross-cutting activities related to key systems-level goals
that are critical for the long term goals articulated by PEPFAR, such as health systems
strengthening and transitioning to sustainability and country ownership.
As described in Part II, in each of these areas the evaluation questions will be addressed
using a mixed methods approach and layers of investigation and analysis, drawing on a range of
available primary and secondary data sources. By applying a mix of methods, data sources, and
analytical techniques, the committee will arrive at findings that can be triangulated to draw
conclusions about the performance of PEPFAR and its contribution to health impact, even when
any one data source is not sufficient or any one methodological approach is not feasible. The
extent to which specific methods can be applied to answer the evaluation questions will depend on
the timely availability of data that is of sufficient quality to lead to reliable findings. Therefore, the
illustrative questions and the methods and data sources that will be used to address them will
undergo further refinement and prioritization as a result of the operational planning phase
activities described previously.
51
OCR for page 52
OCR for page 53
SECTION 1: PEPFAR’S KNOWLEDGE MANAGEMENT
The management of knowledge and information is critical to the success of any program
because it serves to monitor the activities and effects of the program as well as to guide policies,
priorities, and programmatic decisions. Therefore, assessing the performance of PEPFAR’s
knowledge management activities will be an important part of the evaluation of the performance
of PEPFAR as a whole, as well as an assessment of the forward-looking mechanisms that are in
place for continuous M&E of the program’s progress and appropriate responses.
In PEPFAR I, the primary goal of OGAC’s strategic information (SI) efforts for M&E, in
partnership with implementing agencies, was to support PEPFAR through results-based planning
and implementation, focusing on quality information collection, timely data management and
use, evaluation of best practices, and information for decision making. In PEPFAR II, the goals
have been expanded to include support of the larger PEPFAR mission. To this end, the expanded
SI mission includes building the capacity of countries to improve health outcomes by increasing
and strengthening the use of information for effective stewardship of programs and effective
implementation of efficient, high-quality, and sustainable health systems (Bouey, 2010).
During PEPFAR II, the SI goals also include improved harmonization of USG reporting
needs with country-driven M&E efforts through not only strengthening country capacity and
alignment with national data collection, but also through better alignment with global reporting
requirements to lessen the burden on implementing partners and partner governments (OGAC,
2009h). An increased focus on both program coverage and quality will be reflected in SI efforts
to identify indicators that can give an accurate picture of these two areas (OGAC, 2009d).
Finally, although it continues to recognize that PEPFAR is not intended to be a research
initiative, the PEPFAR Five-Year Strategy outlines the additional goals of improving the
program’s efforts to contribute to the evidence base for HIV interventions and to expand the
amount of publicly available data (OGAC, 2009g). This expanded research effort will prioritize
the evaluation and proactive dissemination of topics that PEPFAR is in a unique position to
address as well as studies that focus on methods to improve program delivery (OGAC, 2009h). It
will also increase the tracking of outcomes, cost-effectiveness, innovation, and impacts in order
to identify timely information regarding the program’s effectiveness and impact (OGAC, 2009h).
Strategic Information Management
Structurally, SI activities at the PEPFAR headquarters level are carried out by USG
implementing agencies with coordination through OGAC. These headquarters-level activities
draw from a wide range of data-gathering sources (see Figure 5).
53
OCR for page 54
54 STRATEGIC APPROACH TO THE EVALUATION OF U.S. GLOBAL HIV/AIDS PROGRAMS
FIGURE 5 PEPFAR headquarters-level strategic information partners and headquarters-level data-
gathering sources.
NOTES: CDC = U.S. Centers for Disease Control and Prevention, Census = U.S. Census Bureau, DoD =
U.S. Department of Defense, HRSA = Health Resources and Services Administration of the U.S.
Department of Health and Human Services, M&E = monitoring and evaluation, OGAC = Office of the
U.S. Global AIDS Coordinator, SI = strategic information, TWG = technical working group, USAID =
United States Agency for International Development.
SOURCE: Adapted from Bouey (2010).
The staff at headquarters are responsible for issuing guidance related to COP submission
and reporting processes, as well as for providing technical assistance. This guidance instructs
country teams on how to successfully complete their reporting requirements. In addition to
annual reporting needs, guidance from headquarters can also provide information on collecting,
interpreting, and updating basic epidemiologic profiles as well as information on how to develop
and incorporate efforts to evaluate new initiatives (OGAC, 2009e). While this guidance focuses
primarily on the processes unique to OGAC, it also offers additional information regarding data
collection and target setting that has the potential for broader applicability.
Within headquarters there is also a SI technical working group. These efforts at the
headquarters level also support M&E activities at the country level through the development of
resources such as an M&E Systems Strengthening Tool, which is designed to help partner
countries prioritize their M&E needs and encourage alignment with a national M&E strategy,
and a Data Quality Assessment Tool (MEASURE Evaluation, 2007; PEPFAR, 2008a).
While management of data collection, storage, and analysis at the country and project
level varies, the primary mechanism for reporting and aggregating these data is via an electronic,
Internet-based program known as the Country Operational Plan Reporting System (COPRS).
Data that are reported via COPRS are collected at the OGAC headquarters level during the
relevant semi-annual or annual program reporting periods, depending on the country (OGAC,
2009d). A portion of these data is released to the public and is also communicated to Congress
via an annual report. Occasionally, they are used by OGAC to produce additional topic-specific
reports (e.g., Report on Gender-Based Violence and HIV/AIDS) (PEPFAR, 2010c). COPRS is
OCR for page 55
55
PEPFAR’S KNOWLEDGE MANAGEMENT
currently in transition due to two factors. First, the Next Generation Indicators Reference
Guidance was recently released, introducing a limited number of new indicators and redefining
some measures that had previously been in use. This guidance was developed in part to support
PEPFAR’s contribution to global efforts to harmonize reporting requirements for HIV/AIDS
initiatives, which aims to reduce the reporting burden of program implementers and to allow
more flexibility and increased local ownership of the design of M&E plans (OGAC, 2009d).
Second, a new generation of COPRS, COPRS II, is in development and is expected to be
deployed in FY2010 (OGAC, 2009e).
Beyond Information Management
In 2008, PEPFAR began a campaign titled “know your epidemic/know your results”
aimed at using information to more closely align program activities with population needs
(OGAC, 2008b). As a result, a focus was placed on developing sustainable SI systems to
“collect, analyze, critically review, disseminate, interpret, display, and strategically use data at all
levels” (OGAC, 2008b). Continuing with this development of SI, the 2009 headquarters
operational plan allocated funds for the development of a “draft knowledge management
strategy,” perhaps in response to the first IOM PEPFAR evaluation recommendation to develop a
detailed, overall strategy for institutionalizing its efforts to function as a learning organization
and to increase its contributions to the global knowledge base (IOM, 2007; OGAC, 2009f).
Coordination among country staff and dissemination of best practices is also facilitated by the
PEPFAR implementers’ meeting held annually in a PEPFAR country. In addition to sharing
information across countries, this meeting includes a variety of breakout sessions dedicated to SI
issues (PEPFAR, 2009a).
In addition, as described above, the recent goals for PEPFAR II emphasized the
important role of expanding the program’s research portfolio to contribute to the publicly
available evidence base, with an emphasis on operations research to improve program delivery as
well as methods for timely assessments of the program’s effectiveness and impact (OGAC,
2009h). Some research activities are already occurring in individual partner countries. For
example, the Public Health Evaluations (PHE), initiated in 2007, are a PEPFAR activity intended
as a source to inform policy and program-level changes. They currently serve as the primary
mechanism through which PEPFAR supports research within countries, including operations
research (Edgil, 2010; OGAC, 2009h). Some PHEs are single country, while others are multi-site
investigations. The selection of PHE proposals (annually solicited from investigators) is
performed by an interagency technical policy group charged with prioritizing areas in need of
evaluation, overseeing the implementation of evaluations, and recommending approvals and
levels of funding for PHEs. In doing so, priority is given to studies that are driven by locally-
identified country needs as well as those that involve local institutions and investigators in the
research process. 15 As of 2008 there were 195 PHE activities (Edgil, 2010), with the most recent
call for proposals issued by the National Institutes of Health in April 2010. 16
15
Personal communication from OGAC, April 9, 2010.
16
Ibid.
OCR for page 56
56 STRATEGIC APPROACH TO THE EVALUATION OF U.S. GLOBAL HIV/AIDS PROGRAMS
PEPFAR’s Monitoring and Evaluation Framework
To evaluate PEPFAR’s performance in the area of knowledge management, the
committee will be guided by the public health questions approach (see Figure 6), a framework
that is widely used in the global HIV/AIDS M&E community and has been adopted by OGAC
(Bouey, 2010; Rugg et al., 2004; UNAIDS MERG, 2010). The committee will determine the
extent to which M&E activities are meeting the goals laid out in this framework and the extent to
which these activities are contributing to evaluating and improving the performance of the
program and building the capacity of partner countries to use information to improve health
outcomes.
FIGURE 6 A public health questions approach to HIV monitoring and evaluation.
SOURCE: Reprinted, with permission, from UNAIDS MERG (2010).
Illustrative Questions
The evaluation of PEPFAR’s SI activities will be carried out at the level of headquarters
and in those partner countries where PEPFAR has made major investment in M&E. In order to
accomplish the assessment, the committee will use a desk review of OGAC and country-level
M&E strategy and implementation procedures and data management systems since the beginning
of PEPFAR (i.e., including different iterations over time). The committee will also examine
M&E funding allocations and expenditures at global and country levels where available. A
review of national M&E strategies, national M&E assessment reports, and reports on the HIV
epidemic and response, including global reports such as United Nations General Assembly
Special Session (UNGASS) documents or M&E data from UNAIDS’s National Composite
Policy Index, will also provide context for the evaluation.
Due to the limitations in assessing the progress of efforts over time from guidance and
assessment documents, the committee will also rely on structured interviews with key SI and
programmatic PEPFAR staff at all levels (OGAC, USG agencies, OGAC SI technical working
groups, country staff, contractors/implementers) regarding the mechanisms and role of SI in the
PEPFAR program and in informing the national HIV response. The committee may also explore
structured interviews with key M&E and programmatic staff of each country’s national AIDS
program as well as other multilateral and bilateral organizations (including UNAIDS, WHO, the
OCR for page 57
57
PEPFAR’S KNOWLEDGE MANAGEMENT
Global Fund) about the role of PEPFAR as a partner at the local, national, and global levels.
These interviews could be incorporated as a part of the committee’s country visits or conducted
by phone or video conference (or alternative self-completed questionnaire) for those countries
where a country visit is not planned. The committee’s analysis of the completeness and validity
of data requested from OGAC and implementing partners for all areas of this evaluation will also
inform an assessment of SI and M&E performance and progress.
In order to assess PEPFAR’s contribution to the global knowledge base, the committee
will assess PEPFAR’s participation in international M&E development processes. The
committee will also assess current research efforts, such as the PHEs, and associated
dissemination efforts, such as presentations from PEPFAR programs at the annual implementers’
meeting and other international conferences like the meetings of the International AIDS Society.
In addition, a preliminary search of published literature conducted during this planning phase
will provide the foundation for a more extensive review during the evaluation of available
articles, reports, and other publications resulting from PEPFAR-funded activities.
The following are examples of illustrative questions that the committee may consider in
the evaluation. These questions related to knowledge management reflect a fundamental activity
of the program and as such are intended to contribute to addressing all of the areas for
consideration in the congressional mandate, as described in the Statement of Task (see Appendix
A).
To what extent has investment in M&E resulted in effective systems for PEPFAR decision
making and for program management and improvement at both the headquarters and
country levels?
Is data collection and analysis being used for decision making about PEPFAR
program priorities, implementation strategies, effectiveness, and efficiency? For
example, is PEPFAR using data to support evidence-based COP planning and
resource allocation? Does data collection lead to timely identification of
implementation problems, and does this result in corrective action? Does data on
targets determine whether programs are implemented on a large enough scale to have
an impact on the epidemic?
Are the data collated, analyzed, interpreted, presented, and disseminated in a manner
that allows for use in decision making? What are the mechanisms used to assure the
validity and quality of data? What mechanisms are in place to facilitate the translation
of information produced by M&E systems into action? Are lessons learned accessible
and are changes applied across the whole of the program where appropriate?
Have operations research and other research activities supported by PEPFAR, such as
PHEs, had an impact on service delivery and led to improved outcomes of prevention,
treatment, and care programs supported by PEPFAR? Are operations and other
research activities using appropriate methodologies and resulting in information that
is shared across sites, programs, and countries to optimize and inform policy and
program decisions? Are these research activities addressing the issues most in need of
evaluation? What should the priorities be for future PHEs and other research
activities?
OCR for page 58
58 STRATEGIC APPROACH TO THE EVALUATION OF U.S. GLOBAL HIV/AIDS PROGRAMS
What has been the impact of PEPFAR reporting requirements for accountability
purposes (i.e., reporting to the U.S. Congress) on continuation of PEPFAR funding
and on decision making for priority investments as well as program management and
improvement?
What progress has been made on PEPFAR’s intentions to develop indicators where
there are currently limited mechanisms for tracking progress, such as gender and
health systems strengthening?
Are there sufficient M&E mechanisms and capacity to evaluate whether the program
meets new goals for sustainability and country ownership, as well as the resulting
outcomes and impact of changes made to address those goals?
Does OGAC draw on sources of data outside of PEPFAR to inform programmatic
and policy decisions?
To what extent is PEPFAR contributing to the global knowledge base?
Are PEPFAR-funded activities resulting in research that is contributing to the
scientific knowledge base? Are research findings, lessons learned, and best practices
from PEPFAR available in the published literature? What other mechanisms are used
to disseminate knowledge not only within but also beyond PEPFAR? To what extent
is that process encouraged or facilitated?
To what extent is PEPFAR engaging with other international stakeholders around SI
activities? To what extent is PEPFAR contributing to the development of state-of-the-
art practices in M&E at the global level?
To what extent has PEPFAR built/is PEPFAR building capacity at the country level,
including national M&E systems, 17 to support an appropriate, effective, and efficient
national HIV response?
What is PEPFAR’s approach to supporting long-term sustainability of the national
M&E system? How is PEPFAR translating the commitment of the United States to
the “Third One” adopted by most donors—one national M&E system to reduce
reporting burden and transactional costs of business for countries with multiple
donors?
To what extent is the PEPFAR M&E strategy aligned with and incorporated in the
national M&E strategy/plan? What are the positive and negative effects of the
headquarters-level PEPFAR M&E strategy on national M&E systems? To what
extent and how are PEPFAR M&E data (program planning, routine program
17
National M&E system refers to M&E at the national, sub-national, and service-delivery levels.
OCR for page 59
59
PEPFAR’S KNOWLEDGE MANAGEMENT
monitoring, findings from special studies) shared with the national M&E system to
ensure a coordinated HIV response and to guide program improvement?
What mechanisms are used by PEPFAR for M&E capacity building and to ensure
effective partnerships for technical cooperation and technology transfer? What are the
effects of the PEPFAR M&E capacity building activities on national M&E system
strengthening and data use for decision making?
To what extent has PEPFAR built/is PEPFAR building M&E capacity within partner
organizations implementing programs at the country level, including data analysis
and management?
OCR for page 122
122 STRATEGIC APPROACH TO THE EVALUATION OF U.S. GLOBAL HIV/AIDS PROGRAMS
Partnership Frameworks to Promote Sustainable Approaches
The Lantos–Hyde Act of 2008 permitted the USG to establish framework documents
(Partnership Frameworks) with countries to promote a more sustainable approach of the USG’s
global efforts against HIV/AIDS, malaria, and TB “that is characterized by strengthened country
capacity, ownership, and leadership” (OGAC, 2009b, p. 3). Further, these 5-year joint strategic
frameworks between the USG and partner governments are meant to intensify focus on
cooperation through “technical assistance and support for service delivery, policy reform, and
coordinated financial commitments” (OGAC, 2009b, p. 3). At the end of the 5-year time frame,
the expectation is that “country governments will be better positioned to assume primary
responsibility for the national response to HIV/AIDS in terms of management, strategic
direction, performance monitoring, decision making, coordination, and where possible, financial
support and service delivery” (OGAC, 2009b, p. 3). The axiom of “do no harm” has been
adopted by OGAC for continued support of existing implementing partner service delivery
systems to continue to provide quality services while this transition to county ownership occurs
over time. There are additional expectations of transparency, accountability, and engagement of
multiple stakeholders in the country.
As for policy reform and financial commitments, the Partnership Frameworks are
supposed to emphasize policy areas identified by the government and civil society that require
additional or focused attention and overall accountability for resources and appropriate
budgeting in HIV programs. They also provide a capacity building opportunity for the USG to
assist countries in managing multiple funding sources. Some countries, based on their resources,
are expected to increase their financial contributions over time. The Partnership Frameworks also
provide an opportunity for the USG to provide technical assistance to countries for improved
monitoring and tracking of overall health spending (including HIV/AIDS) from different
sources, including financial monitoring and reporting systems (OGAC, 2009b).
Although the reauthorization legislation did not define sustainability per se, PEPFAR did
define how to promote sustainability in the Partnership Frameworks and Partnership Framework
Implementation Plans guidance issued in September 2009—based on the principles of the Three
Ones, the Paris Declaration, and the Monterey Accord of 2002 (activities that facilitate financing
for development): “For purposes of the Partnership Frameworks, promoting sustainability means
supporting the partner government in growing its capacity to lead, manage, and ultimately
finance its health system with indigenous resources (including its civil society sector), rather than
external resources, to the greatest extent possible” (OGAC, 2009b, p. 4). The Partnership
Frameworks are distinct from the annual work plans for USG-supported intervention, the COPs.
However, COPs are expected to reflect the Partnership Framework principles and the transition
strategy outlined in the Partnership Framework Implementation Plans. The implementation plans
have minimal required elements, including “an analysis of how the existing portfolio of USG-
supported, NGO-implemented programs will transition to the partner government, remain NGO-
based, or be terminated within the 5-year timeframe” (OGAC, 2009b, p. 6) and “a description of
the approach to supporting increased country ownership, baseline data, specific strategies for
achieving the 5-year goals and objectives, and a monitoring and evaluation plan” (OGAC,
2009b, p. 8). It also seems reasonable that cost efficiencies for the future national response can
be identified by both PEPFAR and the partner government during the implementation of the
Partnership Frameworks and by the end of the 5-year performance period, when countries are
OCR for page 123
123
KEY SYSTEMS-LEVEL GOALS AND ACTIVITIES
expected to assume primary responsibility for the national HIV/AIDS response, even if the
partner government does not assume full responsibility of financing its health system.
The goals and objectives would be measureable goals for the USG and all partners in the
Partnership Framework. As such, the Partnership Frameworks are expected to identify indicators
to assess progress toward meeting the goals, objectives, and programmatic and financial
commitments—with an eye toward international efforts to harmonize indicators. PEPFAR-
specific reporting systems are expected to be transitioned to nationally- and country-owned
systems. This step would be in full support of the Third One—one agreed HIV/AIDS country-
level M&E system (OGAC, 2009b). Lastly, the guidance also states that country governments
should be developing the “capacity to support all relevant components … of a multi-sector health
system” (OGAC, 2009b, p. 4). These components are the six areas of the WHO six-building
block framework for effective health systems (see Figure 16) that has been endorsed by
PEPFAR.
Program and Health System Interaction and Integration
The most widely accepted definition of a health system was proposed by the WHO—“all
organizations, people, and actions whose primary intent is to promote, restore, or maintain
health” (WHO, 2007a, p. 2). A health system therefore includes both public and private (for-
profit and not-for-profit) providers, which may be either community- or facility-based.
Identifying both well-planned synergies and unintended antagonisms with national health
systems that may have resulted from the implementation of a program like PEPFAR requires a
systems model, as interventions in one part of the health system may have impacts in others. The
WHO Positive Synergies working group took such an approach, resulting in several
recommendations that are relevant to PEPFAR: prioritize health system strengthening, agree on
and track health system strengthening indicators, align resource allocation between global health
initiatives and country health systems, generate more reliable data for the costs and benefits of
strengthening health systems, and commit to increased national and global health financing that
is more predictable to support sustainable and equitable growth of health systems (Samb et al.,
2009).
While global HIV prevalence stabilized in 2000 (Bongaarts et al., 2008; UNAIDS, 2008),
the absolute burden of HIV treatment demand will continue to grow each year for the foreseeable
future. (UNAIDS, 2008; UNAIDS and WHO, 2009). With newly-infected people still outpacing
both AIDS-related deaths and numbers of people being put on ART, it is increasingly clear that
treatment alone will not be a sufficient response to control the epidemic (Bertozzi et al., 2009).
Achieving sustainable HIV programs not only requires health system strengthening, but also
successful scale-up of effective HIV prevention strategies to avert continued growth in the HIV
treatment burden in health systems that are already overburdened. As previously mentioned in
the Mapping of PEPFAR Funding section, scaling-up does not necessarily mean just increased
spending, so understanding unit costs of HIV-program delivery may help elucidate variation in
health system capacity, efficiencies, and quality. Also program and data management and health
information systems are needed to more completely assess population impact rather than just
process measures.
OCR for page 124
124 STRATEGIC APPROACH TO THE EVALUATION OF U.S. GLOBAL HIV/AIDS PROGRAMS
Capacity Building
As has been noted, the capacity to scale up programs in low-income countries is limited
by resource constraints, including lack of trained health workers and fragile health systems.
PEPFAR II therefore added an additional target of training 140,000 new health workers to
support the capacity of countries to improve “the overall quality of their services … and build
capacity to plan, manage and sustainably finance their health systems” (OGAC, 2009h).
Investing in healthcare workers and health systems increases the likelihood of more people
receiving prevention and treatment services for HIV and of achieving a broader health impact
(PEPFAR Reauthorization Action Team, 2010).
Joint Activities Between PEPFAR and the Global Fund
As seen in the statement of task in Appendix A, Congress mandates the committee to
evaluate the impact on health of prevention, treatment, and care efforts that are supported by
U.S. funding, including multilateral and bilateral programs involving joint operations. Further
discussions among IOM staff, OGAC, and congressional staff 48 clarified “multilateral and
bilateral programs involving joint operations” to mean programs operated in conjunction with
bilateral funding through PEPFAR and the Global Fund (Bressler, 2009; Marsh, 2009). In the
preliminary research of the planning committee, this includes financing of ART and procurement
and supply management of ARVs and other commodities for its Voluntary Pooled Procurement
(discussed in the Adult and Pediatric Treatment section of this report), as well as prevention and
care activities (PMTCT and TB treatment, respectively discussed in the Prevention and Care and
Support Services sections). It also includes time-limited, outcome-oriented technical assistance
through centrally-funded grants from OGAC to Global Fund recipients with active grants (which
are not necessarily PEPFAR recipients). The USG Global Fund Technical Support Advisory
Panel advises OGAC headquarters on these technical assistance activities.
Begun in 2005 under the Grants Management Solutions project, this technical support is
intended to (1) improve the functioning of Global Fund grants, (2) strengthen local capacity, and
(3) alleviate specific bottlenecks to address under-performing Global Fund grants. The main
areas identified with inadequate or poor performance were organizational development
(including governance and leadership), financial management, procurement and supply
management, and M&E (Coleman, 2007; PEPFAR, 2009b). These four areas correspond to four
of the six blocks in the WHO building-block model for effective health systems (see Figure 16).
There are stated goals for improvement for each of the four areas of technical assistance. In July
2008, a USG Global Fund Technical Support Evaluation was conducted, which showed
achievements through 2008 in each of the areas from the $12 million available in FY2005
technical support through the $31 million available in FY2007 (Coleman, 2010). The committee
will examine the 2008 evaluation, including its methods and findings and will also request
available country/recipient progress reports as well as copies of the grant requests from recipient
countries.
48
Supra., note 10.
OCR for page 125
125
KEY SYSTEMS-LEVEL GOALS AND ACTIVITIES
Evaluation Strategy
As part of an evaluation of the impact of PEPFAR, it is necessary to determine partner
country readiness to make this transition to sustainability. To assess partner country readiness,
the evaluation committee aims to examine the partnership frameworks with partner countries. In
addition, the committee will assess PEPFAR efforts and synergy with other global stakeholders
for capacity building and technical assistance, including activities similar to those identified by
the International Health Partnership, such as donor funding harmonization and collaboration
between international actors and developing countries to develop and implement national health
plans.
The committee will also establish working definitions, define data sources, and identify
evaluation methodologies for the key parameters of sustainability that include country ownership
and local capacity building, including health system strengthening and healthcare workforce
expansion. However, it has been noted that “little [global] consensus has emerged to provide
uniform guidance” to indicators that can track these activities, although the development of these
indicators is an area of intense international interest and activity. PEPFAR defines indicator and
reporting requirements in health systems strengthening “to reflect a more narrow scope of
interest tied to PEPFAR’s focus on HIV” (OGAC, 2009d, p. 199)—resulting in selection of two
indicators to be reported centrally to OGAC that reflect laboratory and health workforce
strengthening. Evaluating data for other parameters will require metrics not currently available
from routine data sources or the PEPFAR indicator database (OGAC, 2009d), including extra-
health sector factors such as good governance (Dybul, 2009), to permit the committee to assess
performance of current or past PEPFAR activities. The committee will also examine Partnership
Frameworks and Implementation Plans to assess what the countries and OGAC are responsible
for, how it is being measured or tracked, and how the processes evolve for PEPFAR-related
responsibilities and activities to be transitioned to country leadership for sustainable programs
and positive impacts on individual and population health.
Health System Frameworks
PEPFAR’s increased focus on health systems is shared with other global health
initiatives, including The Global Alliance for Vaccines and Immunisation (2010) and the Global
Fund, as well as the World Bank and the WHO. The committee will benefit from and incorporate
deliberations among these agencies that will address both how to define health systems and how
to measure progress toward strengthened health systems (Frenk, 2010; Shakarishvili, 2009).
Adopting a common conceptual framework is a key requirement for the evaluation of
health systems strengthening and is important to framing such measures. The most widely
adopted framework, proposed by the WHO in 2007, is based on operational “building blocks” of
the health system (Figure 16)—services, workforce, information, commodities and technologies,
financing, and leadership/governance (WHO, 2007a). These building blocks serve several
functions including describing what a health system should have the capacity to do in each of
these blocks to lead to improved health, system responsiveness, improved efficiency, and social
and financial risk protection. It also lends itself well to a descriptive set of activities that may be
undertaken through PEPFAR programs. The importance of “systems thinking” has also been
emphasized, because health systems are complex, dynamic, and non-linear systems whose
OCR for page 126
126 STRATEGIC APPROACH TO THE EVALUATION OF U.S. GLOBAL HIV/AIDS PROGRAMS
function is dependent on the interplay of all of its components, including “the spaces in between”
(Atun and Menabde, 2008; de Savigny and Taghreed, 2009).
FIGURE 16 Representation of WHO’s six building blocks for effective health systems.
SOURCE: Adapted from WHO, 2007a.
OCR for page 127
127
KEY SYSTEMS-LEVEL GOALS AND ACTIVITIES
Program Impact Pathway
The committee will utilize an impact pathway (Figure 17) to assess PEPFAR inputs,
output, and outcomes where measurable, within the context of the WHO building blocks
framework for health systems strengthening. The pathway can also help assess the same block
elements for technical assistance and other activities related to country readiness for assuming
increased to total responsibility for their HIV/AIDS response. Although a useful evaluation tool,
the linear nature of an impact pathway is a simplified representation of the reality of PEPFAR
programs and their impact, and the committee found this to be particularly challenging in the
area of systems-level activities. Activities at the systems level are intended to result in an
outcome or impact on the health system, but these are also critical inputs to all other
programmatic areas. Thus, the functioning of the health system can be both a starting and end
point for a pathway. The committee grappled with illustrating the complexities, dynamism, and
non-linear nature of health systems with this linear pathway—analogous to forcing a “square peg
in a round hole” —but ultimately recognized the utility of the impact pathway to evaluate the
process by helping to frame the areas of inquiry and the measures that may be undertaken to
assess health system strength. The findings will be interpreted in light of the more complex
realities when the evaluation committee draws conclusions and makes recommendations.
As previously mentioned, PEPFAR II (as well as the GHI) has adopted the WHO six
building block framework to assess its capacity building and programmatic impact on health
systems, categorizing contribution to (1) core HIV activities, (2) secondary benefits or
intentional spillover effects of PEPFAR activities on other programs, and (3) targeted leveraging
including partnerships.
The six building block elements are denoted in the figure with blocks in all uppercase
letters. The committee’s working definition of an intervention that strengthens health systems is
one that improves the activities and processes within the six health system building blocks, and
manages interactions within the “building blocks” over time to achieve more equitable and
sustained improvements in population health outcomes. These effects may be either short-term
(e.g., better trained workforce delivering higher quality care) or longer-term (e.g., higher-quality
care results in a healthier population that is economically more productive), thus growing
resources that feed back into the causal chain as an input to the health system. Additional
outcomes of an effective health system (e.g., social and financial risk protection for beneficiaries
accessing and utilizing services or increased block or system efficiency) can result in increased
responsiveness and potentially expanded service coverage when the activities within and among
the blocks dynamically interact (see Figure 17). As previously mentioned, interventions in one
part of the health system may have impacts in others. Since any intervention may affect the
entire system, evaluation is, by definition, context-specific and requires a mix of evaluation
strategies, both quantitative and qualitative. Although improved health is an important impact of
health systems strengthening, additional measures regarding health system efficiency and
equitable distribution of benefits may likewise be important to track as they are developed and
adopted by international working groups.
OCR for page 128
128
FIGURE 17 Program impact pathway for evaluation of PEPFAR’s health systems strengthening activities at the country level.
NOTES: Upper case headers indicate the WHO’s six building blocks. All of the elements listed in the pathway can be categorized under one of the
six building blocks, but due to space limitations, select elements were chosen to illustrate blocks within the pathway. ARV = antiretroviral drugs;
ART = antiretroviral therapy; CD4 = cluster of differentiation 4; EQA/PT = external quality assurance/proficiency testing; M&E = monitoring and
evaluation.
OCR for page 129
129
KEY SYSTEMS-LEVEL GOALS AND ACTIVITIES
Illustrative Questions
The main questions that the committee will consider in the evaluation include the
potential positive and negative impact of PEPFAR activities on country-level health system
functioning, with regards to both HIV and non-HIV programs. Examples of the kinds of
questions the committee may ask that specifically link the WHO six building blocks to our
results chain framework include the following. These questions are intended to contribute to
addressing part B, item ii of the areas for consideration in the congressional mandate. 49 Because
health systems are a fundamental aspect of all program activities, these questions are also
intended to contribute to addressing all of the areas for consideration described in the Statement
of Task (see Appendix A).
Finance: To what extent has PEPFAR funding and technical assistance for sound public
finance systems resulted in more efficient and equitable care in HIV and non-HIV health
systems? To what extent do the joint funds from the Global Fund support the six building
blocks? To what extent have the technical support funds from PEPFAR to the Global
Fund improved the performance of grant recipients in general and how is this measured?
For the four main areas of technical assistance which are among the six WHO building
blocks and how are they measured?
Commodities and Procurement: How have PEPFAR technical assistance and training
affected HIV laboratory diagnostic capacity, pharmacy infrastructure, and supply chain
management for reagents and drugs?
Information Systems: What is the evidence that PEPFAR-supported health information
systems are resulting in higher functioning, quality-driven health systems performance?
How will new and existing information officers be trained to meet country needs to
strengthen higher functioning and increased quality?
Service Delivery: What elements of more efficient, equitable, and effective service
delivery should be expected to result in improved population health over time with better
integration of HIV- and non-HIV health care and use of continuous quality improvement
methods?
Leadership and Governance: To what extent will partnership frameworks, jointly funded
Global Fund activities, and other expressions of country ownership lead to improved and
accountable governance as well as transfer of oversight, management, guidance, and
financing for HIV-related services in health systems?
Health Workforce: How will the pre-service education target of 140,000 new workers
affect health system equity of access (e.g., rural versus urban) and health system
strengthening such as increased skill capacity and retention of workforce? Are there
other types of workers, (e.g., social service workers and program managers) who also
need to be trained? What proportion of newly trained new health or social workers is
49
(B)(ii) an assessment of the effects on health systems, including on the financing and management of health
systems and the quality of service delivery and staffing
OCR for page 130
130 STRATEGIC APPROACH TO THE EVALUATION OF U.S. GLOBAL HIV/AIDS PROGRAMS
retained one year later? How does PEPFAR support the country’s existing health
resources development plan or help develop those where absent?
The committee will also assess progress toward the goals of increased country ownership
to transition the HIV/AIDS response to national governments for long-term sustainability.
Although there are high-volume discussions within the global development assistance
community in this area, there appear to be few to no meaningful metrics to measure increased
ownership or country readiness, no accepted logic models to describe and illustrate the transition
process, and no indicators for when a country has sufficiently strengthened governance/
leadership or built and operated a financially-sound public finance system to implement and
oversee a national health plan. Illustrative questions for this area of the evaluation include the
following. Country ownership and sustainability cut across all PEPFAR activities and as such
these questions are intended to contribute to addressing all of the areas for consideration in the
congressional mandate, as described in the Statement of Task (see Appendix A).
How is country ownership defined by the country? By PEPFAR? By relevant global
stakeholders in aid development? Are there differences? Have they been reconciled? Do
the Partnership Frameworks represent an “agreement” on the definition between the
country and PEPFAR?
To what extent have PEPFAR capacity-building, technical assistance, and financing
activities to countries contributed to country readiness for transitioning the knowledge
management, decision making responsibility, financing, and accountability/oversight of
the PEPFAR-funded HIV/AIDS response to the national government? How is it being
measured? Are plans in place for future activities needed to improve or increase country
readiness if PEPFAR were absent?
How will achievements or milestones by country and/or PEPFAR be measured in the
Partnership Frameworks? Will these measures adequately reflect country readiness for
program, policy, and financial transitions?
Is the country transitional process explained adequately? Will it be standard for all
countries or adapted for each country? Will the transition be parceled over time? Which
pieces might be transitioned over time, how, and why?
What are the achievements and lessons learned from the Technical Support Grants to the
Global Fund from PEPFAR? How are they measuring whether bottlenecks have been
opened or bypassed? What has been the impact on the functioning of programs?
How do Global Fund and PEPFAR measure their relative and unique contributions to
programs they jointly fund or operate? How are access, equity, and quality measured for
these programs?
A more extensive listing of illustrative questions for this complex issue that the
committee may attempt to address, given the timely availability of quality data, can be found in
Appendix H. These may help guide assessment of whether and how PEPFAR has helped
OCR for page 131
131
KEY SYSTEMS-LEVEL GOALS AND ACTIVITIES
countries to mount a stronger response to their HIV epidemics, plan for transition to country
ownership including effective leadership and oversight of multilateral donor activities, and focus
on affordable care that meets population health needs.
Sustainability and its associated elements must be considered for all of PEPFAR’s
support. PEPFAR activities will be evaluated relative to their contribution of inputs and support
of processes, and the resulting outputs and outcomes. These should ultimately result in the
desired outcome of stronger health systems that can more adequately respond not only to HIV
but to other serious causes of morbidity and mortality, as well as any emerging challenges to the
health and safety of their respective populations.
OCR for page 132
