Click for next page ( 50


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 49
PART III Illustrative Evaluation Details for Assessment of PEPFAR’s Performance and Impact

OCR for page 49

OCR for page 49
The IOM was mandated by Congress in the Lantos–Hyde Act of 2008 to conduct a study that includes an assessment of the performance of U.S.-assisted global HIV/AIDS programs and an evaluation of the impact on health of prevention, treatment, and care efforts that are supported by U.S. funding. Part of the charge to the planning committee in developing a plan for this evaluation was to be cognizant of the requirements and charges mandated for the evaluation (see Appendix A). To augment the overview of the evaluation design presented in Part II, this part of the plan partitions and elaborates the areas of interest laid out in the congressional mandate. The guiding framework of the program impact pathway is applied to each of these areas, reflecting the committee’s understanding of the rationale for how PEPFAR’s specific inputs and activities can be plausibly linked to PEPFAR’s contribution to effects on HIV-specific health outcomes and impacts. This part of the report illustrates the types of questions that will guide the evaluation of PEPFAR’s activities in prevention, adult and pediatric treatment, care and support, child and adolescent wellbeing (including orphans and vulnerable children), and gender-related vulnerability and risk activities. The evaluation will also consider other fundamental activities in the areas of knowledge management and funding flows; these are considered first in this part of the report because they underlie the success of all other programmatic areas. This part of the report culminates with a discussion of cross-cutting activities related to key systems-level goals that are critical for the long term goals articulated by PEPFAR, such as health systems strengthening and transitioning to sustainability and country ownership. As described in Part II, in each of these areas the evaluation questions will be addressed using a mixed methods approach and layers of investigation and analysis, drawing on a range of available primary and secondary data sources. By applying a mix of methods, data sources, and analytical techniques, the committee will arrive at findings that can be triangulated to draw conclusions about the performance of PEPFAR and its contribution to health impact, even when any one data source is not sufficient or any one methodological approach is not feasible. The extent to which specific methods can be applied to answer the evaluation questions will depend on the timely availability of data that is of sufficient quality to lead to reliable findings. Therefore, the illustrative questions and the methods and data sources that will be used to address them will undergo further refinement and prioritization as a result of the operational planning phase activities described previously. 51

OCR for page 49

OCR for page 49
SECTION 1: PEPFAR’S KNOWLEDGE MANAGEMENT The management of knowledge and information is critical to the success of any program because it serves to monitor the activities and effects of the program as well as to guide policies, priorities, and programmatic decisions. Therefore, assessing the performance of PEPFAR’s knowledge management activities will be an important part of the evaluation of the performance of PEPFAR as a whole, as well as an assessment of the forward-looking mechanisms that are in place for continuous M&E of the program’s progress and appropriate responses. In PEPFAR I, the primary goal of OGAC’s strategic information (SI) efforts for M&E, in partnership with implementing agencies, was to support PEPFAR through results-based planning and implementation, focusing on quality information collection, timely data management and use, evaluation of best practices, and information for decision making. In PEPFAR II, the goals have been expanded to include support of the larger PEPFAR mission. To this end, the expanded SI mission includes building the capacity of countries to improve health outcomes by increasing and strengthening the use of information for effective stewardship of programs and effective implementation of efficient, high-quality, and sustainable health systems (Bouey, 2010). During PEPFAR II, the SI goals also include improved harmonization of USG reporting needs with country-driven M&E efforts through not only strengthening country capacity and alignment with national data collection, but also through better alignment with global reporting requirements to lessen the burden on implementing partners and partner governments (OGAC, 2009h). An increased focus on both program coverage and quality will be reflected in SI efforts to identify indicators that can give an accurate picture of these two areas (OGAC, 2009d). Finally, although it continues to recognize that PEPFAR is not intended to be a research initiative, the PEPFAR Five-Year Strategy outlines the additional goals of improving the program’s efforts to contribute to the evidence base for HIV interventions and to expand the amount of publicly available data (OGAC, 2009g). This expanded research effort will prioritize the evaluation and proactive dissemination of topics that PEPFAR is in a unique position to address as well as studies that focus on methods to improve program delivery (OGAC, 2009h). It will also increase the tracking of outcomes, cost-effectiveness, innovation, and impacts in order to identify timely information regarding the program’s effectiveness and impact (OGAC, 2009h). Strategic Information Management Structurally, SI activities at the PEPFAR headquarters level are carried out by USG implementing agencies with coordination through OGAC. These headquarters-level activities draw from a wide range of data-gathering sources (see Figure 5). 53

OCR for page 49
54 STRATEGIC APPROACH TO THE EVALUATION OF U.S. GLOBAL HIV/AIDS PROGRAMS FIGURE 5 PEPFAR headquarters-level strategic information partners and headquarters-level data- gathering sources. NOTES: CDC = U.S. Centers for Disease Control and Prevention, Census = U.S. Census Bureau, DoD = U.S. Department of Defense, HRSA = Health Resources and Services Administration of the U.S. Department of Health and Human Services, M&E = monitoring and evaluation, OGAC = Office of the U.S. Global AIDS Coordinator, SI = strategic information, TWG = technical working group, USAID = United States Agency for International Development. SOURCE: Adapted from Bouey (2010). The staff at headquarters are responsible for issuing guidance related to COP submission and reporting processes, as well as for providing technical assistance. This guidance instructs country teams on how to successfully complete their reporting requirements. In addition to annual reporting needs, guidance from headquarters can also provide information on collecting, interpreting, and updating basic epidemiologic profiles as well as information on how to develop and incorporate efforts to evaluate new initiatives (OGAC, 2009e). While this guidance focuses primarily on the processes unique to OGAC, it also offers additional information regarding data collection and target setting that has the potential for broader applicability. Within headquarters there is also a SI technical working group. These efforts at the headquarters level also support M&E activities at the country level through the development of resources such as an M&E Systems Strengthening Tool, which is designed to help partner countries prioritize their M&E needs and encourage alignment with a national M&E strategy, and a Data Quality Assessment Tool (MEASURE Evaluation, 2007; PEPFAR, 2008a). While management of data collection, storage, and analysis at the country and project level varies, the primary mechanism for reporting and aggregating these data is via an electronic, Internet-based program known as the Country Operational Plan Reporting System (COPRS). Data that are reported via COPRS are collected at the OGAC headquarters level during the relevant semi-annual or annual program reporting periods, depending on the country (OGAC, 2009d). A portion of these data is released to the public and is also communicated to Congress via an annual report. Occasionally, they are used by OGAC to produce additional topic-specific reports (e.g., Report on Gender-Based Violence and HIV/AIDS) (PEPFAR, 2010c). COPRS is

OCR for page 49
55 PEPFAR’S KNOWLEDGE MANAGEMENT currently in transition due to two factors. First, the Next Generation Indicators Reference Guidance was recently released, introducing a limited number of new indicators and redefining some measures that had previously been in use. This guidance was developed in part to support PEPFAR’s contribution to global efforts to harmonize reporting requirements for HIV/AIDS initiatives, which aims to reduce the reporting burden of program implementers and to allow more flexibility and increased local ownership of the design of M&E plans (OGAC, 2009d). Second, a new generation of COPRS, COPRS II, is in development and is expected to be deployed in FY2010 (OGAC, 2009e). Beyond Information Management In 2008, PEPFAR began a campaign titled “know your epidemic/know your results” aimed at using information to more closely align program activities with population needs (OGAC, 2008b). As a result, a focus was placed on developing sustainable SI systems to “collect, analyze, critically review, disseminate, interpret, display, and strategically use data at all levels” (OGAC, 2008b). Continuing with this development of SI, the 2009 headquarters operational plan allocated funds for the development of a “draft knowledge management strategy,” perhaps in response to the first IOM PEPFAR evaluation recommendation to develop a detailed, overall strategy for institutionalizing its efforts to function as a learning organization and to increase its contributions to the global knowledge base (IOM, 2007; OGAC, 2009f). Coordination among country staff and dissemination of best practices is also facilitated by the PEPFAR implementers’ meeting held annually in a PEPFAR country. In addition to sharing information across countries, this meeting includes a variety of breakout sessions dedicated to SI issues (PEPFAR, 2009a). In addition, as described above, the recent goals for PEPFAR II emphasized the important role of expanding the program’s research portfolio to contribute to the publicly available evidence base, with an emphasis on operations research to improve program delivery as well as methods for timely assessments of the program’s effectiveness and impact (OGAC, 2009h). Some research activities are already occurring in individual partner countries. For example, the Public Health Evaluations (PHE), initiated in 2007, are a PEPFAR activity intended as a source to inform policy and program-level changes. They currently serve as the primary mechanism through which PEPFAR supports research within countries, including operations research (Edgil, 2010; OGAC, 2009h). Some PHEs are single country, while others are multi-site investigations. The selection of PHE proposals (annually solicited from investigators) is performed by an interagency technical policy group charged with prioritizing areas in need of evaluation, overseeing the implementation of evaluations, and recommending approvals and levels of funding for PHEs. In doing so, priority is given to studies that are driven by locally- identified country needs as well as those that involve local institutions and investigators in the research process. 15 As of 2008 there were 195 PHE activities (Edgil, 2010), with the most recent call for proposals issued by the National Institutes of Health in April 2010. 16 15 Personal communication from OGAC, April 9, 2010. 16 Ibid.

OCR for page 49
56 STRATEGIC APPROACH TO THE EVALUATION OF U.S. GLOBAL HIV/AIDS PROGRAMS PEPFAR’s Monitoring and Evaluation Framework To evaluate PEPFAR’s performance in the area of knowledge management, the committee will be guided by the public health questions approach (see Figure 6), a framework that is widely used in the global HIV/AIDS M&E community and has been adopted by OGAC (Bouey, 2010; Rugg et al., 2004; UNAIDS MERG, 2010). The committee will determine the extent to which M&E activities are meeting the goals laid out in this framework and the extent to which these activities are contributing to evaluating and improving the performance of the program and building the capacity of partner countries to use information to improve health outcomes. FIGURE 6 A public health questions approach to HIV monitoring and evaluation. SOURCE: Reprinted, with permission, from UNAIDS MERG (2010). Illustrative Questions The evaluation of PEPFAR’s SI activities will be carried out at the level of headquarters and in those partner countries where PEPFAR has made major investment in M&E. In order to accomplish the assessment, the committee will use a desk review of OGAC and country-level M&E strategy and implementation procedures and data management systems since the beginning of PEPFAR (i.e., including different iterations over time). The committee will also examine M&E funding allocations and expenditures at global and country levels where available. A review of national M&E strategies, national M&E assessment reports, and reports on the HIV epidemic and response, including global reports such as United Nations General Assembly Special Session (UNGASS) documents or M&E data from UNAIDS’s National Composite Policy Index, will also provide context for the evaluation. Due to the limitations in assessing the progress of efforts over time from guidance and assessment documents, the committee will also rely on structured interviews with key SI and programmatic PEPFAR staff at all levels (OGAC, USG agencies, OGAC SI technical working groups, country staff, contractors/implementers) regarding the mechanisms and role of SI in the PEPFAR program and in informing the national HIV response. The committee may also explore structured interviews with key M&E and programmatic staff of each country’s national AIDS program as well as other multilateral and bilateral organizations (including UNAIDS, WHO, the

OCR for page 49
57 PEPFAR’S KNOWLEDGE MANAGEMENT Global Fund) about the role of PEPFAR as a partner at the local, national, and global levels. These interviews could be incorporated as a part of the committee’s country visits or conducted by phone or video conference (or alternative self-completed questionnaire) for those countries where a country visit is not planned. The committee’s analysis of the completeness and validity of data requested from OGAC and implementing partners for all areas of this evaluation will also inform an assessment of SI and M&E performance and progress. In order to assess PEPFAR’s contribution to the global knowledge base, the committee will assess PEPFAR’s participation in international M&E development processes. The committee will also assess current research efforts, such as the PHEs, and associated dissemination efforts, such as presentations from PEPFAR programs at the annual implementers’ meeting and other international conferences like the meetings of the International AIDS Society. In addition, a preliminary search of published literature conducted during this planning phase will provide the foundation for a more extensive review during the evaluation of available articles, reports, and other publications resulting from PEPFAR-funded activities. The following are examples of illustrative questions that the committee may consider in the evaluation. These questions related to knowledge management reflect a fundamental activity of the program and as such are intended to contribute to addressing all of the areas for consideration in the congressional mandate, as described in the Statement of Task (see Appendix A). To what extent has investment in M&E resulted in effective systems for PEPFAR decision making and for program management and improvement at both the headquarters and country levels? Is data collection and analysis being used for decision making about PEPFAR program priorities, implementation strategies, effectiveness, and efficiency? For example, is PEPFAR using data to support evidence-based COP planning and resource allocation? Does data collection lead to timely identification of implementation problems, and does this result in corrective action? Does data on targets determine whether programs are implemented on a large enough scale to have an impact on the epidemic? Are the data collated, analyzed, interpreted, presented, and disseminated in a manner that allows for use in decision making? What are the mechanisms used to assure the validity and quality of data? What mechanisms are in place to facilitate the translation of information produced by M&E systems into action? Are lessons learned accessible and are changes applied across the whole of the program where appropriate? Have operations research and other research activities supported by PEPFAR, such as PHEs, had an impact on service delivery and led to improved outcomes of prevention, treatment, and care programs supported by PEPFAR? Are operations and other research activities using appropriate methodologies and resulting in information that is shared across sites, programs, and countries to optimize and inform policy and program decisions? Are these research activities addressing the issues most in need of evaluation? What should the priorities be for future PHEs and other research activities?

OCR for page 49
58 STRATEGIC APPROACH TO THE EVALUATION OF U.S. GLOBAL HIV/AIDS PROGRAMS What has been the impact of PEPFAR reporting requirements for accountability purposes (i.e., reporting to the U.S. Congress) on continuation of PEPFAR funding and on decision making for priority investments as well as program management and improvement? What progress has been made on PEPFAR’s intentions to develop indicators where there are currently limited mechanisms for tracking progress, such as gender and health systems strengthening? Are there sufficient M&E mechanisms and capacity to evaluate whether the program meets new goals for sustainability and country ownership, as well as the resulting outcomes and impact of changes made to address those goals? Does OGAC draw on sources of data outside of PEPFAR to inform programmatic and policy decisions? To what extent is PEPFAR contributing to the global knowledge base? Are PEPFAR-funded activities resulting in research that is contributing to the scientific knowledge base? Are research findings, lessons learned, and best practices from PEPFAR available in the published literature? What other mechanisms are used to disseminate knowledge not only within but also beyond PEPFAR? To what extent is that process encouraged or facilitated? To what extent is PEPFAR engaging with other international stakeholders around SI activities? To what extent is PEPFAR contributing to the development of state-of-the- art practices in M&E at the global level? To what extent has PEPFAR built/is PEPFAR building capacity at the country level, including national M&E systems, 17 to support an appropriate, effective, and efficient national HIV response? What is PEPFAR’s approach to supporting long-term sustainability of the national M&E system? How is PEPFAR translating the commitment of the United States to the “Third One” adopted by most donors—one national M&E system to reduce reporting burden and transactional costs of business for countries with multiple donors? To what extent is the PEPFAR M&E strategy aligned with and incorporated in the national M&E strategy/plan? What are the positive and negative effects of the headquarters-level PEPFAR M&E strategy on national M&E systems? To what extent and how are PEPFAR M&E data (program planning, routine program 17 National M&E system refers to M&E at the national, sub-national, and service-delivery levels.

OCR for page 49
59 PEPFAR’S KNOWLEDGE MANAGEMENT monitoring, findings from special studies) shared with the national M&E system to ensure a coordinated HIV response and to guide program improvement? What mechanisms are used by PEPFAR for M&E capacity building and to ensure effective partnerships for technical cooperation and technology transfer? What are the effects of the PEPFAR M&E capacity building activities on national M&E system strengthening and data use for decision making? To what extent has PEPFAR built/is PEPFAR building M&E capacity within partner organizations implementing programs at the country level, including data analysis and management?

OCR for page 49
122 STRATEGIC APPROACH TO THE EVALUATION OF U.S. GLOBAL HIV/AIDS PROGRAMS Partnership Frameworks to Promote Sustainable Approaches The Lantos–Hyde Act of 2008 permitted the USG to establish framework documents (Partnership Frameworks) with countries to promote a more sustainable approach of the USG’s global efforts against HIV/AIDS, malaria, and TB “that is characterized by strengthened country capacity, ownership, and leadership” (OGAC, 2009b, p. 3). Further, these 5-year joint strategic frameworks between the USG and partner governments are meant to intensify focus on cooperation through “technical assistance and support for service delivery, policy reform, and coordinated financial commitments” (OGAC, 2009b, p. 3). At the end of the 5-year time frame, the expectation is that “country governments will be better positioned to assume primary responsibility for the national response to HIV/AIDS in terms of management, strategic direction, performance monitoring, decision making, coordination, and where possible, financial support and service delivery” (OGAC, 2009b, p. 3). The axiom of “do no harm” has been adopted by OGAC for continued support of existing implementing partner service delivery systems to continue to provide quality services while this transition to county ownership occurs over time. There are additional expectations of transparency, accountability, and engagement of multiple stakeholders in the country. As for policy reform and financial commitments, the Partnership Frameworks are supposed to emphasize policy areas identified by the government and civil society that require additional or focused attention and overall accountability for resources and appropriate budgeting in HIV programs. They also provide a capacity building opportunity for the USG to assist countries in managing multiple funding sources. Some countries, based on their resources, are expected to increase their financial contributions over time. The Partnership Frameworks also provide an opportunity for the USG to provide technical assistance to countries for improved monitoring and tracking of overall health spending (including HIV/AIDS) from different sources, including financial monitoring and reporting systems (OGAC, 2009b). Although the reauthorization legislation did not define sustainability per se, PEPFAR did define how to promote sustainability in the Partnership Frameworks and Partnership Framework Implementation Plans guidance issued in September 2009—based on the principles of the Three Ones, the Paris Declaration, and the Monterey Accord of 2002 (activities that facilitate financing for development): “For purposes of the Partnership Frameworks, promoting sustainability means supporting the partner government in growing its capacity to lead, manage, and ultimately finance its health system with indigenous resources (including its civil society sector), rather than external resources, to the greatest extent possible” (OGAC, 2009b, p. 4). The Partnership Frameworks are distinct from the annual work plans for USG-supported intervention, the COPs. However, COPs are expected to reflect the Partnership Framework principles and the transition strategy outlined in the Partnership Framework Implementation Plans. The implementation plans have minimal required elements, including “an analysis of how the existing portfolio of USG- supported, NGO-implemented programs will transition to the partner government, remain NGO- based, or be terminated within the 5-year timeframe” (OGAC, 2009b, p. 6) and “a description of the approach to supporting increased country ownership, baseline data, specific strategies for achieving the 5-year goals and objectives, and a monitoring and evaluation plan” (OGAC, 2009b, p. 8). It also seems reasonable that cost efficiencies for the future national response can be identified by both PEPFAR and the partner government during the implementation of the Partnership Frameworks and by the end of the 5-year performance period, when countries are

OCR for page 49
123 KEY SYSTEMS-LEVEL GOALS AND ACTIVITIES expected to assume primary responsibility for the national HIV/AIDS response, even if the partner government does not assume full responsibility of financing its health system. The goals and objectives would be measureable goals for the USG and all partners in the Partnership Framework. As such, the Partnership Frameworks are expected to identify indicators to assess progress toward meeting the goals, objectives, and programmatic and financial commitments—with an eye toward international efforts to harmonize indicators. PEPFAR- specific reporting systems are expected to be transitioned to nationally- and country-owned systems. This step would be in full support of the Third One—one agreed HIV/AIDS country- level M&E system (OGAC, 2009b). Lastly, the guidance also states that country governments should be developing the “capacity to support all relevant components … of a multi-sector health system” (OGAC, 2009b, p. 4). These components are the six areas of the WHO six-building block framework for effective health systems (see Figure 16) that has been endorsed by PEPFAR. Program and Health System Interaction and Integration The most widely accepted definition of a health system was proposed by the WHO—“all organizations, people, and actions whose primary intent is to promote, restore, or maintain health” (WHO, 2007a, p. 2). A health system therefore includes both public and private (for- profit and not-for-profit) providers, which may be either community- or facility-based. Identifying both well-planned synergies and unintended antagonisms with national health systems that may have resulted from the implementation of a program like PEPFAR requires a systems model, as interventions in one part of the health system may have impacts in others. The WHO Positive Synergies working group took such an approach, resulting in several recommendations that are relevant to PEPFAR: prioritize health system strengthening, agree on and track health system strengthening indicators, align resource allocation between global health initiatives and country health systems, generate more reliable data for the costs and benefits of strengthening health systems, and commit to increased national and global health financing that is more predictable to support sustainable and equitable growth of health systems (Samb et al., 2009). While global HIV prevalence stabilized in 2000 (Bongaarts et al., 2008; UNAIDS, 2008), the absolute burden of HIV treatment demand will continue to grow each year for the foreseeable future. (UNAIDS, 2008; UNAIDS and WHO, 2009). With newly-infected people still outpacing both AIDS-related deaths and numbers of people being put on ART, it is increasingly clear that treatment alone will not be a sufficient response to control the epidemic (Bertozzi et al., 2009). Achieving sustainable HIV programs not only requires health system strengthening, but also successful scale-up of effective HIV prevention strategies to avert continued growth in the HIV treatment burden in health systems that are already overburdened. As previously mentioned in the Mapping of PEPFAR Funding section, scaling-up does not necessarily mean just increased spending, so understanding unit costs of HIV-program delivery may help elucidate variation in health system capacity, efficiencies, and quality. Also program and data management and health information systems are needed to more completely assess population impact rather than just process measures.

OCR for page 49
124 STRATEGIC APPROACH TO THE EVALUATION OF U.S. GLOBAL HIV/AIDS PROGRAMS Capacity Building As has been noted, the capacity to scale up programs in low-income countries is limited by resource constraints, including lack of trained health workers and fragile health systems. PEPFAR II therefore added an additional target of training 140,000 new health workers to support the capacity of countries to improve “the overall quality of their services … and build capacity to plan, manage and sustainably finance their health systems” (OGAC, 2009h). Investing in healthcare workers and health systems increases the likelihood of more people receiving prevention and treatment services for HIV and of achieving a broader health impact (PEPFAR Reauthorization Action Team, 2010). Joint Activities Between PEPFAR and the Global Fund As seen in the statement of task in Appendix A, Congress mandates the committee to evaluate the impact on health of prevention, treatment, and care efforts that are supported by U.S. funding, including multilateral and bilateral programs involving joint operations. Further discussions among IOM staff, OGAC, and congressional staff 48 clarified “multilateral and bilateral programs involving joint operations” to mean programs operated in conjunction with bilateral funding through PEPFAR and the Global Fund (Bressler, 2009; Marsh, 2009). In the preliminary research of the planning committee, this includes financing of ART and procurement and supply management of ARVs and other commodities for its Voluntary Pooled Procurement (discussed in the Adult and Pediatric Treatment section of this report), as well as prevention and care activities (PMTCT and TB treatment, respectively discussed in the Prevention and Care and Support Services sections). It also includes time-limited, outcome-oriented technical assistance through centrally-funded grants from OGAC to Global Fund recipients with active grants (which are not necessarily PEPFAR recipients). The USG Global Fund Technical Support Advisory Panel advises OGAC headquarters on these technical assistance activities. Begun in 2005 under the Grants Management Solutions project, this technical support is intended to (1) improve the functioning of Global Fund grants, (2) strengthen local capacity, and (3) alleviate specific bottlenecks to address under-performing Global Fund grants. The main areas identified with inadequate or poor performance were organizational development (including governance and leadership), financial management, procurement and supply management, and M&E (Coleman, 2007; PEPFAR, 2009b). These four areas correspond to four of the six blocks in the WHO building-block model for effective health systems (see Figure 16). There are stated goals for improvement for each of the four areas of technical assistance. In July 2008, a USG Global Fund Technical Support Evaluation was conducted, which showed achievements through 2008 in each of the areas from the $12 million available in FY2005 technical support through the $31 million available in FY2007 (Coleman, 2010). The committee will examine the 2008 evaluation, including its methods and findings and will also request available country/recipient progress reports as well as copies of the grant requests from recipient countries. 48 Supra., note 10.

OCR for page 49
125 KEY SYSTEMS-LEVEL GOALS AND ACTIVITIES Evaluation Strategy As part of an evaluation of the impact of PEPFAR, it is necessary to determine partner country readiness to make this transition to sustainability. To assess partner country readiness, the evaluation committee aims to examine the partnership frameworks with partner countries. In addition, the committee will assess PEPFAR efforts and synergy with other global stakeholders for capacity building and technical assistance, including activities similar to those identified by the International Health Partnership, such as donor funding harmonization and collaboration between international actors and developing countries to develop and implement national health plans. The committee will also establish working definitions, define data sources, and identify evaluation methodologies for the key parameters of sustainability that include country ownership and local capacity building, including health system strengthening and healthcare workforce expansion. However, it has been noted that “little [global] consensus has emerged to provide uniform guidance” to indicators that can track these activities, although the development of these indicators is an area of intense international interest and activity. PEPFAR defines indicator and reporting requirements in health systems strengthening “to reflect a more narrow scope of interest tied to PEPFAR’s focus on HIV” (OGAC, 2009d, p. 199)—resulting in selection of two indicators to be reported centrally to OGAC that reflect laboratory and health workforce strengthening. Evaluating data for other parameters will require metrics not currently available from routine data sources or the PEPFAR indicator database (OGAC, 2009d), including extra- health sector factors such as good governance (Dybul, 2009), to permit the committee to assess performance of current or past PEPFAR activities. The committee will also examine Partnership Frameworks and Implementation Plans to assess what the countries and OGAC are responsible for, how it is being measured or tracked, and how the processes evolve for PEPFAR-related responsibilities and activities to be transitioned to country leadership for sustainable programs and positive impacts on individual and population health. Health System Frameworks PEPFAR’s increased focus on health systems is shared with other global health initiatives, including The Global Alliance for Vaccines and Immunisation (2010) and the Global Fund, as well as the World Bank and the WHO. The committee will benefit from and incorporate deliberations among these agencies that will address both how to define health systems and how to measure progress toward strengthened health systems (Frenk, 2010; Shakarishvili, 2009). Adopting a common conceptual framework is a key requirement for the evaluation of health systems strengthening and is important to framing such measures. The most widely adopted framework, proposed by the WHO in 2007, is based on operational “building blocks” of the health system (Figure 16)—services, workforce, information, commodities and technologies, financing, and leadership/governance (WHO, 2007a). These building blocks serve several functions including describing what a health system should have the capacity to do in each of these blocks to lead to improved health, system responsiveness, improved efficiency, and social and financial risk protection. It also lends itself well to a descriptive set of activities that may be undertaken through PEPFAR programs. The importance of “systems thinking” has also been emphasized, because health systems are complex, dynamic, and non-linear systems whose

OCR for page 49
126 STRATEGIC APPROACH TO THE EVALUATION OF U.S. GLOBAL HIV/AIDS PROGRAMS function is dependent on the interplay of all of its components, including “the spaces in between” (Atun and Menabde, 2008; de Savigny and Taghreed, 2009). FIGURE 16 Representation of WHO’s six building blocks for effective health systems. SOURCE: Adapted from WHO, 2007a.

OCR for page 49
127 KEY SYSTEMS-LEVEL GOALS AND ACTIVITIES Program Impact Pathway The committee will utilize an impact pathway (Figure 17) to assess PEPFAR inputs, output, and outcomes where measurable, within the context of the WHO building blocks framework for health systems strengthening. The pathway can also help assess the same block elements for technical assistance and other activities related to country readiness for assuming increased to total responsibility for their HIV/AIDS response. Although a useful evaluation tool, the linear nature of an impact pathway is a simplified representation of the reality of PEPFAR programs and their impact, and the committee found this to be particularly challenging in the area of systems-level activities. Activities at the systems level are intended to result in an outcome or impact on the health system, but these are also critical inputs to all other programmatic areas. Thus, the functioning of the health system can be both a starting and end point for a pathway. The committee grappled with illustrating the complexities, dynamism, and non-linear nature of health systems with this linear pathway—analogous to forcing a “square peg in a round hole” —but ultimately recognized the utility of the impact pathway to evaluate the process by helping to frame the areas of inquiry and the measures that may be undertaken to assess health system strength. The findings will be interpreted in light of the more complex realities when the evaluation committee draws conclusions and makes recommendations. As previously mentioned, PEPFAR II (as well as the GHI) has adopted the WHO six building block framework to assess its capacity building and programmatic impact on health systems, categorizing contribution to (1) core HIV activities, (2) secondary benefits or intentional spillover effects of PEPFAR activities on other programs, and (3) targeted leveraging including partnerships. The six building block elements are denoted in the figure with blocks in all uppercase letters. The committee’s working definition of an intervention that strengthens health systems is one that improves the activities and processes within the six health system building blocks, and manages interactions within the “building blocks” over time to achieve more equitable and sustained improvements in population health outcomes. These effects may be either short-term (e.g., better trained workforce delivering higher quality care) or longer-term (e.g., higher-quality care results in a healthier population that is economically more productive), thus growing resources that feed back into the causal chain as an input to the health system. Additional outcomes of an effective health system (e.g., social and financial risk protection for beneficiaries accessing and utilizing services or increased block or system efficiency) can result in increased responsiveness and potentially expanded service coverage when the activities within and among the blocks dynamically interact (see Figure 17). As previously mentioned, interventions in one part of the health system may have impacts in others. Since any intervention may affect the entire system, evaluation is, by definition, context-specific and requires a mix of evaluation strategies, both quantitative and qualitative. Although improved health is an important impact of health systems strengthening, additional measures regarding health system efficiency and equitable distribution of benefits may likewise be important to track as they are developed and adopted by international working groups.

OCR for page 49
128 FIGURE 17 Program impact pathway for evaluation of PEPFAR’s health systems strengthening activities at the country level. NOTES: Upper case headers indicate the WHO’s six building blocks. All of the elements listed in the pathway can be categorized under one of the six building blocks, but due to space limitations, select elements were chosen to illustrate blocks within the pathway. ARV = antiretroviral drugs; ART = antiretroviral therapy; CD4 = cluster of differentiation 4; EQA/PT = external quality assurance/proficiency testing; M&E = monitoring and evaluation.

OCR for page 49
129 KEY SYSTEMS-LEVEL GOALS AND ACTIVITIES Illustrative Questions The main questions that the committee will consider in the evaluation include the potential positive and negative impact of PEPFAR activities on country-level health system functioning, with regards to both HIV and non-HIV programs. Examples of the kinds of questions the committee may ask that specifically link the WHO six building blocks to our results chain framework include the following. These questions are intended to contribute to addressing part B, item ii of the areas for consideration in the congressional mandate. 49 Because health systems are a fundamental aspect of all program activities, these questions are also intended to contribute to addressing all of the areas for consideration described in the Statement of Task (see Appendix A). Finance: To what extent has PEPFAR funding and technical assistance for sound public finance systems resulted in more efficient and equitable care in HIV and non-HIV health systems? To what extent do the joint funds from the Global Fund support the six building blocks? To what extent have the technical support funds from PEPFAR to the Global Fund improved the performance of grant recipients in general and how is this measured? For the four main areas of technical assistance which are among the six WHO building blocks and how are they measured? Commodities and Procurement: How have PEPFAR technical assistance and training affected HIV laboratory diagnostic capacity, pharmacy infrastructure, and supply chain management for reagents and drugs? Information Systems: What is the evidence that PEPFAR-supported health information systems are resulting in higher functioning, quality-driven health systems performance? How will new and existing information officers be trained to meet country needs to strengthen higher functioning and increased quality? Service Delivery: What elements of more efficient, equitable, and effective service delivery should be expected to result in improved population health over time with better integration of HIV- and non-HIV health care and use of continuous quality improvement methods? Leadership and Governance: To what extent will partnership frameworks, jointly funded Global Fund activities, and other expressions of country ownership lead to improved and accountable governance as well as transfer of oversight, management, guidance, and financing for HIV-related services in health systems? Health Workforce: How will the pre-service education target of 140,000 new workers affect health system equity of access (e.g., rural versus urban) and health system strengthening such as increased skill capacity and retention of workforce? Are there other types of workers, (e.g., social service workers and program managers) who also need to be trained? What proportion of newly trained new health or social workers is 49 (B)(ii) an assessment of the effects on health systems, including on the financing and management of health systems and the quality of service delivery and staffing

OCR for page 49
130 STRATEGIC APPROACH TO THE EVALUATION OF U.S. GLOBAL HIV/AIDS PROGRAMS retained one year later? How does PEPFAR support the country’s existing health resources development plan or help develop those where absent? The committee will also assess progress toward the goals of increased country ownership to transition the HIV/AIDS response to national governments for long-term sustainability. Although there are high-volume discussions within the global development assistance community in this area, there appear to be few to no meaningful metrics to measure increased ownership or country readiness, no accepted logic models to describe and illustrate the transition process, and no indicators for when a country has sufficiently strengthened governance/ leadership or built and operated a financially-sound public finance system to implement and oversee a national health plan. Illustrative questions for this area of the evaluation include the following. Country ownership and sustainability cut across all PEPFAR activities and as such these questions are intended to contribute to addressing all of the areas for consideration in the congressional mandate, as described in the Statement of Task (see Appendix A). How is country ownership defined by the country? By PEPFAR? By relevant global stakeholders in aid development? Are there differences? Have they been reconciled? Do the Partnership Frameworks represent an “agreement” on the definition between the country and PEPFAR? To what extent have PEPFAR capacity-building, technical assistance, and financing activities to countries contributed to country readiness for transitioning the knowledge management, decision making responsibility, financing, and accountability/oversight of the PEPFAR-funded HIV/AIDS response to the national government? How is it being measured? Are plans in place for future activities needed to improve or increase country readiness if PEPFAR were absent? How will achievements or milestones by country and/or PEPFAR be measured in the Partnership Frameworks? Will these measures adequately reflect country readiness for program, policy, and financial transitions? Is the country transitional process explained adequately? Will it be standard for all countries or adapted for each country? Will the transition be parceled over time? Which pieces might be transitioned over time, how, and why? What are the achievements and lessons learned from the Technical Support Grants to the Global Fund from PEPFAR? How are they measuring whether bottlenecks have been opened or bypassed? What has been the impact on the functioning of programs? How do Global Fund and PEPFAR measure their relative and unique contributions to programs they jointly fund or operate? How are access, equity, and quality measured for these programs? A more extensive listing of illustrative questions for this complex issue that the committee may attempt to address, given the timely availability of quality data, can be found in Appendix H. These may help guide assessment of whether and how PEPFAR has helped

OCR for page 49
131 KEY SYSTEMS-LEVEL GOALS AND ACTIVITIES countries to mount a stronger response to their HIV epidemics, plan for transition to country ownership including effective leadership and oversight of multilateral donor activities, and focus on affordable care that meets population health needs. Sustainability and its associated elements must be considered for all of PEPFAR’s support. PEPFAR activities will be evaluated relative to their contribution of inputs and support of processes, and the resulting outputs and outcomes. These should ultimately result in the desired outcome of stronger health systems that can more adequately respond not only to HIV but to other serious causes of morbidity and mortality, as well as any emerging challenges to the health and safety of their respective populations.

OCR for page 49