Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 67
Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary 3 Designing an Evaluation That Incorporates the Guiding Principles of Coordination, Harmonization, and Capacity Building Workshop participants focused their discussions not only on what should be evaluated, but on the ideal process of designing and conducting the evaluation itself. Participants stressed the importance of an earnest effort to design and conduct the evaluation in a way that truly incorporates the principles of coordinating evaluation efforts among global partners, harmonizing with evaluation needs of country partners, and contributing to strengthened local evaluative capacity. Many lessons for designing the process of evaluation can be learned, participants noted, from previous experience with harmonization, coordination, and capacity building in the context of implementation of human immunodeficiency virus/acquired immunodeficiency syndrome (HIV/AIDS) interventions through the President’s Emergency Plan for AIDS Relief (PEPFAR). This chapter summarizes the benefits, costs, and opportunities of coordination and harmonization, as well as for capacity building in evaluation. BENEFITS, COSTS, AND OPPORTUNITIES OF COORDINATION AND HARMONIZATION IN EVALUATION Benefits of Coordination and Harmonization in Evaluation Workshop participants outlined the value and benefit of coordination and harmonization in impact evaluation. Discussant Jim Sherry of George Washington University noted that coordination and harmonization in evaluation design and implementation are important in influencing
OCR for page 68
Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary others’ work, which is in turn critical for addressing strategic questions at a broader policy- or program-practice level. It is useful to distinguish, he noted, between an internal, institution-specific evaluation that relates to tracking needs for program survival and direction change and a shared international evaluation that relates to influencing how other actors use resources and to strengthening overall capacity. Given the high program transaction costs of evaluation for countries and partners, collaborating as much as possible also minimizes work and ensures more efficient use of funds, observed speakers Julia Compton of the UK Department for International Development (DFID) and Agnes Binagwaho of the Rwanda National AIDS Control Commission. Speaker Sara Pacqué-Margolis of the Elizabeth Glaser Pediatric AIDS Foundation further underscored the importance of coordination to ensure maximum use of evaluation dollars, noting that successful completion of impact evaluations involves high human and financial resource costs, an extensive time frame, and serious commitment. Speaker Mary Lyn Field-Nguer of John Snow, Inc., pointed out that the strengths and perspectives brought to the table by focus-country government partners, implementing partners, and other stakeholders are a further benefit. She and other workshop participants outlined the value each stakeholder brings to the evaluation process. Partner Countries The value of engaging partner countries is their accountability to their citizens, observed Field-Nguer. Government partners were dealing with the challenges of service delivery to their populations long before PEPFAR and can provide a critical perspective on health system issues, such as health care workforce and supply chain issues, and how PEPFAR is addressing these, she added. Even in the context of an emergency situation, involvement of partner countries in the design of evaluation questions and methodology can improve the quality of the overall evaluation design and the interpretation of evaluation findings, noted speaker Binagwaho. Speaker Kathy Marconi of the Office of the U.S. Global AIDS Coordinator (OGAC) stressed that involvement of partner countries can support the process of developing evaluation priorities that are actually meaningful to the countries, and this has implications for sustainability. Implementing Partners The value of engaging implementing partners is their familiarity with program data and lessons and their understanding of the challenges of delivering services across a continuum of care, stated Field-Nguer. Coordination with implementing partners, with their knowledge of the lessons of
OCR for page 69
Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary decades of AIDS work, for example, can contribute to avoiding mistakes, such as repetition of research to answer questions already answered and implementation of program strategies that do not work. Implementers add value not just as reporters of data and information, but also as users of data and information, and hence need to be involved in the evaluation design process, observed speaker Nils Daulaire of the Global Health Council. Beneficiaries and Other Stakeholders Local people add value to the evaluation process because they can place change in context, said workshop participant Joanne Capper of the U.S. Peace Corps. Speaker Field-Nguer added that service beneficiaries are the experts on program impact and are an audience that can define the prevention, care, and treatment service parameters of acceptability, accessibility, and affordability. In the end, patients, clients, and community members define the effectiveness of care services. However, she noted, few studies are available about the community level of knowledge. Field-Nguer identified specific groups of stakeholders who can add value to the design phase of impact evaluation through choosing the questions and the methodologies. These groups include the following: People living with HIV/AIDS Women Youth, including those in and out of school Other populations at risk Government ministries—beyond ministries of health—and local management units Health care workers at all levels in urban and rural settings Health facilities Nongovernmental and community organizations working inside and outside facilities Community leaders Religious leaders Constraints of Coordination and Harmonization in Evaluation High transaction costs are among the greatest constraints to coordination and harmonization in evaluation, workshop participants said. There is tension between the benefits of coordinating and taking advantage of existing synergies by linking with others’ evaluation work and the costs of that coordination, remarked workshop moderator Ruth Levine of the Center for Global Development. “The mere sharing of information is a huge task that can be all consuming,” observed Ambassador Jimmy Kolker, OGAC.
OCR for page 70
Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary In addition to being costly, the processes of design, consensus building, commitment, planning, and working with host countries are very complex, asserted speaker Pacqué-Margolis. Speaker Paul De Lay of the Joint United Nations Programme on HIV/ AIDS (UNAIDS) pointed out that the transaction costs of inclusiveness and consensus building are particularly pronounced when immediate results are desired, as was the case for PEPFAR. However, he noted, in order to sustain desired results, harmonization, integration, and sharing across partners will be necessary. Overtaxing or overextending evaluators with efficiency-level questions was a further constraint to coordination identified by discussant Sherry. Questions at this level prevent partners from influencing others’ work and being influenced. He emphasized that the focus of coordination and harmonization should be on strategic, broad, or program-level questions. Institutionalizing accountability to program beneficiaries through coordination may be constrained when there is a large power imbalance such as that existing between donors and partner countries, observed another workshop participant. When countries depend on donor resources, which could potentially be removed, there is concern that countries may not stand up to donors and speak up for what they want, particularly if a strong coordinating plan and leadership are not in place. Institutionalization of such accountability, noted speaker Norm Daniels of the Harvard School of Public Health, depends heavily on the effectiveness of the national coordinator and the national plan. Donors who view country ownership as a desirable objective also must be open to acknowledging what counts as fair and reasonable at the national level. Opportunities for Coordination and Harmonization in Evaluation Efforts Workshop participants described a number of opportunities for coordination and harmonization with global partners and country partners in evaluation design and implementation. Coordinating Evaluation Design, Conduct, and Results with Global Partners Workshop participants strongly articulated the need for PEPFAR to coordinate its evaluation efforts with other global partners. PEPFAR should not just be learning from its own evaluations, said speaker Rachel Glennerster of the Abdul Latif Jameel Poverty Action Laboratory, it should be looking at work being done elsewhere. Another participant stressed that such coordination should include divulging and
OCR for page 71
Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary sharing negative results with global partners. Speaker De Lay noted that in parallel to PEPFAR’s evaluation efforts, The Global Fund to Fight AIDS, Malaria, and Tuberculosis (The Global Fund); the World Bank; and UNAIDS are conducting evaluative efforts, and bilateral efforts, such as DFID’s AIDS program, are also undergoing evaluation. Although these different evaluations address unique issues, constituencies, and time frames, there is strong potential for sharing data, approaches, and evaluation research. De Lay suggested that outcomes of PEPFAR evaluative efforts be tied to and shared with other global and bilateral evaluations of AIDS institutions and initiatives. Several of these efforts are described in greater detail below. The Global Fund. Speakers De Lay, Compton, Marconi, and John Novak of the U.S. Agency for International Development (USAID) supported the suggestion that PEPFAR collaborate with The Global Fund evaluation, particularly with regard to broader questions such as systemwide effects. As De Lay noted, The Global Fund is just starting a 5-year evaluation with a specific set of questions about the effectiveness of the funding models, level of partner support, and technical assistance. The Global Fund evaluation will devote nearly $15 million of the $17 million total evaluation budget to research, using prospective study survey evaluation research methods. Discussant Kolker noted that OGAC is currently in close contact with The Global Fund impact study organizers. Speaker Theresa Diaz of the U.S. Centers for Disease Control and Prevention (CDC) added that an OGAC technical group will work with The Global Fund impact evaluation technical group to review all methodologies and analyses used. In every PEPFAR country, a group of government contacts has been designated as part of the task force working with The Global Fund on the impact evaluation, she said. UNAIDS. Speakers De Lay, Marconi, and Stefano Bertozzi of the National Institute of Public Health, Mexico, along with discussant Kolker, suggested that PEPFAR engage collaboratively with UNAIDS. According to De Lay, UNAIDS had a major evaluation in 2001–2002 and will now start a second evaluation. These evaluations have focused on the role and impact of UNAIDS cosponsors in a changing environment. Bertozzi noted the particular value of engaging with the economics reference group at UNAIDS; Marconi mentioned the strengths of the UNAIDS modeling reference group in developing global-level impact measures, such as those for stigma and gender discrimination. Kolker noted that the United States is the main supporter of the UNAIDS monitoring and evaluation (M&E) reference group.
OCR for page 72
Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary World Bank. PEPFAR collaboration with the World Bank on evaluation may be beneficial in developing an integrated HIV/AIDS research agenda, noted speaker Jody Kusek of the World Bank. She said the World Bank evaluation will focus on the impact of technical assistance such as treatment, scale-up of treatment facilities, prevention, programs that affect policy instruments (such as cash transfer policy instruments as incentives for behavioral change), and socioeconomic impacts of HIV/AIDS; the effectiveness of HIV/AIDS programs in achieving goals; and the design of new investments to ensure that impact can be assessed. OECD. Collaboration with the Organization for Economic Cooperation and Development (OECD) on evaluation may be helpful in the area of HIV/ AIDS that deals with gap analysis and meta-evaluations, speaker Compton said. WHO. Discussant Kolker reported that OGAC has been involved in the annual meeting on the World Health Organization’s (WHO’s) HIV/AIDS impact evaluation. Coordinating with partners beyond the AIDS community. Discussant Sherry suggested that drawing on evaluation expertise beyond the AIDS community, such as in the areas of sustainability and broader development issues, may be beneficial to the evaluation of PEPFAR. Coordinating Evaluation Design, Conduct, and Results with Country-Level Partners Workshop participants described several opportunities for harmonization of evaluation with country partners. Speaker Pacqué-Margolis asserted that the evaluation function should be prioritized in country operation plans; partners should be included in evaluation planning; and funding mechanisms should promote harmonization. Workshop discussant Phillip Nieburg of the Center for Strategic and International Studies envisioned that country partner organizations would be given an opportunity to review drafts and submit comments and concerns in the evaluation process. Speaker Binagwaho and workshop participant David Stanton of USAID highlighted the importance of community-level interpretation of evaluation results. Several speakers pointed out that impact evaluation should mirror, and perhaps draw lessons from, the experience in coordinating and harmonizing with partner countries in program implementation. Field-Nguer reminded participants that the guiding philosophy of PEPFAR, the “three ones,” includes the concept of a single monitoring and evaluation plan shared with partner countries. Speaker Daulaire commented that lessons about
OCR for page 73
Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary coordination and harmonization also can be drawn from the experience in universal childhood immunization and smallpox eradication, in which stakeholder engagement was done effectively and with a view toward sustainability. Specific opportunities for country-partner harmonization on evaluation are described below. Joint field evaluations. Discussant Kolker remarked that aid effectiveness could be improved if common program and sectoral approaches used joint evaluation visits based on a single national plan and a single monitoring and evaluation system. Speaker Binagwaho offered the Rwandan experience—in which partners from PEPFAR, The Global Fund, and the government of Rwanda routinely conduct joint field visits for evaluation purposes—as a potential model. Centralized funding and data aggregation. Echoing points made in Shannon Hader’s presentation about the value of a central coordinating hub for impact evaluation, speaker Pacqué-Margolis argued that a dedicated, central funding source is operationally better suited to coordinate the evaluation effort across countries. She noted that funding mechanisms that send money to the country level do not promote harmonization on evaluation because countries often have little money left over for M&E. Pacqué-Margolis further stated that evaluation findings need to be aggregated across countries—also by a central coordinating unit—to be meaningful and to see patterns. Knowledge management. Speaker Field-Nguer spoke of the opportunity to use evaluation coordination to develop a mechanism for knowledge management, both for existing data and for methodologies to gather new data. Harmonization of fairness-monitoring approaches. Speaker Daniels suggested that harmonization be used to work toward agreement on which aspects of equity, accountability, and efficiency will be part of the M&E program. He noted that such discussions provide an opportunity to clarify the overall program objectives. BENEFITS, CONSTRAINTS, AND OPPORTUNITIES OF BUILDING CAPACITY IN EVALUATION Benefits of Building Capacity in Evaluation Workshop participants emphasized the importance of designing an evaluation that itself strengthens local capacity. In-country capacity is
OCR for page 74
Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary needed not just for service delivery, noted speaker Pacqué-Margolis, but for M&E and for continuous quality improvement, advocacy, national planning, and budgeting. Building local evaluative capacity has particular benefit as PEPFAR undergoes a transition from an emergency program to a long-term, sustained program, speakers noted. In an emergency response, observed speaker Bertozzi, the time to build local capacity on evaluation issues is limited. In contrast, a long-term response involves, for example, not just training existing health practitioners but also educating new ones, and not just involving local researchers in a project, but also building the capacity of local health researchers to do prospective evaluations. Field-Nguer added that local capacity and systems to collect, analyze, and use program information are of critical importance to program success and sustainability and should be built into the process of PEPFAR program implementation and impact evaluation. Constraints to Building Local Capacity in Evaluation Lack of systems for gathering data, inadequate funding mechanisms, and poor stakeholder engagement are among the constraints to building local capacity for evaluation, workshop participants said. Speaker Kusek observed that many countries are unable to take advantage of the 5 percent to 10 percent of total project budget available for evaluation under World Bank loans because they often don’t have systems in place for gathering data. There is powerful competition for resources between the development of monitoring systems and the implementation of the program, she said. Pacqué-Margolis observed that although funding mechanisms should develop and promote capacity for evaluation, information dissemination, and advocacy, they often fall short. Funding earmarks and directives limit the prioritization of M&E, operations research, clinical research, and advocacy in Country Operation Plan planning. Poor or distorted engagement of stakeholders is another constraint to building local capacity for evaluation. Many partners, including country-level stakeholders, do not have a place at the table in defining the research agenda and conceptual frameworks, noted Pacqué-Margolis. In contrast, researchers tend to drive the evaluation agenda according to their own interests, noted Kusek, using resources to conduct more narrowly focused impact evaluations that are less helpful. Opportunities for Strengthening Evaluative Capacity Workshop participants suggested a number of opportunities for strengthening local capacity to conduct impact evaluation. Diaz suggested that donors could disseminate suggested methodologies, offer technical assistance, conduct training workshops, and engage in one-on-one mentor-
OCR for page 75
Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary ing. Binagwaho added that the recruitment and training of independent consultants in partner countries is another way to develop competency in evaluation methods that will remain in the country. Country-driven decision making and priority-setting processes were also suggested as potential models or tools for strengthening local evaluative capacity. Binagwaho suggested that the experience of comanaged decision making in Rwanda potentially could be applied to evaluation design. Workshop participant Laura Porter from CDC suggested that a country-driven evaluation priority-setting process—drawn up by a multidisciplinary team—could be another mechanism for building local capacity. She noted that a need for such a process was articulated at a recent meeting of the M&E Reference Group Evaluation Subgroup.
OCR for page 76
Design Considerations for Evaluating the Impact of PEPFAR: Workshop Summary This page intentionally left blank.