Click for next page ( 18


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 17
PART II Proposed Evaluation Approach

OCR for page 17

OCR for page 17
CONGRESSIONAL CHARGE AND PLANNING PHASE APPROACH Under the Lantos–Hyde Act of 2008, 9 Congress mandated that the IOM conduct a study that includes an assessment of the performance of U.S.-assisted global HIV/AIDS programs and an evaluation of the impact on health of prevention, treatment, and care efforts that are supported by U.S. funding, including multilateral and bilateral programs involving joint operations (see Appendix A). Based on clarifications with congressional staff and OGAC, 10 the charge is intended to focus on the performance and impact of bilaterally funded PEPFAR programs in the current partner countries (see Table 1 in Part III, Section 2 for a list of countries). This will include programs and activities that are operated jointly with both bilateral funding through PEPFAR and funding through the Global Fund. Consistent with the clarified congressional intent, U.S. contributions to the Global Fund that are not a part of activities jointly funded or implemented by PEPFAR will not be the focus of the evaluation, and the evaluation will not compare the performance of bilateral PEPFAR programs to that of Global Fund programs (Bressler, 2009; Marsh, 2009). The study will consider PEPFAR’s performance and impact since funding first became available in 2004. The timing of the study, with a final report to be delivered in 2012, dictates that the evaluation will consider data that are now, or will become, available through 2011. As the first phase of this study, the IOM was charged to form an ad hoc committee to develop a strategic plan for the assessment and evaluation of HIV/AIDS programs implemented under the Lantos–Hyde Act of 2008 and to issue a short report to the U.S. Congress describing the plan’s proposed design, taking into consideration the requirements for the congressionally mandated study. These requirements and the charge for developing the evaluation plan can be found in full in the Statement of Task in Appendix A. More information about the members of the planning committee can be found in Appendix B. This report documenting the proposed evaluation approach is the product for the first phase of the study. To produce this report, the planning committee met three times to deliberate in person, conducted two teleconferences, and engaged in additional deliberations in smaller working groups by telephonic and electronic communications as needed. In the development of its plan, the committee consulted widely, and remains open to receiving input from, the broad range of parties interested in and affected by PEPFAR. To solicit input and gather information from a wide range of stakeholders, public sessions were held in conjunction with the first and second committee meetings, and delegations from the committee and IOM project staff also held information-gathering meetings with a range of global stakeholders, including UNAIDS, WHO, the United Nations Children’s Fund (UNICEF), and the Global Fund. The primary purpose of these meetings was to establish working relationships with these stakeholders and to discuss potential data sources and methodologies, as well as strategies and lessons learned from large- scale programmatic or organizational evaluations. The agendas for these activities can be found in Appendix C. In addition, one staff member attended the PEPFAR Annual Implementers’ Meeting in 2009. The committee also consulted the available literature on PEPFAR, global HIV/AIDS, and the state of the art in large-scale program evaluation, including the summary of a 9 Supra., note 6 at §101(c), 22 U.S.C. 7611(c). 10 Personal communications from Congressional Staff at the U.S. House Committee on Foreign Affairs and U.S. Senate Committee on Foreign Relations and OGAC, 2009. 19

OCR for page 17
20 STRATEGIC APPROACH TO THE EVALUATION OF U.S. GLOBAL HIV/AIDS PROGRAMS workshop convened by the IOM, “Design Considerations for Evaluating the Impact of PEPFAR,” which focused on methodological, policy, and practical considerations (IOM, 2008). The committee and staff also conducted an initial scan of potential data sources for the forthcoming evaluation using a range of sources, including a preliminary review of documents from OGAC and other bilateral and multilateral agencies and of relevant published literature, as well as communications with a wide range of staff from OGAC, implementing partners, and multilateral stakeholders. The committee used this information to assess the methods that could be employed to answer evaluation questions based on the charge in the statement of task, focusing on data and methodology that will be robust, available, feasible, and appropriate to the questions. Through this information gathering and deliberation, the planning committee developed a conceptual framework for the evaluation that is based on both the committee’s expertise and current standards in evaluation methodologies for large-scale programs. This report is intended to provide Congress and OGAC with an overview of the strategic plan for the forthcoming evaluation. As agreed upon contractually with the study sponsor, the planning process for the evaluation will culminate with a transitional period for operational planning that will take place between the delivery of this report and the implementation of the evaluation itself in the fall of 2010. During this operational planning phase, IOM staff, planning committee members, and consultants will carry out activities to further develop and refine the plan described here. These activities, which will inform the implementation of the evaluation, are described as part of the work plan later in this section. This structure for the study, with a report describing strategic elements of the plan delivered to Congress before detailed operational planning is complete, was intentionally designed to allow uninterrupted progress in preparation for the evaluation during the time necessary for review of the report and budget planning by the sponsor and for subsequent preparation of the contract for the evaluation. After congressional review of the plan’s proposed design and budget, the final phase of the project will be to carry out the assessment/evaluation of the program. The IOM will convene a new ad-hoc committee to conduct the evaluation as a consensus study. The intent is for the evaluation committee and staff to have considerable overlap from the planning committee. Standard IOM procedures will be followed to ensure that the evaluation committee and project staff have the appropriate expertise to conduct the evaluation activities described in this report. EVALUATION GOALS AND CONCEPTUAL FRAMEWORK FOR EVALUATION DESIGN The legislative mandate to evaluate PEPFAR is a complex challenge. As described above, PEPFAR is a large, multifaceted program with many activities carried out by many different partners in a diverse group of countries. In addition, PEPFAR activities are being implemented in the context of programs supported by other funders that have the same ultimate aim. PEPFAR is also by necessity a dynamic program; the ability to change the program over time can be beneficial, but makes evaluation difficult as it presents a “moving target.” Therefore, this report not only outlines an approach for evaluating the performance of PEPFAR, but also delineates the challenges in evaluating the impact of such a complex, large-scale foreign assistance program and provides information about reasonable and appropriate expectations for an evaluation of this kind. The committee has endeavored to present a plan that is thorough and well-defined in its approach yet maintains ample flexibility. This will allow the evaluation to be adapted in response both to the evolving goals of the program and to the additional information

OCR for page 17
21 PROPOSED EVALUATION PLAN the evaluation committee will gather during operational planning and as the evaluation itself proceeds. The proposed conceptual framework for the evaluation and its limitations will be described briefly here, followed by a more thorough discussion of the methods and data sources that will be used. The subsequent sections in Part III of this report address specific components of the evaluation in greater detail. Evaluation Goals and Assumptions The planning committee understood the mandate from Congress as a charge to develop a plan to assess the program with two primary goals. The first of these is an assessment of the success of the program in meeting the performance goals and targets laid out in two sources: the reauthorization legislation and the new PEPFAR Five-Year Strategy. Although the statement of task was written before the new strategic plan was available, the committee interpreted the charge to take this document into account because it articulates the current guiding principles and the future direction for the program. Therefore, the evaluation will include a careful review and comparison of these guiding documents in order to more clearly define the targets and goals of the program. The second goal of the committee’s charge is to evaluate the health impact of PEPFAR, including impact of treatment, care, and prevention programs; effects on health systems; efforts to address gender-specific aspects of HIV/AIDS; impact of programs on child mortality; and impact of interventions on behalf of orphans and vulnerable children. The findings and conclusions of the evaluation of PEPFAR’s progress toward its stated goals and the impact of the program will then be used to make recommendations for improving the USG response to global HIV/AIDS, in particular through PEPFAR programs. It is important to note that the IOM is being charged to conduct an evaluation early in the implementation of changes to the program in response to the reauthorization legislation and the new PEPFAR Five-Year Strategy. These changes reflect a progressive transition to a new era of challenges and goals for the program, which include efforts to improve sustainability of the response over time, to enhance coordination with partner governments and other global funding partners, and to support accountable ownership of HIV program delivery by countries themselves. They also reflect efforts to give greater consideration to the relationship of PEPFAR to broader health and development needs in partner countries. The timing of this evaluation, with data collection extending through 2011 for a final report due in 2012, will make it difficult to evaluate the outcomes or impact of these most recent changes so soon after implementation. For example, it could take several years or even decades for a full effect to be realized from some efforts to strengthen health systems, such as the training and retention of new health care workers or the strengthening of health information systems to support M&E efforts. However, the evaluation will assess efforts, process, and initial results in these areas to provide insight into whether PEPFAR is making reasonable progress toward these new goals and to lead to recommendations for how the program can be improved to ensure that these evolving goals for the program can be met. As part of this, the evaluation will assess whether there is sufficient M&E capacity in place to eventually evaluate whether the program has met these goals as well as the resulting outcomes and impact. The legislative mandate calls for the assessment of PEPFAR to be delivered in 2012; this would coincide with reauthorization discussions for the program, which the current legislation extends through 2013. It is of course not possible to predict the future needs and priorities of

OCR for page 17
22 STRATEGIC APPROACH TO THE EVALUATION OF U.S. GLOBAL HIV/AIDS PROGRAMS Congress and OGAC with complete accuracy, but the planning committee’s goal was to design an evaluation approach that, to the extent possible, looks ahead to anticipate the evolution of the program and therefore produce findings that address key issues under consideration at the time of the report release, including discussions about possible future legislative reauthorization. Conceptual Framework for Evaluation Design The planning committee developed an overall conceptual framework for the evaluation, which calls for the use of a program impact pathway to guide an assessment of the contribution of PEPFAR to changes in health impact within the context of multiple international and national funding streams. This program impact pathway, described in more detail below, illustrates the committee’s understanding of how PEPFAR programs are currently structured and intended to ultimately translate into health impacts, laying out a plausible pathway for causal effects. It represents the theory of change that underlies the program—in other words, the rationale for how the combination of activities supported by PEPFAR are logically expected to produce intermediate outcomes, which are then expected to collectively contribute, along with programs funded by other sources, to the desired population health impact. The use of a program impact pathway, which is also referred to as a logic model or results chain, has become well established as a method for evaluating complex, large-scale development assistance programs and is becoming widely accepted as a standard in the global HIV/AIDS community (Leeuw and Vaessen, 2009; UNAIDS MERG, 2010). 11 Guided by the program impact pathway, the evaluation committee will use a mixed methods approach that will draw on a combination of analytical techniques and on a range of both quantitative and qualitative data sources. By assessing whether there is convergence and consistency among different data sources and methods, the evaluation committee will seek to triangulate findings that support reasonable, plausible linkages to outcomes and impact (Greene et al., 1989; Leeuw and Vaessen, 2009). The methods and data in the mix will complement each other, and will each have different strengths and limitations. This approach helps to account for the reality that, even given access to all potential data sources and extensive evaluation resources, there still would not be direct measures to answer many of the evaluation questions posed in the charge to the IOM. However, when taken together, the totality of evidence will allow the evaluation committee to draw conclusions and make recommendations for the program as a whole. Program Impact Pathway Figure 3 shows the program impact pathway that the planning committee developed to represent a plausible causal chain of results for PEPFAR. The pathway begins with a series of investments or inputs to the program. For PEPFAR, these inputs include not only funding and other resources but also strategic planning, programmatic and policy guidance, technical assistance, and knowledge transfer and research that represent the evolving evidence base. These inputs support activities that provide services and support to children, adolescents, and adults in need. Although these services are described by PEPFAR in categories like prevention, treatment, 11 Many of the terms used in the program impact pathway have different meanings in different fields of research. In this report, the terms correspond to definitions that reflect the current consensus in program evaluation. Definitions can be found in the glossary (Appendix D).

OCR for page 17
23 PROPOSED EVALUATION PLAN and care and support, the conceptual framework acknowledges that they are all part of an interrelated and overlapping approach, which also includes activities around gender issues and capacity building. These activities result in outputs that are measureable proximal effects. When the program is implemented well, these outputs are expected to produce outcomes as intermediate effects on the pathway to the ultimate goal of health impact. These intermediate outcomes include, for example, the delivery of high-quality, efficient services that are available and accessible to the targeted populations and that are achieving the intended and appropriate coverage. Other target outcomes include, for example, health systems strengthening; changes in individual risk behavior; and changes in knowledge, norms, and attitudes that affect sexual behavior, stigma, and gender issues. Ultimately, the program is intended to operate through this pathway to contribute to an impact on individual and population health and well-being, including HIV incidence, HIV prevalence, morbidity, and mortality. Data will not be available to directly measure all of the outcomes and impacts illustrated in the impact pathway, and when available may need to come from sources other than PEPFAR. In some cases, such as assessing effects on incidence, proxy measures or modeling data will have to be used. A critical advantage of the program impact pathway approach is that it identifies the intermediate steps between the inputs invested in the program and the ultimate impact on health. This allows the evaluation to consider not just the beginning and endpoints, but also to assess whether the program is performing in the way it is intended along the full range of its implementation. Thus, even when it is not possible to assess impact directly, the evaluation committee will be able to state plausible findings about the effects of the program and draw conclusions that provide more refined and useful information about elements of the program that are functioning well or that could be improved in order to result in a greater impact on health. For each of the programmatic areas that will be assessed in the evaluation, the committee will work from more specific program impact pathways. These are described in Part III of this report, along with illustrative evaluation questions based on the committee’s interpretation of its charge to assess PEPFAR’s performance and impact. All of the specific program impact pathways are oriented to describe outcomes that contribute to the HIV-related health impacts shown in Figure 3, which represent the stated overall goals of PEPFAR.

OCR for page 17
24 FIGURE 3 Program impact pathway for evaluation of PEPFAR’s effects on HIV-related health impact for children and adults. In the case of joint PEPFAR and Global Fund programs, some inputs may be provided by the Global Fund.

OCR for page 17
25 PROPOSED EVALUATION PLAN Although it provides a critical guide for developing evaluation questions, assessing data sources, and selecting methodologies, the program impact pathway is of course a simplified view of PEPFAR programs and their impact. Of particular importance for this evaluation’s conceptual framework is the reality that in any country that receives PEPFAR support, the program operates within the context of a wide range of other factors that affect the implementation of the program as well as health outcomes (see Figure 4). Investments from a range of other sources support programs that are aimed at the same desired outcomes, and the proportion of total HIV/AIDS support that is provided by PEPFAR varies from country to country. In some cases, multiple funding sources may be co-mingled to support the same programs. Therefore, changes in population health that can be used to reflect program impact cannot be separated by specific programs or investments. Even individual measures can be difficult to attribute directly, as an individual or household may be receiving different services from different programs funded through different sources, all of which have an impact on the health outcomes of the beneficiary. Health outcomes are also influenced by a wide range of cultural, societal, geographical, and political factors and influences that the program cannot control. In addition, as PEPFAR programs increasingly operate with an emphasis on country ownership and harmonization with national plans, the extent to which central USG guidance and authority can influence all levels of priority-setting, decision making, and implementation can be quite limited. Finally, with a foreign assistance program that is implemented as broadly and on the scale of PEPFAR, there is rarely an appropriate comparison available in order to attribute outcomes to the program based on what would have happened in the absence of the investment. Therefore, although the ideal goal in a program impact assessment may be to determine to what extent a desired outcome can be attributed directly to the program or policy investment, the realities of a large-scale program such as PEPFAR can make it difficult to determine the extent to which successes or failures in achieving the intended effect can be attributed directly to the program. Thus, the aim of this proposed evaluation approach is not to attempt to determine the direct attribution of PEPFAR funds to health outcomes. Rather, the aim is to assess the plausible contribution of PEPFAR to changes in health impact, both globally and by country, within the landscape of broader funding, programs, and other factors that influence health. This contribution analysis approach is consistent with the guidance given to the committee by congressional staff about expectations for the evaluation (Bressler, 2009; Marsh, 2009). It is also accepted as an appropriate standard for large-scale development assistance programs (Leeuw and Vaessen, 2009). There may be some areas in which attribution can be more readily determined or approximated, as in the direct relationship at the first step of the impact pathway between inputs and the activities they support, or in the case of controlled experimental studies that assess the effects of intervention components that are distinct to PEPFAR, or in countries where PEPFAR is or has been the nearly exclusive funder of all national HIV/AIDS activities. If feasible, when these opportunities arise, the evaluation committee will consider whether a finding of attribution may be plausible.

OCR for page 17
26 STRATEGIC APPROACH TO THE EVALUATION OF U.S. GLOBAL HIV/AIDS PROGRAMS FIGURE 4 Context for PEPFAR program implementation. NOTES: M&E = monitoring and evaluation; USG = U.S. government; NGOs = non-governmental organizations.

OCR for page 17
27 PROPOSED EVALUATION PLAN OVERARCHING EVALUATION CHALLENGES AND LIMITATIONS There are a number of overarching challenges to carrying out this evaluation. These are described here, while more specific challenges and limitations are described in more detail in the following sections on evaluation methodologies and data sources as well as in the subsequent sections in Part III of this report that address specific components of the evaluation. One of the primary challenges to the evaluation is that there are limited data to address health impact and other evaluation questions about the whole of the PEPFAR program. Therefore, many of the evaluation questions will require additional sources of data, analytical approaches, and methodologies. In the mixed methods approach described in this evaluation plan some limitations with readily obtained impact data may be overcome by using other proposed comparison methodologies, ancillary studies, key informant interviews, and site visits. These methods are described in more detail in the subsequent sections on evaluation methodologies and data sources. The type of methodology and analytical approach used to answer specific questions requested by Congress will differ depending on the rigor and feasibility of collecting existing data or the feasibility of gathering new information—as the committee intends to do during country visits and during interviews with OGAC staff, implementers, and other key stakeholders that will occur outside of country visits. This poses limitations on the evaluation approach and on the interpretation of the findings. There are indicators that are reported centrally to OGAC across the entire PEPFAR program; however, these provide only limited answers to the evaluation charge. Although data from within PEPFAR that go beyond the centrally-reported indicators may be available, a preliminary scan of sources revealed that these data will have to come from disparate sources that are not currently catalogued or coordinated. These data sources, such as recommended indicators not reported to OGAC and data collected by the major USG implementing agencies and other implementation partners, are not managed through a process that allows for easy cataloguing of what is available from whom. Therefore, accessing the data to answer many of the evaluation questions will require a significant data-mapping and data-gathering effort that adds to the resource requirements of the evaluation. Requests from the IOM will also likely impose a burden of time and resources on staff at OGAC and other implementing agencies as well as on country teams and implementing partners; this introduces a dependence on the timely efforts of many different actors who, in many cases, are already overburdened. Another possible source is data that are collected through other multilateral organizations and can be made available to the IOM. However, some of these data, as well as data from OGAC and implementing partners, are not already analyzed in a way that answers the questions posed by this evaluation in response to the congressional mandate. Therefore, to make full use of these data would require new analyses. In addition to questions requiring new analyses, some questions would require developing and using new data collection tools, which may or may not be feasible during the time period of the evaluation. Each of these approaches could enhance the quality of the evaluation, but each will also require a greater investment of resources than restricting the evaluation to existing data and analyses. Another important limitation to note is that most of the additional data sources will not be PEPFAR-wide data or population data but will instead be country-specific, program-specific, or component-specific. When data are not collected systematically across all PEPFAR countries, this will limit the ability to generalize findings to the whole of the program. There is also considerable heterogeneity in the implementation of PEPFAR across different countries and

OCR for page 17
38 STRATEGIC APPROACH TO THE EVALUATION OF U.S. GLOBAL HIV/AIDS PROGRAMS available infrastructure and capacity prior to the introduction of PEPFAR. In addition, many countries where PEPFAR has not been implemented may have implemented similar interventions to achieve the same objective through programs with support from other external or national funding sources. When this is the case, comparisons cannot evaluate the presence or absence of the intervention activities supported by PEPFAR per se, but rather the implementation and delivery strategy used by PEPFAR. As a result, the approach of comparing PEPFAR to non- PEPFAR countries would require special care in implementation, analysis, and interpretation. For this kind of comparison approach to be useful for this evaluation, it would be critical to identify control countries that can be suitably compared to PEPFAR countries. The control counties would be selected from the same geographic regions as the PEPFAR countries (e.g., sub-Saharan Africa, Asia, the Russian Federation or Eurasia, and the Caribbean). The evaluation committee’s work would begin by assembling a database of baseline country-level variables in both PEPFAR and non-PEPFAR countries that might relate to the course of the epidemic. The committee will also document investments in HIV activities from country governments and external donors. The validity of using this comparison approach to draw reliable inferences about the effects of PEPFAR will depend on whether the analyses can be adequately adjusted to make fair comparisons between PEPFAR and the candidate non-PEPFAR control countries. Adjusted analyses that statistically control for differences will be considered. The dependent variables in the adjusted analyses will be the relative or percentage changes in key indicators before and after introduction of PEPFAR. The before-after percentage changes in PEPFAR countries will be compared to non-PEPFAR countries, adjusting for differences in baseline variables and taking into account HIV activities supported from other sources. As with the comparisons among countries within PEPFAR, a major limitation is that there are important uncertainties in some of the key benchmark indicators used as the dependent variables, such as HIV prevalence and numbers of HIV-related deaths. In addition, there are a number of measures of interest for this evaluation for which data are not collected across PEPFAR and non-PEPFAR countries, which would limit the scope of this approach in addressing many of the evaluation questions drawn from the statement of task. DATA SOURCES AND ANALYSIS The extent to which the goals of this evaluation can be met depends on the availability of relevant and timely data. As described previously, most evaluation questions will require the evaluation committee to draw on data that go beyond the indicators that are reported centrally to OGAC. These data will have to come from a range of disparate sources. The availability of this data will partly depend on the feasibility of access within the timeframe of the evaluation. There will also be challenges of sampling and interpretation due to heterogeneous data sources with different data collection systems and criteria, as well as the potential for reporting bias in the responses to data requests from the committee. The approach for collecting and assessing data that could be used for the evaluation is described here and in the subsequent section on the workplan for the evaluation. Mapping of Data Sources The time and resources available for the planning phase did not allow for a complete mapping of all currently available and anticipated data sources in time for this report. In the

OCR for page 17
39 PROPOSED EVALUATION PLAN operational planning period the IOM staff, under the guidance of the planning committee, will continue an extensive data-mapping effort, expanding on the preliminary scan of data sources conducted during this strategic planning phase. The mapping will occur through document review, informant interviews, information obtained from domestic and international data requests, and qualitative methods used during three pilot country visits. The timing of this evaluation, with a final report to be delivered in 2012, dictates that the committee will be considering only data that are or will become available to the committee through 2011. This mapping will determine what data are available for each of the PEPFAR countries, providing the evaluation committee with a data matrix similar to the template that can be found in Appendix G. Some of these data sources will also be mapped for non-PEPFAR countries to inform the feasibility of the comparison approaches described earlier—these approaches would rely on the availability of data from these countries, and on the willingness and capacity of stakeholders in non-PEPFAR countries to participate in country visits and other data-gathering requests. This mapping of available data will also include an assessment of the feasibility of collecting data from each source, taking into consideration the burden that additional data requests would place on each source’s resources and staff time. In addition, this data mapping will assess whether data from each source would require new data analysis in order to answer the evaluation questions posed by the committee. The categories and some examples of available data sources that will be mapped and, if available, used for the evaluation are listed in Boxes 2a–d. These include central USG data sources, data from multilateral organizations, country-level data from both PEPFAR and other sources, and data from additional sources, which may be from single countries or multiple countries. The applicability of specific data sources to address illustrative evaluation questions in some specific programmatic areas will be discussed in the subsequent sections of this report.

OCR for page 17
40 STRATEGIC APPROACH TO THE EVALUATION OF U.S. GLOBAL HIV/AIDS PROGRAMS BOX 2a Central U.S. Government Data Sources Office of the U.S. Global AIDS Coordinator (OGAC) Reports and Planning/Guidance Documents: OGAC periodically releases reports of its activities as well as programmatic, policy, and reporting guidance for field programs. Most of the reports are requested by Congress or required under federal regulations. Guidance for field programs includes both formal guidance documents and other communications from headquarters to implementing partners and country teams. 5-year strategic plans Country Operational Plan guidance PEPFAR indicators reference guide (including the Next Generation Indicators Reference Guide) Programmatic guidance Partnership Frameworks and Partnership Framework Implementation Plans Guidance Public health evaluation guidance Reporting guidance for the annual program results (APRs)/semi-annual progress results (SPRs) PEPFAR annual reports and other reports to Congress PEPFAR operational plans Obligation and outlay reports PEPFAR State of the Program Area News to the Field Data Reported to OGAC through the Country Operational Plan Reporting System (COPRS II): As part of PEPFAR’s monitoring and evaluation (M&E) of activities, countries are required to report program data through COPRS II. Countries submit two reports to OGAC annually (APRs and SAPRs), which include data from the essential and reported PEPFAR indicators* collected from implementing partners on all technical areas. Congressional Appropriations Bills and Conference Reports: The U.S. House of Representatives and U.S. Senate Committees on Appropriations and their appropriate subcommittees have the broad responsibility over the discretionary budget for global HIV/AIDS bilateral funding and the U.S. government funding for multilateral organizations such as the Global Fund to Fight AIDS, Tuberculosis and Malaria. Office of Management and Budget (OMB): OMB Circulars are instructions or information issued by OMB to federal agencies. These are expected to have a continuing effect of 2 years or more. PEPFAR funding for HIV/AIDS is subjected to OMB Circulars. PEPFAR Implementing Agencies Data: Program monitoring, evaluation, and research data as well as other relevant information over which these agency have oversight (e.g., principally the Office of HIV/AIDS within the Global Health Bureau at the United States Agency for International Development [USAID] and the Global AIDS Program at the U.S. Centers for Disease Control and Prevention).

OCR for page 17
41 PROPOSED EVALUATION PLAN PEPFAR External Evaluations: Reports of evaluations of PEPFAR conducted by other U.S. government agencies, including the Government Accountability Office. Congressional Research Service, OMB, and the Offices of the Inspector General for the Department of State, USAID, and the Department of Health and Human Services. These include reports on topics such as program management and implementation, coordination, funding allocations and oversight, technical assistance, harmonization, and program efficiency and effectiveness.a * PEPFAR guidance classifies indicators in three ways: by degree of importance and aggregation level (i.e., essential and reported to headquarters, essential and not reported to headquarter, or recommended indicators), by reporting level (i.e., direct program or national indicators), and by standard M&E classification (i.e., output, outcome, or impact indicators). SOURCE: Compiled from U.S. Government publicly available information and PEPFAR’s website (www.pepfar.gov). a For example: CRS (2005,2007, 2008a, 2008b, 2009); DoS OIG (2008, 2009); GAO (2004, 2005, 2006, 2008, 2009).

OCR for page 17
42 STRATEGIC APPROACH TO THE EVALUATION OF U.S. GLOBAL HIV/AIDS PROGRAMS BOX 2b Multilateral Donor and Other International Data Sources Multilateral donors and international organizations play an active role in implementing global commitments on HIV/AIDS and supporting these through funding and technical assistance. Data available from multilateral donors and international organizations are reported by national governments, which are generally required to report on the progress of externally supported HIV/AIDS programs. The following are examples of these types of sources of data: Joint United Nations Programme on HIV/AIDS (UNAIDS) Global HIV statistics and other estimates (e.g., “Report on the Global AIDS Epidemic” and “AIDS Epidemic Update” reports) Frameworks and Indexes (e.g., National Composite Policy Index and Stigma Index) National AIDS spending assessments United Nations General Assembly Special Session on HIV and AIDS country progress reports Project and indicator data collected, analyzed, and reported through the Country Response Information System or the Indicator Registry United Nations Children’s Fund (UNICEF) HIV Statistics and other socio-economic statistics affecting child well-being (i.e., “State of the World’s Children” annual reports) Frameworks (e.g., Five-year global campaign on children and AIDS) Publications (e.g., the “Children and AIDS Stocktaking Reports”) Technical and policy documents Review of status of programs (addressing focus areas: preventing mother-to-child transmission of HIV; providing pediatric treatment; preventing infection among adolescents and young people; protecting and supporting children affected by HIV and AIDS) World Health Organization (WHO) Data and statistics (e.g., data on testing and counselling, mother-to-child transmission of HIV, antiretroviral therapy, and pediatric HIV) National Health Accounts Country-specific antiretroviral drug costs HIV drug resistance monitoring reports and literature WHO normative guidance and publications The International Health Regulations 2005 The Global Fund to Fight AIDS, Tuberculosis and Malaria (Global Fund) National Health Accounts Progress reports including technical support grants from PEPFAR Key performance indicators Global Fund five-year evaluation: Study area 3 reporta Global Fund evaluation country case studies

OCR for page 17
43 PROPOSED EVALUATION PLAN The World Bank Public expenditure reviews Project documents Analytic work/research Health and HIV/AIDS project evaluations Evaluation of HIV/AIDS support Bank-wideb Country assistance evaluations Organisation for Economic Co-operation and Development (OECD) HIV-related funding data Country surveys Evaluation studies Other Multilateral or International Data Sources: aids2031 reports and working papers Committee on the Rights of the Child, States reports Millennium Development Goals reports UNITAID reports European HIV/AIDS Funders Group Interagency Group for Mortality Estimation Funders Concerned about AIDS SOURCE: Compiled from the Global Fund, OECD, UNAIDS, UNICEF, WHO, and World Bank publicly available information and personal communications with individuals at these organizations. a TERG (2009). b For example: IEG World Bank (2009) and World Bank (2007).

OCR for page 17
44 STRATEGIC APPROACH TO THE EVALUATION OF U.S. GLOBAL HIV/AIDS PROGRAMS BOX 2c Country Data Sources PEPFAR Country Sources: Program data and other information generated at the country level. Country operational plan (fiscal years 2004–2011) Partnership framework and implementation plan Prime and sub-prime partner reports OGAC indicators not centrally reported* HIV Programs costing data Other communications among country teams and implementing partners National Policy Documents and other National AIDS Response Information: Relevant national policy documents, strategies, and plans of action supporting PEPFAR activities and/or beneficiaries of PEPFAR-funded activities. National AIDS Coordinating Authority’s strategy and framework Agencies or departments policy documents and plans (e.g., Ministries of Health, Finance) Country harmonization and alignment tool surveys National Health Information Systems: National health information systems play an important role in ensuring that reliable and timely health information is available for operational and strategic decision making about HIV/AIDS country programs. Census data Civil registration and vital statistics Ministries of Health and Finance data Health services records Population surveys (e.g., Multiple Indicator Cluster Survey, Demographic and Health Survey, AIDS Indicator Survey, Behavioral Surveillance Survey, and Biologic and Behavioral Surveillance Survey) Antenatal care surveillance data Facility surveys (e.g., Service Provision Assessment and Service Availability Mapping) *Additional essential/not reported to headquarters and recommended indicators beyond the 25 essential indicators reported to headquarters collected at the country and partner level. SOURCE: Compiled from publicly available information and personal communications.

OCR for page 17
45 PROPOSED EVALUATION PLAN BOX 2d Other Data Sources (Single or Multi-Country) Public Health Evaluations: Concept papers, protocols, and/or progress reports for each approved PEPFAR public health evaluation (PHE). PHEs are investigator-initiated studies intended to guide the PEPFAR program and future policy development, to provide evidence to the HIV/AIDS community on programs that work, and to identify gaps in knowledge that can be filled with timely program evaluation and research. Published Literature: Peer-reviewed journal articles, grey literature, and other reports relevant to PEPFAR’s activities. These will address country-specific or program-specific studies as well as technical areas such as operations research of HIV programs, prevention of mother-to-child transmission, sexual prevention, blood safety, injection safety, intravenous and non-intravenous drug use, male circumcision, adult and pediatric care and treatment, tuberculosis and HIV co- infection, counseling and testing, health systems strengthening, gender-related HIV issues. Existing Modeling Data Sources for Costing: PEPFAR works with countries in estimating resources needs for interventions. At the country-level, PEPFAR uses several models including the ART Cost Project, HIV/AIDS Program Sustainability Analysis Tool, and Spectrum Policy Modeling System. Existing Modeling Data Sources for HIV Infections Averted: Since the numbers of HIV infections averted due to the implementation of a specific intervention(s) cannot be measured directly, modeling approaches provide a proxy to measure impact (e.g., models that estimate the efficacy of the intervention on incident HIV infection, models that describe how HIV infections are mediated by behavior, and models that compare incidence trends with the expected or baseline HIV incidence trends). SOURCE: Compiled from publicly available information and personal communications. Analysis and Interpretation of Data The evaluation committee will guide the implementation of the evaluation and data analysis, interpret the data, and deliberate to come to consensus on the findings, conclusions, and recommendations. Primary data and secondary data that require additional analysis will be analyzed, using appropriate statistical methodologies, by the members of the evaluation committee and, with the committee’s guidance, by the study staff team, which will be augmented for the implementation of the evaluation with additional staff trained in statistical analysis and data management. In addition, the committee will use specific subcontractor services for some areas where there is specialized knowledge needed with a substantial time commitment above what the volunteer committee members can provide. For example, expert consultation will contribute to the design of the tools and methods for qualitative data collection and to oversight of the analysis of primary qualitative data collected during country visits, other structured interviews, and other qualitative methods. Expert consultation will also be used to advise and assist in designing and supervising appropriate data requests and quantitative/qualitative analysis

OCR for page 17
46 STRATEGIC APPROACH TO THE EVALUATION OF U.S. GLOBAL HIV/AIDS PROGRAMS of secondary data. The committee will oversee all analyses performed by subcontractors to ensure validity and rigor as well as integration with the overall evaluation methodology. The committee, staff, and consultants will take steps to quantify the quality and completeness of the data used for the evaluation. For the primary data collected by the committee, the methods used to assure the quality of the data will be described in full in the final report. When existing data analyses are used, the committee will review and assess the methodology and quality of the data in the original analyses. When secondary data are requested and used for new analyses conducted by the committee, a request will also be made for a description of the data management plan, and the committee will assess the procedures in place to assure the quality of the data, including, whenever possible, parameters such as reporting rates, sampling frame, and data completeness. This information will allow the evaluation committee to assess possible reporting bias and data quality and take these factors into account to inform the evaluation committee’s interpretation of the data based on the likely reliability and quality. In its final report, the committee will include an accounting of data requests as well as a summary and analysis of the data quality and completeness. This will include the number of data requests made and the extent to which these requests were completed as requested. For data requests within PEPFAR, this will also afford the committee an opportunity to assess the completeness and validity of data as a metric of program progress toward sufficient data collection capacity for M&E, a critical component of sustainability. Although the assessment of data requests and data quality will be reported in the aggregate, data request outcomes will not be linked in the report to the specific organizations that receive data requests to avoid inhibiting the reporting of data. WORKPLAN Operational Planning Phase As the culmination of the planning phase for this study, a transitional operational planning period will take place between the delivery of this report and the implementation of the evaluation itself. As described previously, the operational planning activities were intentionally structured and approved by Congress and OGAC as part of an ongoing planning phase, after delivery of the report, so that work on the evaluation could continue uninterrupted and so that the evaluation committee would not be starting de novo with respect to data availability and cataloguing, pilot-testing of instruments and methods, and development of relationships with relevant stakeholders. The results of these operational planning activities will be detailed in staff- authored planning documents for the evaluation committee as part of their background information and preparation to implement the evaluation. Activities in this period will be carried out by IOM staff and planning committee members and will be designed to further develop and refine the plan described here and to inform the implementation of the evaluation. The operational planning will focus on data mapping (sources and availability of relevant data); mapping of methods and data sources, including key indicators, to the mandated tasks and illustrative questions in order to refine and prioritize key evaluation questions and identify key indicators; developing procedures for data requests; initiating data requests; designing and initiating data quality review methods for data received; refining and testing country visit selection criteria; preparing country timelines and other background materials for PEPFAR countries, and developing country study frameworks

OCR for page 17
47 PROPOSED EVALUATION PLAN and methods for country visits. Some initial structured interviews with key informants will also take place during the operational planning period, and a final operational planning task will be continued relationship-building with relevant stakeholders such as contacts in PEPFAR countries and at implementing partner organizations. Initial discussions with OGAC staff about data availability and the preliminary data scan conducted during this planning phase revealed that much of the data that the committee will need are not available through the headquarters level. In addition, the committee learned that implementing partners, agencies, and countries do not necessarily have to share a lot of their available data with OGAC. In light of this, OGAC agreed to partner with the IOM to help facilitate, to the extent they are able, access to these data by making introductions to field, headquarters, and agency staff and by disseminating information about the purpose of the evaluation. An initial introduction was sent in a News-to-the-Field posting from OGAC on June 4, 2010, which explained the mandate for the study, the progress of the planning committee as of the posting date, the proposed data-mapping activities and pilot country visits during the operational planning phase, and that the IOM data requests, country visits, site visits, and interview would be entirely independent of the relationship of implementing partners and other country-level stakeholders with OGAC and other USG implementing agencies. It also assured that the evaluation is not a financial audit or evaluation of specific programs; that findings, examples, and comments would not be attributed without expressed permission; and that participation in the evaluation is voluntary. 14 Operational planning activities will also include the use of a qualitative research and evaluation consultant. This consultant will help to develop and refine data collection instruments and processes for country visits, in-depth studies, and other qualitative data collection; to determine what design issues, options, and qualitative methods beyond interviewing might be appropriate and feasible (e.g., content/thematic, statistical, or combination analyses; systematic triangulation; focus groups; direct observations for contextual information; town halls; photovoice); to plan logistics for field work; to test illustrative questions for refinement; to make determinations about the balance of breadth versus depth within the design options and data collection instruments; to develop audit trails to assure rigor of the fieldwork; and to train IOM staff in qualitative methods and the use of qualitative analytical software. In addition, pilot testing and refinement of field research methods and data collection instruments, with the qualitative consultant, will occur during visits to three PEPFAR countries, one bilaterally- and two multilaterally-funded. Implementation Phase The evaluation committee will produce one consensus report with its findings and recommendations. This report is targeted for delivery to Congress by Fall of 2012. The overall time line for the evaluation will be approximately 24 months. The first 18 months will be data collection and analysis, building on the activities of the operational planning phase. This will also include consultation with relevant domestic and international stakeholders, implementing partners, and others with relevant expertise. The remaining six months will include final data analysis and interpretation of findings, determination of conclusions and recommendations by consensus among the committee members, finalization of the committee’s report, an 14 Personal communication from OGAC, June 4, 2010.

OCR for page 17
48 STRATEGIC APPROACH TO THE EVALUATION OF U.S. GLOBAL HIV/AIDS PROGRAMS institutionally-overseen peer-report review, report production, and briefings for the sponsor as requested. Over the course of the evaluation, the full committee will meet at least four times in person, with participation of the subcontractors and consultants. Additional virtual meetings will be conducted as needed using videoconferencing, teleconferencing, and web-based conferencing tools. In addition, working groups within the committee focused on specific content areas will hold in-person and virtual meetings, as needed, for ongoing deliberations as well as data analysis and interpretation. These committee activities will be augmented by ongoing communications, by telephone and electronic mail among the committee members, staff, and subcontractors and consultants. A summary schematic of the proposed work plan and timeline for the evaluation can be found in Appendix F. Adjustments may be needed to the timeline and work plan due to any delay in the start time of the evaluation phase or to uncontrollable external shocks such as man-made or natural disasters (e.g., Haitian earthquake), political instability that could jeopardize the safety of members in countries that are identified for committee visits, or unforeseen scheduling problems for traveling (e.g., the Icelandic volcano eruption in 2010).