This document was prepared by MEASURE Evaluation at the request of the PEPFAR OVC Technical Working Group and reflects a legislative mandate to monitor and evaluate programs funded by the U.S. President’s Emergency Plan for AIDS Relief (PEPFAR).
As part of its new monitoring, evaluation, and reporting (MER) guidance, launched a set of outcome indicators for PEPFAR OVC programs. These outcome indicators reflect internationally-accepted developmental milestones and collectively measure holistic wellbeing for children and their families over time. Indicators track the ways OVC programs gain from and contribute to the broader HIV and child protection response. These outcome indicators are designated as “level 2”, meaning that PEPFAR requires countries to collect Level 2 indicators biennially. These indicators are held in country to be used to inform program planning and review, both by country and visiting headquarters staff.
The purpose of this document is to provide U.S. government staff and others with a high-level understanding of outcomes monitoring and approaches to outcomes monitoring to enable effective procurement of data collection services for these new outcome indicators.
Two appropriate methods for outcomes monitoring are briefly described: cluster sample surveys and Lot Quality Assurance Sampling (LQAS) surveys. Like evaluation, outcomes monitoring should be carried out by a group that is independent and external to program delivery. The methodology for outcomes monitoring must be documented in a data collection protocol. Unlike evaluation, outcomes monitoring protocol may be exempt from full ethical review. However, protocols should be submitted to an ethics review board to certify and document exemption. A data collection tool and analysis guidance is provided.
A Case Study to Measure National HIV Monitoring and Evaluation System Strengthening: Côte d’Ivoire
This Côte d’Ivoire study uses interviews with stakeholders and key informants and other evidence to verify real change in routine health information system strengthening through indicator and data collection tool harmonization, development of an electronic patient record system, and improvement in data quality through the collection, collation, and reporting data cycle. The study also shows that although much has been accomplished, considerations for future monitoring and evaluation (M&E) system strengthening assessment and implementation should account for a systems thinking approach to strengthen a country-led M&E system.
Future interventions also will need to collaboratively and actively identify the work of each national agency and development partner to build partnerships that use substantive, productive feedback on what is and is not working.
Mozambique Program Assessment: Community Care for Vulnerable Children in an Integrated Vulnerable Children and Home-Based Care Program
In Mozambique, the Community Care Program (Programa de Cuidados Comunitários, or PCC in Portuguese) is a five-year project (2010-2015) that seeks to strengthen the response to HIV and AIDS, specifically support to orphans and vulnerable children (OVC) and home-based care (HBC) service provision for people living with HIV and AIDS (PLHIV), in seven focus provinces through a network of community-based organizations (CBOs). Prior to PCC, OVC support and HBC had usually been provided by different activistas (community workers) and/or different CBOs, even when the same household had both HBC clients and OVC.
PCC integrated OVC support with HBC service provision: a single activista would provide integrated support to all people living with HIV (PLHIV), OVC, and pre/post-partum women living in the household through HBC, as well as supporting households that may have only one beneficiary type (e.g., households with vulnerable children and no HBC client). Integration of services and service providers involved devising a schedule of visits based on those needing more frequent care and those needing less frequent care (“intensive” versus “maintenance” stages of care).
Stakeholders expected that this integrated approach would offer a more efficient model of service provision. The U.S. Agency for International Development (USAID) in Mozambique asked MEASURE Evaluation to assess the integrated model to better understand what integration of HBC and OVC services means for OVC beneficiaries.The assessment outlined in this report found consensus regarding the value of having one activista provide services to all clients in one household for the efficiencies it offers, including offering holistic support to households, the potential for cost savings, and the integrated activista’s ability to reach OVC that may not have been previously reached.
Dr. Azim explains the HMIS process used in collaboration with the Government of Ethiopia that led to a country-owned system.
Evaluation Design for Complex Global Initiatives: Workshop summary
Board on Global Health, Institute of Medicine and Steve Olson, editor. Washington, DC: National Academies Press, 2014.
A workshop was convened by the Institute of Medicine in January 2014 to explore recent evaluation experiences for some of the world’s largest multi-national health initiatives. This summary considers the lessons learned from how these evaluations were designed, carried out, and used. It also explores the relative benefits and limitations of different quantitative and qualitative approaches within the mixed methods designs used for these complex and costly evaluations.
Institutional Capacity Building in Nigeria Sustainability Assessment Report: Obafemi Awolowo University, Ahmadu Bello University
In 2012, MEASURE Evaluation conducted a sustainability assessment of its two monitoring and evaluation (M&E) training partners in Nigeria, Ahmadu Bello University (ABU) and Obafemi Awolowo University (OAU). The assessment evaluated the capacity at ABU and OAU to offer workshops that meet M&E workforce needs, develop other M&E products including relevant higher education coursework and certification, and generally serve as Nigeria’s key M&E training and resource institutions.
This sustainability assessment was conducted in two phases and focused on the Population and Reproductive Health Programme (PRHP) and Department of Community Health at OAU and the Department of Community Medicine at ABU. The findings show that the two universities have achieved significant progress toward institutionalizing both M&E training workshops and their own M&E academic programs.
The Future of Long-Acting and Permanent Methods of Contraception in Bangladesh: A Policy Brief
This policy brief gives a systematic review of the trend of demand for and use of long-acting and permanent methods (LAPM), such as IUDs, implants, and female or male sterilizations, and programmatic and socio-cultural factors associated with the low use of LAPM, and combines the findings from the evaluation of the Mayer Hashi project that provided interventions to increase the use of LAPM in 21 districts of Bangladesh during 2010-2013. Recommendations that will help family planning programs strategize and prioritize the future LAPM program actions in Bangladesh are provided.