Background
Development aid is a salient topic throughout the Western world. People recognize the importance of international aid programs but many have called for greater accountability of spending and demonstration of the impact that their money has made. This is particularly important at a time when governments are forced to make cuts to their domestic social welfare programs. The following case study presents our approach to using survey data and GIS (Geographic Information Systems) to evaluating the performance of an aid project.
The Aid Program
The Nigerian health system has been characterized by indices of poor performance with health indicators among the worst in sub-Saharan Africa. In 2008 DfID launched a six-year program called Partnerships for Transforming Health Systems in Nigeria (PATHS2) aimed at improving health delivery services for vulnerable populations, predominantly women and children in five states.
The Evaluation
ORB conducted a survey of 10,000 households and 600 health facilities (comprising interviews with facility managers and care providers). Surveys were undertaken in focal and non-focal areas to allow us to compare KPIs (Key Performance Indicators) between areas in receipt of program funding and the control areas which have not received any program funding.
The Process
ORB trained local interviewers from each of the five states to conduct the interviews. Household interviews were undertaken with GPS enabled PDAs to ensure high levels of quality control with daily monitoring of questionnaires and progress, a faster delivery of data (than that achieved with pen and paper interviews) and allowed us to map the data and provide spatial analysis.
Program Evaluation
The research provided robust data upon which the performance of the program against the KPIs could be effectively measured. Across the five states the program was shown to have fulfilled its commitment to achieving its objectives at this interim stage of the process and continued funding was secured.
Spatial Analysis
In addition to provising a robust measure of the program’s performance ORB used GIS software ORB were able to calculate the effective reach of the program facilities. For example one objective of the program was to reduce the number of homebirths through improved facility provision and outreach work in the local communities. Figure 2 shows the percentage of births at a health facility against distance form the nearest facility. The chart effectively shows that amongst mothers whose nearest facility has a program funded scheme a significantly higher prioportion give birth withen a health facility.
Figure 2 Distance and Program Effectiveness
The chart also shows that at a distance of approximately 10km from the facilities the results from those living nearest a funded and non-funded facility converge. This convergence shows that the average reach of the program is 10km at which point the impact becomes negligible. This data can be analysed by state and urban and rural areas to help determine the distribution of funded facilities required to achieve program objectives when rolling out a scheme into new areas.
GIS Mapping
In addition to a statistical analysis of KPIs between focal and non-focal facilities ORB used GIS to provide a mapping of the how the states overall perform against the KPIs (states contain both focal and non-focal areas).
Figure 1 (above) clearly shows the difference between the proportion of births given at home between the predominantly Muslim North (with red representing areas of poor performance) and the prominantly Christian South (with blue representing areas of high performance).
Focusing on the three northern states the GIS mapping allows us to identify differences in the KPI scores within the states. Figure 3 shows higher levels of performance in southern areas of Kaduna State and specific urban areas within the three states..
Interactive Geo-analytics
ORB has also developed an interative web-based program which allows clients to view the data for selected variables, zoom in to specific areas to view the results in more detail and to automtically benchmark these results with survey-wide averages. This program can be particulalrly useful when assessing the effectiveness of a program at the local level as well as helping to plan future programming. A screen shot of the data for a KPI around the city of Kaduna has been presented in Figure 4.
Figure 4 Web-based GIS Program
“The research undertaken by ORB on behalf of PATHS2 was very useful in showing the impact of the project in Nigeria. However, the insight provided by the geospatial analysis was the area that provided actionable intelligence to the PATHS 2 team. The result provided by the research was a central piece of evidence for the project evaluation, which showed that this vital program is impacting positively on the Nigerian health sector, and saving lives, thus justifying ongoing funding”
Dr Ibrahim Ozovehe Yisa
National Monitoring and Evaluation Advisor, PATHS2 Project.
For more information on ORB’s GIS mapping services please email contactus@orb-international.com.