Senior Researcher, Ghazi Ghazi, presenting ORB’s ‘funnel approach’ at ANDE Brazil
This fall, ORB had the pleasure of presenting aspects of our Impact Assessment for the Tony Elumelu Foundation on three different continents. We attended the Aspen Network of Development Entrepreneurs (ANDE) annual conference in Ghana, the World Association for Public Opinion Research (WAPOR) Annual Conference in Austria, and ANDE’s Metrics Conference in Brazil. Each conference boasted a wide range of industry professionals: impact investors, philanthropic foundations, measurement and evaluation experts, and research specialists. We had the opportunity to share our methods and results for measuring impact and United Nations’ Sustainable Development Goal (SDG) centric research. At the same time, we learned new and innovative ways of designing and implementing research focusing on impact investing and entrepreneurship.
First stop: Accra
Our first stop was at the ANDE Annual Conference in Accra, Ghana. At the ANDE Annual Conference, ORB and the Tony Elumelu Foundation (TEF) debuted the findings from the Entrepreneurship Programme’s Impact Assessment. We explained the study design, shared TEF’s impact story, and illustrated TEF’s overall contributions towards the Sustainable Development Goals (SDGs). In addition, we were joined by two TEF beneficiaries who provided a first-hand account of the program’s impact on their lives. Our presentation ended with a discussion on lessons learned from our experience conducting this evaluation and measuring progress towards SDGs. The most important lessons were:
- Indicators for each SDG are highly subjective. For example, SDG 8: Decent Work and Economic Growth, aims for “full and productive employment and decent work for all.” However, “decent work” can be interpreted in many different ways. Therefore, ORB collaborated with TEF to set parameters for decent work determined by hours of work per week and pay relative to country context. This allowed us to make more objective benchmarks to measure TEF’s contribution towards the goal.
- SDGs are difficult to apply on a micro scale, but it is necessary if we are to measure specific program impacts towards these goals. For example, SDG 1: No Poverty uses macro-economic indicators such as “proportion of the population living below the international poverty line.” Given the small scope of each TEF- supported start-up (TEF issues $5,000 USD to participants), we adapted the indicators to local and micro-level outputs such as increase in household income, beneficiary income versus national average, and full-time jobs provided by the business. We assume these micro-level contributions may aggregate to macro-level outputs.
- Progress on the SDGs, whether good or bad, can provide insights to your evaluation and highlights areas of program improvement and recommendations. For example, when measuring outcomes against SDG 17: Partnership for the Goals, only half of TEF’s beneficiaries reached out to alumni using TEF’s network. This knowledge gives TEF the opportunity to strengthen its alumni network programming.
It is essential to apply these lessons to future impact measurement and management (IMM) frameworks in order to accurately report organization-level progress and contribution towards the SDGs.
Our time in Accra allowed us to have discussions with a wide range of stakeholders in the development entrepreneurship space, including investors, program implementers, and industry experts from many different countries and regions.
View from Austria
Next stop: Salzburg
In September, the World Association for Public Opinion Research (WAPOR) held their 76th Annual Conference in Salzburg, Austria—birthplace of Mozart. Our presentation at WAPOR focused on the utilization of survey methods and operationalization for measuring progress towards SDGs, outlined below. Our main focus of this presentation was on SDG 1 (No Poverty), SDG 8 (Decent Work and Economic Growth), and SDG 11 (Sustainable Cities and Communities). Furthermore, our presentation focused on operationalization of SDG targets and indicators to meet program evaluation standards. To do this, we take the SDG indicators and turn them into relevant survey questions. We then gather responses to these questions using quantitative surveys. Finally, we analyze the responses to measure progress towards the original indicators. Figure 1 illustrates this process using our evaluation of the TEF Programme regarding SDG 8:
Figure 1: ORB’s illustrative process to operationalized SDG 8
This process allows ORB researchers to directly tie all parts of our research to the SDGs we intend to study.
Additionally, attending WAPOR not only offered ORB a chance to present and discuss our research with our colleagues in the survey and public opinion space, but also the opportunity to learn from leading experts through a variety of breakout sessions and methodology workshops. For example, our team attended workshops on using The Survey Quality Predictor (SQP), an open-source AI tool that predicts the quality of survey questions. These workshops allowed ORB researchers to learn new tools and methods to strengthen future research.
Our time in Salzburg also allowed us to engage with some of our in-country teams and local partners, fostering stronger connections with our research community. Knowledge and insights gained from this conference will allow us to strengthen our research process with new, innovative, and rigorous method for a variety of research topics.
Last stop: Rio
Our final stop took us to Rio de Janeiro, Brazil for the ANDE ‘Metrics from the Ground Up’ Conference. This conference, in contrast to the Annual Conference held the previous month in Accra, focused mainly on impact measurement and management (IMM). At the Metrics Conference, ORB presented a session which concentrated on the multi-mode data collection method we used for TEF’s evaluation, the ‘funnel approach.’
The ‘funnel approach’ uses web, phone, and in-person interview methods in stages. Potential respondents are first contacted by email, then by phone, and finally by in-person site visit where most necessary. Our approach, visualized below, has shown us we can increase survey response rates and increase coverage of respondents through multi-stage, multi-mode approaches.
Figure 2: ORB’s Funnel Approach
When looking deeper into the data and differentiating how survey participants completed the survey (whether online, on the phone, or in-person), we found beneficiaries who were harder to reach may provide stakeholders with a better understanding of different subgroups and their varied experiences. This allows program evaluators to be more equipped to make assumptions for program recommendations to stakeholders.
Homecoming: ORB moving forward
ORB is excited to incorporate what we learned at these conferences into our research processes moving forward. In addition to engaging with industry partners to gain feedback on our methodologies, we expanded our circle of industry partners pushing for sustainable global development and rigorous measurement of impact. This circle will encourage us to innovate further to build better, more targeted impact measurement and management approaches for our clients. We’re looking forward to next Conference Season to keep learning and expanding our network!
___________________________________
Cover Photo by ANDE Brasil