Corporate-level impact measurement – IFAD’s experience

Assessing impact requires attribution, which refers to the ability to claim that impact on an indicator of success is the result of a particular investment. Identifying impact entails creating a counterfactual that allows comparison of what has happened as the results of an intervention and what would have happened in the absence of that intervention. As seen in the other articles in this issue, identifying impact at the project level is well understood. Experimental (randomised controlled trials) and non-experimental approaches are becoming widely used to assess impact. These approaches create a counterfactual through a combination of careful data collection and statistical methods which provide confidence that impact estimates are unbiased and thus can be attributed to the intervention.

Attributing corporate-level impact for a development institution, such as the International Fund for Agricultural Development (IFAD), is more complicated and less straightforward. Nevertheless, bodies that govern development institutions are expanding demand for impact estimates that can be attributed to the activities of these institutions. For donor countries, corporate-level impact estimate can be crucial for justifying funding to an institution since it can address taxpayers’ and parliaments’ questions about whether development assistance is effective. For this reason, governing bodies are increasingly asking for Results Measurement Frameworks (RMFs), which lay out the indicators of institutional success and include attributable impact indicators.

The aggregation challenge

Along with the issue of attribution, an additional consideration for corporate-level impact is aggregation. Corporate impact measurement requires having indicators that can be aggregated across a range of interventions as well as a means to add up the overall impacts across those interventions. If every project had the same objective along with the same indicators of success and if every project had an impact evaluation, aggregation would be straightforward; measures of impact could simply be added up. But projects vary in their objectives based on local development needs and country priorities and impact evaluations are costly and cannot be undertaken for every project.

To address the need for attribution and aggregation, IFAD has developed a system for measuring corporate results (see upper Figure above). Based on a theory of change, every IFAD project invests in inputs that are expected to lead to outputs. Provided that beneficiaries of project funding behave in an anticipated manner, this should lead to anticipated outcomes and ultimately impact. For each project, a logical framework (log-frame) is developed with indicators for inputs, outputs, outcomes and impacts. For inputs, outputs and some outcomes attribution is not an issue since the institution knows where its funds go and what their immediate effect is. Data on these indicators is collected based on a monitoring and evaluation (M&E) plan that is developed at the initiation of the project. Since projects necessarily differ because they address distinct development problems in differing contexts, these indicators vary by project. This presents the challenge for aggregation to the corporate level since it is not possible to aggregate different indicators. But there are sufficient similarities among IFAD projects to allow for similar indicators – what we refer to as Core Indicators. These are mandatory when relevant for all projects and can be aggregated for corporate results reporting.

Indicators reflect the fund’s strategic objectives

As noted, getting attribution at the impact level is more complicated and costly since it requires substantial data collection efforts on indicators for a treatment (beneficiaries) and control (counterfactual) group. It is therefore difficult to justify impact assessments for all projects. For this reason, the impact level measurement is done in 15 per cent of projects. Of course, this creates a problem in that corporate-level impact should be an estimate of total impact, not just 15 per cent of the portfolio. For this reason, IFAD has devised an aggregation “methodology”. The first step in this process is the identification of the indicators and targets to measure in order to reflect corporate success. The overarching goal of IFAD’s Strategic Framework 2016-2025 is to invest in rural people to enable them to overcome poverty and achieve food security through remunerative, sustainable and resilient livelihoods. To achieve this goal, the Fund identifies three strategic objectives: SO1 – increase rural people’s productive capacities; SO2 – increase rural people’s benefits from market participation; and SO3 – strengthen the environmental sustainability and climate resilience of rural people’s economic activities. In accordance with this strategy, the following impact indicators are used in the RMF and defined in such a way that they can be aggregated across projects:

1. Number of people experiencing economic mobility (goal)
2. Number of people with improved production (SO1)
3. Number of people with improved market access (SO2)
4. Number of people with greater resilience (SO3)

As with Core Indicators not all indicators for an impact evaluation are RMF indicators since projects may include other context-relevant indicators.

While impact evaluations provide estimates of the average project impact, these indicators are reported in terms of a number of people receiving a benefit. The lower Figure summarises the approach to aggregating from the 15 per cent of projects with impact evaluations to corporate reporting of the number of people benefiting.

Step one in aggregating impact is to understand the portfolio of investments. This includes investment projects that are being completed during a particular period of interest, which for IFAD is the three year replenishment cycles (e.g. IFAD10: 2016–2018). This means having a sense not only of the amount of investment, but also of the types of investment. The second step is to select the 15 per cent of projects suitable for an impact evaluation. These need to be selected to represent the types of projects in the portfolio, so that any aggregation reflects the whole portfolio. The third step is to conduct the impact evaluations of individual selected projects and estimate project-specific impact. This provides the average impact of each project on project beneficiaries. The final step is to take these estimates and use a methodology, for instance meta-analysis, to obtain a global average impact estimate, and employ a set of assumptions that allow one to project such estimate to the overall portfolio. This requires a clear understanding of the portfolio and a set of assumptions regarding how average impacts translate into number of people benefiting. The details of this are beyond the scope of this article, but the logic should be clear. The basis for the estimates are impact evaluations that allow for attribution, carefully selected aggregable indicators and a clear understanding of the portfolio that allows for aggregation.

Considering the entire portfolio

While relying on a number of assumptions, this system enables reporting corporate-level impact measurement, something that has not been done by many development institutions. Although this is important for accountability and reporting to governing bodies, it allows for a systematic review of the portfolio and for learning on how to improve projects. Standard practice for impact evaluation is to select projects based on the ability to learn lessons but also reflecting the feasibility of assessing impact and the interests of researchers. A corporate-level approach ensures that an entire portfolio is considered and that lessons learned on whether an institution is doing things right as well as doing the right things can be gathered in an efficient manner.

Of course, there are numerous challenges in doing this, and the system needs to be reconsidered and improved. Along with methodological issues, it requires technical expertise and significant resources that could be used elsewhere. This means that the value of the impact evaluations needs to be sufficient to justify the costs. The entire process must draw lessons that feed into future decisions on investments and help improve approaches to development.

Paul Winters is Associate Vice President, a.i., Strategy and Knowledge Department and Director, Research and Impact Assessment Division at the International Fund for Agricultural Development (IFAD). Previously, he was a Professor of Economics at American University in Washington, DC.
Contact: p.winters@ifad.org

Alessandra Garbero is a Senior Econometrician in the Research and Impact Assessment Division at IFAD. She previously worked for the International Institute for Applied Systems Analysis, on population forecasting, demographic modelling and vulnerability to climate change.
Contact: a.garbero@ifad.org 

Further reading:

IFAD Research Series Issue 7 – Measuring IFAD's Impact
https://www.ifad.org/web/knowledge/publication/asset/39318659