27.03.2018

Impact evaluation is a ‘with versus without’ analysis: what happened with the programme (which is a factual record) compared to what would have happened in the absence of the programme (which requires a counterfactual). Most development agencies produce reports about implementation and results at the time of project closure, if not earlier. Why are these usually misleading when it comes to results? These reports typically rely solely on information and monitoring data provided by the programme, and thus quite frequently fall victim to the before-after fallacy. Consider measuring an outcome both before the programme starts and after it has been implemented for a while.

Typically, if there is an improvement, the programme manager considers the intervention a success. But over the period of any programme, many other factors come into play, not least of which all the other programmes that are being implemented in the same country. Without a valid counterfactual, there is no way of knowing whether the improvement can be attributed to the programme’s activities or may have happened in spite of these.