The ex-post mortem

In a completely dark room, even a dim light can be very helpful.

We in the MSD community would probably say that we’re not a ‘fire and forget’ crowd, but we don’t look back as often as we should. We get excited about impact ‘after the programme ends’, but do we have enough evidence to stand behind this claim?

We need to invest in – and continue to top up – the good faith that we have all placed in the ‘future impact’ sentiment over the years. We need to do far more ex-post reviews!

In this spirit, the ILO’s Sida-funded Systems Change Initiative put its money (well, more its time actually) where its mouth is, producing a ‘rough-and-ready’ ex-post review in Afghanistan – on the cheap. Indeed, programme-instigated changes in the carpet sector appear to have stood the rather complicated test of time, though others in the livestock and dairy sectors fell short of the longevity mark.

How do you know when MSD has ‘worked’?

Perhaps almost too simply, if the number of people benefiting from the new-look system is the same or higher 2-3 years after the programme ended than it was on the day it ended, could we say that MSD has ‘worked’? Well, maybe it’s a good starting point, but other considerations are certainly valid:

• Who, specifically, continues to benefit? Is it the people that the project was created to help?
• What is the magnitude of the benefit? Similar, greater, or watered-down?

What about situations where big shocks to systems have occurred? Not just the usual ‘dramatic’ shocks – a conflict, a severe climate event, a natural disaster – but perhaps the ripple effects caused by price spikes, rapid technological changes, or even political scandals. Indeed, the number of beneficiaries might be lower than previous totals, but maybe more resilient systems have staved off a complete collapse in numbers, or even supported the post-shock rebound? Isn’t this MSD ‘working’ too?

Indeed, ex-post reviewers must channel their inner investigative journalist – they need to be interested in counterfactuals, key decision points, and the alternative options available to managers (at the time). They must understand why decisions were made, not just what happened. For example, were any of these shocks foreseeable, and could something else have been done at the time? All’s to say, we surely need a more nuanced approach to making sense of the fate of the innovations instigated by past programmes and the contexts they were launched in. Looking at surface-level numbers only would be an injustice.

What kind of evidence is ‘good enough’?

To those asking: “where is the evidence?” – should the research follow a specific methodology?

No. Different people will be satisfied with different types and degrees of evidence. What do you and they want to use it for (or to claim)? Gathering some evidence that programmes using the MSD approach have produced changes that last is actually pretty straightforward… and fairly cheap.

Notwithstanding, trade-offs certainly need to be acknowledged. For instance, ex-post reviews can be done and paid for by the original implementing organisation, rather than via a contracted third party. You can rely on narratives, the direction of change, and ‘rough number’ estimates, instead of seeking precise figures. You can use a narrower range of sources and undertake fewer triangulation and verification activities. All of these decision points impact the review’s independence and accuracy, and therefore its integrity and trustworthiness.

Simply detail your method, state who was involved, and let readers decide for themselves whether the evidence is ‘good enough’ for them and for the purposes they’d like it for. At present, it is not so much whether an ex-post review is performed to a ‘gold standard’ – it’s rather more about getting into the habit of doing any kind of ex-post review at all!

Let learning be the objective

Ex-post reviews may be the only meaningful way to better understand whether or not 'doing MSD' is worth the effort.

Occasionally, MSD interventions produce extraordinary results within their own programmes’ lifetimes. But we also know that things don’t always ‘catch fire’ immediately, or perhaps not at all. The intervention portfolios of MSD programmes are always mixed bags, with equal parts long-term successes, initial successes that subsequently fall by the wayside, and a spoonful of immediate failures.

However, despite the urge to uncover these important ‘proof-related’ storylines, there is a bigger prize here – ex-post studies can be immensely helpful in supporting us to improve as a field, and to make better decisions more of the time.

Indeed, there is a great deal of learning still to be unearthed by digging into the past. Can we get closer to identifying the most critical ingredients in the ‘sustainability and scale’ recipe? Can we better understand the conditions within which the MSD approach works best, and conversely, where it might make less sense to use it? What do we regularly get wrong? What should we absolutely avoid doing?

A final call to action

To close, we applaud all ex-post work in our small universe. USAID’s recent mini ex-post series examining past efforts in Ghana and Senegal was as brave as it was revealing. We enjoyed the long-view of Katalyst's work.

Five years ago, the ILO commissioned its own metaphorical ‘archaeological dig’ of the Enter Growth programme… a decade after the curtains had been drawn.

Though not focused on programmes using an MSD approach, SDC recently funded several ex-post evaluations, and we’re aware of other implementing organisations, such as Swisscontact, who have undertaken their own in-house ex-post studies for learning purposes.

Itad issued a call for more ex-post studies in 2019, The Springfield Centre carried the torch in 2022, and the recent BEAM Exchange Evidence Review repeated the sentiment to a wider audience in 2024.

Let’s all do many, many more of these. Let them be done internally, roughly and on the cheap. But also let them be occasionally well-funded, thorough and undertaken with greater independence.

And, before signing off, if anyone would like a copy of our own short ‘how to guide’ for doing thrifty ex-posts, please do get in touch