Why MSD4E needs more ex-post evaluations

We close programmes confident that change is happening, but we rarely go back to check.

In the first blog of this series we explored the consequences of not comparing like-for-like across MSD4E interventions. The second blog looked at how incentives shape market actors’ behaviour during implementation. In this third, and final, blog we make the case for ex-post evaluations - and show how to make them more feasible.

We stop learning at the point of closure

In most MSD4E programmes learning ends when the most important part of the story begins. Monitoring wraps up as implementation closes, and we rarely return to see what happened once subsidies and technical support have been withdrawn, before employers have internalised costs, and before workers can tell us whether new positions or improved conditions actually lasted.

We hope that our programmes have left behind lasting positive outcomes. But we don’t know for sure whether employers stayed committed, whether skills providers, regulators or job-matching services continued to offer improved services, or whether target segments benefitted from more and better employment over the longer term.

Without this follow-up, temporary shifts can look like lasting change, and hopeful narratives can appear stronger than they are. The reverse can also be true, when changes that looked modest during implementation have wider ripple effects over time.

Why this matters

Weak evidence of what happens after closure undermines credibility. Job numbers mean less when we can't tell whether work proved genuinely productive and lasting, or lower quality than expected.

It also limits learning. Blog 2 showed how difficult it can be to align incentives among market actors that influence the employment system. Ex-post assessments can reveal which incentives held once the programme ended, and where behaviours didn't last under real-world pressures.

Better evidence also guides future policy and programming. Governments, donors and implementers all benefit from knowing which delivery models work without subsidy, and which don't. Accountability is weaker, too, when nobody checks the sustainability of employment outcomes, because untested claims can move forward without challenge, shaping policies and strategy in ways that might not match reality.

What ex-posts can help reveal

Ex-post evaluations are generally conducted two or more years after programme closure, traditionally by independent evaluators, though leaner self-administered models are also emerging. When done well, ex-posts can shed light on three things:

1. Did the behaviours that mattered stick? Did the market actors that changed behaviours during the programme – employers, training providers, job platforms, associations – continue to work in new ways once support ended? For employers, this might mean assessing whether commercial or reputational incentives still hold. For training providers, it may depend on funding flows, political priorities or performance metrics. The core question is whether the underlying incentives remain strong enough for improved practices to stick.
2. Did change spread, deepen or disappear? Systems change depends on whether new practices spread beyond initial partners. Ex-posts can look for signs of crowding-in, replication and policy uptake. Did competitors adopt the model? Did ministries or associations integrate it? Were reforms effectively implemented?
3. Did outcomes last for people? The most important test lies with workers themselves. Ex-posts can follow jobseekers and new hires to see whether improvements lasted. Did they stay employed, move into better roles or earn more? And did women, rural youth or low-income groups continue to benefit?

Following up with workers can be costly and complicated, but practical options exist. Phone surveys, qualitative follow-ups and administrative or platform data can all yield useful evidence. Tracer studies from implementation can be adapted and given a longer-term institutional home, and in some settings digital records now allow providers to track retention and wages without major new survey costs.

Why ex-posts are still rare

Despite the value they can bring, ex-posts are still quite rare in MSD4E. This is partly because it’s a relatively new field. But it can also be challenging when budget allocations end with programme closure. Partner and beneficiary contact data can be lost or incomplete, and finding a senior ‘sponsor’ at donor or implementer HQ to commission follow-up studies can be difficult. So post-closure learning becomes nobody’s clear responsibility.

But these barriers are solvable. One remedy is to plan ex-posts from the beginning. This means securing consent, maintaining contact data, and building follow-up into MEL frameworks. Ring-fencing a small portion of MEL budgets enables follow-ups after closure.

There are also ways to keep costs manageable if a full-blown ex-post isn’t feasible. Mixed-methods approaches, for example combining firm data with qualitative interviews or short phone surveys, can still yield useful results. Pooling funding across portfolios, sharing sampling frames, and partnering with trusted local research providers can increase efficiency. Over time, standardised ‘lean’ ex-post tools could help normalise the practice.

The ILO's ex-post studies of Road to Jobs (R2J) and the follow-on Road to Decent Jobs for All Afghans (R2DJ4ALL) in northern Afghanistan – looking at the carpet sector and livestock and dairy – show what lean approaches can achieve. Dan Nippard’s blog offers thoughtful reflections on the trade-offs involved.

Who needs to act

Funders can help shift incentives by budgeting for ex-post evaluations upfront and sending a clear message to implementers that honest, nuanced results are preferred to big but untested numbers. SDC’s recent commissioning of an ex-post evaluation for the MarketMakers programme in Bosnia & Herzegovina is a good example of this in action.

Implementers can prepare by identifying traceable partners and beneficiaries, ensuring consent and data systems are in place. Teaming up with local and international research partners who share an interest in labour market outcomes, systems change or particular sectors and geographies also helps.

The design and delivery of MSD4E interventions have made real progress. Knowing more about what lasts, what doesn't, and why is essential if policy and practice is to keep improving.

See also our:

• first blog in this series: Apples and oranges: making sense of employment outcomes
• second blog in this series: Cracking the incentives puzzle in MSD4E