While traditional evaluations focus on immediate outputs and outcomes during a programme's lifespan, ex-post evaluations offer a critical perspective by assessing long-term and systemic impacts after a programme has concluded.
Ex-post is Latin for ‘after the event’, and ex-post evaluations are best left for several years after a programme has concluded, if possible.
Why ex-post evaluations matter in MSD
MSD programmes operate on the principle that sustainable change requires altering the underlying dynamics of market systems leading to outcomes that endure beyond direct interventions. Ex-post evaluations offer a great way to verify whether or not this has happened by:
- Assessing sustainability: they can determine whether changes initiated by a programme have persisted and become embedded in the market system after external support is withdrawn. This involves examining if market actors continue to adopt new practices, invest in innovations and operate without reliance on donor funding.
- Measuring systemic impacts: ex-post evaluations go beyond immediate beneficiary impacts to assess the broader, systemic effects of the programme on the market system. This includes examining changes in market structure, rules and regulations, supporting functions and the overall competitiveness and inclusiveness of the market.
- Identifying unintended consequences: over time, unintended consequences – both positive and negative – may emerge. Ex-post evaluations provide an opportunity to identify these unanticipated effects and understand their implications for the market system and its stakeholders.
- Informing future programme design: lessons learned from ex-post evaluations can inform the design of future MSD programmes, helping to refine strategies, target interventions more effectively and avoid repeating past mistakes. They provide valuable insights into what works, what doesn't and why, in the context of specific market systems.
- Ensuring accountability: ex-post evaluations demonstrate accountability to donors, governments and other stakeholders by providing a comprehensive assessment of the long-term value and impact of MSD investments. They help justify the resources allocated to MSD programmes and demonstrate their contribution to sustainable development.
Do the impacts of MSD activities sustain and scale after close? presents findings from two ex-post evaluations of USAID-funded MSD activities (Senegal and Ghana) conducted three or more years after programme closure. It includes data-rich findings on sustainability, scale and benefits for target populations (e.g. yields, incomes, the scale of engagement and satisfaction with the business model).
What does evidence from two recent ex-posts tell us provides evidence about the efficacy and utility of undertaking ex-posts.
Challenges of conducting ex-post evaluations in MSD
While ex-post evaluations are a vital tool for MSD evaluation, they also present several methodological and practical challenges:
- Attribution difficulties: establishing a direct causal link between a programme's interventions and long-term outcomes can be complex. Market systems are influenced by numerous factors, and disentangling the programme's contribution from other external influences can be challenging.
- Data availability: gathering reliable and comprehensive data several years after the programme's conclusion can be difficult. Programme records may be incomplete or inaccessible, and key stakeholders may be difficult to locate or unwilling to participate.
- Resource constraints: ex-post evaluations are often resource-intensive, requiring significant time, expertise and financial investment. Securing funding for these evaluations can be challenging, particularly as they are conducted after a programme has officially ended.
- Defining systemic change: systemic change is a complex and multifaceted concept, making it difficult to define, measure and evaluate. Identifying appropriate indicators and developing robust methodologies for assessing systemic impacts is not always straightforward.
- Time horizon: long term time horizons may influence the extent and type of data collected.
Is the MSD hockey stick fable or fact? provides reflections on the evidence that MSD works, drawing on ex-post studies.
Designing effective ex-post evaluations
To ensure the success of ex-post evaluations, careful planning and design are essential:
- Define clear evaluation questions: the evaluation should be guided by clear and specific questions that address the key objectives and hypotheses related to the programme's long-term and systemic impacts.
- Engage stakeholders: stakeholder engagement is crucial for ensuring the relevance and credibility of the evaluation. Involve key stakeholders, including donors, implementing partners, market actors and the target population in the design and implementation of the evaluation.
- Develop a robust sampling strategy: develop a sampling strategy that ensures the representation of diverse stakeholders and captures the variations in experiences and outcomes across different segments of the market system.
- Collect longitudinal data: if possible, collect data at multiple time points to track changes over time and assess the sustainability of outcomes. This may involve revisiting baseline data or collecting new data from a representative sample of stakeholders.
- Triangulate data sources and methods: use a combination of quantitative and qualitative methods to triangulate findings and strengthen the validity of the evaluation.
- Consider unintended consequences: actively seek to identify unintended consequences of the programme, both positive and negative. This may involve using participatory methods to solicit feedback from a wide range of stakeholders.
Uganda CPM ex-post evaluation retrospectively evaluates the spillover and sustained impact of the CPM Activity after it ended in 2018.