Evaluating MSD programmes presents unique challenges due to the complexity of market systems, the indirect nature of interventions and the long time horizons required for systemic change to occur.
2024 Evidence review provides a narrative synthesis of current evidence. It asks: what does the current evidence base say about the impact of programmes using the market systems approach?
Timing your evaluation
One important consideration for undertaking an evaluation of MSD programmes is deciding on when these should occur. For example the mid-term and end-of-programme offer distinct opportunities to assess progress, identify challenges and inform future strategies. Yet they differ in their scope, focus and timing.
Key differences between mid-term and end-of-programme evaluations
| Feature | Mid-term | End-of-programme |
|---|---|---|
Timing |
Roughly half-way through the programme cycle. |
At the formal conclusion of a programme. |
Purpose |
To improve programme implementation . |
To assess overall achievement, capture lessons learned, and demonstrate accountability. |
Data requirements |
Primarily process data, early outcome data and stakeholder perceptions. |
Both process and outcome/impact data, stakeholder perspectives and (if possible) longitudinal data. |
Recommendations |
Action-oriented, focusing on adjustments to improve programme implementation. |
Strategic, focusing on broader lessons learned for future programmes and policy. |
Use of impact evaluation results looks into several factors that contribute to effective evaluation use.
Evaluation designs and methods
Typically, a mix of evaluation designs and methods are required to comprehensively evaluate MSD programmes. These may include, for example:
- Contribution analysis: this approach helps to assess the programme's contribution to observed outcomes, recognising that multiple factors may have influenced the results. It involves identifying key outcomes, gathering evidence to support their occurrence, and assessing the programme's role in achieving those outcomes while also considering other contributing factors.
- Process tracing: this method involves examining the causal pathways and mechanisms through which the programme's interventions are expected to lead to long-term impacts. It involves gathering evidence to verify the links in the causal chain and assess the strength of the evidence for specific causal relationships.
- Social network analysis (SNA): SNA can help to map the relationships and interactions between different actors in the market system and assess how these networks have changed over time as a result of the programme's interventions. It can provide insights into the diffusion of innovations, the strengthening of market linkages, and the emergence of new forms of collaboration.
- Systems dynamics modelling: this approach involves developing a computer-based model of the market system to simulate the effects of different interventions and scenarios. It can help to understand the complex and dynamic interactions between different elements of the system and assess the potential for long-term and systemic impacts.
- Participatory approaches: these methods involve engaging a wide range of stakeholders (e.g. market actors, beneficiaries, government officials) in the evaluation process. Participatory techniques, such as focus group discussions and key informant interviews, can help understand diverse perspectives, validate findings and identify unexpected outcomes.
Evaluating programmes provides synopses describing the methods used for evaluation, the challenges and the lessons learned – by both the programme teams and evaluators. They also explain how to evaluate programmes that are adaptive, flexible and systemic and how evaluation works with the monitoring function of the programme.