There will be circumstances during the course of a programme when you will need to make a rigorous assessment of what an intervention is achieving. These might include:
- Where a decision to scale-up, modify or cancel an intervention needs to be made.
- Where robust evidence that an innovation piloted by the programme works is needed to persuade other market actors to adopt it.
Whether or not an intervention is proving successful often hinges on the question of causality. If an innovation is taking off in a market or exports are increasing, does this mean the programme is having a demonstrable impact, or would this have happened anyway?
Causality and attribution in market systems development sheds light on the current theoretical/academic and practitioner understanding of attribution and causality. It then suggests a typology for attribution to create a better understanding of when one method is more appropriate than another.
Approaches to establishing causality
Two main approaches to establishing causality in market systems programmes are:
- Approaches that allow impacts to be attributed to the intervention. These compare the effects of an intervention to a counterfactual situation, or what would have otherwise happened. Where appropriate methods are used carefully, these approaches make it possible to make causal claims about the intervention being the cause of an impact, and to measure how much of the impact can be linked to the intervention.
- Approaches that show whether and how an intervention has contributed to observed impacts, along with other factors. These approaches set out to make a plausible argument for causality, identifying outcomes and then tracing the mechanisms through which interventions may have influenced them, while paying careful attention to the context.
Contribution, causality, causality and context provides an overview of the approaches and methods used to assess systemic change and the resulting insights can guide adaptive management. It acknowledges the limitations of attributing outcomes to programmes alone and proposes a way to generalise effectiveness where outcomes are highly contingent on a specific contextual embedding.
Attribution-based approaches: using a counterfactual
Experimental and quasi-experimental methods compare the results for a treatment group of beneficiaries, who participate in an intervention, and a control group who do not. The approaches work on the assumption that in all other respects, the treatment and control groups have the same characteristics, and that differences in outcomes between them can therefore be attributed to the intervention. Examples include:
- Randomised Controlled Trials (RCTs): RCTS are an example of an experimental research design, in which beneficiaries are randomly assigned to either the treatment or control group. While RCTs have been used in some market systems programmes, the characteristics of these programmes means that it is often difficult or impossible to select a sample in a truly random manner.
- Quasi- and non-experimental methods: in quasi-experimental approaches, comparison groups are not selected randomly but are constructed using other characteristics to control for observed differences. So-called non-experimental approaches are based on comparing the results of an intervention with a counterfactual, but without the use of a control group. These approaches use hypothetical predictions about what would have happened in the absence of the intervention to establish the counterfactual.
Challenges with counterfactual-based designs
Market systems development programmes face a number of challenges when designing studies to establish causality using counterfactuals:
- Difficulties in establishing a counterfactual. The aim of diffusing innovations throughout a whole market system can make it difficult to establish a counterfactual with people that have not been affected in one way or another by the programme
- Self-selection bias. The aim of programmes is often to encourage forward-looking market actors to adopt a piloted innovation. Their willingness to do so makes them qualitatively different from other actors, so comparisons between treatment and control groups may not be meaningful.
- Simultaneous interventions carried out in parallel. Market systems programmes often implement parallel interventions at different levels, such that it is hard to disentangle the effects of each. For instance, trying to influence policy at a national macro level, while also intervening at the micro level in a particular sector.
- The need for adaptive implementation hinders the establishment of baselines or control groups, such as when the geographical focus of an intervention changes.
Establishing causality without a counterfactual
Several methods exist which embody a similar underlying philosophy and are typically used in 'theory-based evaluation'. These examine the causal chain of events (as set out in the theory of change LINK) that connect an intervention with observed outcomes. Evidence is then gathered and reviewed to assess whether the causal mechanisms specified in the theory of change are plausibly responsible for the result, or whether competing hypotheses provide a better explanation. The concept of contribution, rather than attribution, is at the heart of the claims these approaches make about causality. Examples include outcome harvesting and contribution analysis.
Outcome harvesting
Outcome harvesting is an approach that collects evidence of an intervention's achievements in a given period. It starts by identifying a range of outcomes associated with an intervention. Both qualitative and quantitative evidence is then collected in order to consider how far the intervention contributed to observed changes. Outcome harvesting is particularly appropriate for an 'outwards-in' approach that starts from observed changes in the market system and attempts to trace these back to programme interventions.
Outcome harvesting describes the process, carried out for Helvetas, in three countries in the Western Balkans - Albania, Bosnia and Herzegovina and Kosovo - to better understand the effects and effectiveness of three of Helvetas' most mature MSD programmes.
Contribution analysis
Contribution analysis also looks at the theory of change, and aims to build up evidence that demonstrates the contribution made by an intervention 'beyond reasonable doubt', while also establishing the relative importance of other influences. The approach draws on the idea that an intervention's theory of change can be used to infer causation by assessing whether the processes that it aims to initiate have in fact occurred. In contrast to outcome harvesting, contribution analysis takes the intervention as the point of departure, and then works its way 'outwards'.
Contribution analysis offers a step-by-step approach designed to help managers, researchers and policy makers arrive at conclusions about the contribution their programme has made (or is currently making) to particular outcomes.
Developing a narrative of transformation
A similar approach to establishing causal links between an intervention and observed effects is to undertake a detailed analysis and explanation of how the intervention has worked with market actors and within the wider context. A rich narrative can then be constructed to describe transformations in market systems. Such narratives communicate the team’s understanding of the role of a programme in such transformations.
Causality counts challenges us to not get too bogged down in the contribution vs attribution debate and instead focus on assessing what changes and why, and use that information for decision making.