Assessing system change: you may not know it but you are probably already doing it, so why not go the extra mile?

New guidance on operationalising system change assessment helps programme teams progressively collect and document system change data.

I recently visited a programme which involved a four-hour round trip to one of the implementation sites. I was accompanied by the programme intervention managers. In that commute I learnt more about the programme - what they were doing, why they were doing it, how their initial strategy was going and how they have had to adapt it - than I did from the two days that I spent reviewing all their documents in preparation for the trip. 

This is not a first. In fact, it happens a lot. So, what does this have to do with assessing system change? I suspect we are doing more than we realise to assess system change but we fail to articulate it and so end up reporting less than what we already know.  

In early 2023, Aly Miehlbradt, Phitcha Wanitphon and I started preparing for the Advanced Training Workshop in Results Measurement for PSD, held biannually to bring together experienced results measurement practitioners. The 2023 workshop focused explicitly on assessing system change. As we were building the course content, we struggled to find enough ‘documented’ examples of how programmes were tracking system change in the sectors they targeted. 

We found many documented examples of programmes analysing individual business models or innovations they had introduced to see if other market actors had replicated or responded to them. However, we struggled to find examples of programmes going one step further - investigating what the innovations mean in the grander scheme of things, and how they influence the bigger system the programme targets. 

For example, take a programme working with seed companies to introduce affordable, small packs of vegetable seeds and improve last-mile distribution. When it came to assessing system change, they were only trying to track if other seed companies were replicating this model. They were not systematically collecting data on how the innovations in seeds and in links to markets were influencing the vegetable system. They were also overlooking how other factors, such as availability of other inputs, land, climate change adaptation, produce quality and market information, were affecting the sector.

Does this mean that programme implementers are lacking this bigger picture overview? Rarely so. In my experience, working as a MEL consultant, I often find that the answer is out there (and yes, I am a nineties kid, I love The Matrix!), buried somewhere in the intervention managers’ heads. When asked the right questions, the intervention managers can explain - just as they did on my four-hour car ride. But these conversations are often not systematic and aren’t been documented.

So how can programme managers start systematically investigating and documenting what they already know, or are only assessing sporadically? How can they use this knowledge to confidently showcase how they may be contributing to system change, or not, for whatever reasons?

A Pragmatic Approach to Assessing System Change (2020) provides guidance on how to use a ‘helicopter lens’ as well as an ‘intervention lens’ to assess system change. The helicopter lens assessment uses that bigger picture perspective to assess what influences the main system a programme is targeting - for example, the vegetable sector - and how those factors are changing and influencing that main system. The intervention lens assessment, which is used more commonly in programmes, uses a bottom-up perspective on how interventions trigger results beyond the direct partners.

In May 2024, new guidance on Operationalising system change assessment was published, with accompanying user-friendly tools that can help programme teams to collect and document information that shows their progress in contributing to system change. I particularly like that the guidance builds on programme information often already there - and supplements it with some ‘add-ons’ to turn it into a solid framework to track progress towards system change. Here are some highlights from it: 

• It builds on a programme’s market assessment and strategy.
  Most programmes begin with market research. A good sector assessment identifies the end-market opportunities and the key building blocks that can enable market actors to tap into those opportunities (and constraints that block them from doing so). For example, for farmers to access premium markets for their produce they need quality seeds, effective fertilisers and reliable market information etc.
  
  The Strategy table template is a tool designed to easily capture an overview of the entire system a programme aims to influence, including its connected building-blocks or sub-systems. The information is derived from market research and analysis of the desired objectives.
   
• It explains how to develop a helicopter lens assessment plan which provides a framework for the regular gathering of information on the broader perspective that intervention managers often implicitly track.
  The helicopter lens assessment plan builds on the strategy table by collecting information for a more comprehensive view on how the vegetable sector is evolving. Intervention managers often discuss this information informally amongst themselves, and when talking to partners. The plan features a neat and concise way of tracking this evolving picture of a system which can feed into programme review meetings to inform decision-making. 
   
• Finally, the guidance emphasises pragmatism over getting it perfect.
  One of its key messages is to approach system change assessment incrementally – the research effort should remain manageable, and data should build over time through triangulation and checking multiple sources. This process helps to turn anecdotal information into more rigorous evidence.

So what is stopping you? 
You have the information and you have the tools to collect and document it. Why not use it to gain both recognition and insights from your work in influencing system change?