Monitoring frameworks are likely to need to use a number of designs, each combining different methods and tools. For instance:
- A 'light touch' approach based on key informant interviews and the recording of observations by programme staff might be used to check that an intervention was on course.
- An experimental or statistical approach might be used to evaluate its impact on incomes or employment, in order to assess the merits of scaling it up.
- Review of secondary data (e.g. market statistics and press reports) might be combined with a programme of key informant interviews in order to understand what is happening in the wider market system.
The designs will need to be specified according to the context and the specific issues to be investigated.
Research methods for systemic change programmes aims to describe the toolkit of methods which is useful for systemic change programmes. It briefly covers what the method is, where it should be deployed, and some strengths and weaknesses.
Techniques for understanding market systems change
Different programmes have experimented with a range of techniques to understand how the market systems they are working in are changing, and the kinds of impact that their interventions are having. Three examples include:
- Social network analysis
Social network analysis assesses social relations, interactions and connections between people, organisations and other networks using quantitative data obtained through surveys. The method is not restricted to social networks in a narrow sense but can be used to analyse any network of connections. The networks are visualised in network maps with nodes (the elements of the network) and ties or links (the connections). In market systems approaches, social network analysis can be a powerful tool to monitor changes in the relationships among different market actors in a value chain or sector. It can also be used to map market transactions, including frequency of sales and satisfaction with a deal.
USAID Ethiopia resilience learning brief presents a baseline social network analysis conducted by USAID/Ethiopia's Resilience Learning Activity (RLA).
- Most Significant Change
Most Significant Change is a participatory technique in which stakeholders and staff collectively decide which changes are significant. In this technique, personal stories and narratives are collected from stakeholders at ground level. These stories are analysed first by the stakeholders themselves and then by staff and other stakeholders, to decide which changes are most significant, and why. Most Significant Change can provide an effective evaluation tool to help assessing what beneficiaries and other participants think the real successes of a programme are. It may therefore be particularly helpful in evaluating systemic initiatives, because of its potential to identify issues which actors from different parts of the market system agree are important. - SenseMaker
SenseMaker is a proprietary software programme that allows large numbers of personal stories and other narrative fragments to be collected and analysed in a quantitative way. It aims to avoid researchers introducing their own biases into the analysis by asking respondents to interpret or ‘signify’ their own stories by answering a carefully crafted set of questions, called a signification framework. The signification framework allows respondents to interpret shared experiences and also provides quantitative data for statistical analysis. The results of this analysis can then be presented back to respondents or other stakeholders for further discussion of the conclusions.
Methods and tools for measuring systemic change provides a shortlist of seven tools to measure systemic change, including: outcome mapping, outcome harvesting, SenseMaker, social network analysis, systemic action research, and standard measurement tools to capture key indicators.
Data collection methods/tools
Regardless of your data collection approach, there are a number of common tools. For example:
Surveys
Surveys take a sample of opinions from a wider population, normally using a carefully structured questionnaire or a looser interview topic guide. They are useful for gauging how businesses, households, individual producers or wage labourers respond to a market systems intervention, and therefore for assessing an intervention’s effects.
Supplier and customer insight surveys provides guidance for light-touch, firm-led insight surveys designed to expand understanding about supplier or customer’s preferences and capacities, and enhance the likelihood of a successful partnership.
Structured, semi-structured, unstructured and key informant interviews
Structured, unstructured and key informant interviews are typically used for qualitative surveys. They are particularly helpful in identifying a wider range of effects from an intervention, some of which the researcher may not know about in advance.
- In a structured interview, all the questions (including the exact wording and order) are determined beforehand and the interviewer does not alter them.
- Semi-structured interviews are useful for exploring issues of interest in more depth and typically include both closed and open-ended questions. As a data collection method it is also more personal and can be helpful for identifying individual reactions towards an intervention and how it is working.
- Unstructured interviews do not make use of strict interview guides, but work with an interviewer keeping the overall goal of the interview in mind while asking open-ended questions.
- Key informant interviews are semi- or unstructured interviews with people who have in-depth knowledge about the issue being studied. In a market systems programme these could be individuals in an enterprise or other market actor with detailed knowledge of an industry or sector, or other observers who understand the local political economy.
Interview guides for a market systems assessment includes interview guides for different market actors and key informants. Questions are formulated in such a way to identify the dynamics of the market system in question, including trends, farmer/producer constraints, challenges that firms face to improve/expand products, services and support for farmers/producers etc.
Focus group discussions
Focus groups are group interviews where information is gathered from participants with similar characteristics or interests. As a small number of participants are selected to take part, a focus group is not a random sample and is not a substitute for a large scale survey. It cannot, for instance, provide reliable conclusions about what a large population (e.g. customers for a particular product or service) think. A focus group can however be very useful to explore in detail issues identified by a broader survey, or for uncovering the early signs of change resulting from an intervention.
Case studies
Case studies can be used to examine a specific situation in an intervention and determine the different factors that influenced the outcome. They typically use a mix of qualitative and quantitative information. Case studies may be valuable to allow deeper investigation of specific issues identified in other monitoring activities. They can also illustrate statistical findings for audiences that respond better to narrative explanations of results, and to share learning within a programme or its partners.
Monitoring programme progress describes how the one programme structured monitoring to make it more efficient and effective and provides examples of tools that the programme used.