Unpacking Bias: Most Significant Change and Language Influence on the Selection Process
A little while back, we had an interesting exchange in our Microsoft Teams channel about the inherent bias in the story selection process in the Most Significant Change (MSC) technique and what can be done to address it. In honour of the approaching 20-year anniversary of the MSC method (yes, it’s really been around for nearly 20 years!) we thought we should shine a light on how we help clients and learners mitigate the effects of their story selection bias to make the most of the MSC method.
First things first, what is the Most Significant Change Method?
The MSC (Most Significant Change) method is an engaging and insightful approach to monitoring and understanding the impact of your projects or programs. It was co-authored by Jess Dart and Rick Davis 20 years ago and is used widely today in development work across the globe. Rather than relying solely on quantitative data, this method emphasises the power (and importance) of storytelling. You gather stories from individuals involved in or affected by your project, focusing on the most significant changes they have experienced. These stories are then reviewed and discussed to identify the most significant ones, providing rich, qualitative insights into what is truly making a difference, and how we value these different types of impact. This approach not only yields valuable information but also fosters a deeper connection and understanding among stakeholders.
The MSC technique consists of three key parts:
1. Collecting stories
In this phase, stories are gathered from individuals involved in or affected by the project or program. These stories focus on significant changes that have occurred, highlighting personal experiences and the impact of the project.
2. Selecting the most significant stories
A panel of stakeholders, which might include project staff, beneficiaries, and other relevant parties, reviews the collected stories. They discuss and select the stories that represent the most significant changes, providing a qualitative assessment of the project’s impact. (this is the part we’ll focus on in this article!)
3. Learning and feedback
The selected stories are then used to facilitate learning and improvement. The insights gained from these stories are shared with all stakeholders, informing decision-making, program adjustments, and strategic planning. This phase ensures that the stories’ lessons are actively used to enhance the program or project’s effectiveness.
Why is it important?
The MSC method is incredibly valuable because it captures rich, qualitative insights through personal stories, providing a deeper understanding of a project’s impact. It humanises evaluation, encouraging participation and ownership among stakeholders while uncovering unexpected outcomes that traditional metrics might miss. This method facilitates continuous learning and adaptation by regularly collecting feedback, supports decision-making with comprehensive insights, and builds a positive narrative that boosts morale and stakeholder engagement. Overall, it offers a powerful way to understand and enhance the true impact of our work.
Gabrielle Chamberland
(Mentor in our Complete Guide to Measurement, Evaulation, and Learning course)
Hi, a learner in our MEL course asked a tricky question today. Looking for some help!
Going through an MSC selection process as part of the course activity, one learner noticed that each MSC story used a different language/flow (because we use the storyteller’s own words). The learner asked if this may pose a problem—for example, creating bias in which stories are selected because one story flows or reads better than another or because some people may be biased towards language or a way of talking similar to theirs, etc. Would anyone have a response to that? Or has anyone experienced this and if so, how did you overcome it?
Sophie Pinwill
(Lead Mentor in our Most Significant Change course):
Hi Gabrielle, I guess the first thing to mention is that MSC is inherently biased, so it’s important to recognise that, rather than avoid the issue or hope that it won’t matter too much. On the flip side of bias is values, and MSC is all about surfacing these.
The most important thing to do is be transparent about that bias by acknowledging it and making the selection process transparent. This means recording the discussions and reasons for the story selection. While you can never eliminate bias (and don’t really want to because MSC is about surfacing values), you can mitigate to some extent by asking selection panel members to focus on the actual change described in the story, not the storyteller or the way the story is written/recorded. In practice, the success of this varies, particularly when the panel members know the storytellers (which, in some cases, is unavoidable). Guiding the selection panel to focus on the change itself is more successful when they are not familiar with the storytellers.
Other tips include telling the selection panel that there are no “right” answers. It is also helpful to ask the question Why do we think that particular change is the most significant, rather than What do you think is the best story? It’s also a good idea not to frame the selection exercise as a “competition” with a “winner” because that’s not what it is. It’s a way of surfacing instances of impact and values in order to generate engagement and discussion. It’s critical to note that MSC is not a “stand-alone” tool and should always be used in combination with other MEL methods, which helps to balance or offset the bias to gain a broader picture of impact for evaluation purposes.
Here is a short summary of Sophie’s tips to mitigate bias in the story selection process:
- Be transparent about bias by acknowledging it and making the selection process transparent.
- Record the discussions and reasons for the story selection.
- Mitigate bias by asking selection panel members to focus on the actual change described in the story, not the storyteller or the way the story is written/recorded.
- Remember that success in mitigating bias varies, particularly when panel members know the storytellers.
- Guiding the selection panel to focus on the change itself is more successful when they are not familiar with the storytellers.
- Tell the selection panel that there are no “right” answers.
- Ask the question “Why do we think that particular change is the most significant?” rather than “What do you think is the best story?”
- Do not frame the selection exercise as a “competition” with a “winner.”
- Emphasise that the exercise is about surfacing instances of impact and values to generate engagement and discussion.
Note that MSC is not a “stand-alone” tool and should always be used in combination with other MEL methods to gain a broader picture of impact for evaluation purposes.
Helping you make the most of the MSC method
Many people find the MSC Selection Process the most challenging of the three MSC parts for several reasons, one of which is the bias element. People can often feel uncomfortable “judging” other people’s stories to arrive at a decision about which one represents the most significant change. But MSC is not simply a “feel good” exercise! It’s an excellent and unique way to share, test, and challenge assumptions and values and to engage people in important conversations about what impacts are being seen and what is valued and why. Unlike other approaches where the end result is a story or group of stories that may or may not have been analysed for MEL purposes, MSC draws out these conversations through the selection process.
Join our upcoming MSC course
Are you keen to make the most of this qualitative data collection method for your change projects? We warmly invite you to join our upcoming course here. Sophie Pinwill is our lead mentor in our MSC course, joined by fellow mentor, Callisha Gregg-Rowan. To continue exploring this valuable storytelling technique right away, you can also browse our free resources, ‘Most Significant Change in Five Minutes’ or ‘The Most Significant Change User Guide’ in our free learning hub.
