“Oh Great, No Chips Left!” How AI is Transforming Qualitative Analysis: Navigating the Challenges of Ambiguity and Context
By: Ethel Karskens and Mathilde Wennevold
Clear Horizon’s Data and Insights Lead Ethel Karskens held an energetic and captivating presentation at the 2024 AES Conference where she took a deep dive into her current (but ongoing) hyper fixation: Artificial Intelligence. In her presentation, she talked about context, data quality, the road ahead, and… potato chips? We caught up with Ethel for a walk and talk in the sun, and this article is the write-up. If you want to hear Ethel’s takeaways directly, make sure you keep an eye out for snippets of the chat on our LinkedIn page over the next week!
In recent years, AI has made remarkable strides in analysing large sets of qualitative data, providing exciting new opportunities for researchers, evaluators, and other professionals who work with complex information. While AI’s potential in qualitative analysis is undeniable, it comes with unique challenges, particularly when handling ambiguity and context.
One illustrative example Ethel brought up in her presentation involved something as simple as potato chips. Imagine Ethel coming home to find no potato chips left in the bowl. From her body language, it’s clear that she’s frustrated, and an AI can easily determine that this is a negative sentiment. However, things become more complicated when the story introduces sarcasm. If Ethel instead comes home and sarcastically says,
“Oh great, no chips left! Thanks, mate,”
the AI might misinterpret this statement, thinking Ethel is happy about the situation. The subtlety and complexity of sarcasm are often lost on AI, which is why context becomes so crucial in ensuring the accuracy of qualitative analysis.
The Importance of Contextualising Data
One of the most important lessons for anyone working with AI in qualitative analysis is the need to properly contextualise the data. AI can be a powerful tool, but it still requires thoughtful preparation of inputs to deliver useful and accurate insights. When dealing with transcripts, for instance, it’s important to clean up the data, check for inconsistencies, and choose the right model for the specific task. Different AI models excel in different areas, and being precise about what you need from the AI is key to ensuring high-quality outputs.
This process doesn’t stop at input preparation. Ensuring quality through rigorous QA (quality assurance) checks is another essential step. Since AI has been known to “hallucinate”—producing information that wasn’t present in the original transcript—it’s important to manually review its outputs. Some professionals recommend comparing outputs from two different AI models, but ultimately, human oversight is indispensable. No matter how advanced AI becomes, you must always verify that the information generated is accurate and relevant to the context.
How AI Might Change Evaluation Work
Looking ahead, AI holds exciting potential for transforming how we work with qualitative data. In the field of evaluation, for example, AI could play a significant role in automating the more tedious aspects of qualitative analysis, such as cleaning and preparing data. By taking over these time-consuming tasks, AI could free up evaluators to focus on what truly matters: making sense of the data and advising on its implications.
One common concern is whether AI will replace evaluators altogether, but this is unlikely. Rather than eliminating the human element, AI is more likely to enhance it, giving professionals more time to concentrate on the meaningful, nuanced aspects of their work. The hope is that AI will allow evaluators to spend more energy on interpreting data, helping organisations make better-informed decisions, rather than getting bogged down in the mechanics of data preparation.
Overcoming the Intimidation Factor
Despite the exciting possibilities, many evaluators are still uncertain about how to get started with AI. For those feeling daunted, it’s important to begin with a solid foundation. Start by reviewing your organisation’s governance, contracts, and policies to ensure you can safely use AI. Understand how the AI models you choose to handle data, and if necessary, undergo basic AI training to get comfortable with the tools. There are plenty of free resources available online—platforms like DataCamp or OpenAI Academy provide valuable training in AI fundamentals, including prompt engineering.
Once you’ve taken these preliminary steps, it’s time to dive in. Experiment with AI, even if it means making some mistakes along the way. Every attempt will bring you closer to understanding how to leverage AI effectively in your work. The key is to keep trying, learning from your failures, sharing insights and lessons learned with your coworkers, and refining your approach.
Final Thoughts
The future of AI in qualitative analysis and evaluation is full of potential, but it comes with both opportunities and challenges. While AI can help us automate the more tedious aspects of data work, it requires careful handling to ensure that it accurately interprets context and avoids missteps like misunderstanding sarcasm or generating false information. Evaluators and other professionals must continue to play a central role in guiding the AI process, ensuring that its outputs are not only correct but also meaningful.
Putting AI to the test
Out of curiosity, we asked ChatGPT’s DALLE to create a rendition of the scene Ethel described in her AES presentation.
DALLE impressively got the sarcasm spot on, picking up on Ethel’s frustration, despite her words. Instead, surprisingly, the aspect DALLE could not quite get right was the missing chips. No matter our efforts to point out that there should be no chips in the bowl, DALLE instead provided more and more chips until they eventually spilled out of the bowl.
So, while it’s great to be excited about the current and future uses of AI to enhance our work, it remains crucial to keep a close eye on the data input and result output.
Talk to us about AI!
At Clear Horizon, we’re excited about AI’s possibilities and have already started using it in our work. If you have any questions about how AI might benefit your organisation and work or how to use it safely, don’t hesitate to reach out to us—we’re happy to help guide you on your AI journey.
