Why qualitative wins: The future of human experience insights

The growth trifecta: why qualitative research is growing so fast

Tell me more. And more. And more.

We want the full story. The colors, the shapes, the sounds, and the feelings. It’s no longer enough to sift data, crunch numbers and plot charts. We want to investigate our customers’ hearts and minds, to see what lies beneath the surface and understand at a human level. We want to understand the human experience.

But how? We need a new approach to qualitative research, that sweeps away the clouds covering our understanding and uses modern tools that bring out the best in each qualitative research method. In this eBook, we look at what’s driving the exponential growth in the sector, what you can use to power up your qualitative research and most importantly examine what’s coming next for the future of qualitative research. So you can be ready to make the most of this new era.

Great growth = great opportunity

Qualitative research is an essential method for developing rich, human-centered understanding of customers, users, and other audiences. It is also growing fast.

Global revenues for qualitative grew by 37% over these two years, while other market research categories (primarily quantitative research and reporting) grew by 26%.

But industry experts estimate that advertised vacancies for UX researcher roles – a rough proxy for category growth – are increasing at an annual rate of 20%3.

Finally, qualitative research is a hotbed of technology innovation. You can see the breadth and volume of creativity in this sector in the market landscape for qualitative research technology4 (ResTech) rendering below.

As can be seen, “qualitative research“ spans an enormous variety of different software tools and products in categories such as video research, online focus groups, communities, mobile ethnography and more.

Many of these software tools are relatively highly specialized; some, such as Forsta, are present in multiple categories. But the overall picture is one of a healthy qualitative technology ecosystem with a mixture of both new startups and established players, and both niche application and broad capability.

So what factors are behind the increasing demand, revenue growth and technology innovation in qualitative research?

1

The five key growth drivers for qualitative research

• 1.1 Data overload
• 1.2 Growing mistrust of data and metrics
• 1.3 The importance of storytelling
• 1.4 The growth of the experience economy
• 1.5 The rise of empathy in design and innovation

There are many different reasons for the increasing popularity of qualitative research. In this section, we highlight five of the most important: overload, mistrust, storytelling, experience, and empathy.

1.1 Data overload

The availability of data is transforming the way organizations understand and communicate with their audiences. Digital platforms, CRM systems, programmatic advertising: these and dozens of other sources have given marketing, experience, and customer support teams new insights to support their work.

But there is a downside to all this abundant information. The sheer volume of data available can be paralyzing.

Two-thirds of CMOs are overwhelmed by data

A major survey of 300 CMOs in North American and Europe5 revealed that two-thirds feel overwhelmed by the volume of data they deal with. The same survey showed that the average number of data sources used by marketing teams has more than doubled in the space of two years.

It’s not just marketing teams who suffer from this three V’s phenomenon – increasing volume, variety, and velocity of data. Customer experience, support and sales teams are also victims, which often leads to analysis paralysis within the team and the wider organization.

Set against this data and metrics overload, qualitative research provides tangible, accessible and understandable human insights. Many organizations are increasing their investment in ”small data” as both an antidote and a complement to their “big data” sources.

1.2 Growing mistrust of data and metrics

People are overwhelmed by large volumes of data; and increasingly, they wonder whether they should believe data at all.

Digital data – particularly where it relates to programmatic advertising – is increasingly polluted and unreliable. Up to 88% of clicks on digital ads are potentially fraudulent6, and one study concluded that nearly half of the reviews on Amazon.com were fake7. By 2025, digital ad fraud could be worth $150 billion globally.

Up to 88% of clicks on digital ads are potentially fraudulent

Quantitative survey data is also susceptible to fraud. Research technology, data and service providers constantly invest in countering this threat – but in some surveys, as many as 30 to 50% of responses can be bogus: a combination of active fraud, inattentive respondents and bots or click farms that automatically answer surveys8.

If you use quantitative data sources like these for insights into brands, consumer preferences or online user behavior, you may feel increasingly skeptical about how valid your conclusions actually are.

But when it comes to insights based on qualitative research, “seeing is believing.” Interviews and observations with participants – whether in person or online – supply the tangibility, human depth, and trustworthiness that large data sets can lack.

1.3 The importance of storytelling

In recent years, an enormous number of business books about storytelling have been published. Building a Story Brand, Stories that Stick, Brand Storytelling and dozens of other titles reflect the recognition that narrative is an essential part of marketing, sales, and experience management.

Human stories serve to illustrate hard numbers and enhance the impact of data-driven evidence. From a neuroscience perspective, stories are a highly effective mechanism for triggering emotion, encoding memory, and driving action.

These are all elements that help to persuade audiences in marketing communications; engage customers in experiences; and influence colleagues and decision-makers inside organizations.

And what is one of the richest sources of human stories? Qualitative research, of course.

1.4 The growth of the experience economy

The fourth of our five factors driving growth in qualitative research is the Experience Economy.

This term was coined in the late nineties in a Harvard Business Review article that identifies experience management as the most highly differentiated and premium strategy for companies to adopt:

In the early 2020s, experience management is a core strategy for organizations as they seek to attract users, customers, and employees.

Customer Experience (CX) management and Voice of Customer (VoC) programs are firmly established in many companies. User Experience (UX) is now core to the design of services, digital interfaces and – increasingly – physical products. And Employee Experience (EX) programs and Voice of Employee (VoE) tracking are also commonplace in larger organizations.

But where do the rich, human insights come from to help inform these experiences?

More and more, the answers come from qualitative research. It is a foundational element of UX research and design, and modern experience tracking programs include qualitative ”deep dive” components that enrich the insights from survey metrics.

1.5 The rise of empathy in design and innovation

Finally, and closely related to experience management, is the rise of empathy in design, innovation, and product creation.

The design thinking framework10 increasingly underpins the innovation process for services, digital experiences, and physical products.

The design thinking framework starts with empathy: deep understanding of a target user’s context, needs and behaviors. Empathy is the very first step, and precedes all other steps: problem definition, idea generation, prototyping or testing.

And where does empathy originate? Research to understand users is an overwhelmingly qualitative activity. Both attitudinal and behavioral research methods are needed: observations to identify natural behaviors and interviews to understand users’ perceptions. But the core methodology is almost always qualitative.

So far, we have highlighted five key factors that are driving demand for qualitative research: data overload, mistrust of metrics, the growth of storytelling, the rise of experience management and the increasing importance of empathy in product and service design.

But historically, qualitative research had some significant constraints because it relied largely almost exclusively on in-person fieldwork. Despite the availability of online qualitative tools – and the rapid online migration of other research methods – many research practitioners and clients preferred the richness of face-to-face research.

This meant that planning, conducting, and analyzing results was time-consuming. Compared to other research methods such as surveys, it could appear high cost to budget-holders. And accessing target audiences in different geographic locations required significant resources. COVID lockdowns, however, forced a significant re-think for many qualitative researchers.

Online methods were suddenly the only choice – at a time when understanding consumer needs was of paramount importance. Digital qualitative tools saw exponential growth in adoption during 2020, and researchers found new and creative ways to generate human insights.

But now, in the post pandemic era, which of these digital research methods will endure?

And what will the future of qualitative research look like?

2

The modern qualitative research toolbox

• 2.1 Remote video
• 2.2 Online communities
• 2.3 Mobile diaries and journey mapping
• 2.4 Qualitative analytics
• 2.5 Human expertise

As in the offline world, online research can involve dozens of niche methods, tools, and frameworks. In this section, we explore five foundational components of modern qualitative research: remote video, online communities, mobile diaries, qualitative analytics, and human experts.

2.1 Remote video

Remote video has multiple applications for qualitative research, including interviews and focus groups, online communities, and mobile diaries and journey mapping.

Interviews and focus groups

One-to-one interviews and group discussions are the most common use cases. But this type of ‘remote’ video research has two distinct flavors.

1. Fully online: all participants are in different locations.
2. Live-streamed: a focus group or interview takes place in-person; remote video technology allows colleagues, observers, or co-moderators to watch or take part even if they are not physically present with participants

Focus groups and interviews are very familiar from the offline world of research and are effective ways to explore issues and generate ideas. The distinctive value in modern, remote video methods is in bringing human stories to life across different locations, at greater speed and lower cost than offline approaches.

Generic online meeting platforms can be suitable for remote interviews and group discussions. They are widely available, familiar to many people and low cost – or even free in some cases.

However, they will not be suitable in many situations. Specialist online platforms have advanced features such privacy participant and compliance management; hiding observers in virtual backrooms; and in-built analytics capabilities.

As we saw earlier, User Experience (UX) or User Research is a rapidly growing capability in many organizations. Remote UX testing is a common method of evaluating live and prototype websites and digital applications.

A typical test will involve setting tasks for a participant to achieve whilst navigating a website or app, and optionally responding to a series of interview questions. Tests may be moderated by an interview or unmoderated (fully automated). Video is a core element of the process: the user’s screen activity is recorded to capture navigation, clicks, and pauses; and their front-facing camera and microphone are activated to record video of their facial expressions and verbal comments as they interact.

This approach captures both attitudinal feedback (to interview questions) and behavioral feedback (task completion, unspoken responses in facial expressions) to uncover opportunities and challenges in the UX design.

2.2 Online communities

Online communities – also known as insight communities, customer advisory boards, private panels or market research online communities (MROCs) – involve recruiting participants who agree to take part in research over a certain time. Communities can theoretically include thousands of participants over several years – or, more commonly for qualitative research, dozens of people over several weeks or months.

Qualitative communities offer the ability to conduct a wide range of activities on a variety of topics with an engaged group of participants. This can always be achieved in separate projects – but there are three distinct advantages with communities:

1. Immediacy: participants are available and willing to take part at short notice
2. Cost effectiveness: an on-demand ‘pool’ of participants reduces on-going recruitment costs and project overheads3. Iteration: research topics can be explored repeatedly over time. Communities are particularly effective for co-creation activities, complex research briefs and rapidly validating ideas.

They almost always need specialist technology, as these platforms support the full range of capabilities.

These include email communications, participant database, privacy and compliance tools, incentives, reporting and a wide variety of research activity types to meet different research briefs and keep engagement: real time or asynchronous group discussions; video feedback; polls; whiteboards and brainstorming tasks; and many others.

2.3 Mobile diaries and journey mapping

Smartphones have revolutionized qualitative research. Mobile devices support both the remote video and online community applications covered earlier.

Here, we explore the mobile-specific capabilities of diaries – collecting longitudinal feedback – and the related aspect of journey mapping.

For both types of projects, participants are recruited, opt in, and download a smartphone app through which they interact over time. They complete tasks set by a researcher such as self-ethnography (documenting habits and behaviors), uploading images or videos, providing text-based feedback and other activities.

Tasks can be set manually or triggered automatically at certain times, locations or even in response to specified on-device behaviors. Participants receive ‘push’ notifications directly on their device that may prompt them to complete virtual or in-person missions: visiting a store, preparing a meal, or documenting daily habits.

Mobile diary applications are particularly helpful in capturing in-the-moment insights from retail point of purchase, in-venue experience, or other activities; and gathering insights over time from routines such as applying makeup, consuming media, or caring for children.

The same underlying technology helps to document journeys.

Journey mapping can help to reveal key insights into user, customer, or patient experiences; and to document path-to-purchase behaviors for prepare for major investments such as vacation planning, home renovations or buying a new car.

2.4 Qualitative analytics

Data data everywhere, and not a byte to eat.

Qualitative research can generate large volumes of messy data in diverse formats: transcripts, videos, images, whiteboards, concept feedback, diary submissions, long and short form text inputs … content analysis is typically painstaking, time consuming and relies heavily on experts.

So the fourth key component in our modern digital qualitative toolbox is qualitative analytics.

Automation and technology don’t replace human expertise; but it helps make life easier for qualitative researchers by conducting the “heavy lifting” and freeing them to identify insights, patterns, and connections that only human minds can find.

Examples of qualitative analytics include:

a. Text analysis

At its most basic, this involves simply counting the number of mentions of terms and presenting the results in tabular or word cloud form.

But modern qualitative platforms go well beyond this, and now include thematic analysis, rich visualisations, affinity mapping to identify relationships between concepts and some even automatically summarize large blocks of text from interview transcripts or other sources.

b. Concept feedback

Online focus groups and community tasks often involve feedback on visual stimulus or concepts. Participants may highlight or ‘pin’ positive or negative areas of an image. Concept analysis tools aggregate this ‘pinned’ feedback to create heatmap reports that quickly reveal key results.

c. Image analysis

For simple reporting, images can be clustered into groups based on descriptive tags created by participants or by a researcher. In more advanced solutions, AI-based visual analytics can identify the content of images automatically and classify according to naturally emerging or pre-defined topics.

d. Video analysis

The most common video analytics for qualitative research involve automatic transcription of audio. This allows researchers to see summaries of the most frequently mentioned terms; search for video clips using specific keywords; or select all videos that mention a certain topic or brand. Advanced tools allow users to create video ‘showreels’ by simply selecting text from transcripts; and some use the same machine learning features highlighted above to identify visual content in clips.

e. Research repositories

Research repositories are storage and knowledge management solutions. Videos, interview transcripts, images and all other project outputs can be stored in a centralized platform to help surface deeper insights over time, conduct cross-project meta-analysis and provide a source of continuous learning. These are particularly valuable for in-house research departments and insights teams.

2.5 Human expertise

In a world where tools and technology play an increasing role in the qualitative research process, we need to recognize the paramount importance of human expertise. No matter how much automation we have, expert support from experts is needed different stages of the research process.

Professional services, technical support or customer success teams help with research platform setup and maintenance.

Project management teams help with participant recruitment, multilingual projects that require simultaneous translation, or even preparing outputs to make life easier for researchers.

And, of course, experts in qualitative research design, moderation and analysis will always be needed to choose the right methods, engage participants effectively and analyze results using appropriate frameworks.

Niche experts will be needed for methodologies such as ethnography, semiotics, or cultural insights; or to help understand audiences in new categories, countries or ethnic groups.

Technology is never just technology: it’s rather a part of a human process, part of the symbiotic relationship between people and software. The modern digital toolbox may include game-changing innovations in remote video, online communities, mobile diaries, and qualitative analytics; but none of these add value without the essential skills of the people who use them.

3

The five key future trends for qualitative research

• 3.1 Observation
• 3.2 Automation
• 3.3 Integration
• 3.4 Democratization
• 3.5 Craft

So where does qualitative research go from here? In this final section, we pick five key trends that will feature heavily in the future evolution of this category. The trends are observation, automation, integration, democratization, and craft.

3.1 Observation

Asking questions – in interviews, focus groups, communities or even through mobile apps – is a long-established research technique. But these methods all suffer to some extent from bias14 and recall issues. People forget the details of experiences; they want to answer questions ‘correctly’ or in ‘pleasing’ ways; or the question ordering can encourage certain answers.

Observational research, on the other hand, helps to capture people’s natural behaviors and surface unanticipated insights. But relying on this approach alone isn’t enough.

So qualitative research will increasingly be about watching people as well as asking questions to ensure that insights are as complete as they can be. Combining both methods helps to narrow the “say-do gap” between what people say in research and what they do in practice.

Remote user testing, described above, is a good example of observational research combined with feedback. This approach is now being extended beyond digital testing into the realm of physical products: using remote video to capture usage of a test product or unboxing experiences to inform packaging design.

Newer applications of this principle are in the use of virtual reality-based research for high value or niche applications. Examples include virtual car clinics to test new automotive models digitally rather than shipping confidential prototype vehicles around the world at high cost for in-person feedback. Simulated retail environments help to understand how people behave when choosing products at point-of-sale. And complex B2B research into highly technical products or industrial designs has also been completed using virtual reality headsets.

Extending this further is research in the metaverse: understanding how children behave in environments such as Roblox™, or research to understanding behavior and emotion amongst gamers.

Qualitative research using virtual reality headsets and in the metaverse is still very niche and new. But the underlying principles of combining observation with feedback is far broader and will continue to grow.

3.2 Automation

Automation using AI (or, more accurately, machine learning) is a feature of many existing qualitative research methods explored earlier in this white paper. One example is text analytics applied to posts in online communities.

But in this section, we will focus specifically on one advanced feature of automation: large language models.

These models are based in machine learning and use advanced Natural Language Processing techniques trained on billions of inputs – typically online conversations, social media posts, transcripts, digitized books, news articles and many other sources.

The most well-known of these models are OpenAI’s GPT315 and Google’s LaMDA16. These advanced technologies are not used directly by qualitative researchers; instead, builders of qualitative research tools will increasingly use these models as inputs to software applications with a range of uses, for example:

• Chatbot interviews with greater flexibility than structured surveys
• Reporting tools that summarize the content of qualitative interviews automatically
• Avatar-based moderators for interview or focus groups capable of simulating empathy, intuition, and inductive reasoning.

Even further applications build on the language models to generate images or video content. The most well-known is DALL-E17 which uses text-based input to create images. In the future, such tools may play a role in co-creation, projective exercises, or concept development as part of a qualitative research project

3.3 Integration

Integration has many aspects, but here we consider deepening the connections between qualitative and quantitative research.

This is already happening in creative ways at a project level. Surveys are used to profile and recruit participants for qualitative communities. Video feedback question types can be embedded directly into a survey. Voice of Customer tracking programs have regular ‘deep dive’ qualitative activities to explore specific issues.

Organization design is also starting to reflect the need to combine “big data” with “small data” from qualitative research. UX researchers frequently work alongside digital analytics experts to combine qualitative observations and feedback with quantified measurement of product behaviors. Consumer insights teams increasingly feature both primary market research and data science experts.

In the future, this trend will continue with improved software features, smarter project designs and more hybrid team structures to ensure that qualitative research is better connected to larger scale data sources.

One high potential area is conversational AI, which uses the large language models covered earlier. These approaches blur the boundaries between qualitative and quantitative research.

3.4 Democratization

More people in more departments want to do their own research with users, customers, employees, suppliers, and consumers.

This trend towards greater democratization of research – supported by technology – is already underway in many organizations. PWDR – People Who Do Research – is increasingly used as a blanket term to describe the broader research community of designers, product managers, HR teams, CX teams and marketers who conduct their own qualitative research.

These ‘lay’ researchers have distinct needs and are rarely experts in qualitative methodology or analysis. They need good software tools, clear process and coaching from expert researchers.

But there’s tension in this trend. On one hand, there is a risk that teams who are not research experts may conduct flawed research or interpret results incorrectly. On the other, research teams can benefit by supporting other departments to conduct some of their own research and amplify their own effectiveness.

This debate is in its early stages. But democratization of qualitative research will only increase. The job of expert researchers is to ensure that it happens in the most effective way.

3.5 Craft

The final trend in our collection is craft. Alongside increased tech-led automation and democratization of research, there will continue to be a strong role for human experts who provide value with deep qualitative expertise.

Advanced software, large language models and integrated analytics are important features of the future; but qualitative research will remain a deeply human activity.

Consultants, planners, and strategic advisors will be needed to connect qualitative insight with action and advice to stakeholders.

Experts in different audience groups, categories and geographies will be in increasing demand as companies search for new sources of growth in increasingly competitive markets.

And practitioners of advanced methods such as ethnography, psychology or semiotics will keep their status as niche experts.

Conclusion

If you want the full story – it’s yours for the taking. We’ve seen how the roaring appetite for qualitative research is growing. That’s because of the overwhelming tsunami of data washing through organizations combined with the growing mistrust of data and metrics. Changing organizational cultures play a key role too, as professionals in all business areas demand a more human experience when it comes to customer narratives. There’s also the growth of the experience economy where companies realize that to differentiate, they must deliver exceptional experiences, across the board. And perhaps most compelling, is the rise of empathy as a core value for developing, designing, and delivering products and services.

Luckily, we are living in a new era. One where new technology bolstered by human expertise offers an unprecedented opportunity to understand each person in your audience, intimately. These technology tools are the foundation of modern qualitative research. Combined with quantitative methods and tools you can amplify your research and experience programs to uncover insights from both broad patterns and personal pictures. Your customer’s human experience is waiting for you to understand it.