In my previous article, I explored how AI can help us create functional personas more efficiently. We looked at building personas that focus on what users are trying to accomplish rather than demographic profiles that look good on posters but rarely change design decisions.

But creating personas is only half the battle. The bigger challenge is getting those insights into the hands of people who need them, at the moment they need them.

Every day, people across your organization make decisions that affect user experience. Product teams decide which features to prioritize. Marketing teams craft campaigns. Finance teams design invoicing processes. Customer support teams write response templates. All of these decisions shape how users experience your product or service.

And most of them happen without any input from actual users.

The Problem With How We Share User Research

You do the research. You create the personas. You write the reports. You give the presentations. You even make fancy infographics. And then what happens?

The research sits in a shared drive somewhere, slowly gathering digital dust. The personas get referenced in kickoff meetings and then forgotten. The reports get skimmed once and never opened again.

When a product manager is deciding whether to add a new feature, they probably do not dig through last year’s research repository. When the finance team is redesigning the invoice email, they almost certainly do not consult the user personas. They make their best guess and move on.

This is not a criticism of those teams. They are busy. They have deadlines. And honestly, even if they wanted to consult the research, they probably would not know where to find it or how to interpret it for their specific question.

The knowledge stays locked inside the heads of the UX team, who cannot possibly be present for every decision being made across the organization.

What If Users Could Actually Speak?

What if, instead of creating static documents that people need to find and interpret, we could give stakeholders a way to consult all of your user personas at once?

Imagine a marketing manager working on a new campaign. Instead of trying to remember what the personas said about messaging preferences, they could simply ask: “I’m thinking about leading with a discount offer in this email. What would our users think?”

And the AI, drawing on all your research data and personas, could respond with a consolidated view: how each persona would likely react, where they agree, where they differ, and a set of recommendations based on their collective perspectives. One question, synthesized insight across your entire user base.

This is not science fiction. With AI, we can build exactly this kind of system. We can take all of that scattered research (the surveys, the interviews, the support tickets, the analytics, the personas themselves) and turn it into an interactive resource that anyone can query for multi-perspective feedback.

Building the User Research Repository

The foundation of this approach is a centralized repository of everything you know about your users. Think of it as a single source of truth that AI can access and draw from.

If you have been doing user research for any length of time, you probably have more data than you realize. It is just scattered across different tools and formats:

• Survey results sitting in your survey platform,
• Interview transcripts in Google Docs,
• Customer support tickets in your helpdesk system,
• Analytics data in various dashboards,
• Social media mentions and reviews,
• Old personas from previous projects,
• Usability test recordings and notes.

The first step is gathering all of this into one place. It does not need to be perfectly organized. AI is remarkably good at making sense of messy inputs.

If you are starting from scratch and do not have much existing research, you can use AI deep research tools to establish a baseline.

These tools can scan the web for discussions about your product category, competitor reviews, and common questions people ask. This gives you something to work with while you build out your primary research.

Creating Interactive Personas

Once you have your repository, the next step is creating personas that the AI can consult on behalf of stakeholders. This builds directly on the functional persona approach I outlined in my previous article, with one key difference: these personas become lenses through which the AI analyzes questions, not just reference documents.

The process works like this:

1. Feed your research repository to an AI tool.
2. Ask it to identify distinct user segments based on goals, tasks, and friction points.
3. Have it generate detailed personas for each segment.
4. Configure the AI to consult all personas when stakeholders ask questions, providing consolidated feedback.

Here is where this approach diverges significantly from traditional personas. Because the AI is the primary consumer of these persona documents, they do not need to be scannable or fit on a single page. Traditional personas are constrained by human readability: you have to distill everything down to bullet points and key quotes that someone can absorb at a glance. But AI has no such limitation.

This means your personas can be considerably more detailed. You can include lengthy behavioral observations, contradictory data points, and nuanced context that would never survive the editing process for a traditional persona poster. The AI can hold all of this complexity and draw on it when answering questions.

You can also create different lenses or perspectives within each persona, tailored to specific business functions. Your “Weekend Warrior” persona might have a marketing lens (messaging preferences, channel habits, campaign responses), a product lens (feature priorities, usability patterns, upgrade triggers), and a support lens (common questions, frustration points, resolution preferences). When a marketing manager asks a question, the AI draws on the marketing-relevant information. When a product manager asks, it pulls from the product lens. Same persona, different depth depending on who is asking.

The personas should still include all the functional elements we discussed before: goals and tasks, questions and objections, pain points, touchpoints, and service gaps. But now these elements become the basis for how the AI evaluates questions from each persona’s perspective, synthesizing their views into actionable recommendations.

Implementation Options

You can set this up with varying levels of sophistication depending on your resources and needs.

The Simple Approach

Most AI platforms now offer project or workspace features that let you upload reference documents. In ChatGPT, these are called Projects. Claude has a similar feature. Copilot and Gemini call them Spaces or Gems.

To get started, create a dedicated project and upload your key research documents and personas. Then write clear instructions telling the AI to consult all personas when responding to questions. Something like:

You are helping stakeholders understand our users. When asked questions, consult all of the user personas in this project and provide: (1) a brief summary of how each persona would likely respond, (2) an overview highlighting where they agree and where they differ, and (3) recommendations based on their collective perspectives. Draw on all the research documents to inform your analysis. If the research does not fully cover a topic, search social platforms like Reddit, Twitter, and relevant forums to see how people matching these personas discuss similar issues. If you are still unsure about something, say so honestly and suggest what additional research might help.

This approach has some limitations. There are caps on how many files you can upload, so you might need to prioritize your most important research or consolidate your personas into a single comprehensive document.

The More Sophisticated Approach

For larger organizations or more ongoing use, a tool like Notion offers advantages because it can hold your entire research repository and has AI capabilities built in. You can create databases for different types of research, link them together, and then use the AI to query across everything.

The benefit here is that the AI has access to much more context. When a stakeholder asks a question, it can draw on surveys, support tickets, interview transcripts, and analytics data all at once. This makes for richer, more nuanced responses.

What This Does Not Replace

I should be clear about the limitations.

Virtual personas are not a substitute for talking to real users. They are a way to make existing research more accessible and actionable.

“

There are several scenarios where you still need primary research:

• When launching something genuinely new that your existing research does not cover;
• When you need to validate specific designs or prototypes;
• When your repository data is getting stale;
• When stakeholders need to hear directly from real humans to build empathy.

In fact, you can configure the AI to recognize these situations. When someone asks a question that goes beyond what the research can answer, the AI can respond with something like: “I do not have enough information to answer that confidently. This might be a good question for a quick user interview or survey.”

And when you do conduct new research, that data feeds back into the repository. The personas evolve over time as your understanding deepens. This is much better than the traditional approach, where personas get created once and then slowly drift out of date.

The Organizational Shift

If this approach catches on in your organization, something interesting happens.

The UX team’s role shifts from being the gatekeepers of user knowledge to being the curators and maintainers of the repository.

“

Instead of spending time creating reports that may or may not get read, you spend time ensuring the repository stays current and that the AI is configured to give helpful responses.

Research communication changes from push (presentations, reports, emails) to pull (stakeholders asking questions when they need answers). User-centered thinking becomes distributed across the organization rather than concentrated in one team.

This does not make UX researchers less valuable. If anything, it makes them more valuable because their work now has a wider reach and greater impact. But it does change the nature of the work.

Getting Started

If you want to try this approach, start small. If you need a primer on functional personas before diving in, I have written a detailed guide to creating them. Pick one project or team and set up a simple implementation using ChatGPT Projects or a similar tool. Gather whatever research you have (even if it feels incomplete), create one or two personas, and see how stakeholders respond.

Pay attention to what questions they ask. These will tell you where your research has gaps and what additional data would be most valuable.

As you refine the approach, you can expand to more teams and more sophisticated tooling. But the core principle stays the same: take all that scattered user knowledge and give it a voice that anyone in your organization can hear.

In my previous article, I argued that we should move from demographic personas to functional personas that focus on what users are trying to do. Now I am suggesting we take the next step: from static personas to interactive ones that can actually participate in the conversations where decisions get made.

Because every day, across your organization, people are making decisions that affect your users. And your users deserve a seat at the table, even if it is a virtual one.

Further Reading On SmashingMag

• “A Closer Look At Personas: What They Are And How They Work | 1”, Shlomo Goltz
• “How To Improve Your Design Process With Data-Based Personas”, Tim Noetzel
• “How To Make Your UX Research Hard To Ignore”, Vitaly Friedman
• “How To Build Strong Customer Relationships For User Research”, Renaissance Rachel

(yk)

In the early days of my career, I believed that nothing wins an argument more effectively than strong and unbiased research. Surely facts speak for themselves, I thought.

If I just get enough data, just enough evidence, just enough clarity on where users struggle — well, once I have it all and I present it all, it alone will surely change people’s minds, hearts, and beliefs. And, most importantly, it will help everyone see, understand, and perhaps even appreciate and commit to what needs to be done.

Well, it’s not quite like that. In fact, the stronger and louder the data, the more likely it is to be questioned. And there is a good reason for that, which is often left between the lines.

Research Amplifies Internal Flaws

Throughout the years, I’ve often seen data speaking volumes about where the business is failing, where customers are struggling, where the team is faltering — and where an urgent turnaround is necessary. It was right there, in plain sight: clear, loud, and obvious.

But because it’s so clear, it reflects back, often amplifying all the sharp edges and all the cut corners in all the wrong places. It reflects internal flaws, wrong assumptions, and failing projects — some of them signed off years ago, with secured budgets, big promotions, and approved headcounts. Questioning them means questioning authority, and often it’s a tough path to take.

As it turns out, strong data is very, very good at raising uncomfortable truths that most companies don’t really want to acknowledge. That’s why, at times, research is deemed “unnecessary,” or why we don’t get access to users, or why loud voices always win big arguments.

So even if data is presented with a lot of eagerness, gravity, and passion in that big meeting, it will get questioned, doubted, and explained away. Not because of its flaws, but because of hope, reluctance to change, and layers of internal politics.

This shows up most vividly in situations when someone raises concerns about the validity and accuracy of research. Frankly, it’s not that somebody is wrong and somebody is right. Both parties just happen to be right in a different way.

What To Do When Data Disagrees

We’ve all heard that data always tells a story. However, it’s never just a single story. People are complex, and pointing out a specific truth about them just by looking at numbers is rarely enough.

When data disagrees, it doesn’t mean that either is wrong. It’s just that different perspectives reveal different parts of a whole story that isn’t completed yet.

In digital products, most stories have 2 sides:

• Quantitative data ← What/When: behavior patterns at scale.
• Qualitative data ← Why/How: user needs and motivations.
• ↳ Quant usually comes from analytics, surveys, and experiments.
• ↳ Qual comes from tests, observations, and open-ended surveys.

Risk-averse teams overestimate the weight of big numbers in quantitative research. Users exaggerate the frequency and severity of issues that are critical for them. As Archana Shah noted, designers get carried away by users’ confident responses and often overestimate what people say and do.

And so, eventually, data coming from different teams paints a different picture. And when it happens, we need to reconcile and triangulate. With the former, we track what’s missing, omitted, or overlooked. With the latter, we cross-validate data — e.g., finding pairings of qual/quant streams of data, then clustering them together to see what’s there and what’s missing, and exploring from there.

And even with all of it in place and data conflicts resolved, we still need to do one more thing to make a strong argument: we need to tell a damn good story.

Facts Don’t Win Arguments, Stories Do

Research isn’t everything. Facts don’t win arguments — powerful stories do. But a story that starts with a spreadsheet isn’t always inspiring or effective. Perhaps it brings a problem into the spotlight, but it doesn’t lead to a resolution.

The very first thing I try to do in that big boardroom meeting is to emphasize what unites us — shared goals, principles, and commitments that are relevant to the topic at hand. Then, I show how new data confirms or confronts our commitments, with specific problems we believe we need to address.

When a question about the quality of data comes in, I need to show that it has been reconciled and triangulated already and discussed with other teams as well.

A good story has a poignant ending. People need to see an alternative future to trust and accept the data — and a clear and safe path forward to commit to it. So I always try to present options and solutions that we believe will drive change and explain our decision-making behind that.

They also need to believe that this distant future is within reach, and that they can pull it off, albeit under a tough timeline or with limited resources.

And: a good story also presents a viable, compelling, shared goal that people can rally around and commit to. Ideally, it’s something that has a direct benefit for them and their teams.

These are the ingredients of the story that I always try to keep in my mind when working on that big presentation. And in fact, data is a starting point, but it does need a story wrapped around it to be effective.

Wrapping Up

There is nothing more disappointing than finding a real problem that real people struggle with and facing the harsh reality of research not being trusted or valued.

We’ve all been there before. The best thing you can do is to be prepared: have strong data to back you up, include both quantitative and qualitative research — preferably with video clips from real customers — but also paint a viable future which seems within reach.

And sometimes nothing changes until something breaks. And at times, there isn’t much you can do about it unless you are prepared when it happens.

“Data doesn’t change minds, and facts don’t settle fights. Having answers isn’t the same as learning, and it for sure isn’t the same as making evidence-based decisions.”

— Erika Hall

Meet “How To Measure UX And Design Impact”

You can find more details on UX Research in Measure UX & Design Impact (8h), a practical guide for designers and UX leads to measure and show your UX impact on business. Use the code 🎟 IMPACT to save 20% off today. Jump to the details.

Useful Resources

• “How to Present Research So Stakeholders Sit Up and Take Action”, by Nikki Anderson
• “What To Do When Data Disagrees”, by Subhasree Chatterjee, Archana Shah, Sanket Shukl, and Jason Bressler
• “Mixed-Method UX Research”, by Raschin Fatemi
• “A Step-by-Step Framework For Mixed-Method Research”, by Jeremy Williams
• “The Ultimate Guide To Mixed Methods”, by Ben Wiedmaier
• Survey Design Cheatsheet, by yours truly
• Useful Calculators For UX Research, by yours truly
• Beyond Measure, by Erika Hall

Useful Books

• Just Enough Research, by Erika Hall
• Designing Surveys That Work, by Caroline Jarrett
• Designing Quality Survey Questions, by Sheila B. Robinson

(yk)

New technologies and innovative concepts frequently enter the product development lifecycle, promising to revolutionize user experiences. However, even the most ingenious ideas risk failure without a fundamental grasp of user interaction with these new experiences.

Consider the plight of the Nintendo Power Glove. Despite being a commercial success (selling over 1 million units), its release in late 1989 was followed by its discontinuation less than a full year later in 1990. The two games created solely for the Power Glove sold poorly, and there was little use for the Glove with Nintendo’s already popular traditional console games.

A large part of the failure was due to audience reaction once the product (which allegedly was developed in 8 weeks) was cumbersome and unintuitive. Users found syncing the glove to the moves in specific games to be extremely frustrating, as it required a process of coding the moves into the glove’s preset move buttons and then remembering which buttons would generate which move. With the more modern success of Nintendo’s WII and other movement-based controller consoles and games, we can see the Power Glove was a concept ahead of its time.

If Power Glove’s developers wanted to conduct effective research prior to building it out, they would have needed to look beyond traditional methods, such as surveys and interviews, to understand how a user might truly interact with the Glove. How could this have been done without a functional prototype and slowing down the overall development process?

Enter the Wizard of Oz method, a potent tool for bridging the chasm between abstract concepts and tangible user understanding, as one potential option. This technique simulates a fully functional system, yet a human operator (“the Wizard”) discreetly orchestrates the experience. This allows researchers to gather authentic user reactions and insights without the prerequisite of a fully built product.

The Wizard of Oz (WOZ) method is named in tribute to the similarly named book by Frank L. Baum. In the book, the Wizard is simply a man hidden behind a curtain, manipulating the reality of those who travel the land of Oz. Dorothy, the protagonist, exposes the Wizard for what he is, essentially an illusion or a con who is deceiving those who believe him to be omnipotent. Similarly, WOZ takes technologies that may or may not currently exist and emulates them in a way that should convince a research participant they are using an existing system or tool.

WOZ enables the exploration of user needs, validation of nascent concepts, and mitigation of development risks, particularly with complex or emerging technologies.

The product team in our above example might have used this method to have users simulate the actions of wearing the glove, programming moves into the glove, and playing games without needing a fully functional system. This could have uncovered the illogical situation of asking laypeople to code their hardware to be responsive to a game, show the frustration one encounters when needing to recode the device when changing out games, and also the cumbersome layout of the controls on the physical device (even if they’d used a cardboard glove with simulated controls drawn in crayon on the appropriate locations.

Jeff Kelley credits himself (PDF) with coining the term WOZ method in 1980 to describe the research method he employed in his dissertation. However, Paula Roe credits Don Norman and Allan Munro for using the method as early as 1973 to conduct testing on an airport automated travel assistant. Regardless of who originated the method, both parties agree that it gained prominence when IBM later used it to conduct studies on a speech-to-text tool known as The Listening Typewriter (see Image below).

In this article, I’ll cover the core principles of the WOZ method, explore advanced applications taken from practical experience, and demonstrate its unique value through real-world examples, including its application to the field of agentic AI. UX practitioners can use the WOZ method as another tool to unlock user insights and craft human-centered products and experiences.

The Yellow Brick Road: Core Principles And Mechanics

The WOZ method operates on the premise that users believe they are interacting with an autonomous system while a human wizard manages the system’s responses behind the scenes. This individual, often positioned remotely (or off-screen), interprets user inputs and generates outputs that mimic the anticipated functionality of the experience.

Cast Of Characters

A successful WOZ study involves several key roles:

• The User
  The participant who engages with what they perceive as the functional system.
• The Facilitator
  The researcher who guides the user through predefined tasks and observes their behavior and reactions.
• The Wizard
  The individual manipulates the system’s behavior in real-time, providing responses to user inputs.
• The Observer (Optional)
  An additional researcher who observes the session without direct interaction, allowing for a secondary perspective on user behavior.

Setting The Stage For Believability: Leaving Kansas Behind

Creating a convincing illusion is key to the success of a WOZ study. This necessitates careful planning of the research environment and the tasks users will undertake. Consider a study evaluating a new voice command system for smart home devices. The research setup might involve a physical mock-up of a smart speaker and predefined scenarios like “Play my favorite music” or “Dim the living room lights.” The wizard, listening remotely, would then trigger the appropriate responses (e.g., playing a song, verbally confirming the lights are dimmed).

Or perhaps it is a screen-based experience testing a new AI-powered chatbot. You have users entering commands into a text box, with another member of the product team providing responses simultaneously using a tool like Figma/Figjam, Miro, Mural, or other cloud-based software that allows multiple users to collaborate simultaneously (the author has no affiliation with any of the mentioned products).

The Art Of Illusion

Maintaining the illusion of a genuine system requires the following:

• Timely and Natural Responses
  The wizard must react to user inputs with minimal delay and in a manner consistent with expected system behavior. Hesitation or unnatural phrasing can break the illusion.
• Consistent System Logic
  Responses should adhere to a predefined logic. For instance, if a user asks for the weather in a specific city, the wizard should consistently provide accurate information.
• Handling the Unexpected
  Users will inevitably deviate from planned paths. The wizard must possess the adaptability to respond plausibly to unforeseen inputs while preserving the perceived functionality.

Ethical Considerations

Transparency is crucial, even in a method that involves a degree of deception. Participants should always be debriefed after the session, with a clear explanation of the Wizard of Oz technique and the reasons for its use. Data privacy must be maintained as with any study, and participants should feel comfortable and respected throughout the process.

Distinguishing The Method

The WOZ method occupies a unique space within the UX research toolkit:

• Unlike usability testing, which evaluates existing interfaces, Wizard of Oz explores concepts before significant development.
• Distinct from A/B testing, which compares variations of a product’s design, WOZ assesses entirely new functionalities that might otherwise lack context if shown to users.
• Compared to traditional prototyping, which often involves static mockups, WOZ offers a dynamic and interactive experience, enabling observation of real-time user behavior with a simulated system.

This method proves particularly valuable when exploring truly novel interactions or complex systems where building a fully functional prototype is premature or resource-intensive. It allows researchers to answer fundamental questions about user needs and expectations before committing significant development efforts.

Let’s move beyond the foundational aspects of the WOZ method and explore some more advanced techniques and critical considerations that can elevate its effectiveness.

Time Savings: WOZ Versus Crude Prototyping

It’s a fair question to ask whether WOZ is truly a time-saver compared to even cruder prototyping methods like paper prototypes or static digital mockups.

While paper prototypes are incredibly fast to create and test for basic flow and layout, they fundamentally lack dynamic responsiveness. Static mockups offer visual fidelity but cannot simulate complex interactions or personalized outputs.

The true time-saving advantage of the WOZ emerges when testing novel, complex, or AI-driven concepts. It allows researchers to evaluate genuine user interactions and mental models in a seemingly live environment, collecting rich behavioral data that simpler prototypes cannot. This fidelity in simulating a dynamic experience, even with a human behind the curtain, often reveals critical usability or conceptual flaws far earlier and more comprehensively than purely static representations, ultimately preventing costly reworks down the development pipeline.

Additional Techniques And Considerations

While the core principle of the WOZ method is straightforward, its true power lies in nuanced application and thoughtful execution. Seasoned practitioners may leverage several advanced techniques to extract richer insights and address more complex research questions.

Iterative Wizardry

The WOZ method isn’t necessarily a one-off endeavor. Employing it in iterative cycles can yield significant benefits. Initial rounds might focus on broad concept validation and identifying fundamental user reactions. Subsequent iterations can then refine the simulated functionality based on previous findings.

For instance, after an initial study reveals user confusion with a particular interaction flow, the simulation can be adjusted, and a follow-up study can assess the impact of those changes. This iterative approach allows for a more agile and user-centered exploration of complex experiences.

Managing Complexity

Simulating complex systems can be difficult for one wizard. Breaking complex interactions into smaller, manageable steps is crucial. Consider researching a multi-step onboarding process for a new software application. Instead of one person trying to simulate the entire flow, different aspects could be handled sequentially or even by multiple team members coordinating their responses.

Clear communication protocols and well-defined responsibilities are essential in such scenarios to maintain a seamless user experience.

Measuring Success Beyond Observation

While qualitative observation is a cornerstone of the WOZ method, defining clear metrics can add a layer of rigor to the findings. These metrics should match research goals. For example, if the goal is to assess the intuitiveness of a new navigation pattern, you might track the number of times users express confusion or the time it takes them to complete specific tasks.

Combining these quantitative measures with qualitative insights provides a more comprehensive understanding of the user experience.

Integrating With Other Methods

The WOZ method isn’t an island. Its effectiveness can be amplified by integrating it with other research techniques. Preceding a WOZ study with user interviews can help establish a deeper understanding of user needs and mental models, informing the design of the simulated experience. Following a WOZ study, surveys can gather broader quantitative feedback on the concepts explored. For example, after observing users interact with a simulated AI-powered scheduling tool, a survey could gauge their overall trust and perceived usefulness of such a system.

When Not To Use WOZ

WOZ, as with all methods, has limitations. A few examples of scenarios where other methods would likely yield more reliable findings would be:

• Detailed Usability Testing
  Humans acting as wizards cannot perfectly replicate the exact experience a user will encounter. WOZ is often best in the early stages, where prototypes are rough drafts, and your team is looking for guidance on a solution that is up for consideration. Testing on a more detailed wireframe or prototype would be preferable to WOZ when you have entered the detailed design phase.
• Evaluating extremely complex systems with unpredictable outputs
  If the system’s responses are extremely varied, require sophisticated real-time calculations that exceed human capacity, or are intended to be genuinely unpredictable, a human may struggle to simulate them convincingly and consistently. This can lead to fatigue, errors, or improvisations that don’t reflect the intended system, thereby compromising the validity of the findings.

Training And Preparedness

The wizard’s skill is critical to the method’s success. Training the individual(s) who will be simulating the system is essential. This training should cover:

• Understanding the Research Goals
  The wizard needs to grasp what the research aims to uncover.
• Consistency in Responses
  Maintaining consistent behavior throughout the sessions is vital for user believability.
• Anticipating User Actions
  While improvisation is sometimes necessary, the wizard should be prepared for common user paths and potential deviations.
• Remaining Unbiased
  The wizard must avoid leading users or injecting their own opinions into the simulation.
• Handling Unexpected Inputs
  Clear protocols for dealing with unforeseen user actions should be established. This might involve having a set of pre-prepared fallback responses or a mechanism for quickly consulting with the facilitator.

All of this suggests the need for practice in advance of running the actual session. We shouldn’t forget to have a number of dry runs in which we ask our colleagues or those who are willing to assist to not only participate but also think about possible responses that could stump the wizard or throw things off if the user might provide them during a live session.

I suggest having a believable prepared error statement ready to go for when a user throws a curveball. A simple response from the wizard of “I’m sorry, I am unable to perform that task at this time” might be enough to move the session forward while also capturing a potentially unexpected situation your team can address in the final product design.

Was This All A Dream? The Art Of The Debrief

The debriefing session following the WOZ interaction is an additional opportunity to gather rich qualitative data. Beyond asking “What did you think?” effective debriefing involves sharing the purpose of the study and the fact that the experience was simulated.

Researchers should then conduct psychological probing to understand the reasons behind user behavior and reactions. Asking open-ended questions like “Why did you try that?” or “What were you expecting to happen when you clicked that button?” can reveal valuable insights into user mental models and expectations.

Exploring moments of confusion, frustration, or delight in detail can uncover key areas for design improvement. Think about the potential information the Power Gloves’ development team could have uncovered if they’d asked participants what the experience of programming the glove and trying to remember what they’d programmed into which set of keys had been.

Case Studies: Real-World Applications

The value of the WOZ method becomes apparent when examining its application in real-world research scenarios. Here is an in-depth review of one scenario and a quick summary of another study involving WOZ, where this technique proved invaluable in shaping user experiences.

Unraveling Agentic AI: Understanding User Mental Models

A significant challenge in the realm of emerging technologies lies in user comprehension. This was particularly evident when our team began exploring the potential of Agentic AI for enterprise HR software.

Agentic AI refers to artificial intelligence systems that can autonomously pursue goals by making decisions, taking actions, and adapting to changing environments with minimal human intervention. Unlike generative AI that primarily responds to direct commands or generates content, Agentic AI is designed to understand user intent, independently plan and execute multi-step tasks, and learn from its interactions to improve performance over time. These systems often combine multiple AI models and can reason through complex problems. For designers, this signifies a shift towards creating experiences where AI acts more like a proactive collaborator or assistant, capable of anticipating needs and taking the initiative to help users achieve their objectives rather than solely relying on explicit user instructions for every step.

Preliminary research, including surveys and initial interviews, suggested that many HR professionals, while intrigued by the concept of AI assistance, struggled to grasp the potential functionality and practical implications of truly agentic systems — those capable of autonomous action and proactive decision-making. We saw they had no reference point for what agentic AI was, even after we attempted relevant analogies to current examples.

Building a fully functional agentic AI prototype at this exploratory stage was impractical. The underlying algorithms and integrations were complex and time-consuming to develop. Moreover, we risked building a solution based on potentially flawed assumptions about user needs and understanding. The WOZ method offered a solution.

Setup

We designed a scenario where HR employees interacted with what they believed was an intelligent AI assistant capable of autonomously handling certain tasks. The facilitator presented users with a web interface where they could request assistance with tasks like “draft a personalized onboarding plan for a new marketing hire” or “identify employees who might benefit from proactive well-being resources based on recent activity.”

Behind the scenes, a designer acted as the wizard. Based on the user’s request and the (simulated) available data, the designer would craft a response that mimicked the output of an agentic AI. For the onboarding plan, this involved assembling pre-written templates and personalizing them with details provided by the user. For the well-being resource identification, the wizard would select a plausible list of employees based on the general indicators discussed in the scenario.

Crucially, the facilitator encouraged users to interact naturally, asking follow-up questions and exploring the system’s perceived capabilities. For instance, a user might ask, “Can the system also schedule the initial team introductions?” The wizard, guided by pre-defined rules and the overall research goals, would respond accordingly, perhaps with a “Yes, I can automatically propose meeting times based on everyone’s calendars” (again, simulated).

As recommended, we debriefed participants following each session. We began with transparency, explaining the simulation and that we had another live human posting the responses to the queries based on what the participant was saying. Open-ended questions explored initial reactions and envisioned use. Task-specific probing, like “Why did you expect that?” revealed underlying assumptions. We specifically addressed trust and control (“How much trust…? What level of control…?”). To understand mental models, we asked how users thought the “AI” worked. We also solicited improvement suggestions (“What features…?”).

By focusing on the “why” behind user actions and expectations, these debriefings provided rich qualitative data that directly informed subsequent design decisions, particularly around transparency, human oversight, and prioritizing specific, high-value use cases. We also had a research participant who understood agentic AI and could provide additional insight based on that understanding.

Key Insights

This WOZ study yielded several crucial insights into user mental models of agentic AI in an HR context:

• Overestimation of Capabilities
  Some users initially attributed near-magical abilities to the “AI”, expecting it to understand highly nuanced or ambiguous requests without explicit instruction. This highlighted the need for clear communication about the system’s actual scope and limitations.
• Trust and Control
  A significant theme revolved around trust and control. Users expressed both excitement about the potential time savings and anxiety about relinquishing control over important HR processes. This indicated a need for design solutions that offered transparency into the AI’s decision-making and allowed for human oversight.
• Value in Proactive Assistance
  Users reacted positively to the AI proactively identifying potential issues (like burnout risk), but they emphasized the importance of the AI providing clear reasoning and allowing human HR professionals to review and approve any suggested actions.
• Need for Tangible Examples
  Abstract explanations of agentic AI were insufficient. Users gained a much clearer understanding through these simulated interactions with concrete tasks and outcomes.

Resulting Design Changes

Based on these findings, we made several key design decisions:

• Emphasis on Transparency
  The user interface would need to clearly show the AI’s reasoning and the data it used to make decisions.
• Human Oversight and Review
  Built-in approval workflows would be essential for critical actions, ensuring HR professionals retain control.
• Focus on Specific, High-Value Use Cases
  Instead of trying to build a general-purpose agent, we prioritized specific use cases where agentic capabilities offered clear and demonstrable benefits.
• Educational Onboarding
  The product onboarding would include clear, tangible examples of the AI’s capabilities in action.

Exploring Voice Interaction for In-Car Systems

In another project, we used the WOZ method to evaluate user interaction with a voice interface for controlling in-car functions. Our research question focused on the naturalness and efficiency of voice commands for tasks like adjusting climate control, navigating to points of interest, and managing media playback.

We set up a car cabin simulator with a microphone and speakers. The wizard, located in an adjacent room, listened to the user’s voice commands and triggered the corresponding actions (simulated through visual changes on a display and audio feedback). This allowed us to identify ambiguous commands, areas of user frustration with voice recognition (even though it was human-powered), and preferences for different phrasing and interaction styles before investing in complex speech recognition technology.

These examples illustrate the versatility and power of the method in addressing a wide range of UX research questions across diverse product types and technological complexities. By simulating functionality, we can gain invaluable insights into user behavior and expectations early in the design process, leading to more user-centered and ultimately more successful products.

The Future of Wizardry: Adapting To Emerging Technologies

The WOZ method, far from being a relic of simpler technological times, retains relevance as we navigate increasingly sophisticated and often opaque emerging technologies.

The WOZ method’s core strength, the ability to simulate complex functionality with human ingenuity, makes it uniquely suited for exploring user interactions with systems that are still in their nascent stages.

“

WOZ In The Age Of AI

Consider the burgeoning field of AI-powered experiences. Researching user interaction with generative AI, for instance, can be effectively done through WOZ. A wizard could curate and present AI-generated content (text, images, code) in response to user prompts, allowing researchers to assess user perceptions of quality, relevance, and trust without needing a fully trained and integrated AI model.

Similarly, for personalized recommendation systems, a human could simulate the recommendations based on a user’s stated preferences and observed behavior, gathering valuable feedback on the perceived accuracy and helpfulness of such suggestions before algorithmic development.

Even autonomous systems, seemingly the antithesis of human control, can benefit from WOZ studies. By simulating the autonomous behavior in specific scenarios, researchers can explore user comfort levels, identify needs for explainability, and understand how users might want to interact with or override such systems.

Virtual And Augmented Reality

Immersive environments like virtual and augmented reality present new frontiers for user experience research. WOZ can be particularly powerful here.

Imagine testing a novel gesture-based interaction in VR. A researcher tracking the user’s hand movements could trigger corresponding virtual events, allowing for rapid iteration on the intuitiveness and comfort of these interactions without the complexities of fully programmed VR controls. Similarly, in AR, a wizard could remotely trigger the appearance and behavior of virtual objects overlaid onto the real world, gathering user feedback on their placement, relevance, and integration with the physical environment.

The Human Factor Remains Central

Despite the rapid advancements in artificial intelligence and immersive technologies, the fundamental principles of human-centered design remain as relevant as ever. Technology should serve human needs and enhance human capabilities.

The WOZ method inherently focuses on understanding user reactions and behaviors and acts as a crucial anchor in ensuring that technological progress aligns with human values and expectations.

“

It allows us to inject the “human factor” into the design process of even the most advanced technologies. Doing this may help ensure these innovations are not only technically feasible but also truly usable, desirable, and beneficial.

Conclusion

The WOZ method stands as a powerful and versatile tool in the UX researcher’s toolkit. The WOZ method’s ability to bypass limitations of early-stage development and directly elicit user feedback on conceptual experiences offers invaluable advantages. We’ve explored its core mechanics and covered ways of maximizing its impact. We’ve also examined its practical application through real-world case studies, including its crucial role in understanding user interaction with nascent technologies like agentic AI.

The strategic implementation of the WOZ method provides a potent means of de-risking product development. By validating assumptions, uncovering unexpected user behaviors, and identifying potential usability challenges early on, teams can avoid costly rework and build products that truly resonate with their intended audience.

I encourage all UX practitioners, digital product managers, and those who collaborate with research teams to consider incorporating the WOZ method into their research toolkit. Experiment with its application in diverse scenarios, adapt its techniques to your specific needs and don’t be afraid to have fun with it. Scarecrow costume optional.

(yk)

This article is sponsored by Lyssna

We’ve all been there: you pour your heart and soul into conducting meticulous user research. You gather insightful data, create detailed reports, and confidently deliver your findings. Yet, months later, little has changed. Your research sits idle on someone’s desk, gathering digital dust. It feels frustrating, like carefully preparing a fantastic meal, only to have it left uneaten.

There are so many useful tools (like Lysnna) to help us run incredible user research, and articles about how to get the most from them. However, there’s much less guidance about ensuring our user research gets adopted and brings about real change. So, in this post, I want to answer a simple question: How can you make sure your user research truly transforms your organization?

Introduction

User research is only as valuable as the impact it has.

When research insights fail to make their way into decisions, teams miss out on opportunities to improve products, experiences, and ultimately, business results. In this post, we’ll look at:

• Why research often fails to influence organizational change;
• How to ensure strategic alignment so research matters from day one;
• Ways to communicate insights clearly so stakeholders stay engaged;
• How to overcome practical implementation barriers;
• Strategies for realigning policies and culture to support research-driven changes.

By covering each of these areas, you’ll have a clear roadmap for turning your hard-won research into genuine action.

Typical Reasons For Failure

If you’ve ever felt your research get stuck, it probably came down to one (or more) of these issues.

Strategic Misalignment

When findings aren’t tied to business objectives or ROI, they struggle to gain traction. Sharing a particular hurdle that users face will fall on deaf ears if stakeholders cannot see how that problem will impact their bottom line.

Research arriving too late is another hurdle. If you share insights after key decisions are made, stakeholders assume your input won’t change anything. Finally, research often competes with other priorities. Teams might have limited resources and focus on urgent deadlines rather than long-term user improvements.

Communication Issues

Even brilliant research can get lost in translation if it’s buried in dense reports. I’ve seen stakeholders glaze over when handed 30-page documents full of jargon. When key takeaways aren’t crystal clear, decision-makers can’t quickly act on your findings.

Organizational silos can make communication worse. Marketing might have valuable insights that product managers never see, or designers may share findings that customer support doesn’t know how to use. Without a way to bridge those gaps, research lives in a vacuum.

Implementation Challenges

Great insights require a champion. Without a clear owner, research often lives with the person who ran it, and no one else feels responsible. Stakeholder skepticism also plays a role. Some teams doubt the methods or worry the findings don’t apply to real customers.

Even if there is momentum, insufficient follow-up or progress tracking can stall things. I’ve heard teams say, “We started down that path but ran out of time.” Without regular check-ins, good ideas fade away.

Policy And Cultural Barriers

Legal, compliance, or tech constraints can limit what you propose. I once suggested a redesign to comply with new accessibility standards, but the existing technical stack couldn’t support it. Resistance due to established culture is also common. If a company’s used to launching fast and iterating later, they might see research-driven change as slowing them down.

Now that we understand what stands in the way of effective research implementation, let’s explore practical solutions to overcome these challenges and drive real organizational change.

Ensuring Strategic Alignment

When research ties directly to business goals, it becomes impossible to ignore. Here’s how to do it.

Early Stakeholder Engagement

Invite key decision-makers into the research planning phase. I like to host a kickoff session where we map research objectives to specific KPIs, like increasing conversions by 10% or reducing support tickets by 20%. When your stakeholders help shape those objectives, they’re more invested in the results.

Research Objectives Aligned With Business KPIs

While UX designers often focus on user metrics like satisfaction scores or task completion rates, it’s crucial to connect our research to business outcomes that matter to stakeholders. Start by identifying the key business metrics that will demonstrate the value of your research:

• Identify which metrics matter most to the organization (e.g., conversion rate, churn, average order value).
• Frame research questions to directly address those metrics.
• Make preliminary hypotheses about how insights may affect the bottom line.

Develop Stakeholder-Specific Value Propositions

When presenting user research to groups, it’s easy to fall into the trap of delivering a one-size-fits-all message that fails to truly resonate with anyone. Instead, we need to carefully consider how different stakeholders will receive and act on our findings.

The real power of user research emerges when we can connect our insights directly to what matters most for each specific audience:

• For the product team: Show how insights can reduce development time by eliminating guesswork.
• For marketing: Demonstrate how understanding user language can boost ad copy effectiveness.
• For executives: Highlight potential cost savings or revenue gains.

ROI Framework Development

Stakeholders want to see real numbers. Develop simple templates to estimate potential cost savings or revenue gains. For example, if you uncover a usability issue that’s causing a 5% drop-off in the signup flow, translate that into lost revenue per month.

I also recommend documenting success stories from similar projects within your own organization or from case studies. When a stakeholder sees that another company boosted revenue by 15% after addressing a UX flaw, they’re more likely to pay attention.

Research Pipeline Integration

Integrate research tasks directly into your product roadmap. Schedule user interviews or usability tests just before major feature sprints. That way, findings land at the right moment — when teams are making critical decisions.

Regular Touchpoints with Strategic Teams

It’s essential to maintain consistent communication with strategic teams through regular research review meetings. These sessions provide a dedicated space to discuss new insights and findings. To keep everyone aligned, stakeholders should have access to a shared calendar that clearly marks key research milestones. Using collaborative tools like Trello boards or shared calendars ensures the entire team stays informed about the research plan and progress.

Resource Optimization

Research doesn’t have to be a massive, months-long effort each time. Build modular research plans that can scale. If you need quick, early feedback, run a five-user usability test rather than a full survey. For deeper analysis, you can add more participants later.

Addressing Communication Issues

Making research understandable is almost as important as the research itself. Let’s explore how to share insights so they stick.

Create Research One-Pagers

Condense key findings into a scannable one-pager. No more than a single sheet. Start with a brief summary of the problem, then highlight three to five top takeaways. Use bold headings and visual elements (charts, icons) to draw attention.

Implement Progressive Disclosure

Avoid dumping all details at once. Start with a high-level executive summary that anyone can read in 30 seconds. Then, link to a more detailed section for folks who want the full methodology or raw data. This layered approach helps different stakeholders absorb information at their own pace.

Use Visual Storytelling

Humans are wired to respond to stories. Transform data into a narrative by using journey maps, before/after scenarios, and user stories. For example, illustrate how a user feels at each step of a signup process, then show how proposed changes could improve their experience.

Regular Stakeholder Updates

Keep the conversation going. Schedule brief weekly or biweekly “research highlights” emails or meetings. These should be no more than five minutes and focus on one or two new insights. When stakeholders hear snippets of progress regularly, research stays top of mind.

Interactive Presentations

Take research readouts beyond slide decks. Host workshop-style sessions where stakeholders engage with findings hands-on. For instance, break them into small groups to discuss a specific persona and brainstorm solutions. When people physically interact with research (sticky notes, printed journey maps), they internalize it better.

Overcome Implementation Challenges

Now that stakeholders understand and value your research, let’s make sure they turn insights into action.

Establish Clear Ownership

Assign a dedicated owner for each major recommendation. Use a RACI matrix to clarify who’s Responsible, Accountable, Consulted, and Informed. I like to share a simple table listing each initiative, the person driving it, and key milestones.

When everyone knows who’s accountable, progress is more likely.

RACI Matrix Example

Build Implementation Roadmaps

Break recommendations down into phases. For example,

• Phase 1: Quick usability tweaks (1–2 weeks).
• Phase 2: Prototype new design (3–4 weeks).
• Phase 3: Launch A/B test (2–3 weeks).

Each phase needs clear timelines, success metrics, and resources identified upfront.

Address Stakeholder Skepticism

Be transparent about your methods. Share your recruitment screeners, interview scripts, and a summary of analysis steps. Offer validation sessions where stakeholders can ask questions about how the data was collected and interpreted. When they understand the process, they trust the findings more.

Create Support Systems

Even when stakeholders agree, they need help executing. Establish mentorship or buddy programs where experienced researchers or designers guide implementation. Develop training materials, like short “how-to” guides on running usability tests or interpreting survey data. Set up feedback channels (Slack channels, shared docs) where teams can ask questions or share roadblocks.

Monitor And Track Progress

Establish regular progress reviews weekly or biweekly. Use dashboards to track metrics such as A/B test performance, error rates, or user satisfaction scores. Even a more complicated dashboard can be built using no-code tools and AI, so you no longer need to rely on developer support.

Realign Policies and Culture

Even the best strategic plans and communication tactics can stumble if policies and culture aren’t supportive. Here’s how to address systemic barriers.

Create a Policy Evolution Framework

First, audit existing policies for anything that blocks research-driven changes. Maybe your data security policy requires months of legal review before you can recruit participants. Document those barriers and work with legal or compliance teams to create flexible guidelines. Develop a process for policy exception requests — so if you need a faster path for a small study, you know how to get approval without massive delays.

Technical Infrastructure Adaptation

Technology can be a silent killer of good ideas. Before proposing changes, work with IT to understand current limitations. Document technical requirements clearly so teams know what’s feasible. Propose a phased approach to any necessary infrastructure updates. Start with small changes that have an immediate impact, then plan for larger upgrades over time.

Build Cultural Buy-In

Culture shift doesn’t happen overnight. Share quick wins and success stories from early adopters in your organization. Recognize and reward change pioneers. Send a team-wide shout-out when someone successfully implements a research-driven improvement. Create a champions network across departments, so each area has at least one advocate who can spread best practices and encourage others.

Develop a Change Management Strategy

Change management is about clear, consistent communication. Develop tailored communication plans for different stakeholder groups. For example, executives might get a one-page impact summary, while developers get technical documentation and staging environments to test new designs. Establish feedback channels so teams can voice concerns or suggestions. Finally, provide change management training for team leaders so they can guide their direct reports through transitions.

Measure Cultural Impact

Culture can be hard to quantify, but simple pulse surveys go a long way. Ask employees how they feel about recent changes and whether they are more confident using data to make decisions. Track employee engagement metrics like survey participation or forum activity in research channels. Monitor resistance patterns (e.g., repeated delays or rejections) and address the root causes proactively.

Conclusions

Transforming user research into organizational change requires a holistic approach. Here’s what matters most:

• Strategic Alignment: Involve stakeholders early, tie research to KPIs, and integrate research into decision cycles.
• Effective Communication: Use one-pagers, progressive disclosure, visual storytelling, regular updates, and interactive presentations to keep research alive.
• Implementation Frameworks: Assign clear ownership, build phased roadmaps, address skepticism, offer support systems, and track progress.
• Culture and Policy: Audit and update policies, adapt infrastructure gradually, foster cultural buy-in, and employ change management techniques.

When you bring all of these elements together, research stops being an isolated exercise and becomes a driving force for real, measurable improvements. Keep in mind:

• Early stakeholder engagement drives buy-in.
• Clear research-to-ROI frameworks get attention.
• Ongoing, digestible communication keeps momentum.
• Dedicated ownership and phased roadmaps prevent stalls.
• Policy flexibility and cultural support enable lasting change.

This is an iterative, ongoing process. Each success builds trust and opens doors for more ambitious research efforts. Be patient, stay persistent, and keep adapting. When your organization sees research as a core driver of decisions, you’ll know you’ve truly succeeded.

(yk, il)

Unmoderated usability testing has been steadily growing more popular with the assistance of online UX research tools. Allowing participants to complete usability testing without a moderator, at their own pace and convenience, can have a number of advantages.

The first is the liberation from a strict schedule and the availability of moderators, meaning that a lot more participants can be recruited on a more cost-effective and quick basis. It also lets your team see how users interact with your solution in their natural environment, with the setup of their own devices. Overcoming the challenges of distance and differences in time zones in order to obtain data from all around the globe also becomes much easier.

However, forgoing the use of moderators also has its drawbacks. The moderator brings flexibility, as well as a human touch into usability testing. Since they are in the same (virtual) space as the participants, the moderator usually has a good idea of what’s going on. They can react in real-time depending on what they witness the participant do and say. A moderator can carefully remind the participants to vocalize their thoughts. To the participant, thinking aloud in front of a moderator can also feel more natural than just talking to themselves. When the participant does something interesting, the moderator can prompt them for further comment.

Meanwhile, a traditional unmoderated study lacks such flexibility. In order to complete tasks, participants receive a fixed set of instructions. Once they are done, they can be asked to complete a static questionnaire, and that’s it.

The feedback that the research & design team receives will be completely dependent on what information the participants provide on their own. Because of this, the phrasing of instructions and questions in unmoderated testing is extremely crucial. Although, even if everything is planned out perfectly, the lack of adaptive questioning means that a lot of the information will still remain unsaid, especially with regular people who are not trained in providing user feedback.

If the usability test participant misunderstands a question or doesn’t answer completely, the moderator can always ask for a follow-up to get more information. A question then arises: Could something like that be handled by AI to upgrade unmoderated testing?

Generative AI could present a new, potentially powerful tool for addressing this dilemma once we consider their current capabilities. Large language models (LLMs), in particular, can lead conversations that can appear almost humanlike. If LLMs could be incorporated into usability testing to interactively enhance the collection of data by conversing with the participant, they might significantly augment the ability of researchers to obtain detailed personal feedback from great numbers of people. With human participants as the source of the actual feedback, this is an excellent example of human-centered AI as it keeps humans in the loop.

There are quite a number of gaps in the research of AI in UX. To help with fixing this, we at UXtweak research have conducted a case study aimed at investigating whether AI could generate follow-up questions that are meaningful and result in valuable answers from the participants.

Asking participants follow-up questions to extract more in-depth information is just one portion of the moderator’s responsibilities. However, it is a reasonably-scoped subproblem for our evaluation since it encapsulates the ability of the moderator to react to the context of the conversation in real time and to encourage participants to share salient information.

Experiment Spotlight: Testing GPT-4 In Real-Time Feedback

The focus of our study was on the underlying principles rather than any specific commercial AI solution for unmoderated usability testing. After all, AI models and prompts are being tuned constantly, so findings that are too narrow may become irrelevant in a week or two after a new version gets updated. However, since AI models are also a black box based on artificial neural networks, the method by which they generate their specific output is not transparent.

Our results can show what you should be wary of to verify that an AI solution that you use can actually deliver value rather than harm. For our study, we used GPT-4, which at the time of the experiment was the most up-to-date model by OpenAI, also capable of fulfilling complex prompts (and, in our experience, dealing with some prompts better than the more recent GPT-4o).

In our experiment, we conducted a usability test with a prototype of an e-commerce website. The tasks involved the common user flow of purchasing a product.

Note: See our article published in the International Journal of Human-Computer Interaction for more detailed information about the prototype, tasks, questions, and so on).

In this setting, we compared the results with three conditions:

1. A regular static questionnaire made up of three pre-defined questions (Q1, Q2, Q3), serving as an AI-free baseline. Q1 was open-ended, asking the participants to narrate their experiences during the task. Q2 and Q3 can be considered non-adaptive follow-ups to Q1 since they asked participants more directly about usability issues and to identify things that they did not like.
2. The question Q1, serving as a seed for up to three GPT-4-generated follow-up questions as the alternative to Q2 and Q3.
3. All three pre-defined questions, Q1, Q2, and Q3, each used as a seed for its own GPT-4 follow-up.

The following prompt was used to generate the follow-up questions:

To assess the impact of the AI follow-up questions, we then compared the results on both a quantitative and a qualitative basis. One of the measures that we analyzed is informativeness — ratings of the responses based on how useful they are at elucidating new usability issues encountered by the user.

As seen in the figure below, the informativeness dropped significantly between the seed questions and their AI follow-up. The follow-ups rarely helped identify a new issue, although they did help elaborate further details.

The emotional reactions of the participants offer another perspective on AI-generated follow-up questions. Our analysis of the prevailing emotional valence based on the phrasing of answers revealed that, at first, the answers started with a neutral sentiment. Afterward, the sentiment shifted toward the negative.

In the case of the pre-defined questions Q2 and Q3, this could be seen as natural. While question Seed 1 was open-ended, asking the participants to explain what they did during the task, Q2 and Q3 focused more on the negative — usability issues and other disliked aspects. Curiously, the follow-up chains generally received even more negative receptions than their seed questions, and not for the same reason.

Frustration was common as participants interacted with the GPT-4-driven follow-up questions. This is rather critical, considering that frustration with the testing process can sidetrack participants from taking usability testing seriously, hinder meaningful feedback, and introduce a negative bias.

A major aspect that participants were frustrated with was redundancy. Repetitiveness, such as re-explaining the same usability issue, was quite common. While pre-defined follow-up questions yielded 27-28% of repeated answers (it’s likely that participants already mentioned aspects they disliked during the open-ended Q1), AI-generated questions yielded 21%.

That’s not that much of an improvement, given that the comparison is made to questions that literally could not adapt to prevent repetition at all. Furthermore, when AI follow-up questions were added to obtain more elaborate answers for every pre-defined question, the repetition ratio rose further to 35%. In the variant with AI, participants also rated the questions as significantly less reasonable.

Answers to AI-generated questions contained a lot of statements like “I already said that” and “The obvious AI questions ignored my previous responses.”

The prevalence of repetition within the same group of questions (the seed question, its follow-up questions, and all of their answers) can be seen as particularly problematic since the GPT-4 prompt had been provided with all the information available in this context. This demonstrates that a number of the follow-up questions were not sufficiently distinct and lacked the direction that would warrant them being asked.

Insights From The Study: Successes And Pitfalls

To summarize the usefulness of AI-generated follow-up questions in usability testing, there are both good and bad points.

Successes:

• Generative AI (GPT-4) excels at refining participant answers with contextual follow-ups.
• Depth of qualitative insights can be enhanced.

Challenges:

• Limited capacity to uncover new issues beyond pre-defined questions.
• Participants can easily grow frustrated with repetitive or generic follow-ups.

While extracting answers that are a bit more elaborate is a benefit, it can be easily overshadowed if the lack of question quality and relevance is too distracting. This can potentially inhibit participants’ natural behavior and the relevance of feedback if they’re focusing on the AI.

Therefore, in the following section, we discuss what to be careful of, whether you are picking an existing AI tool to assist you with unmoderated usability testing or implementing your own AI prompts or even models for a similar purpose.

Recommendations For Practitioners

Context is the end-all and be-all when it comes to the usefulness of follow-up questions. Most of the issues that we identified with the AI follow-up questions in our study can be tied to the ignorance of proper context in one shape or another.

Based on real blunders that GPT-4 made while generating questions in our study, we have meticulously collected and organized a list of the types of context that these questions were missing. Whether you’re looking to use an existing AI tool or are implementing your own system to interact with participants in unmoderated studies, you are strongly encouraged to use this list as a high-level checklist. With it as the guideline, you can assess whether the AI models and prompts at your disposal can ask reasonable, context-sensitive follow-up questions before you entrust them with interacting with real participants.

Without further ado, these are the relevant types of context:

• General Usability Testing Context.
  The AI should incorporate standard principles of usability testing in its questions. This may appear obvious, and it actually is. But it needs to be said, given that we have encountered issues related to this context in our study. For example, the questions should not be leading, ask participants for design suggestions, or ask them to predict their future behavior in completely hypothetical scenarios (behavioral research is much more accurate for that).
• Usability Testing Goal Context.
  Different usability tests have different goals depending on the stage of the design, business goals, or features being tested. Each follow-up question and the participant’s time used in answering it are valuable resources. They should not be wasted on going off-topic. For example, in our study, we were evaluating a prototype of a website with placeholder photos of a product. When the AI starts asking participants about their opinion of the displayed fake products, such information is useless to us.
• User Task Context.
  Whether the tasks in your usability testing are goal-driven or open and exploratory, their nature should be properly reflected in follow-up questions. When the participants have freedom, follow-up questions could be useful for understanding their motivations. By contrast, if your AI tool foolishly asks the participants why they did something closely related to the task (e.g., placing the specific item they were supposed to buy into the cart), you will seem just as foolish by association for using it.
• Design Context.
  Detailed information about the tested design (e.g., prototype, mockup, website, app) can be indispensable for making sure that follow-up questions are reasonable. Follow-up questions should require input from the participant. They should not be answerable just by looking at the design. Interesting aspects of the design could also be reflected in the topics to focus on. For example, in our study, the AI would occasionally ask participants why they believed a piece of information that was very prominently displayed in the user interface, making the question irrelevant in context.
• Interaction Context.
  If Design Context tells you what the participant could potentially see and do during the usability test, Interaction Context comprises all their actual actions, including their consequences. This could incorporate the video recording of the usability test, as well as the audio recording of the participant thinking aloud. The inclusion of interaction context would allow follow-up questions to build on the information that the participant already provided and to further clarify their decisions. For example, if a participant does not successfully complete a task, follow-up questions could be directed at investigating the cause, even as the participant continues to believe they have fulfilled their goal.
• Previous Question Context.
  Even when the questions you ask them are mutually distinct, participants can find logical associations between various aspects of their experience, especially since they don’t know what you will ask them next. A skilled moderator may decide to skip a question that a participant already answered as part of another question, instead focusing on further clarifying the details. AI follow-up questions should be capable of doing the same to avoid the testing from becoming a repetitive slog.
• Question Intent Context.
  Participants routinely answer questions in a way that misses their original intent, especially if the question is more open-ended. A follow-up can spin the question from another angle to retrieve the intended information. However, if the participant’s answer is technically a valid reply but only to the word rather than the spirit of the question, the AI can miss this fact. Clarifying the intent could help address this.

When assessing a third-party AI tool, a question to ask is whether the tool allows you to provide all of the contextual information explicitly.

If AI does not have an implicit or explicit source of context, the best it can do is make biased and untransparent guesses that can result in irrelevant, repetitive, and frustrating questions.

Even if you can provide the AI tool with the context (or if you are crafting the AI prompt yourself), that does not necessarily mean that the AI will do as you expect, apply the context in practice, and approach its implications correctly. For example, as demonstrated in our study, when a history of the conversation was provided within the scope of a question group, there was still a considerable amount of repetition.

The most straightforward way to test the contextual responsiveness of a specific AI model is simply by conversing with it in a way that relies on context. Fortunately, most natural human conversation already depends on context heavily (saying everything would take too long otherwise), so that should not be too difficult. What is key is focusing on the varied types of context to identify what the AI model can and cannot do.

The seemingly overwhelming number of potential combinations of varied types of context could pose the greatest challenge for AI follow-up questions.

For example, human moderators may decide to go against the general rules by asking less open-ended questions to obtain information that is essential for the goals of their research while also understanding the tradeoffs.

In our study, we have observed that if the AI asked questions that were too generically open-ended as a follow-up to seed questions that were open-ended themselves, without a significant enough shift in perspective, this resulted in repetition, irrelevancy, and — therefore — frustration.

The fine-tuning of the AI models to achieve an ability to resolve various types of contextual conflict appropriately could be seen as a reliable metric by which the quality of the AI generator of follow-up questions could be measured.

Researcher control is also key since tougher decisions that are reliant on the researcher’s vision and understanding should remain firmly in the researcher’s hands. Because of this, a combination of static and AI-driven questions with complementary strengths and weaknesses could be the way to unlock richer insights.

A focus on contextual sensitivity validation can be seen as even more important while considering the broader social aspects. Among certain people, the trend-chasing and the general overhype of AI by the industry have led to a backlash against AI. AI skeptics have a number of valid concerns, including usefulness, ethics, data privacy, and the environment. Some usability testing participants may be unaccepting or even outwardly hostile toward encounters with AI.

Therefore, for the successful incorporation of AI into research, it will be essential to demonstrate it to the users as something that is both reasonable and helpful. Principles of ethical research remain as relevant as ever. Data needs to be collected and processed with the participant’s consent and not breach the participant’s privacy (e.g. so that sensitive data is not used for training AI models without permission).

Conclusion: What’s Next For AI In UX?

So, is AI a game-changer that could break down the barrier between moderated and unmoderated usability research? Maybe one day. The potential is certainly there. When AI follow-up questions work as intended, the results are exciting. Participants can become more talkative and clarify potentially essential details.

To any UX researcher who’s familiar with the feeling of analyzing vaguely phrased feedback and wishing that they could have been there to ask one more question to drive the point home, an automated solution that could do this for them may seem like a dream. However, we should also exercise caution since the blind addition of AI without testing and oversight can introduce a slew of biases. This is because the relevance of follow-up questions is dependent on all sorts of contexts.

Humans need to keep holding the reins in order to ensure that the research is based on actual solid conclusions and intents. The opportunity lies in the synergy that can arise from usability researchers and designers whose ability to conduct unmoderated usability testing could be significantly augmented.

Humans + AI = Better Insights

The best approach to advocate for is likely a balanced one. As UX researchers and designers, humans should continue to learn how to use AI as a partner in uncovering insights. This article can serve as a jumping-off point, providing a list of the AI-driven technique’s potential weak points to be aware of, to monitor, and to improve on.

(yk)

Google Analytics is great, but not everyone in your organization will be granted access. In many places I’ve worked, it was on a kind of “need to know” basis.

In this article, I’m gonna flip that on its head and show you how I wrote a GitHub Action that queries Google Analytics, generates a top ten list of the most frequently viewed pages on my site from the last seven days and compares them to the previous seven days to tell me which pages have increased in views, which pages have decreased in views, which pages have stayed the same and which pages are new to the list.

The report is then nicely formatted with icon indicators and posted to a public Slack channel every Friday at 10 AM.

Not only would this surfaced data be useful for folks who might need it, but it also provides an easy way to copy and paste or screenshot the report and add it to a slide for the weekly company/department meeting.

Here’s what the finished report looks like in Slack, and below, you’ll find a link to the GitHub Repository.

GitHub

To use this repository, follow the steps outlined in the README.

• https://github.com/PaulieScanlon/smashing-weekly-analytics
  

Prerequisites

To build this workflow, you’ll need admin access to your Google Analytics and Slack Accounts and administrator privileges for GitHub Actions and Secrets for a GitHub repository.

Customizing the Report and Action

Naturally, all of the code can be changed to suit your requirements, and in the following sections, I’ll explain the areas you’ll likely want to take a look at.

Customizing the GitHub Action

The file name of the Action weekly-analytics.report.yml isn’t seen anywhere other than in the code/repo but naturally, change it to whatever you like, you won’t break anything.

The name and jobs: names detailed below are seen in the GitHub UI and Workflow logs.

The cron syntax determines when the Action will run. Schedules use POSIX cron syntax and by changing the numbers you can determine when the Action runs.

You could also change the secrets variable names; just make sure you update them in your repository Settings.

# .github/workflows/weekly-analytics-report.yml

name: Weekly Analytics Report

on:
  schedule:
    - cron: '0 10 * * 5' # Runs every Friday at 10 AM UTC
  workflow_dispatch: # Allows manual triggering

jobs:
  analytics-report:
    runs-on: ubuntu-latest

    env:
      SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}
      GA4_PROPERTY_ID: ${{ secrets.GA4_PROPERTY_ID }}
      GOOGLE_APPLICATION_CREDENTIALS_BASE64: ${{ secrets.GOOGLE_APPLICATION_CREDENTIALS_BASE64 }}

    steps:
      - name: Checkout repository
        uses: actions/checkout@v4

      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '20.x'

      - name: Install dependencies
        run: npm install

      - name: Run the JavaScript script
        run: node src/services/weekly-analytics.js

Customizing the Google Analytics Report

The Google Analytics API request I’m using is set to pull the fullPageUrl and pageTitle for the totalUsers in the last seven days, and a second request for the previous seven days, and then aggregates the totals and limits the responses to 10.

You can use Google’s GA4 Query Explorer to construct your own query, then replace the requests.

// src/services/weekly-analytics.js#L75

const [thisWeek] = await analyticsDataClient.runReport({
  property: `properties/${process.env.GA4_PROPERTY_ID}`,
  dateRanges: [
    {
      startDate: '7daysAgo',
      endDate: 'today',
    },
  ],
  dimensions: [
    {
      name: 'fullPageUrl',
    },
    {
      name: 'pageTitle',
    },
  ],
  metrics: [
    {
      name: 'totalUsers',
    },
  ],
  limit: reportLimit,
  metricAggregations: ['MAXIMUM'],
});

Creating the Comparisons

There are two functions to determine which page views have increased, decreased, stayed the same, or are new.

The first is a simple reduce function that returns the URL and a count for each.

const lastWeekMap = lastWeekResults.reduce((items, item) => {
  const { url, count } = item;
  items[url] = count;
  return items;
}, {});

The second maps over the results from this week and compares them to last week.

// Generate the report for this week
const report = thisWeekResults.map((item, index) => {
  const { url, title, count } = item;
  const lastWeekCount = lastWeekMap[url];
  const status = determineStatus(count, lastWeekCount);

  return {
    position: (index + 1).toString().padStart(2, '0'), // Format the position with leading zero if it's less than 10
    url,
    title,
    count: { thisWeek: count, lastWeek: lastWeekCount || '0' }, // Ensure lastWeekCount is displayed as '0' if not found
    status,
  };
});

The final function is used to determine the status of each.

// Function to determine the status
const determineStatus = (count, lastWeekCount) => {
  const thisCount = Number(count);
  const previousCount = Number(lastWeekCount);

  if (lastWeekCount === undefined || lastWeekCount === '0') {
    return NEW;
  }

  if (thisCount > previousCount) {
    return HIGHER;
  }

  if (thisCount < previousCount) {
    return LOWER;
  }

  return SAME;
};

I’ve purposely left the code fairly verbose, so it’ll be easier for you to add console.log to each of the functions to see what they return.

Customizing the Slack Message

The Slack message config I’m using creates a heading with an emoji, a divider, and a paragraph explaining what the message is.

Below that I’m using the context object to construct a report by iterating over comparisons and returning an object containing Slack specific message syntax which includes an icon, a count, the name of the page and a link to each item.

You can use Slack’s Block Kit Builder to construct your own message format.

// src/services/weekly-analytics.js#151 

    const slackList = report.map((item, index) => {
      const {
        position,
        url,
        title,
        count: { thisWeek, lastWeek },
        status,
      } = item;

      return {
        type: 'context',
        elements: [
          {
            type: 'image',
            image_url: `${reportConfig.url}/images/${status}`,
            alt_text: 'icon',
          },
          {
            type: 'mrkdwn',
            text: `${position}.  <${url}|${title}> | *\`${`x${thisWeek}`}\`* / x${lastWeek}`,
          },
        ],
      };
    });

Before you can run the GitHub Action, you will need to complete a number of Google, Slack, and GitHub steps.

Ready to get going?

Creating a Google Cloud Project

Head over to your Google Cloud console, and from the dropdown menu at the top of the screen, click Select a project, and when the modal opens up, click NEW PROJECT.

Project name

On the next screen, give your project a name and click CREATE. In my example, I’ve named the project smashing-weekly-analytics.

Enable APIs & Services

In this step, you’ll enable the Google Analytics Data API for your new project. From the left-hand sidebar, navigate to APIs & Services > Enable APIs & services. At the top of the screen, click + ENABLE APIS & SERVICES.

Enable Google Analytics Data API

Search for “Google analytics data API,” select it from the list, then click ENABLE.

Create Credentials for Google Analytics Data API

With the API enabled in your project, you can now create the required credentials. Click the CREATE CREDENTIALS button at the top right of the screen to set up a new Service account.

A Service account allows an “application” to interact with Google APIs, providing the credentials include the required services. In this example, the credentials grant access to the Google Analytics Data API.

Service Account Credentials Type

On the next screen, select Google Analytics Data API from the dropdown menu and Application data, then click NEXT.

Service Account Details

On the next screen, give your Service account a name, ID, and description (optional). Then click CREATE AND CONTINUE.

In my example, I’ve given my service account a name and ID of smashing-weekly-analytics and added a short description that explains what the service account does.

Service Account Role

On the next screen, select Owner for the Role, then click CONTINUE.

Service Account Done

You can leave the fields blank in this last step and click DONE when you’re ready.

Service Account Keys

From the left-hand navigation, select Service Accounts, then click the “more dots” to open the menu and select Manage keys.

Service Accounts Add Key

On the next screen, locate the KEYS tab at the top of the screen, then click ADD KEY and select Create new key.

Service Accounts Download Keys

On the next screen, select JSON as the key type, then click CREATE to download your Google Application credentials .json file.

Google Application Credentials

If you open the .json file in your code editor, you should be looking at something similar to the one below.

In case you’re wondering, no, you can’t use an object as a variable defined in an .env file. To use these credentials, it’s necessary to convert the whole file into a base64 string.

Note: I wrote a more detailed post about how to use Google Application credentials as environment variables here: “How to Use Google Application .json Credentials in Environment Variables.”

From your terminal, run the following: replace name-of-creds-file.json with the name of your .json file.

cat name-of-creds-file.json | base64

If you’ve already cloned the repo and followed the Getting started steps in the README, add the base64 string returned after running the above and add it to the GOOGLE_APPLICATION_CREDENTIALS_BASE64 variable in your .env file, but make sure you wrap the string with double quotation makes.

GOOGLE_APPLICATION_CREDENTIALS_BASE64="abc123"

That completes the Google project side of things. The next step is to add your service account email to your Google Analytics property and find your Google Analytics Property ID.

Google Analytics Properties

Whilst your service account now has access to the Google Analytics Data API, it doesn’t yet have access to your Google Analytics account.

Get Google Analytics Property ID

To make queries to the Google Analytics API, you’ll need to know your Property ID. You can find it by heading over to your Google Analytics account. Make sure you’re on the correct property (in the screenshot below, I’ve selected paulie.dev — GA4).

Click the admin cog in the bottom left-hand side of the screen, then click Property details.

On the next screen, you’ll see the PROPERTY ID in the top right corner. If you’ve already cloned the repo and followed the Getting started steps in the README, add the property ID value to the GA4_PROPERTY_ID variable in your .env file.

Add Client Email to Google Analytics

From the Google application credential .json file you downloaded earlier, locate the client_email and copy the email address.

In my example, it looks like this: smashing-weekly-analytics@smashing-weekly-analytics.iam.gserviceaccount.com.

Now navigate to Property access management from the left hide side navigation and click the + in the top right-hand corner, then click Add users.

On the next screen, add the client_email to the Email addresses input, uncheck Notify new users by email, and select Viewer under Direct roles and data restrictions, then click Add.

That completes the Google Analytics properties section. Your “application” will use the Google application credentials containing the client_email and will now have access to your Google Analytics account via the Google Analytics Data API.

Slack Channels and Webhook

In the following steps, you’ll create a new Slack channel that will be used to post messages sent from your “application” using a Slack Webhook.

Creating The Slack Channel

Create a new channel in your Slack workspace. I’ve named mine #weekly-analytics-report. You’ll need to set this up before proceeding to the next step.

Creating a Slack App

Head over to the slack api dashboard, and click Create an App.

On the next screen, select From an app manifest.

On the next screen, select your Slack workspace, then click Next.

On this screen, you can give your app a name. In my example, I’ve named my Weekly Analytics Report. Click Next when you’re ready.

On step 3, you can just click Done.

With the App created, you can now set up a Webhook.

Creating a Slack Webhook

Navigate to Incoming Webhooks from the left-hand navigation, then switch the Toggle to On to activate incoming webhooks. Then, at the bottom of the screen, click Add New Webook to Workspace.

On the next screen, select your Slack workspace and a channel that you’d like to use to post messages, too, and click Allow.

You should now see your new Slack Webhook with a copy button. Copy the Webhook URL, and if you’ve already cloned the repo and followed the Getting started steps in the README, add the Webhook URL to the SLACK_WEBHOOK_URL variable in your .env file.

Slack App Configuration

From the left-hand navigation, select Basic Information. On this screen, you can customize your app and add an icon and description. Be sure to click Save Changes when you’re done.

If you now head over to your Slack, you should see that your app has been added to your workspace.

That completes the Slack section of this article. It’s now time to add your environment variables to GitHub Secrets and run the workflow.

Add GitHub Secrets

Head over to the Settings tab of your GitHub repository, then from the left-hand navigation, select Secrets and variables, then click Actions.

Add the three variables from your .env file under Repository secrets.

A note on the base64 string: You won’t need to include the double quotes!

Run Workflow

To test if your Action is working correctly, head over to the Actions tab of your GitHub repository, select the Job name (Weekly Analytics Report), then click Run workflow.

If everything worked correctly, you should now be looking at a nicely formatted list of the top ten page views on your site in Slack.

Finished

And that’s it! A fully automated Google Analytics report that posts directly to your Slack. I’ve worked in a few places where Google Analytics data was on lockdown, and I think this approach to sharing Analytics data with Slack (something everyone has access to) could be super valuable for various people in your organization.

(yk)

Do you recognize this situation? The marketing and business teams talk about their customers, and each team thinks they have the same understanding of the problem and what needs to be done. Then, they’re including the Product and UX team in the conversation around how to best serve a particular customer group and where to invest in development and marketing efforts. They’ve done their initial ideation and are trying to prioritize, but this turns into a long discussion with the different teams favoring different areas to focus on. Suddenly, an executive highlights that instead of this customer segment, there should be a much higher focus on an entirely different segment — and the whole discussion starts again.

This situation often arises when there is no joint-up understanding of the different customer segments a company is serving historically and strategically. And there is no shared understanding beyond using the same high-level terms. To reach this understanding, you need to dig deeper into segment definitions, goals, pain points, and jobs-to-be-done (JTBD) so as to enable the organization to make evidence-based decisions instead of having to rely on top-down prioritization.

The hardest part about doing the right thing for your user or customers (please note I’m aware these terms aren’t technically the same, but I’m using them interchangeably in this article so as to be useful to a wider audience) often starts inside your own company and getting different teams with diverging goals and priorities to agree on where to focus and why.

But how do you get there — thinking user-first AND ensuring teams are aligned and have a shared mental model of primary and secondary customer segments?

Personas vs Segments

To explore that further, let’s take a brief look at the most commonly applied techniques to better understand customers and communicate this knowledge within organizations.

Two frequently employed tools are user personas and user segmentation.

Product/UX (or non-demographic) personas aim to represent the characteristics and needs of a certain type of customer, as well as their motivations and experience. The aim is to illustrate an ideal customer and allow teams to empathize and solve different use cases. Marketing (or demographic) personas, on the other hand, traditionally focus on age, socio-demographics, education, and geography but usually don’t include needs, motivations, or other contexts. So they’re good for targeting but not great for identifying new potential solutions or helping teams prioritize.

In contrast to personas, user segments illustrate groups of customers with shared needs, characteristics, and actions. They are relatively high-level classifications, deliberately looking at a whole group of needs without telling a detailed story. The aim is to gain a broader overview of the wider market’s wants and needs.

Tony Ulwick, creator of the “jobs-to-be-done” framework, for example, creates outcome-based segmentations, which are quite similar to what this article is proposing. Other types of segmentations include geographic, psychographic, demographic, or needs-based segmentations. What all segmentations, including the user segmentation matrix, have in common is that the segments are different from each other but don‘t need to be mutually exclusive.

As Simon Penny points out, personas and segments are tools for different purposes. While customer segments help us understand a marketplace or customer base, personas help us to understand more about the lived experience of a particular group of customers within that marketplace.

Both personas and segmentations have their applications, but this article argues that using a matrix will help you prioritize between the different segments. In addition, the key aspect here is the co-creation process that fosters understanding across departments and allows for more transparent decision-making. Instead of focusing only on the outcome, the process of getting there is what matters for alignment and collaboration across teams. Let’s dig deeper into how to achieve that.

User Segmentation Matrix: 101

At its core, the idea of the user segmentation matrix is meant to create a shared mental model across teams and departments of an organization to enable better decision-making and collaboration.

And it does that by visualizing the relevance and differences between a company’s customer segments. Crucially, input into the matrix comes from across teams as the process of co-creation plays an essential part in getting to a shared understanding of the different segments and their relevance to the overall business challenge.

Additionally, this kind of matrix follows the principle of “just enough, not too much” to create meaning without going too deep into details or leading to confusion. It is about pulling together key elements from existing tools and methods, such as User Journeys or Jobs-to-be-done, and visualizing them in one place.

For a high-level first overview, see the matrix scaffolding below.

Case Study: Getting To A Shared Mental Model Across Teams

Let’s look at the problem through a case study and see how building a user segmentation matrix helped a global data products organization gain a much clearer view of its customers and priorities.

Here is some context. The organization was partly driven by NGO principles like societal impact and partly by economic concerns like revenue and efficiencies. Its primary source of revenue was raw data and data products, and it was operating in a B2B setting. Despite operating for several decades already, its maturity level in terms of user experience and product knowledge was low, while the amount of different data outputs and services was high, with a whole bouquet of bespoke solutions for individual clients. The level of bespoke solutions that had to be maintained and had grown organically over time had surpassed the “featuritis” stage and turned utterly unsustainable.

And you probably guessed it: The business focus had traditionally been “What can we offer and sell?” instead of “What are our customers trying to solve?”

That means there were essentially two problems to figure out:

1. Help executives and department leaders from Marketing through Sales, Business, and Data Science see the value of customer-first product thinking.
2. Establish a shared mental model of the key customer segments to start prioritizing with focus and reduce the completely overgrown service offering.

For full disclosure, here’s a bit about my role in this context: I was there in a fractional product leader role at first, after running a discovery workshop, which then developed into product strategy work and eventually a full evaluation of the product portfolio according to user & business value.

Approach

So how did we get to that outcome? Basically, we spent an afternoon filling out a table with different customer segments, presented it to a couple of stakeholders, and everyone was happy — THE END. You can stop reading…

Or not, because from just a few initial conversations and trying to find out if there were any existing personas, user insights, or other customer data, it became clear that there was no shared mental model of the organization’s customer segments.

At the same time, the Business and Account management teams, especially, had a lot of contact with new and existing customers and knew the market and competition well. And the Marketing department had started on personas. However, they were not widely used and weren’t able to act as that shared mental model across different departments.

So, instead of thinking customer-first the organization was operating “inside-out first,” based on the services they offered. With the user segmentation matrix, we wanted to change this perspective and align all teams around one shared canvas to create transparency around user and business priorities.

But How To Proceed Quickly While Taking People Along On The Journey?

Here’s the approach we took:

1. Gather All Existing Research

First, we gathered all user insights, customer feedback, and data from different parts of the organization and mapped them out on a big board (see below). Initially, we really tried to map out all existing documentation, including links to in-house documents and all previous attempts at separating different user groups, analytics data, revenue figures, and so on.

The key here was to speak to people in different departments to understand how they were currently thinking about their customers and to include the terms and documentation they thought most relevant without giving them a predefined framework. We used the dimensions of the matrix as a conversation guide, e.g., asking about their definitions for key user groups and what makes them distinctly different from others.

2. Start The Draft Scaffolding

Secondly, we created the draft matrix with assumed segments and some core elements that have proven useful in different UX techniques.

In this step, we started to make sense of all the information we had collected and gave the segments “draft labels” and “draft definitions” based on input from the teams, but creating this first draft version within the small working group. The aim was to reduce complexity, settle on simple labels, and introduce primary vs secondary groups based on the input we received.

We then made sure to run this summarized draft version past the stakeholders for feedback and amends, always calling out the DRAFT status to ensure we had buy-in across teams before removing that label. In addition to interviews, we also provided direct access to the workboard for stakeholders to contribute asynchronously and in their own time and to give them the option to discuss with their own teams.

3. Refine

In the next step, we went through several rounds of “joint sense-making” with stakeholders from across different departments. At this stage, we started coloring in the scaffolding version of the matrix with more and more detail. We also asked stakeholders to review the matrix as a whole and comment on it to make sure the different business areas were on board and to see the different priorities between, e.g., primary and secondary user groups due to segment size, pain points, or revenue numbers.

4. Prompt

We then promoted specifically for insights around segment definitions, pain points, goals, jobs to be done, and defining differences to other segments. Once the different labels and the sorting into primary versus secondary groups were clear, we tried to make sure that we had similar types of information per segment so that it would be easy to compare different aspects across the matrix.

5. Communicate

Finally, we made sure the core structure reached different levels of leadership. While we made sure to include senior stakeholders in the process throughout, this step was essential prior to circulating the matrix widely across the organization.

However, due to the previous steps, we had gone through, at this point, we were able to assure senior leadership that their teams had contributed and reviewed several times, so getting that final alignment was easy.

We did this in a team of two external consultants and three in-house colleagues, who conducted the interviews and information gathering exercises in tandem with us. Due to the size and global nature of the organization and various different time zones to manage, it took around 3 weeks of effort, but 3 months in time due to summer holidays and alignment activities. So we did this next to other work, which allowed us to be deeply plugged into the organization and avoid blind spots due to having both internal and external perspectives.

Building on in-house advocates with deep organizational knowledge and subject-matter expertise was a key factor and helped bring the organization along much better than purely external consultants could have done.

User Segmentation Matrix: Key Ingredients

So, what are the dimensions we included in this mapping out of primary and secondary user segments?

The dimensions we used were the following:

1. Segment definition
   Who is this group?
   Define it in a simple, straightforward way so everyone understands — NO acronyms or abbreviations. Further information to include that’s useful if you have it: the size of the segment and associated revenue.
2. Their main goals
   What are their main goals?
   Thinking outside-in and from this user groups perspective these would be at a higher level than the specific JTBD field, big picture and longer term.
3. What are their “Jobs-to-be-done”?
   Define the key things this group needs in order to get their own work done (whether that’s currently available in your service or not; if you don’t know this, it’s time for some discovery). Please note this is not a full JTBD mapping, but instead seeks to call out exemplary practical tasks.
4. How are they different from other segments?
   Segments should be clearly different in their needs. If they’re too similar, they might not be a separate group.
5. Main pain points
   What are the pain points for each segment? What issues are they currently experiencing with your service/product? Note the recurring themes.
6. Key contacts in the organization
   Who are the best people holding knowledge about this user segment?
   Usually, these would be the interview partners who contributed to the matrix, and it helps to not worry too much about ownership or levels here; it could be from any department, and often, the Business or Product org are good starting points.

This is an example of a user segmentation matrix:

Outcomes & Learning

What we found in this work is that seeing all user segments mapped out next to each other helped focus the conversation and create a shared mental model that switched the organization’s perspective to outside-in and customer-first.

Establishing the different user segment names and defining primary versus secondary segments created transparency, focus, and a shared understanding of priorities.

Building this matrix based on stakeholder interviews and existing user insights while keeping the labeling in DRAFT mode, we encouraged feedback and amends and helped everyone feel part of the process. So, rather than being a one-time set visualization, the key to creating value with this matrix is to encourage conversation and feedback loops between teams and departments.

In our case, we made sure that every stakeholder (at different levels within the organization, including several people from the executive team) had seen this matrix at least twice and had the chance to input. Once we then got to the final version, we were sure that we had an agreement on the terminology, issues, and priorities.

Below is the real case study example (with anonymized inputs):

Takeaways And What To Watch Out For

So what did this approach help us achieve?

1. It created transparency and helped the Sales and Business teams understand how their asks would roughly be prioritized — seeing the other customer segments in comparison (especially knowing the difference between primary vs secondary segments).
2. It shifted the thinking to customer-first by providing an overview for the executive team (and everyone else) to start thinking about customers rather than business units and see new opportunities more clearly.
3. It highlighted the need to gather more customer insights and better performance data, such as revenue per segment, more detailed user tracking, and so on.

In terms of the challenges we faced when conducting and planning this work, there are a few things to watch out for:

We found that due to the size and global nature of the organization, it took several rounds of feedback to align with all stakeholders on the draft versions. So, the larger the size of your organization, the more buffer time to include (or the ability to change interview partners at short notice).

If you’re planning to do this in a startup or mid-sized organization, especially if they’ve got the relevant information available, you might need far less time, although it will still make sense to carefully select the contributors.

Having in-house advocates who actively contributed to the work and conducted interviews was a real benefit for alignment and getting buy-in across the organization, especially when things started getting political.

Gathering information from Marketing, Product, Business, Sales and Leadership and sticking with their terms and definitions initially was crucial, so everyone felt their inputs were heard and saw it reflected, even if amended, in the overall matrix.

And finally, a challenge that’s not to be underestimated is the selection of those asked to input — where it’s a tightrope walk between speed and inclusion.

We found that a “snowball system” worked well, where we initially worked with the C-level sponsor to define the crucial counterparts at the leadership level and have them name 3-4 leads in their organization, looking after different parts of the organization. These leaders were asked for their input and their team’s input in interviews and through asynchronous access to the joint workboard.

What’s In It For You?

To summarize, the key benefits of creating a user segmentation matrix in your organization are the following:

• Thinking outside-in and user-first.
  Instead of thinking this is what you offer, your organization starts to think about solving real customer problems — the matrix is your GPS view of your market (but like any GPS system, don’t forget to update it occasionally).
• Clarity and a shared mental model.
  Everyone is starting to use the same language, and there’s more clarity about what you offer per customer segment. So, from Sales through to Business and Product, you’re speaking to users and their needs instead of talking about products and services (or even worse, your in-house org structure). Shared clarity drastically reduces meeting and decision time and allows you to do more impactful work.
• Focus, and more show than tell.
  Having a matrix helps differentiate between primary, secondary, and other customer segments and visualizes these differences for everyone.

When Not To Use It

If you already have a clearly defined set of customer segments that your organization is in agreement on and working towards — good for you; you won’t need this and can rely on your existing data.

Another case where you will likely not need this full overview is when you’re dealing with a very specific customer segment, and there is good alignment between the teams serving this group in terms of focus, priorities, and goals.

Organizations that will see the highest value in this exercise are those who are not yet thinking outside-in and customer-first and who still have a traditional approach, starting from their own services and dealing with conflicting priorities between departments.

Next Steps

And now? You’ve got your beautiful and fully aligned customer segmentation matrix ready and done. What’s next? In all honesty, this work is never done, and this is just the beginning.

If you have been struggling with creating an outside-in perspective in your organization, the key is to make sure that it gets communicated far and wide.

For example, make sure to get your executive sponsors to talk about it in their rounds, do a road show, or hold open office hours where you can present it to anyone interested and give them a chance to ask questions. Or even better, present it at the next company all-hands, with the suggestion to start building up an insights library per customer segment.

If this was really just the starting point to becoming more product-led, then the next logical step is to assess and evaluate the current product portfolio. The aim is to get clarity around which services or products are relevant for which customers. Especially in product portfolios plagued by “featuritis,” it makes sense to do a full audit, evaluate both user and business value, and clean out your product closet.

If you’ve seen gaps and blind spots in your matrix, another next step would be to do some deep dives, customer interviews, and discovery work to fill those. And as you continue on that journey towards more customer-centricity, other tools from the UX and product tool kit, like mapping out user journeys and establishing a good tracking system and KPIs, will be helpful so you can start measuring customer satisfaction and continue to test and learn.

Like a good map, it helps you navigate and create a shared understanding across departments. And this is its primary purpose: getting clarity and focus across teams to enable better decision-making. The process of co-creating a living document that visualizes customer segments is at least as important here as the final outcome.

Further Reading

• Data-Driven Personas for Enhanced User Understanding: Combining Empathy with Rationality for Better Insights to Analytics, Bernard J. Jansen, Joni O. Salminen, Soon-Gyo Jung (Data and Information Management, 2020)
• “Most Effective Ways To Segment Your Users,” Ilia Lotarev (Adapty.io)
• “The 6 Types Of User Segmentation And What They Mean For Your Product,” Pavel Malos (UX Collective, Medium)
• “Understanding Users By Going Beyond Personas, Demographics, And Affinity Groups,” Kate Matesic (UX Magazine)
• “The Difference Between Customer Segmentation And Customer Personas,” Simon Penny (UX Collective, Medium)

(yk)

Ensuring technology is accessible and inclusive relies heavily on receiving feedback directly from disabled users. You cannot rely solely on checklists, guidelines, and good-faith guesses to get things right. This is often hindered, however, by a lack of accessible prototypes available to use during testing.

Rather than wait for the digital landscape to change, researchers should leverage all the available tools they can use to create and replicate the testing environments they need to get this important research completed. Without it, we will continue to have a primarily inaccessible and not inclusive technology landscape that will never be disrupted.

Note: I use “identity first” disability language (as in “disabled people”) rather than “people first” language (as in “people with disabilities”). Identity first language aligns with disability advocates who see disability as a human trait description or even community and not a subject to be avoided or shamed. For more, review “Writing Respectfully: Person-First and Identity-First Language”.

Accessibility-focused Research In All Phases

When people advocate that UX Research should include disabled participants, it’s often with the mindset that this will happen on the final product once development is complete. One primary reason is because that’s when researchers have access to the most accessible artifact with which to run the study. However,

The real ability to ensure an accessible and inclusive system is not by evaluating a final product at the end of a project; it’s by assessing user needs at the start and then evaluating the iterative prototypes along the way.

“

Prototype Research Should Include Disabled Participants

In general, the iterative prototype phase of a project is when teams explore various design options and make decisions that will influence the final project outcome. Gathering feedback from representative users during this phase can help teams make informed decisions, including key pivots before significant development and testing resources are used.

During the prototype phase of user testing, the representative users should include disabled participants. By collecting feedback and perspectives of people with a variety of disabilities in early design testing phases, teams can more thoughtfully incorporate key considerations and supplement accessibility guidelines with real-world feedback. This early-and-often approach is the best way to include accessibility and inclusivity into a process and ensure a more accessible final product.

If you instead wait to include disabled participants in research until a product is near final, this inevitably leads to patchwork fixes of any critical feedback. Then, for feedback not deemed critical, it will likely get “backlogged” where the item priorities compete with new feature updates. With this approach, you’ll constantly be playing catch-up rather than getting it right up front and in an elegant and integrated way.

Accessibility Research Can’t Wait Until The End

Not only does research with disabled participants often occur too late in a project, but it is also far too often viewed as separate from other research studies (sometimes referred to as the “main research”). It cannot be understated that this reinforces the notion of separate-and-not-equal as compared to non-disabled participants and other stakeholder feedback. This has a severe negative impact on how a team will view the priority of inclusive design and, more broadly, the value of disabled people. That is, this reinforces “ableism”, a devaluing of disabled people in society.

UX Research with diverse participants that include a wide variety of disabilities can go a long way in dismantling ableist views and creating vitally needed inclusive technology.

“

The problem is that even when a team is on board with the idea, it’s not always easy to do inclusive research, particularly when involving prototypes. While discovery research can be conducted with minimal tooling and summative research can leverage fully built and accessible systems, prototype research quickly reveals severe accessibility barriers that feel like they can’t be overcome.

Inaccessible Technology Impedes Accessibility Research

Most technology we use has accessibility barriers for users with disabilities. As an example, the WebAIM Million report consistently finds that 96% of web homepages have accessibility errors that are fixable and preventable.

Just like websites, web, and mobile applications are similarly inaccessible, including those that produce early-stage prototypes. Thus, the artifacts researchers might want to use for prototype testing to help create accessible products are themselves inaccessible, creating a barrier for disabled research participants. It quickly becomes a vicious cycle that seems hard to break.

The Limitations Of Figma

Currently, the most popular industry tool for initial prototyping is Figma. These files become the artifacts researchers use to conduct a research study. However, these files often fall short of being accessible enough for many participants with disabilities.

To be clear, I absolutely applaud the Figma employees who have worked very hard on including screen reader support and keyboard functionality in Figma prototypes. This represents significant progress towards removing accessibility barriers in our core products and should not be overlooked. Nevertheless, there are still limitations and even blockers to research.

For one, the Figma files must be created in a way that will mimic the website layout and code. For example, for screen reader navigation to be successful, the elements need to be in their correct reading order in the Layers panel (not solely look correct visually), include labeled elements such as buttons (not solely items styled to look like buttons), and include alternative text for images. Often, however, designers do not build iterative prototypes with these considerations in mind, which prevents the keyboard from navigating correctly and the screen reader from providing the necessary details to comprehend the page.

In addition, Figma’s prototypes do not have selectable, configurable text. This prevents key visual adjustments such as browser zoom to increase text size, dark mode, which is easier for some to view, and selecting text to have it read aloud. If a participant needs these kinds of adjustments (or others I list in the table below), a Figma prototype will not be accessible to them.

Table: Figma prototype limitations per assistive technology

Inclusive Research Is Needed Regardless

Having accessibility challenges with a prototype doesn’t mean we give up on the research. Instead, it means we need to get creative in our approach. This research is too important to keep waiting for the ideal set-up, particularly when our findings are often precisely what’s needed to create accessible technology.

Part of crafting a research study is determining what artifact to use during the study. Thus, when considering prototype research, it is a matter of creating the artifact best suited for your study. If this isn’t going to be, say, a Figma file you receive from designers, then consider what else can be used to get the job done.

Working Around the Current State

Being able to include diverse perspectives from disabled research participants throughout a project’s creation is possible and necessary. Keeping in mind your research questions and the capabilities of your participants, there are research methods and strategies that can be made accessible to gather authentic feedback during the critical prototype design phase.

With that in mind, I propose five ways you can accomplish prototype research while working around inaccessible prototypes:

1. Use a survey.
2. Conduct a co-design session.
3. Test with a similar system.
4. Build your own rapid prototype.
5. Use the Wizard of Oz method.

Use a Survey Instead

Not all research questions at this phase need a full working prototype to be answered, particularly if they are about the general product features or product wording and not the visual design. Oftentimes, a survey tool or similar type of evaluation can be just as effective.

For example, you can confirm a site’s navigation options are intuitive by describing a scenario with a list of navigation choices while also testing if key content is understandable by confirming the user’s next steps based on a passage of text.

Just be sure you build a WCAG-compliant survey that includes accessible form layouts and question types. This will ensure participants can navigate using their assistive technologies. For example, Qualtrics has a specific form layout that is built to be accessible, or check out these accessibility tips for Google Forms. If sharing a document, note features that will enhance accessibility, such as using the ribbon for styling in Microsoft Word.

Tip: To find accessibility documentation for the software you’re using, search in your favorite search engine for the product name plus the word “accessibility” to find a product’s accessibility documentation.

Conduct Co-design Sessions

The prototyping phase might be a good time to utilize co-design and participatory design methods. With these methods, you can co-create designs with participants using any variety of artifacts that match the capabilities of your participants along with your research goals. The feedback can range from high-level workflows to specific visual designs, and you can guide the conversation with mock-ups, equivalent systems, or more creative artifacts such as storyboards that illustrate a scenario for user reaction.

For the prototype artifacts, these can range from low- to high-fidelity. For instance, participants without mobility or vision impairments can use paper-and-pencil sketching or whiteboarding. People with somewhat limited mobility may prefer a tablet-based drawing tool, such as using an Apple pencil with an iPad. Participants with visual impairments may prefer more 3-dimensional tools such as craft supplies, modeling clay, and/or cardboard. Or you may find that simply working on a collaborative online document offers the best accessibility as users can engage with their personalized assistive technology to jot down ideas.

Notably, the types of artifacts you use will be beneficial across differing user groups. In fact, rather than limiting the artifacts, try to offer a variety of ways to provide feedback by default. By doing this, participants can feel more empowered and engaged by the activity while also reassuring them you have created an inclusive environment. If you’re not sure what options to include, feel free to confirm what methods will work best as you recruit participants. That is, as you describe the primary activity when they are signing up, you can ask if the materials you have will be operable for the participant or allow them to tell you what they prefer to use.

The discussion you have and any supplemental artifacts you use then depend on communication styles. For example, deaf participants may need sign language interpreters to communicate their views but will be able to see sample systems, while blind participants will need descriptions of key visual information to give feedback. The actual study facilitation comes down to who you are recruiting and what level of feedback you are seeking; from there, you can work through the accommodations that will work best.

I conducted two co-design sessions at two different project phases while exploring how to create a wearable blind pedestrian navigation device. Early in the project, when we were generally talking about the feature set, we brought in several low-fidelity supplies, including a Braille label maker, cardboard, clay, Velcro, clipboards, tape, paper, and pipe cleaners. Based on user feedback, I fashioned a clipboard hanging from pipe cleaners as one prototype.

Later in the project when we were discussing the size and weight, we taped together Arduino hardware pieces representing the features identified by the participants. Both outcomes are pictured below and featured in a paper entitled, “What Not to Wearable: Using Participatory Workshops to Explore Wearable Device Form Factors for Blind Users.”

Ultimately, the benefit of this type of study is the participant-led feedback. In this way, participants are giving unfiltered feedback that is less influenced by designers, which may lead to more thoughtful design in the end.

Test With an Equivalent System

Very few projects are completely new creations, and often, teams use an existing site or application for project inspiration. Consider using similar existing systems and equivalent scenarios for your testing instead of creating a prototype.

By using an existing live system, participants can then use their assistive technology and adaptive techniques, which can make the study more accessible and authentic. Also, the study findings can range from the desirability of the available product features to the accessibility and usability of individual page elements. These lessons can then inform what design and code decisions to make in your system.

One caveat is to be aware of any accessibility barriers in that existing system. Particularly for website and web applications, you can look for accessibility documentation to determine if the company has reported any WCAG-conformance accessibility efforts, use tools like WAVE to test the system yourself, and/or mimic how your participants will use the system with their assistive technology. If there are workarounds for what you find, you may be able to avoid certain parts of the application or help users navigate past the inaccessible parts. However, if the site is going to be completely unusable for your participants, this won’t be a viable option for you.

If the system is usable enough for your testing, however, you can take the testing a step further by making updates on the fly if you or someone you collaborate with has engineering experience. For example, you can manipulate a website’s code with developer tools to add, subtract, or change the elements and styling on a page in real-time. (See “About browser developer tools”.) This can further enhance the feedback you give to your teams as it may more closely match your team’s intended design.

Build a Rapid Website Prototype

Notably, when conducting research focused on physical devices and hardware, you will not face the same obstacles to inaccessibility as with websites and web applications. You can use a variety of materials to create your prototypes, from cardboard to fabric to 3D printed material. I’ve sewn haptic vibration modules to a makeshift leather bracelet when working with wearables, for instance.

However, for web testing, it may be necessary to build a rapid prototype, especially to work around inaccessible artifacts such as a Figma file. This will include using a site builder that allows you to quickly create a replica of your team’s website. To create an accessible website, you’ll need a site builder with accessibility features and capabilities; I recommend WordPress, SquareSpace, Webflow, and Google Sites.

I recently used Google Sites to create a replica of a client’s draft pages in a matter of hours. I was adamant we should include disabled participants in feedback loops early and often, and this included after a round of significant visual design and content decisions. The web agency building the client’s site used Figma but not with the required formatting to use the built-in screen reader functionality. Rather than leave out blind user feedback at such a crucial time in the project, I started with a similar Google Sites template, took a best guess at how to structure the elements such as headings, recreated the anticipated column and card layouts as best I could, and used placeholder images with projected alt text instead of their custom graphics.

The screen reader testing turned into an impromptu co-design session because I could make changes in-the-moment to the live site for the participant to immediately test out. For example, we determined that some places where I used headings were not necessary, and we talked about image alt text in detail. I was able to add specific design and code feedback to my report, as well as share the live site (and corresponding code) with the team for comparison.

The downside to my prototype was that I couldn’t create the exact 1-to-1 visual design to use when testing with the other disabled participants who were sighted. I wanted to gather feedback on colors, fonts, and wording, so I also recruited low vision and neurodiverse participants for the study. However, my data was skewed because those participants couldn’t make the visual adjustments they needed to fully take in the content, such as recoloring, resizing, and having text read aloud. This was unfortunate, but we at least used the prototype to spark discussions of what does make a page accessible for them.

You may find you are limited in how closely you can replicate the design based on the tools you use or lack of access to developer assistance. When facing these limitations, consider what is most important to evaluate and determine if a paired-down version of the site will still give you valuable feedback over no site at all.

Use Wizard of Oz

The Wizard of Oz (WoZ) research method involves the facilitators mimicking system interactions in place of a fully working system. With WoZ, you can create your system’s approximate functionality using equivalent accessible tools and processes.

As an example, I’ll refer you to the talk by an Ally Financial research team that used this method for participants who used screen readers. They pre-programmed screen reader prompts into a clickable spreadsheet and had participants describe aloud what keyboard actions they would take to then trigger the corresponding prompt. While not the ideal set-up for the participants or researchers, it at least brought screen reader user feedback (and recognition of the users themselves) to the early design phases of their work. For more, review their detailed talk “Removing bias with wizard of oz screen reader usability testing”.

This isn’t just limited to screen reader testing, however. In fact, I’ve also often used Wizard of Oz for Voice User Interface (VUI) design. For instance, when I helped create an Alexa “skill” (their name for an app on Amazon speech-enabled devices), our prototype wouldn’t be ready in time for user testing. So, I drafted an idea to use a Bluetooth speaker to announce prompts from a clickable spreadsheet instead. When participants spoke a command to the speaker (thinking it was an Alexa device), the facilitator would select the appropriate pre-recorded prompt or a generic “I don’t understand” message.

Any system can be mimicked when you break down its parts and pieces and think about the ultimate interaction for the user. Creating WoZ set-ups can take creativity and even significant time to put together, but the outcomes can be worth it, particularly for longer-term projects. Once the main pieces are created, the prototype set-up can be edited and reused indefinitely, including during the study or between participants. Also, the investment in an easily edited prototype pays off exponentially if it uncovers something prior to finishing the entire product. In fact, that’s the main goal of this phase of testing: to help teams know what to look out for before they go through the hard work of finishing the product.

Inclusive Research Can No Longer Wait

Much has been documented about inclusive design to help teams craft technology for the widest possible audience. From the Web Content Accessibility Guidelines that help define what it means to be accessible to the Microsoft Inclusive Design Toolkits that tell the human stories behind the guidelines, there is much to learn even before a product begins.

However, the best approach is with direct user feedback. With this, we must recognize the conundrum many researchers are facing: We want to include disabled participants in UX research prior to a product being complete, but often, prototypes we have available for testing are inaccessible. This means testing with something that is essentially broken and will negatively impact our findings.

While it may feel like researchers will always be at a disadvantage if we don’t have the tools we need for testing, I think, instead, it’s time for us to push back. I propose we do this on two fronts:

1. We make the research work as best we can in the current state.
2. We advocate for the tools we need to make this more streamlined.

The key is to get disabled perspectives on the record and in the dataset of team members making the decisions. By doing this, hopefully, we shift the culture to wanting and valuing this feedback and bringing awareness to what it takes to make it happen.

Ideally, the awareness raised from our bootstrap efforts will lead to more people helping reduce the current prototype barriers. For some of us, this means urging companies to prioritize accessibility features in their roadmaps. For those working within influential prototype companies, it can mean getting much-needed backing to innovate better in this area.

The current state of our inaccessible digital ecosystem can sometimes feel like an entanglement too big to unravel. However, we must remain steadfast and insist that this does not remain the status quo; disabled users are users, and their diverse and invaluable perspectives must be a part of our research outcomes at all phases.

(yk)

Communication is in everything we do. We communicate with users through our research, our design, and, ultimately, the products and services we offer. UX practitioners and those working on digital product teams benefit from understanding principles of communication and their application to our craft. Treating our UX processes as a mode of communication between users and the digital environment can help unveil in-depth, actionable insights.

In this article, I’ll focus on UX research. Communication is a core component of UX research, as it serves to bridge the gap between research insights, design strategy, and business outcomes. UX researchers, designers, and those working with UX researchers can apply key aspects of communication theory to help gather valuable insights, enhance user experiences, and create more successful products.

Fundamentals of Communication Theory

Communications as an academic field encompasses various models and principles that highlight the dynamics of communication between individuals and groups. Communication theory examines the transfer of information from one person or group to another. It explores how messages are transmitted, encoded, and decoded, acknowledges the potential for interference (or ‘noise’), and accounts for feedback mechanisms in enhancing the communication process.

In this article, I will focus on the Transactional Model of Communication. There are many other models and theories in the academic literature on communication. I have included references at the end of the article for those interested in learning more.

The Transactional Model of Communication (Figure 1) is a two-way process that emphasizes the simultaneous sending and receiving of messages and feedback. Importantly, it recognizes that communication is shaped by context and is an ongoing, evolving process. I’ll use this model and understanding when applying principles from the model to UX research. You’ll find that much of what is covered in the Transactional Model would also fall under general best practices for UX research, suggesting even if we aren’t communications experts, much of what we should be doing is supported by research in this field.

Understanding the Transactional Model

Let’s take a deeper dive into the six key factors and their applications within the realm of UX research:

1. Sender: In UX research, the sender is typically the researcher who conducts interviews, facilitates usability tests, or designs surveys. For example, if you’re administering a user interview, you are the sender who initiates the communication process by asking questions.
2. Receiver: The receiver is the individual who decodes and interprets the messages sent by the sender. In our context, this could be the user you interview or the person taking a survey you have created. They receive and process your questions, providing responses based on their understanding and experiences.
3. Message: This is the content being communicated from the sender to the receiver. In UX research, the message can take various forms, like a set of survey questions, interview prompts, or tasks in a usability test.
4. Channel: This is the medium through which the communication flows. For instance, face-to-face interviews, phone interviews, email surveys administered online, and usability tests conducted via screen sharing are all different communication channels. You might use multiple channels simultaneously, for example, communicating over voice while also using a screen share to show design concepts.
5. Noise: Any factor that may interfere with the communication is regarded as ‘noise.’ In UX research, this could be complex jargon that confuses respondents in a survey, technical issues during a remote usability test, or environmental distractions during an in-person interview.
6. Feedback: The communication received by the receiver, who then provides an output, is called feedback. For example, the responses given by a user during an interview or the data collected from a completed survey are types of feedback or the physical reaction of a usability testing participant while completing a task.
   

Applying the Transactional Model of Communication to Preparing for UX Research

We can become complacent or feel rushed to create our research protocols. I think this is natural in the pace of many workplaces and our need to deliver results quickly. You can apply the lens of the Transactional Model of Communication to your research preparation without adding much time. Applying the Transactional Model of Communication to your preparation should:

• Improve Clarity
  The model provides a clear representation of communication, empowering the researcher to plan and conduct studies more effectively.
• Minimize misunderstanding
  By highlighting potential noise sources, user confusion or misunderstandings can be better anticipated and mitigated.
• Enhance research participant participation
  With your attentive eye on feedback, participants are likely to feel valued, thus increasing active involvement and quality of input.

You can address the specific elements of the Transactional Model through the following steps while preparing for research:

Defining the Sender and Receiver

In UX research, the sender can often be the UX researcher conducting the study, while the receiver is usually the research participant. Understanding this dynamic can help researchers craft questions or tasks more empathetically and efficiently. You should try to collect some information on your participant in advance to prepare yourself for building a rapport.

For example, if you are conducting contextual inquiry with the field technicians of an HVAC company, you’ll want to dress appropriately to reflect your understanding of the context in which your participants (receivers) will be conducting their work. Showing up dressed in formal attire might be off-putting and create a negative dynamic between sender and receiver.

Message Creation

The message in UX research typically is the questions asked or tasks assigned during the study. Careful consideration of tenor, terminology, and clarity can aid data accuracy and participant engagement. Whether you are interviewing or creating a survey, you need to double-check that your audience will understand your questions and provide meaningful answers. You can pilot-test your protocol or questionnaire with a few representative individuals to identify areas that might cause confusion.

Using the HVAC example again, you might find that field technicians use certain terminology in a different way than you expect, such as asking them about what “tools” they use to complete their tasks yields you an answer that doesn’t reflect digital tools you’d find on a computer or smartphone, but physical tools like a pipe and wrench.

Choosing the Right Channel

The channel selection depends on the method of research. For instance, face-to-face methods might use physical verbal communication, while remote methods might rely on emails, video calls, or instant messaging. The choice of the medium should consider factors like tech accessibility, ease of communication, reliability, and participant familiarity with the channel. For example, you introduce an additional challenge (noise) if you ask someone who has never used an iPhone to test an app on an iPhone.

Minimizing Noise

Noise in UX research comes in many forms, from unclear questions inducing participant confusion to technical issues in remote interviews that cause interruptions. The key is to foresee potential issues and have preemptive solutions ready.

Facilitating Feedback

You should be prepared for how you might collect and act on participant feedback during the research. Encouraging regular feedback from the user during UX research ensures their understanding and that they feel heard. This could range from asking them to ‘think aloud’ as they perform tasks or encouraging them to email queries or concerns after the session. You should document any noise that might impact your findings and account for that in your analysis and reporting.

Track Your Alignment to the Framework

You can track what you do to align your processes with the Transactional Model prior to and during research using a spreadsheet. I’ll provide an example of a spreadsheet I’ve used in the later case study section of this article. You should create your spreadsheet during the process of preparing for research, as some of what you do to prepare should align with the factors of the model.

You can use these tips for preparation regardless of the specific research method you are undertaking. Let’s now look closer at a few common methods and get specific on how you can align your actions with the Transactional Model.

Applying the Transactional Model to Common UX Research Methods

UX research relies on interaction with users. We can easily incorporate aspects of the Transactional Model of Communication into our most common methods. Utilizing the Transactional Model in conducting interviews, surveys, and usability testing can help provide structure to your process and increase the quality of insights gathered.

Interviews

Interviews are a common method used in qualitative UX research. They provide the perfect method for applying principles from the Transactional Model. In line with the Transactional Model, the researcher (sender) sends questions (messages) in-person or over the phone/computer medium (channel) to the participant (receiver), who provides answers (feedback) while contending with potential distraction or misunderstanding (noise). Reflecting on communication as transactional can help remind us we need to respect the dynamic between ourselves and the person we are interviewing. Rather than approaching an interview as a unidirectional interrogation, researchers need to view it as a conversation.

Applying the Transactional Model to conducting interviews means we should account for a number of facts to allow for high-quality communication. Note how the following overlap with what we typically call best practices.

Asking Open-ended Questions

To truly harness a two-way flow of communication, open-ended questions, rather than close-ended ones, are crucial. For instance, rather than asking, “Do you use our mobile application?” ask, “Can you describe your use of our mobile app?”. This encourages the participant to share more expansive and descriptive insights, furthering the dialogue.

Actively Listening

As the success of an interview relies on the participant’s responses, active listening is a crucial skill for UX researchers. The researcher should encourage participants to express their thoughts and feelings freely. Reflective listening techniques, such as paraphrasing or summarizing what the participant has shared, can reinforce to the interviewee that their contributions are being acknowledged and valued. It also provides an opportunity to clarify potential noise or misunderstandings that may arise.

Being Responsive

Building on the simultaneous send-receive nature of the Transactional Model, researchers must remain responsive during interviews. Providing non-verbal cues (like nodding) and verbal affirmations (“I see,” “Interesting”) lets participants know their message is being received and understood, making them feel comfortable and more willing to share.

Minimizing Noise

We should always attempt to account for noise in advance, as well as during our interview sessions. Noise, in the form of misinterpretations or distractions, can disrupt effective communication. Researchers can proactively reduce noise by conducting a dry run in advance of the scheduled interviews. This helps you become more fluent at going through the interview and also helps identify areas that might need improvement or be misunderstood by participants. You also reduce noise by creating a conducive interview environment, minimizing potential distractions, and asking clarifying questions during the interview whenever necessary.

For example, if a participant uses a term the researcher doesn’t understand, the researcher should politely ask for clarification rather than guessing its meaning and potentially misinterpreting the data.

Additional forms of noise can include participant confusion or distraction. You should let participants know to ask if they are unclear on anything you say or do. It’s a good idea to always ask participants to put their smartphones on mute. You should only provide information critical to the process when introducing the interview or tasks. For example, you don’t need to give a full background of the history of the product you are researching if that isn’t required for the participant to complete the interview. However, you should let them know the purpose of the research, gain their consent to participate, and inform them of how long you expect the session to last.

Strategizing the Flow

Researchers should build strategic thinking into their interviews to support the Transaction Model. Starting the interview with less intrusive questions can help establish rapport and make the participant more comfortable, while more challenging or sensitive questions can be left for later when the interviewee feels more at ease.

A well-planned interview encourages a fluid dialogue and exchange of ideas. This is another area where conducting a dry run can help to ensure high-quality research. You and your dry-run participants should recognize areas where questions aren’t flowing in the best order or don’t make sense in the context of the interview, allowing you to correct the flow in advance.

While much of what the Transactional Model informs for interviews already aligns with common best practices, the model would suggest we need to have a deeper consideration of factors that we can sometimes give less consideration when we become overly comfortable with interviewing or are unaware of the implications of forgetting to address the factors of context considerations, power dynamics, and post-interview actions.

Context Considerations

You need to account for both the context of the participant, e.g., their background, demographic, and psychographic information, as well as the context of the interview itself. You should make subtle yet meaningful modifications depending on the channel you are conducting an interview.

For example, you should utilize video and be aware of your facial and physical responses if you are conducting an interview using an online platform, whereas if it’s a phone interview, you will need to rely on verbal affirmations that you are listening and following along, while also being mindful not to interrupt the participant while they are speaking.

Power Dynamics

Researchers need to be aware of how your role, background, and identity might influence the power dynamics of the interview. You can attempt to address power dynamics by sharing research goals transparently and addressing any potential concerns about bias a participant shares.

We are responsible for creating a safe and inclusive space for our interviews. You do this through the use of inclusive language, listening actively without judgment, and being flexible to accommodate different ways of knowing and expressing experiences. You should also empower participants as collaborators whenever possible. You can offer opportunities for participants to share feedback on the interview process and analysis. Doing this validates participants’ experiences and knowledge and ensures their voices are heard and valued.

Post-Interview Actions

You have a number of options for actions that can close the loop of your interviews with participants in line with the “feedback” the model suggests is a critical part of communication. Some tactics you can consider following your interview include:

• Debriefing
  Dedicate a few minutes at the end to discuss the participant’s overall experience, impressions, and suggestions for future interviews.
• Short surveys
  Send a brief survey via email or an online platform to gather feedback on the interview experience.
• Follow-up calls
  Consider follow-up calls with specific participants to delve deeper into their feedback and gain additional insight if you find that is warranted.
• Thank you emails
  Include a “feedback” section in your thank you email, encouraging participants to share their thoughts on the interview.

You also need to do something with the feedback you receive. Researchers and product teams should make time for reflexivity and critical self-awareness.

As practitioners in a human-focused field, we are expected to continuously examine how our assumptions and biases might influence our interviews and findings.

“

We shouldn’t practice our craft in a silo. Instead, seeking feedback from colleagues and mentors to maintain ethical research practices should be a standard practice for interviews and all UX research methods.

By considering interviews as an ongoing transaction and exchange of ideas rather than a unidirectional Q&A, UX researchers can create a more communicative and engaging environment. You can see how models of communication have informed best practices for interviews. With a better knowledge of the Transactional Model, you can go deeper and check your work against the framework of the model.

Surveys

The Transactional Model of Communication reminds us to acknowledge the feedback loop even in seemingly one-way communication methods like surveys. Instead of merely sending out questions and collecting responses, we need to provide space for respondents to voice their thoughts and opinions freely. When we make participants feel heard, engagement with our surveys should increase, dropouts should decrease, and response quality should improve.

Like other methods, surveys involve the researcher(s) creating the instructions and questionnaire (sender), the survey, including any instructions, disclaimers, and consent forms (the message), how the survey is administered, e.g., online, in person, or pen and paper (the channel), the participant (receiver), potential misunderstandings or distractions (noise), and responses (feedback).

Designing the Survey

Understanding the Transactional Model will help researchers design more effective surveys. Researchers are encouraged to be aware of both their role as the sender and to anticipate the participant’s perspective as the receiver. Begin surveys with clear instructions, explaining why you’re conducting the survey and how long it’s estimated to take. This establishes a more communicative relationship with respondents right from the start. Test these instructions with multiple people prior to launching the survey.

Crafting Questions

The questions should be crafted to encourage feedback and not just a simple yes or no. You should consider asking scaled questions or items that have been statistically validated to measure certain attributes of users.

For example, if you were looking deeper at a mobile banking application, rather than asking, “Did you find our product easy to use?” you would want to break that out into multiple aspects of the experience and ask about each with a separate question such as “On a scale of 1–7, with 1 being extremely difficult and 7 being extremely easy, how would you rate your experience transferring money from one account to another?”.

Minimizing Noise

Reducing ‘noise,’ or misunderstandings, is crucial for increasing the reliability of responses. Your first line of defense in reducing noise is to make sure you are sampling from the appropriate population you want to conduct the research with. You need to use a screener that will filter out non-viable participants prior to including them in the survey. You do this when you correctly identify the characteristics of the population you want to sample from and then exclude those falling outside of those parameters.

Additionally, you should focus on prioritizing finding participants through random sampling from the population of potential participants versus using a convenience sample, as this helps to ensure you are collecting reliable data.

When looking at the survey itself, there are a number of recommendations to reduce noise. You should ensure questions are easily understandable, avoid technical jargon, and sequence questions logically. A question bank should be reviewed and tested before being finalized for distribution.

For example, question statements like “Do you use and like this feature?” can confuse respondents because they are actually two separate questions: do you use the feature, and do you like the feature? You should separate out questions like this into more than one question.

You should use visual aids that are relevant whenever possible to enhance the clarity of the questions. For example, if you are asking questions about an application’s “Dashboard” screen, you might want to provide a screenshot of that page so survey takers have a clear understanding of what you are referencing. You should also avoid the use of jargon if you are surveying a non-technical population and explain any terminology that might be unclear to participants taking the survey.

The Transactional Model suggests active participation in communication is necessary for effective communication. Participants can become distracted or take a survey without intending to provide thoughtful answers. You should consider adding a question somewhere in the middle of the survey to check that participants are paying attention and responding appropriately, particularly for longer surveys.

This is often done using a simple math problem such as “What is the answer to 1+1?” Anyone not responding with the answer of “2” might not be adequately paying attention to the responses they are providing and you’d want to look closer at their responses, eliminating them from your analysis if deemed appropriate.

Encouraging Feedback

While descriptive feedback questions are one way of promoting dialogue, you can also include areas where respondents can express any additional thoughts or questions they have outside of the set question list. This is especially useful in online surveys, where researchers can’t immediately address participant’s questions or clarify doubts.

You should be mindful that too many open-ended questions can cause fatigue, so you should limit the number of open-ended questions. I recommend two to three open-ended questions depending on the length of your overall survey.

Post-Survey Actions

After collecting and analyzing the data, you can send follow-up communications to the respondents. Let them know the changes made based on their feedback, thank them for their participation, or even share a summary of the survey results. This fulfills the Transactional Model’s feedback loop and communicates to the respondent that their input was received, valued, and acted upon.

You can also meet this suggestion by providing an email address for participants to follow up if they desire more information post-survey. You are allowing them to complete the loop themselves if they desire.

Applying the transactional model to surveys can breathe new life into the way surveys are conducted in UX research. It encourages active participation from respondents, making the process more interactive and engaging while enhancing the quality of the data collected. You can experiment with applying some or all of the steps listed above. You will likely find you are already doing much of what’s mentioned, however being explicit can allow you to make sure you are thoughtfully applying these principles from the field communication.

Usability Testing

Usability testing is another clear example of a research method highlighting components of the Transactional Model. In the context of usability testing, the Transactional Model of Communication’s application opens a pathway for a richer understanding of the user experience by positioning both the user and the researcher as sender and receiver of communication simultaneously.

Here are some ways a researcher can use elements of the Transactional Model during usability testing:

Task Assignment as Message Sending

When a researcher assigns tasks to a user during usability testing, they act as the sender in the communication process. To ensure the user accurately receives the message, these tasks need to be clear and well-articulated. For example, a task like “Register a new account on the app” sends a clear message to the user about what they need to do.

You don’t need to tell them how to do the task, as usually, that’s what we are trying to determine from our testing, but if you are not clear on what you want them to do, your message will not resonate in the way it is intended. This is another area where a dry run in advance of the testing is an optimal solution for making sure tasks are worded clearly.

Observing and Listening as Message Receiving

As the participant interacts with the application, concept, or design, the researcher, as the receiver, picks up on verbal and nonverbal cues. For instance, if a user is clicking around aimlessly or murmuring in confusion, the researcher can take these as feedback about certain elements of the design that are unclear or hard to use. You can also ask the user to explain why they are giving these cues you note as a way to provide them with feedback on their communication.

Real-time Interaction

The transactional nature of the model recognizes the importance of real-time interaction. For example, if during testing, the user is unsure of what a task means or how to proceed, the researcher can provide clarification without offering solutions or influencing the user’s action. This interaction follows the communication flow prescribed by the transactional model. We lose the ability to do this during unmoderated testing; however, many design elements are forms of communication that can serve to direct users or clarify the purpose of an experience (to be covered more in article two).

Noise

In usability testing, noise could mean unclear tasks, users’ preconceived notions, or even issues like slow software response. Acknowledging noise can help researchers plan and conduct tests better. Again, carrying out a pilot test can help identify any noise in the main test scenarios, allowing for necessary tweaks before actual testing. Other forms of noise can be less obvious but equally intrusive. For example, if you are conducting a test using a Macbook laptop and your participant is used to a PC, there is noise you need to account for, given their unfamiliarity with the laptop you’ve provided.

The fidelity of the design artifact being tested might introduce another form of noise. I’ve always advocated testing at any level of fidelity, but you should note that if you are using “Lorem Ipsum” or black and white designs, this potentially adds noise.

One of my favorite examples of this was a time when I was testing a financial services application, and the designers had put different balances on the screen; however, the total for all balances had not been added up to the correct total. Virtually every person tested noted this discrepancy, although it had nothing to do with the tasks at hand. I had to acknowledge we’d introduced noise to the testing. As at least one participant noted, they wouldn’t trust a tool that wasn’t able to total balances correctly.

Encouraging Feedback

Under the Transactional Model’s guidance, feedback isn’t just final thoughts after testing; it should be facilitated at each step of the process. Encouraging ‘think aloud’ protocols, where the user verbalizes their thoughts, reactions, and feelings during testing, ensures a constant flow of useful feedback.

You are receiving feedback throughout the process of usability testing, and the model provides guidance on how you should use that feedback to create a shared meaning with the participants. You will ultimately summarize this meaning in your report. You’ll later end up uncovering if this shared meaning was correctly interpreted when you design or redesign the product based on your findings.

We’ve now covered how to apply the Transactional Model of Communication to three common UX Research methods. All research with humans involves communication. You can break down other UX methods using the Model’s factors to make sure you engage in high-quality research.

Analyzing and Reporting UX Research Data Through the Lens of the Transactional Model

The Transactional Model of Communication doesn’t only apply to the data collection phase (interviews, surveys, or usability testing) of UX research. Its principles can provide valuable insights during the data analysis process.

The Transactional Model instructs us to view any communication as an interactive, multi-layered dialogue — a concept that is particularly useful when unpacking user responses. Consider the ‘message’ components: In the context of data analysis, the messages are the users’ responses. As researchers, thinking critically about how respondents may have internally processed the survey questions, interview discussion, or usability tasks can yield richer insights into user motivations.

Understanding Context

Just as the Transactional Model emphasizes the simultaneous interchange of communication, UX researchers should consider the user’s context while interpreting data. Decoding the meaning behind a user’s words or actions involves understanding their background, experiences, and the situation when they provide responses.

Deciphering Noise

In the Transactional Model, noise presents a potential barrier to effective communication. Similarly, researchers must be aware of snowballing themes or frequently highlighted issues during analysis. Noise, in this context, could involve patterns of confusion, misunderstandings, or consistently highlighted problems by users. You need to account for this, e.g., the example I provided where participants constantly referred to the incorrect math on static wireframes.

Considering Sender-Receiver Dynamics

Remember that as a UX researcher, your interpretation of user responses will be influenced by your understandings, biases, or preconceptions, just as the responses were influenced by the user’s perceptions. By acknowledging this, researchers can strive to neutralize any subjective influence and ensure the analysis remains centered on the user’s perspective. You can ask other researchers to double-check your work to attempt to account for bias.

For example, if you come up with a clear theme that users need better guidance in the application you are testing, another researcher from outside of the project should come to a similar conclusion if they view the data; if not, you should have a conversation with them to determine what different perspectives you are each bringing to the data analysis.

Reporting Results

Understanding your audience is crucial for delivering a persuasive UX research presentation. Tailoring your communication to resonate with the specific concerns and interests of your stakeholders can significantly enhance the impact of your findings. Here are some more details:

• Identify Stakeholder Groups
  Identify the different groups of stakeholders who will be present in your audience. This could include designers, developers, product managers, and executives.
• Prioritize Information
  Prioritize the information based on what matters most to each stakeholder group. For example, designers might be more interested in usability issues, while executives may prioritize business impact.
• Adapt Communication Style
  Adjust your communication style to align with the communication preferences of each group. Provide technical details for developers and emphasize user experience benefits for executives.

Acknowledging Feedback

Respecting this Transactional Model’s feedback loop, remember to revisit user insights after implementing design changes. This ensures you stay user-focused, continuously validating or adjusting your interpretations based on users’ evolving feedback. You can do this in a number of ways. You can reconnect with users to show them updated designs and ask questions to see if the issues you attempted to resolve were resolved.

Another way to address this without having to reconnect with the users is to create a spreadsheet or other document to track all the recommendations that were made and reconcile the changes with what is then updated in the design. You should be able to map the changes users requested to updates or additions to the product roadmap for future updates. This acknowledges that users were heard and that an attempt to address their pain points will be documented.

Crucially, the Transactional Model teaches us that communication is rarely simple or one-dimensional. It encourages UX researchers to take a more nuanced, context-aware approach to data analysis, resulting in deeper user understanding and more accurate, user-validated results.

By maintaining an ongoing feedback loop with users and continually refining interpretations, researchers can ensure that their work remains grounded in real user experiences and needs.

“

Tracking Your Application of the Transactional Model to Your Practice

You might find it useful to track how you align your research planning and execution to the framework of the Transactional Model. I’ve created a spreadsheet to outline key factors of the model and used this for some of my work. Demonstrated below is an example derived from a study conducted for a banking client that included interviews and usability testing. I completed this spreadsheet during the process of planning and conducting interviews. Anonymized data from our study has been furnished to show an example of how you might populate a similar spreadsheet with your information.

You can customize the spreadsheet structure to fit your specific research topic and interview approach. By documenting your application of the transactional model, you can gain valuable insights into the dynamic nature of communication and improve your interview skills for future research.

You can use the suggested columns from this table as you see fit, adding or subtracting as needed, particularly if you use a method other than interviews. I usually add the following additional Columns for logistical purposes:

• Date of Interview,
• Participant ID,
• Interview Format (e.g., in person, remote, video, phone).

Conclusion

By incorporating aspects of communication theory into UX research, UX researchers and those who work with UX researchers can enhance the effectiveness of their communication strategies, gather more accurate insights, and create better user experiences. Communication theory provides a framework for understanding the dynamics of communication, and its application to UX research enables researchers to tailor their approaches to specific audiences, employ effective interviewing techniques, design surveys and questionnaires, establish seamless communication channels during usability testing, and interpret data more effectively.

As the field of UX research continues to evolve, integrating communication theory into research practices will become increasingly essential for bridging the gap between users and design teams, ultimately leading to more successful products that resonate with target audiences.

As a UX professional, it is important to continually explore and integrate new theories and methodologies to enhance your practice. By leveraging communication theory principles, you can better understand user needs, improve the user experience, and drive successful outcomes for digital products and services.

Integrating communication theory into UX research is an ongoing journey of learning and implementing best practices. Embracing this approach empowers researchers to effectively communicate their findings to stakeholders and foster collaborative decision-making, ultimately driving positive user experiences and successful design outcomes.

References and Further Reading

• The Mathematical Theory of Communication (PDF), Shannon, C. E., & Weaver, W.
• From organizational effectiveness to relationship indicators: Antecedents of relationships, public relations strategies, and relationship outcomes, Grunig, J. E., & Huang, Y. H.
• Communication and persuasion: Psychological studies of opinion change, Hovland, C. I., Janis, I. L., & Kelley, H. H. (1953). Yale University Press
• Communication research as an autonomous discipline, Chaffee, S. H. (1986). Communication Yearbook, 10, 243-274
• Interpersonal Communication: Everyday Encounters (PDF), Wood, J. (2015)
• Theories of Human Communication, Littlejohn, S. W., & Foss, K. A. (2011)
• McQuail’s Mass Communication Theory (PDF), McQuail, D. (2010)
• Bridges Not Walls: A Book About Interpersonal Communication, Stewart, J. (2012)

(yk)

In today’s world of shortening attention spans and omnipresent hustle, wasting even a second could mean losing the chance to earn more time from a person you want to impress. If your interests lie in creating good user experiences, there is a fair chance you have heard of five-second testing.

Five-second testing is an established technique of usability research used by UX researchers, designers, product managers, and in a variety of other professions, such as marketing or business analysis.

In short, you show a picture of whatever you are designing (site, app, pair of socks) to a member of your intended audience for exactly five seconds. Then, you hide the picture and ask the participant a couple of questions. The goal is to learn whether the reaction — the participants’s first impression — is what you wanted to see. Did you get the main message across? Do people remember the company’s name? Sounds like an efficient way to test your product without needing to turn to full-on usability testing, right?

Note: The word “participant” in this article is used to refer to users involved in five-second testing or related usability research methods. The word “user” is used in more general contexts since users form first impressions all the time, not just when you are testing it.

Why is it five seconds exactly, though? Are five seconds some magical moment when everything we see should become clear? And if it does not, does it automatically mean that a user experience is bad? Or are five seconds just the right amount of time for first impressions to brew in the user’s mind so that they’re neither undercooked nor overcooked?

These are some of the questions that we asked ourselves. Not satisfied with the answers written by others who covered the topic before us, we kept drilling and conducted an actual peer-reviewed scientific case study, exploring the hidden truths behind the testing of first impressions. The research paper examines the five-second test and discusses the results.

So strap in and read what science has to say about five-second testing. And then, what the implications are for you so that you can take practical advantage of this new knowledge to develop better first impressions of your services or products. But first, let us delve into what we know about five-second testing and its caveats so that you see the greater picture of the focal points of our investigation.

The Mythos Of Five Seconds And Its Gaps

You may be familiar with the well-known statistic that a website has about ten seconds to communicate its key message to the user. Knowing that waiting only five seconds to ask testing participants about their first impressions may suddenly seem like an odd choice. If indeed visitors of a website have about ten seconds to grasp a message, are five seconds really enough time for users? There is an alleged justification, as we explain below.

Cutting a bit forward, though, the factual basis for it is admittedly a bit of a Wild West if you look for hard data to support it. There is an almost uncanny resemblance to another not wholly scientific five-second rule that says it’s okay to eat food off the ground if it’s within five seconds from when it dropped there.

The five-second testing method has its origins as a simplification of usability testing. The first references to five-second testing point to Christine Perfetti, who coined the term for the method in the mid-2000s. The answer to “Why five seconds exactly?” comes largely from anecdotal evidence in the form of the experience of usability researchers.

The common story is that if something is shown to participants for more than five seconds, their first impressions will start to deviate from the actual user’s genuine initial impressions. The participant’s perspective becomes more analytical and less task-driven. The five-second test lets you avoid overtly speculative feedback that nobody would give you under normal circumstances.

Fair enough, that could potentially be true. But five seconds is still quite a short period of time. Consider how different people can be when it comes to their cognitive abilities (and there is nothing wrong with that). For example, one user’s sharp perception may let them realistically form first impressions in five seconds or faster, but another user may barely have the time to blink, much less absorb any meaningful information–they need a moment to take it in at their own tempo.

The reasoning starts to fall apart a bit more at its seams once you also consider the visual complexity of the stimulus (a.k.a., the picture you show to the participants). The nature of the things you may want to test can range from very simple to very complex. If the stimulus is simple, it is possible to take even less than five seconds for participants to form their initial impressions. Would this mean that there would be the risk of them using the remaining time to get over-exceedingly analytical?

Conversely, there is the question of whether five seconds is enough time to let participants realistically visually scan a more complex stimulus. I can already hear staunch proponents of five-second testing saying that this last discrepancy is actually rightfully intentional. It’s a feature, not a bug, if you will.

After all, if a stimulus is too complex, that is exactly why you conduct five-second testing. It allows us to find out about things like complexity. It can help you find out if participants cannot extract the key information you want to communicate so that you can fix it.

However, we need to consider that not all user interfaces are the landing pages of websites. They serve to support different user tasks, some of which cannot avoid having a certain degree of complexity.

Five-second testing guides typically avoid directly addressing testing of these types of user interfaces by saying that the method has the following limitation: it should not be used to test user interfaces with multiple purposes. If the same stimulus serves for more than one task, it is alleged that you should probably conduct full-fledged usability testing, which is technically correct (the best kind of correct).

Giving up on the five-second testing in the inherently more complex user interfaces, however, also gives up on its advantages for measuring and optimizing first impressions. For instance, the idea that a screenshot or a mockup is all you need to quickly find usability problems and iterate your designs. This is where five-second testing really shines.

Usability testing does not tell you accurately what the actual first impressions are without considerably interrupting the participant. And even then, you would encounter the same problem: At what moment from when the participant is exposed to a design should their first impressions be gauged so that they are genuine?

As we have discussed so far, there are certainly a fair number of question marks surrounding five-second testing. The method still undeniably has a number of merits, as proven by our experience at UXtweak, where we also provide our own Five Second Test tool. A lack of proper research on the topic is what drove us at UXtweak Research to conduct our very own case study.

The Science, Abridged

Essentially, what we sought to investigate in our case study are the relations of a number of key factors that are absolutely crucial for five-second testing:

• What are the cognitive abilities of the participant engaging in the five-second test?
• How visually complex is the stimulus shown to the participant?
• For how long is the stimulus shown to the participant?
• What kinds of questions do we ask the participants afterward?
• What is the feedback that participants give you?

As you may have noticed, time — that iconic yet controversial five-second threshold — is considered a variable factor. In our experiment, we investigate the differences in feedback between three separate groups of participants who are shown pictures for either five (5) or alternatively two (2) or (10) seconds (so a bit less and a bit more time, respectively). This means that it would not be correct to refer to it as just a five-second test anymore, but rather an N-second test (or a first impression test, if you do not wish to be too pedantic about the number of seconds).

Each participant first passes not just one but two cognitive ability tests. Human minds are multifaceted, and there is not just a single “cognitive ability” metric that would encompass everything that the mind can do. Among standard tests used by psychologists, we picked two that are linked to abilities that can be found as the most relevant to the formation and testing of first impressions:

• Perceptual speed: How quickly you pick up visual information.
• Working memory: How much information you can mentally process at the same time.

Working memory is the appropriate memory ability to focus on since it operates with information that receives the user’s attention. This distinguishes it from sensory memory (the memory processing information that our senses pick up) and long-term memory, where information is stored persistently for later use.

For the first impression test itself, six website screenshots were used as the stimuli. These screenshots were selected for possessing a broad range of visual complexity, from the simplest with just a few visual elements to the most complex with a number of distinct sections that serve different purposes.

Screenshots of real websites local to Czechia and Slovakia were translated into English, and their logos were replaced with fictional brand names so that, for all intents and purposes, the website screenshots would be authentic yet also unfamiliar to the participants who were recruited in the UK.

Finally, participants were asked to provide feedback by answering practically a complete portfolio of the various types of questions that can be typically asked during a first impression test. Each type of question tests a different aspect of the first impressions that the participants have formed inside their heads:

• Attitudinal questions: Rating a perceived quality of the website (e.g., ugly vs. attractive) on a scale from 1 to 7.
• Target identification questions: Questions directed at specific elements or aspects of the stimulus.
• Memory dump questions: Asking participants to describe everything that they remember about what they saw.

The resulting answers were analyzed both quantitatively (with statistics) and qualitatively (by inspecting the contents of the received answers on an individual level). With it, a number of conclusions can be reached, some expected and some rather surprising.

Now that you have a picture of what our case study was about let’s dive into the actual, interesting implications for developing the first impressions of your product.

Note: If you would like to immerse yourself in further details of how our case study was conducted, you can learn more in our scientific paper.

Takeaways

Statistically, all the variables we experimented with — the time duration of showing pictures to participants, the participants’ innate cognitive abilities, and the visual complexity of pictures — had a significant effect on the first impression answers.

For instance, between the groups that were shown screenshots of websites for two, five, and ten seconds, the number of answers that incorrectly identified what the websites were for dropped as time progressed. Notable is the difference between five and ten seconds. If the participants were really focusing on inconsequential details after five seconds, there should not be differences in recognition of such a key aspect as the website’s entire purpose.

Statistical differences lay the grounds for further observations on how changing the conditions of a test can (or cannot) affect its results:

• Attitudes crystallize faster than in five seconds.
  In attitudinal questions where participants are asked to rate how they view the picture’s various qualities (e.g., from clear to confusing, from captivating to dull), answers stay relatively consistent, regardless of how much time the participant has or how good their cognitive abilities are. If you are laser-focused on assessing participants’ attitudes about your product and nothing else, you could present pictures for two seconds, or possibly even less, as research done by others on a related topic also implies.
• Logos are recognized earlier than in five seconds (with one exception).
  The target identification questions where participants are asked to recall the company name from the logo are, on the whole, impacted by time very little. This is to be expected: when viewing a website, our eyes are usually drawn to the top left corner to find out where we have found ourselves. There is an exception to this rule, however.
  Among participants with slower perceptual speed, significantly fewer identified the company name correctly at two seconds when compared to five seconds. This establishes five seconds as a more inclusive choice for timing your first impression test if you expect your target audience to have, on average, lower perceptual speed than the general populace and if the primary aim is to test contents of the header, such as logo design or company name identification. Otherwise, two seconds is a safe bet.
• Irrelevant nitpicking? Yes, if visual complexity is low.
  In some cases, the popular narrative about five seconds being a good viewing time for testing first impressions is indeed true. Particularly for the simplest website screenshots, once five seconds have elapsed, participants start paying attention to minute details (e.g., the girl’s shirt color in the hero image).
  Curiously, though, having more time does not mean that participants would write longer or more complex answers. Instead, when participants have ten seconds to view the screenshots, the higher visual complexity of the screenshots is reflected in better-quality answers. Participants stay more on-topic–describing how the site is visually structured or justifying their criticisms of the page’s design. Different viewing times may be optimal in different situations. Especially since…
• Low working memory warrants longer viewing time.
  When asked to reiterate what they saw in their own words, participants with low and high working memory provided significantly different answers. With low working memory, answers become shorter, less complex, and recall fewer concepts overall. However, when the viewing time is extended to ten seconds, these differences disappear. This implies that the same information is being processed — memory capacity just dictates how fast it can happen.
  Without knowing where each participant’s memory ability stands, it is difficult to tell what they would actually recall if we left them to work at their own pace. Consequently, assessment of working memory before testing first impressions (and adjusting viewing time accordingly) should be considered a good practice.
• For cognitive powerhouses, five seconds are enough.
  A less practical point maybe, but if you are developing an app for people with reasonably high perceptual speed and working memory — be it the mentally gifted, hyperproductive hustle enthusiasts, or caffeine addicts — you could likely show them your screenshots for just two seconds and get similar results as in a five-second test.
• Give participants the proper amount of time to form a first impression.
  When the visual stimulus is more visually complex in a first impression test, the task of mentally processing it becomes more difficult and time-consuming (just like in any normal scenario). This manifests in test results. Fewer people correctly identify the purpose of a more visually complex website, and they recall fewer elements and aspects of the website.
  This could be seen as a bit of a paradox since more complex stimuli mean there is actually more content that participants could potentially remember and comment on, but only if they had the time to absorb the information properly. Data shows that when participants are given ten seconds, the answers do actually normalize, becoming more similar to stimuli of lower visual complexity.

If the purpose of the particular first impression test is not to remove all visual complexity at any cost outright, we would suggest adjusting the viewing time to reflect the visual complexity of the stimulus.

“

Keep in mind there are still aspects of first impression testing that remain unknown. A reasonable question that you can ask now would be: “Okay, so how exactly do I time my first impression test?” While we can sum up our observations into a conceptual framework of how time can be treated in first-impression test planning, it is not an exact guideline; there may be other interpretations or exceptions.

Take this more as an eye-opener and a call to action. Indeed, in our study, ten seconds yielded more appropriate results for more complex websites than five seconds did. But there is nothing to say that for other websites, the best timing could not be fifteen or even twenty seconds. Even more so, once you also factor in the influence of the cognitive ability of each individual participant.

The key takeaway? When you gauge your audience’s first impressions about something, take a more holistic approach.

Consider your goals for your test. What kind of questions do you want to ask? Use some of the tools that are available to measure the visual complexity of the pictures that you want to present. Give your participants a short working memory test before you start bombarding them with pictures and questions.

Try to adjust the timing in your first impression test to match the situation. To give an analogy, by blindly following a different five-second rule and eating off the floor, you could end up getting sick. Be just as cautious about relying on myths in your usability research methods. This is not to discount five seconds. As we show, it is still good timing for first impression tests in plenty of cases, but it is not the be-all and end-all as far as first impression testing goes. By broadening your perspective, you can do even better.

Resources

• “Cognitive abilities and visual complexity impact first impressions in five-second testing”
  The paper by UXtweak Research investigates the impact of timing, cognitive abilities, and visual complexity on first impression testing. Read for further details about the experiment, its outcomes, and a discussion of findings.
• “How Long Do Users Stay on Web Pages?”
  Jakob Nielsen’s article on how long it takes people to leave a website (a fact to consider when testing first impressions).
• “Attention web designers: You have 50 milliseconds to make a good first impression!”
  The paper by Lindgaard et al. shows that when it comes to attitudes, some first impressions are genuinely formed in a flash.
• “The UX Five-Second Rules: Guidelines for User Experience Design’s Simplest Testing Technique”
  Paul Doncaster’s book on five-second testing provides a thorough summary of existing knowledge about the method and its applications.
• “How to Test a Design Concept for Effectiveness”
  Paul Boag’s walkthrough of key concept testing methods, including five-second testing.
• “5-Second Tests: Measuring Your Site’s Content Pages”
  This article by Christine Perfetti, dating back to 2007, can be seen as the first reference to modern five-second testing.
• Aalto Interface Metrics tool for measuring visual complexity of user interfaces
• Human Benchmark test of visual working memory
• UXtweak’s Five Second Test tool

(yk)

For those involved in digital and physical product development or leadership, consider a Voice of Customer (VoC) program. A VoC program systematically gathers and analyzes customer insights, channeling user opinions into actionable intelligence. VoC programs use surveys, analytics, interviews, and more to capture a broad range of customer sentiment. When implemented effectively, a VoC program transforms raw feedback into a roadmap for strategic decisions, product refinement, and service enhancements.

By proactively identifying issues, optimizing offerings for user satisfaction, and tailoring products to real-time demand, VoC programs keep companies ahead. Moreover, in a world of fleeting consumer loyalty, such programs build trust and enhance the overall brand experience. VoC has been a standard CX practice that UX and product teams can utilize to their advantage. We’ll focus on VoC for digital products for this article. However, the methods and lessons learned are equally applicable to those working with physical products.

Successful product teams and User Experience (UX) practitioners understand that customer feedback is invaluable. It guides decisions and fosters innovation for products and services. Whether it’s e-commerce platforms refining user interfaces based on shopper insights or social media giants adjusting algorithms in response to user sentiments, customer feedback is pivotal for digital success. Listening, understanding, and adapting to the customer’s voice are key to sustainable growth.

The role of UX research in capturing the Voice of the Customer

UX research serves as the bridge that spans the chasm between a company’s offerings and its customers’ perspectives. UX research plays a pivotal role in capturing the multifaceted VoC. Trained UX researchers transform raw feedback into actionable recommendations, guiding product development and design in a direction that resonates authentically with users.

Ultimately, UX research is the translator that converts the diverse, nuanced VoC into a coherent and actionable strategy for digital companies.

“

Setting Up A Voice Of Customer Program

Overview Of Steps

We’ve identified six key steps needed to establish a VoC program. At a high level, these steps are the following:

1. Establishing program objectives and goals.
2. Identifying the target audience and customer segments.
3. Selecting the right research methods and tools.
4. Developing a data collection framework.
5. Analyzing and interpreting customer feedback.
6. Communicating insights to stakeholders effectively.

We’ll discuss each of these steps in more detail below.

Establishing Program Objectives And Goals

Before establishing a VoC program, it’s crucial to define clear objectives and goals. Are you aiming to enhance product usability, gather insights for new feature development, or address customer service challenges? By outlining these goals, you create a roadmap that guides the entire program. You will also avoid taking on too much and maintain a focus on what is critical when you state your specific goals and objectives. Specific objectives help shape research questions, select appropriate methodologies, and ensure that the insights collected align with the strategic priorities of the company.

You should involve a diverse group of stakeholders in establishing your goals. You might have members of your product teams and leadership respond to a survey to help quantify what your team and company hope to get out of a VoC. You might also hold workshops to help gain insight into what your stakeholders consider critical for the success of your VoC. Workshops can help you identify how stakeholders might be able to assist in establishing and maintaining the VoC and create greater buy-in for the VoC from your stakeholders. People like to participate when it comes to having a say in how data will be collected and used to inform decisions. If you come up with a long list of goals that seem overwhelming, you can engage key stakeholders in a prioritization exercise to help determine which goals should be the VoC focus.

Identifying The Target Audience And Customer Segments

Once you create clear objectives and goals, defining the target audience and customer segments will be important. For example, you decide your objective is to understand conversion rates between your various customer segments. Your goal is to increase sign-up conversion. You would want to determine if your target audience should be people who have purchased within a certain time frame, people who have never made a purchase, people who have abandoned carts, or a mix of all three.

Analytics can be critical to help create shortcuts at this point. You might start by looking at analytical data collected on the sign-up page to identify age gaps to set the target audience to understand why that specific age gap(s) are not signing up, whether there is evidence certain segments are more likely to abandon carts, and which segments are less likely to visit your site at all. Then, based on these clear objectives and goals, as well as identifying a target audience and customer segment, you could select the right research method and tools to collect data from the audience segment(s) you’ve identified as critical to collect feedback from.

Selecting The Right Research Methods And Tools

The success of a VoC program hinges on the selection of appropriate research methods and tools. Depending on your objectives, you might employ a mix of quantitative methods like surveys and analytics to capture broad trends, along with qualitative methods like user interviews and usability testing to unearth nuanced insights. Utilizing digital tools and platforms can streamline data collection, aggregation, and analysis. These tools, ranging from survey platforms to sentiment analysis software, enhance efficiency and provide in-depth insights.

The key is to choose methods and tools that align with the program’s goals and allow for a holistic understanding of the customer’s voice.

“

Your UX researcher will be critical in helping to identify the correct methods and tools for collecting data.

For example, a company could be interested in measuring satisfaction with its current digital experience. If there are currently no metrics being captured by the company, then a mixed method approach could be used to try to understand customers’ current attitudes towards the digital experience at a large scale and then dive deeper at a smaller scale after analyzing the survey. The quantitative survey could contain traditional metrics to measure people’s feelings like Net Promoter Score (NPS), which attempts to measure customer loyalty using a single item and/or System Usability Scale (SUS), which attempts to measure system usability using a brief questionnaire, and then based on the data collected, would drive the types of questions asked in a qualitative interview.

To collect the survey information, an online survey tool could be used that can draft and calculate metric questions for you. Many tools have integrated analysis that allows users to do statistical analysis of quantitative data collected and light semantic reviews on qualitative data. You can share the survey data easily with your stakeholder groups and then shape an interview protocol that will allow you to reach out to a smaller group of users to get deeper insight into the findings from the survey.

Developing A Data Collection Framework

Collecting feedback requires a structured approach to ensure consistency and reliability. Developing a data collection framework involves creating standardized surveys, questionnaires, and interview protocols that gather relevant information systematically. A well-designed framework ensures you capture essential data points while minimizing biases or leading questions. This framework becomes the backbone of data collection efforts, enabling robust analysis and comparison of feedback across various touchpoints and customer segments.

Your data collection framework should include the following:

• Objectives and research questions.
• Data sources, whether it’s surveys, user interviews, website analytics, or any other relevant means.
• Data collection methods with an emphasis on reliability and validity.
• A robust data management plan. This includes organizing data in a structured format, setting up appropriate storage systems, and ensuring data security and privacy compliance, especially if dealing with sensitive information.
• Timing and frequency of data collection, as well as the duration of your study. A well-thought-out schedule ensures you gather data when it’s most relevant and over a suitable time frame.
• A detailed data analysis plan that outlines how you will process, analyze, and draw insights from the collected data.

Analyzing And Interpreting Customer Feedback

Collecting data is only half the journey; the real value lies in analyzing and interpreting the data collected. This involves processing both quantitative data (such as survey responses) and qualitative data (such as open-ended comments). Data analysis techniques like sentiment analysis, thematic coding, and pattern recognition help distill valuable insights.

These insights unveil customer preferences, emerging trends, and pain points that might require attention. Your UX researcher(s) can take the lead, with assistance from other team members, in helping to analyze your data and interpret your findings. The interpretation phase transforms raw data into actionable recommendations, guiding decision-making for product improvements and strategic initiatives.

Communicating Insights To Stakeholders Effectively

The insights derived from a VoC program hold significance across various levels of the organization. Effectively communicating these insights to stakeholders is critical for driving change and garnering support. Presenting findings through clear, visually engaging reports and presentations helps stakeholders grasp the significance of customer feedback. Additionally, highlighting actionable recommendations and illustrating how they tie back to strategic objectives empowers decision-makers to make informed choices. Regularly updating stakeholders on progress, outcomes, and improvements reinforces the ongoing value of the VoC program and fosters a culture of customer-centricity within the organization.

Key Components Of A Successful Voice Of Customer Program

Building A Culture Of Feedback Within The Organization

A successful VoC program is rooted in an organizational culture that prioritizes feedback at all levels. This culture begins with leadership setting the example by actively seeking and valuing customer opinions. When employees perceive that feedback is not only encouraged but also acted upon, it fosters an environment of collaboration and innovation. This culture should extend across departments, from marketing to development to customer service, ensuring that every team member understands the role they play in delivering exceptional experiences. By integrating customer insights into the company’s DNA, a feedback culture reinforces the notion that everyone has a stake in the customer’s journey.

Start small and incorporate research activities into product development to start harnessing a user-centric approach. Develop reports that showcase the business purpose, findings, and recommendations that can be presented to the product development team and stakeholders, but also to other departments to show the value of VoC research. Lastly, provide opportunities to collaborate with other departments to help them incorporate VoC into their daily activities. As a result, a culture of incorporating a VoC program becomes reinforced.

There are many ways you can go about building this culture. Some specific examples we’ve used include facilitating cross-product or cross-discipline meetings to plan research and review findings, workshops bringing together stakeholders from various lines of business or roles to help shape the research agenda, and perhaps most importantly, identifying and utilizing a champion of insights to promote findings throughout the organization. Ideally, your champion would hold a position that allows them to have exposure horizontally across your business and vertically up to various key stakeholders and members of leadership. Your champion can help identify who should be attending meetings, and they can also be utilized to present findings or have one-off conversations with leadership to promote buy-in for your culture of feedback.

Implementing User-friendly Feedback Mechanisms

For a VoC program to thrive, feedback mechanisms must be accessible, intuitive, and seamless for customers. Whether it’s a user-friendly feedback form embedded within an app, a chatbot for instant assistance, or social media channels for open conversations, the channels for providing feedback should reflect the digital preferences of your audience. These mechanisms should accommodate both quantitative and qualitative inputs, enabling customers to share their experiences in a manner that suits them best. A key element here is the simplicity of the process; if users find it cumbersome or time-consuming to provide feedback, the program’s effectiveness can be compromised.

Encouraging Customer Participation And Engagement

Engaging customers is essential for gathering diverse perspectives. Incentivizing participation through rewards, gamification, or exclusive offers can increase engagement rates. Moreover, companies can foster a sense of ownership among customers by involving them in shaping future offerings. Beta testing, user panels, and co-creation sessions invite customers to actively contribute to product development, reinforcing the idea that their opinions are not only valued but directly influence the company’s direction. By making customers feel like valued collaborators, a VoC program becomes a mutually beneficial relationship.

Integrating Feedback Into The Decision-making Process

Customer feedback should not remain isolated; it needs to permeate the decision-making process across all departments. This integration demands that insights gathered through the VoC program are systematically channeled to relevant teams. Product teams can use these insights to refine features, marketers can tailor campaigns based on customer preferences, and support teams can address recurring pain points promptly. Creating feedback loops ensures that customer opinions are not only heard but also translated into tangible actions, demonstrating the organization’s commitment to iterative improvement driven by user insights.

Continuous Improvement And Iteration Of The VoC Program

A VoC program is a journey, not a destination. It requires a commitment to continuous improvement and adaptation. As customer behaviors and preferences evolve, the program must evolve in tandem. Regularly reviewing the program’s effectiveness, incorporating new data sources, and updating methodologies keep the program relevant. This also includes analyzing the program’s impact on KPIs such as customer satisfaction scores, retention rates, and revenue growth. By iterating the program itself, businesses ensure that it remains aligned with changing business goals and the ever-evolving needs of their customers.

Best Practices And Tips For An Effective VoC Program

Creating Clear And Concise Surveys And Questionnaires

The success of a VoC program often hinges on the quality of the surveys and questionnaires used to collect feedback. To ensure meaningful responses, it’s essential to design clear and concise questions that avoid ambiguity. Keep the surveys focused on specific topics to prevent respondent fatigue and make sure that the language used is easily understandable by your target audience. Utilize a mix of closed-ended (quantitative) and open-ended (qualitative) questions to capture both statistical data and rich, contextual insights. Prioritize brevity and relevance to encourage higher response rates and more accurate feedback.

Monitoring Feedback Across Multiple Channels

Customer feedback is shared through diverse channels: social media, email, app reviews, support tickets, and more. Monitoring feedback across these channels is essential for capturing a holistic view of customer sentiment. Centralize these feedback streams to ensure that no valuable insights slip through the cracks. By aggregating feedback from various sources, you can identify recurring themes and uncover emerging issues, allowing for proactive responses and continuous improvement. Note we have focused on digital products. However, if there is a physical component of your experience, such as a brick-and-mortar store, you should be collecting similar feedback from those customers in those settings.

Incorporating User Testing And Usability Studies

Incorporating user testing and usability studies is important to help evaluate an experience with users. While upfront activities like in-depth user interviews can articulate users’ desires and needs for an experience, they do not help evaluate the updated experience. Findings and recommendations from user testing and usability studies should be incorporated into development sprints or backlogs. This will ensure that the experience consistently considers and reflects the VoC.

Ensuring Privacy And Data Security In The VoC Program

As you talk to users and develop your VoC program, you will constantly be collecting data. The data that is shared in reports should always be anonymous. Additionally, creating documentation on how to collect consent and data policies will be very important. If data is not stored properly, you could face penalties and lose the trust of participants for future VoC activities.

Challenges Of Starting A Voice Of Customer Program

If you are committed to starting a VoC program from scratch and then maintaining that program, you are likely to encounter many challenges. Gaining buy-in and commitment from stakeholders is a challenge for anyone looking to establish a VoC program. You’ll need to commit to a concerted effort across various departments within an organization. Securing buy-in and commitment from key stakeholders, such as executives, managers, and employees, is crucial for its success. Without their support, the program may struggle to gain traction and achieve its goals.

Resources are always an issue, so you’ll need to work on securing adequate funding for the program. Establishing and maintaining a VoC program can be a costly endeavor. This includes the cost of software, training, and staff time. Organizations must be prepared to allocate the necessary resources to ensure the success of the program.

Allocating sufficient time and resources to collect, analyze, and act on feedback: collecting, analyzing, and acting on customer feedback can be a time-consuming process. Organizations must ensure that they have the necessary staff and resources in place to dedicate to the VoC program.

Case Study: Successful Implementation Of A VoC Program

We worked with a large US insurance company that was trying to transform its customers’ digital experience around purchasing and maintaining policies. At the start of the engagement, the client did not have a VoC program and had little experience with research. As a result, we spent a lot of time initially explaining to key stakeholders the importance and value of research and using the findings to make changes to their product as they started their digital transformation journey.

We created a slide deck and presentation outlining the key components of a VoC program, how a VoC program can be used to impact a product, methods of UX research, what type of data the methods would provide, and when to use certain methods. We also shared our recommendations based on decades of experience with similar companies. We socialized this deck through a series of group and individual meetings with key stakeholders. We had the benefit of an internal champion at the company who was able to identify and schedule time with key stakeholders. We also provided a copy of the material we’d created to socialize with people who were unable to attend our meetings or who wanted to take more time digesting information offline.

After our meetings, we fielded many questions about the process, including who would be involved, the resources required, timelines for capturing data and making recommendations, and the potential limitations of certain methods. We should have accounted for these types of questions in our initial presentation.

One large component of enhancing the customer’s digital experience was implementing a service portal. To help better understand the needs and desires of users for this service portal, we started with executing in-depth user interviews. This first VoC activity helped to show the value of VoC research to the business and how it can be used to develop a product with a user-centric approach.

Our biggest challenge during this first activity was recruiting participants. We were unable to use a third-party service to help recruit participants. As a result, we had to collect a pool of potential participants through the sales division. As mentioned before, the company didn’t have much exposure to VoC work, so while trying to execute our VoC research and implement a VoC program, any time we worked with a division in the company that hadn’t heard of VoC, we spent additional time walking through what VoC is and what we were doing. Once we explained to the sales team what we were doing, they helped with providing a list of participants for recruitment for this activity and future ones.

After we received a list of potential participants, we crafted an email with a link to a scheduling tool where potential participants could sign up for interview slots. The email would be sent through a genetic email address to over 50+ potential participants. Even though we sent multiple reminder emails to this potential list of participants, we could only gather 5–8 participants for each VoC activity.

As we conducted more VoC activities and presented our findings to larger audiences throughout the company, more divisions became interested in participating in the VoC program. For example, we conducted unmoderated concept testing for a division that was looking to redesign some PDFs. Their goal was to understand customers’ needs and preferences to drive the redesign process. Additionally, we also helped a vendor conduct usability testing for the company to understand how user-friendly an application system was. This was one way to help grow the VoC program within the company as well as their relationship with the vendor.

We needed to do more than foster a culture of gathering customer feedback. As we began to execute the VoC program more extensively within the company, we utilized methods that went beyond simply implementing feedback. These methods allowed the VoC program to continue growing autonomously.

We introduced a benchmarking survey for the new portal. This survey’s purpose was to gauge the customer experience with the new portal over time, starting even before the portal’s release. This not only served as a means to measure the customer experience as it evolved but also provided insights into the maturation of the VoC program itself.

The underlying assumption was that if the VoC program were maturing effectively, the data gathered from the customer experience benchmarking survey would indicate that customers were enjoying an improved digital experience due to changes and decisions influenced more by VoC.

Next, we focused on transferring our knowledge to the company so the VoC program could continue to mature over time without us there. From the beginning, we were transparent about our processes and the creation of material for a VoC activity. We wanted to create a collaborative environment to make sure we understand the company’s needs and questions, but also so the company could understand the process for executing a VoC activity. We accomplished this in part by involving our internal champion at the company in all of the various studies we conducted and conversations we were having with various business units.

We’d typically start with a request or hypothesis by a division of the company. For example, once the portal is launched, what are people’s opinions on the new portal, and what functionality should the business focus on? Then, we would craft draft materials of the approach and questions. In this case, we decided to execute in-depth user interviews to be able to dive deep into users’ needs, challenges, and desires.

Next, we would conduct a series of working sessions to align the questions and ensure that they still align with the company’s goals for the activity. Once we had all the materials finalized, we had them reviewed by the legal team and began to schedule and recruit participants. Lastly, we would conduct the VoC activity, synthesize the data, and create a report to present to different divisions within the company.

We started the transfer of knowledge and responsibilities to the company by slowly giving them some of these tasks related to executing a VoC activity. With each additional new task the company was in charge of, we set additional time aside to debrief and provide details on what was done well and what could be improved upon. The goal was for the individuals at the company to learn by doing and giving them incremental new tasks as they felt more comfortable. Lastly, we provided documentation to leave behind, including a help guide they could refer to when continuing to execute VoC activities.

We concluded our role managing the VoC program by handing over duties and maintenance to the internal champion who had worked with us from the beginning. We stayed engaged, offering a few hours of consulting time each month; however, we were no longer managing the program. Months later, the program is still running, with a focus on collecting feedback on updates being made to products in line with their respective roadmaps. The client has used many of the lessons we learned to continue overcoming challenges with recruiting and to effectively socialize the findings across the various teams impacted by VoC findings.

Overall, while helping to build this VoC program, we learned a lot. One of our biggest pain points was participant recruitment. The process of locating users and asking them to participate in studies was new for the company. We quickly learned that their customers didn’t have a lot of free time, and unmoderated VoC activities or surveys were ideal for the customers as they could complete them on their own time. As a result, when possible, we opted to execute a mixed-methods approach with the hope we could get more responses.

Another pain point was technology. Some of the tools we’d hoped to use were blocked by the company’s firewall, which made scheduling interviews a little more difficult. Additionally, some divisions had access to certain quantitative tools, but the licenses couldn’t easily be used across divisions, so workarounds had to be created to implement some surveys. As a result, being creative and willing to think about short-term workarounds was important when developing the VoC program.

Conclusion

Building a successful VoC program is an ongoing effort. It requires a commitment to continuously collecting, analyzing and acting on customer feedback. This can be difficult to sustain over time, as other priorities may take precedence. However, a successful VoC program is essential for any organization that is serious about improving the customer experience.

We’ve covered the importance of VoC programs for companies with digital products or services. We recommend you take the approach that makes the most sense for your team and company. We’ve provided details of starting and maintaining a VoC program, including the upfront work needed to define objectives and goals, targeting the right audience, choosing the right methods, putting this all in a framework, collecting data, data analysis, and communicating your findings effectively.

We suggest you start small and have fun growing your program. When done right, you will soon find yourself overwhelmed with requests from other stakeholders to expand your VoC to include their products or business units. Keep in mind that your ultimate goal is to create a product that resonates with users and meets their needs. A VoC program ensures you are constantly collecting relevant data and taking actionable steps to use the data to inform your product or business’s future. You can refine your VoC as you see what works well for your situation.

Additional Voice of Customer Resources

• “7 tips for an effective voice of the customer program,” Lucia Chung
  Blog post with tips on effective VoC.
• “Three books that highlight the importance of the Voice of the Customer,” Taoufik Massoussi
  Article with a brief review of three books on VoC

(yk)

If you don’t know Lieutenant Columbo, I envy you. I wish I could erase my memory and watch this TV masterpiece for the first time again. Columbo, a Los Angeles homicide detective, has become a cult character in American crime drama in the 1970s. Each episode of this show reveals the murderer from the first minute, and the main mystery is how Columbo proves their guilt and distinguishes between lies and the truth.

When I reflect back on this series, it becomes apparent that the UX area has so much in common with crime scene investigation: the truth is unknown, people tend to disguise their real needs, and you have to discover missing facts as soon as possible to build and launch something useful. I’ve never specialized in UX research, but it has been part of my job as a designer for years. When I started, we rarely had the luxury of a dedicated researcher on a team.

So, let’s see what we can learn from a classical fictional character and apply it in the UX area.

Lesson 1: Understate Your Role To Users

It’s not a secret that people behave differently in the vicinity of police, state officials, or management. Columbo understood that if a suspect or witness realized who he was, they would try to disguise or tweak facts (either consciously or subconsciously). That’s why our hero preferred to blend in and keep his position out of sight as long as possible.

For instance, in the episode “By Dawn’s Early Light” (S4E3), the commandant of a military academy murders the chairman of the board. So, Columbo stayed in the barracks for several days and talked with cadets informally until he exposed the killer.

Sometimes, such an approach has caused funny situations. In the episode “Negative Reaction” (S4E2), Columbo was mistaken for a hobo at St. Matthew’s Mission. Lieutenant patiently accepted the nun’s caring and ate a bowl of stew, and only when she suggested a new raincoat instead of Columbo’s beloved old one, he revealed his purpose.

UX research is no less challenging because we explore human behavior but inevitably influence the findings since we are humans, too. Designers often run the risk of receiving twisted information when they forget to tackle users’ fears and insecurity, for example:

• Interviewees believe their boss sent you to assess their skills;
• Users think you created this design, and now they try not to offend you;
• Customers worry that you’ll judge their computer literacy.

Understating your official role gives you precious moments to talk with people more sincerely. In contrast, here is a perfect intro to annihilate research accuracy: “Hello! I’m a Senior UX Designer and Product Manager. Today, I’ll conduct a usability testing session and jobs-to-be-done interview to identify UX gaps in our design…” After hearing that, people would probably flood you with socially expected answers.

Instead, designers should keep their fancy titles to themselves. Try to start a usability testing session humbly, “My name is <…>, and I was asked to check whether this website is useful and clear to you.” Don’t make people think you designed it (even if you did).

And here is an intro phrase I recommend using for a user interview, “I’m a researcher, and today I’d like to ask you a couple of questions about <…>.” Give a simple description without redundant details that may scare people and increase tension.

Depending on the situation, you can even say, “I didn’t design this, so I won’t be offended if you criticize it; please be honest with your feedback!” But it’s on the thin edge between ensuring less biased research and lying.

Lesson 2: “You Don’t Know My Boss…”

Lieutenant Columbo usually dealt with wealthy and mighty criminals who were sure they would go unpunished. So, he played the role of a “little man” and wasn’t ashamed of it. He realized that exposing his authority would only make people stay within their own shells. Not only did he hide his intellect, but he also encouraged others to feel superior towards him so that people behaved more freely and revealed their true motives.

Columbo looked messy — in a creased beige raincoat, with a cigar, driving an old Peugeot — and concealed his shrewd mind behind this slack appearance and sloppy communication manner. He often told naive stories about his wife and appeared henpecked:

Columbo: I’m a worrier. I mean, little insignificant details, I lose my appetite, I can’t eat. My wife, she says to me, “You know, you can really be a pain.”

Another quote is about the “strict” boss, although it’s apparent from the series that the Lieutenant was a self-organized expert:

Columbo: You’re a celebrity. Because of you, my boss, he won’t let me close up this case until I have covered everything. Every loose end gotta be tied up.

As a newbie designer, I was indoctrinated about the value of presentation skills, making a positive first impression, and the necessity of defending design decisions. However, later, these conventions played a cruel joke on me.

In UX research, a common misconception is that you should look confident and competent in front of users. Let me get this straight: conducting research is not the same as presenting designs to top management. During any research, the goal is to make people feel relaxed so that they tell you the truth. However, at a presentation, the main task is to assure everyone that your decision is well-informed and your input helps steer the business in the right direction.

Research is not meant to show off. You see a user for the first and probably the last time in your life; they won’t influence your career; they aren’t here to be impressed. Behave humbly while staying in control of the session. Yes, you may come across as an ordinary person, but it’ll pay off and bring more insights compared to “boss-subordinate” or “expert-noob” paradigms. I’m not saying one should literally look messy like Columbo. The idea is to blend in, for instance:

• Match interviewees’ dress code (within reason, of course).
  Try not to appear much more official or extravagant than a person in front of you, and you’d better keep that creative “Helvetica” T-shirt and “You ≠ user” pin for a UX meetup.
• Avoid design jargon or terminology you have to explain.
  However, a reasonable dose of your interviewees’ professional lingo will boost communication if you work on a specialized topic.
• Behave neutrally but naturally.
  It means balancing impartiality and separation from the subject with normal human behavior and empathy (simply saying, not being a robot).

Lesson 3: Deep-dive Into A New Topic

We call this approach “user safari” nowadays, but Lieutenant Columbo had been practicing it long before it became designers’ mainstream. If you want to understand your suspects (in our case, users), observe their behavior in a “natural habitat,” and don’t miss a chance to try users’ occupations. It’s better to see once than to hear a thousand times, right?

For example, in the episode “Any Old Port in a Storm” (S3E2), a wine connoisseur kills his brother to prevent him from selling the family winery. Columbo had to turn into a sommelier enthusiast for a while to investigate this crime and recognize unusual evidence, which would have been overlooked without specialized knowledge.

The episode “Negative Reaction” (S4E2) features a talented photographer and Pulitzer Prize winner who kills his wife and blames her death on a failed kidnapping. Columbo gets a camera and learns the basic principles of photography to convict the criminal. The detective had absolutely no proof, but owing to the newly gained knowledge, he set a cunning trap so that the murderer gave himself away.

Now, UX research. Of course, we shouldn’t literally follow the TV series and get expensive equipment just to step into users’ shoes. Fortunately, one can empathize much more easily nowadays. I mean observation studies and contextual inquiries when you can access users or documentaries, YouTube blogs, and professional communities if you want to prepare to face real users and avoid surface-level questions.

For example, several years ago, I was preparing for interviews with drilling engineers — future users of a new app suite for drilling planning. So, I watched “Deepwater Horizon,” a U.S. movie about a historical oil spill disaster in the Gulf of Mexico. This movie was recommended by a subject matter expert from the client’s side; he told me it realistically showed a drilling rig in action. As a result, I understood the technical jargon and used interviews with engineers to figure out really unobvious facts, not Wikipedia-level basics.

Another vivid example is a project I heard about from my former colleagues, who conducted product discovery for a Middle East logistics company several years ago. So, during an on-site, the discovery team observed the actual work of delivery crews and eventually witnessed a problem that couriers didn’t dare to report to their superiors. The app was designed for European address conventions and didn’t consider Middle-Eastern reality. Couriers only simulated using the navigation feature because the app required it to proceed to the next step. Frankly, I don’t believe this could’ve been learned from interviewing users or workshops with the client’s management.

Lesson 4: “Uhh… Just One More Thing!”

I guess Columbo used this catchphrase in each of the 69 episodes. In some cases, Lieutenant sounded like a narrow-minded, forgetful cop; sometimes, the question that followed “just one more thing” made a suspect worry. But does it have anything to do with UX research?

If we translate this phrase into modern language, we are talking about the skill of asking follow-up questions and improvising in pursuit of UX insights. Of course, our task in tech is way simpler than Columbo’s: we don’t have to provoke criminals to obtain irrefutable evidence for trial. But what detectives and UX folks share is the sense of valuable information and information buzz. This feeling pushes us to step aside from protocols and scripts and dig deeper.

“I have always found that plans are useless, but planning is indispensable.”
— Dwight Eisenhower

Even the best script for an interview, usability testing, or workshop won’t take into account all nuances.

In qualitative research, you cannot just read prepared questions out loud and call it a day; otherwise, it would’ve been already outsourced to robots.

“

I learned that what you want to know doesn’t equal the questions you ask.

• Research questions are something you want to learn to make better design decisions. You keep them secret from respondents; they are only for your team’s internal use. For example, Will they buy this app? What is their top problem? Why are we worse than our competitors? In Columbo’s terms, they are equivalent to “Who is the murderer?”
• Interview questions are what you actually ask. They are formulated in a certain way because not every answer can be retrieved directly. For example, Please tell me about the last time you ordered grocery delivery. How often do you buy non-fiction books online? They resemble Columbo’s “What did you do after 10 PM last Friday?”

While research questions are agreed upon with the team in advance, interview questions are left to the researcher’s discretion. For example, in one case, you ask a single “Tell me about the last time…” question and get tons of data from a talkative and relaxed person. But another respondent will give you a tiny piece of a puzzle at a time, and you’ll need to ask more granular questions, “What did you order? How did you choose? What payment did you choose? Why this option?” and so on.

Lesson 5: Don’t Take Words At Face Value

Why is “Columbo” so fun to watch? Because the Lieutenant always allows his suspects to justify themselves and compose plausible explanations in a naive attempt to ward off suspicion. I think the suspects should’ve kept silent instead of trying to divert Columbo’s investigation.

The iconic dialog between Columbo and Paul Gerard shows how early one can recognize lies. The episode “Murder Under Glass” (S7E2) tells about a food critic who extorted money from restaurant owners in exchange for positive reviews and poisoned one of them for fear of exposure.

Paul Gerard: When did you first suspect me?
Columbo: As it happens, sir… about two minutes after I met you.
Paul Gerard: That can’t be possible.
Columbo: Oh, you made it perfectly clear, sir, the very first night when you decided to come to the restaurant directly after you were informed that Vittorio was poisoned.
Paul Gerard: I was instructed to come here by the police.
Columbo: And you came, sir.
Paul Gerard: Yes.
Columbo: After eating dinner with a man that had been poisoned. You didn’t go to a doctor. You came because the police instructed you. You didn’t go to a hospital. You didn’t even ask to have your stomach pumped. Mr. Gerard, that’s the damnedest example of good citizenship I’ve ever seen.

Surprisingly, this strongly relates to UX.

All people lie. Influential stakeholders try to push forward their ideas. Some people desire to appear more knowledgeable than they are. Others are afraid to share opinions if they don’t know how they’ll be used. You can also find yourself in the center of office politics when officially declared messages contradict actual goals.

Due to classical UX doctrines, designers are called “user advocates” and broadcasters of the “user’s voice,” but it doesn’t mean we should listen to people indiscriminately.

“

If a person craves a feature but has zero examples of how something similar has helped them in the past, it might be an exaggeration. If a business owner says an app is successful but has only feedback from her colleagues, it may be overly optimistic. And so on. When we notice information discrepancies, the best choice is to continue asking questions, and then, maybe, your interlocutor will start to doubt their own words. For example,

Product owner: Hey, Ann! We need to have an export feature so that users can download nice-looking PDF reports.
Designer: Just for my understanding. Can you please explain the context of this feature idea?
Product owner: Well, I think it’s pretty clear. Export is a standard thing for engineering applications. Probably, there should be a button or icon above the dashboard; a user clicks, and then a PDF with our logo…
Designer: Jack, sorry for interrupting. I’m asking this not out of curiosity but because I want to get it right. If you remember the user interviews last month, engineers usually copy-paste data from the dashboard into a PowerPoint template with their company’s branding…
Product owner: That’s a very good question. I need to double-check it.

So, Columbo teaches us to trust but verify. Carefully listen to what you’re told, don’t show skepticism or suspicion, and continue asking questions until you reach the root cause of a problem.

Summary

Of course, the lessons I deduced from TV series aren’t even close to being comparable with mature research methodologies and UX culture. Unlike the time when I started my design career, today, I see more and more dedicated researchers who take care of insights that steer businesses in the right direction. So, I hope this article entertains you with unusual parallels between UX and fictional crime investigation.

If Lieutenant Columbo were a UX guru like Don Norman or Jacob Nielsen, he would probably give us the following advice:

1. Don’t flash your fancy UX title without necessity.
2. Don’t show off in front of users; this is not a job interview or top management presentation.
3. Strive to observe users in context, in their “natural habitat.”
4. Have plenty of contextual and follow-up questions up your sleeve.
5. All people lie (often unintentionally). Double-check their words.

Further Reading

• The Mom Test: How to Talk to Customers and Learn If Your Business is a Good Idea when Everyone is Lying to You, a book by Rob Fitzpatrick
• “First Rule of Usability? Don’t Listen to Users,” an article by Jakob Nielsen for Nielsen Norman Group
• Communicating the User Experience: A Practical Guide for Creating Useful UX Documentation, a book by Richard Caddick and Steve Cable

(yk)

While the User Experience practice has been expanding and will continue to balloon in the coming years, so have its sub-disciplines such as content strategy, operations, and user research. As the practice of UX Research matures, scalability will continue to be important in order to meet the rapid needs of iterative product development.

While there are several effective ways to scale user research, such as increasing researcher-to-designer ratios, leveraging big data and real-time analytics, or research democratization, one of the most effective methods is developing a Rapid Research program. In a Rapid Research program, teams are provided quick insight into key problems at an unprecedented operational speed.

Rapid Research-type support has been around for a while and has taken different shapes across different organizations. What remains true, however, is the goal to provide actionable insights from end-users at a quick pace that fits within product sprints and maintains pace with agile development practices.

In this article, I’m going to unpack what a Rapid Research program is, how to build one in your organization, and underscore the unique benefits that a program like this can provide to your team. Given that there is no singular ‘right way’ to scale insights or mature a user research practice, this outline is intended to provide building blocks and considerations that you may take in the context of the culture, opportunities, and challenges of your organization.

What Is Rapid Research?

Rapid research is a relatively recent program where typical user research practices and operations are standardized and templatized to provide a consistent, repeatable cadence of insights. As the name suggests, a core requirement of a rapid research program is that it delivers quicker-than-average insights. In many teams, this means delivering research on a weekly cadence where a confluence of guardrails, templates, and requirements work to ensure a smooth and consistent process.

Programs like Rapid Research may be created out of a necessity to keep up with the pace of development while freeing the bandwidth of expert researchers’ time for more complex discovery work that often takes longer. A rapid research program can be a crucial component of any team’s insight ecosystem, balanced against solving different business problems with flexible levels of support.

Scope

Research Methods

In order to make research more rapid, teams may consider some research methodologies out of the question in their Rapid Research program. Methods such as longitudinal diary studies, surveys, or long-form interviews might suffer from lower quality if done too quickly. When determining the scope of your rapid research program, ask yourself what methods you can easily templatize and, most importantly, which best support the needs of your experience teams.

For example, if your experience teams work on 2-week sprints and need insights in that time, then you will need to consider which research methods can reliably be conducted in 1–2 week increments.

Sample Size And Research Duration

Methods alone won’t ensure a successful implementation of a rapid research program. You will also need to consider sample size and session duration. Even if you decide usability tests are a reasonable methodology for your rapid research framework, you may be introducing too much complexity to run them with 15+ users within 60-min sessions and analyze all that data efficiently. This may require you to narrow your focus to fewer sessions with shorter duration.

Participant Recruitment

While there may be fewer and shorter sessions for each study, you also need to consider your participant pool. Recruitment is one of the most difficult aspects of conducting any user research, and this effort must be considered when determining the scope of the program. Recruitment can jeopardize the pace of your program if you source highly specific participants or if they are harder to reach due to internal bureaucracy or compliance constraints.

In order to simplify recruitment, consider what types of participants are both the easiest to reach and who account for the most use cases or products you expect to be researching. Be careful with this, though, as you don’t want to broaden your customer profiles too much for fear of not getting the helpful feedback you need, as UserZoom says:

“Why is sourcing participants such a challenge? Well, you could probably find as many users as you like by spreading the net as wide as possible and offering generous incentives, but you won’t necessarily find the ‘right’ participants.”

— UserZoom, “Four top challenges UX teams face in 2020 and how to solve them”

Timing

Why Timing Matters

Coupled tightly with scope, the timing of your rapid research end-to-end process will be paramount to the program’s success. Even if you have narrowed the scope to only a handful of research methods with limited sessions at shorter durations and with specific participant profiles, it won’t be ‘rapid’ if your end-to-end project timeline is as long as your average traditional study. Care must be taken to ensure that the project timelines of your rapid research studies are notably quicker than your average studies so that this program feels differentiating and adds value on top of the work your team is already doing.

Reconsidering scope

If your timelines are about the same, or your rapid cadence is less than 50% more efficient than your average study, consider whether or not you’re being judicious enough in your scope above. Always monitor your timelines and identify where you can speed things up or limit the scope in order to reach a quick turnaround, which is acceptable. One way to support shorter project timelines is through compartmentalization.

Compartmentalization

About Compartmentalization

One way to balance scope, timing, and consistency is by breaking up pieces of your average study process into smaller, separate efforts. Consider what your program would look like if you separated project intake from the study kick-off or if discussion guides were not dependent on recruitment or participant types. Splitting out your workflow into separate parts and templating them may eliminate typical dependencies and streamline your processes.

Ways To Compartmentalize

Once you’ve determined the set of research methods and ideal participants to include in your program, you may:

• Templatize the discussion guides to provide a quick starting point for researchers and cut down on upfront preparation time.
• Create a consistent recruitment schedule independent of the study method to start before study intake or kick-off to save upfront time.
• Pre-schedule recurring kick-off and readout sessions to set expectations for all studies while limiting timeline risk when at the mercy of others’ calendars.

There is a myriad of opportunities to do things differently than your typical research study when you reconsider the relationships and interdependencies in the process.

Consistency

Expectability

While a quality rapid research program takes into consideration scope, timing, and compartmentalization, it also needs to consider consistency. It would be difficult to discern whether or not the program was ‘rapid’ if, on one week, a study takes one week, and on another week, a study takes 2.5 weeks. Both may be below your current study average. However, project stakeholders may blur the lines between the differences in your rapid studies and your typical studies due to the variability in approach. In addition, it may be difficult to operationalize compartmentalization or rapid recruitment without some form of expected cadence.

More Agility

As you and your team get used to operating within your rapid cadence, you may identify additional opportunities to templatize, compartmentalize or focus scope. If the program is inconsistent from study to study, it may be more difficult to notice these opportunities for increased agility, hindering your program from becoming even more rapid over time.

A Rapid Research Case Study

While working at one of the largest telecommunications companies in the US, I had the privilege of witnessing the growth of the UX Research team from just four practitioners to over 25 by the time I left. During this time, the company had matured its user experience practice, including the standards, processes, and discipline of user research.

Identifying The Need

As we grew, human insight became a central part of the product development process, which meant an exponential increase in its demand. While this was a great thing and allowed our team to grow, the work we were doing was not sustainable — we were constantly trying to keep pace with product teams who brought us in too late in the process simply to validate their ideas. Not only did we always feel rushed, but we were stuck doing only evaluative work, which not only stifled innovation but also did not satisfy our more senior researchers who wished to do more generative research.

How It Fits In

Once diagnosing this issue, our leadership initiated several new processes to build a more well-rounded research portfolio that supported iterative research while enabling generative research. This included a democratization program, quarterly planning, and my initiative: Rapid Research. We determined that we needed a program that would allow us to take on mid-sized projects at the pace of product development while providing a new opportunity to hire junior researchers who would be a great talent pool for our team and provide a meaningful way for those new to the field to grow their skills.

Getting Started

In order to build the rapid research program, I audited the previous year’s worth of research to determine our average timelines, the most common methodologies used for iterative and mid-sized projects, and to identify our primary customer who we do research with most often. My findings would be the bedrock of the program:

• Most iterative research was lite interviews and brief usability tests.
• Many objectives could be covered in 30-minute sessions.
• Mid-sized projects were often with just a handful of current customers.
• Our average study time was 2–3 weeks, so we’d need to cut this down.
• Given the above constraints, study goals should be highly focused.

Building The Program

At first, we did not have the budget for hiring new junior researchers to staff the program team. What we did have, however, was a contract with a research vendor who we’ve worked with for years, so we decided to partner with researchers from their team to run our rapid research program.

• We created specific templates for ‘rapid’ usability tests and interviews.
• Studies were capped at two objectives and only a handful of questions in order to fit into 30-min sessions.
• Study intake was governed via a simple intake form, required to be filled out by EOD every Wednesday.
• We scheduled standing kick-off and readout sessions every Friday and shared these invites with product teams for visibility.
• To further establish our senior researchers as Portfolio Research Leads and to protect against scope creep, we required teams to formally request ‘rapid’ studies through them first.
• We started our rapid cadence at two weeks and were able to cut it down to just one week after piloting the program for a month.

Strong Results

We saw the incredible value and strong results from building our rapid research program, especially alongside the other processes our team was standing up to support varying insights needs.

• Speed
  We were able to eventually run three research studies simultaneously, enabling us to deliver more research at twice the pace of a traditional study.
• Scale
  Through this enablement of speed, consistent recruitment, and templatized process, we ran over 100 studies & 650+ moderated interviews.
• Impact
  Because we outsourced rapid research to a vendor, our team was freed up to deliver foundational research, which doubled our work capacity.
• Growth
  Eventually, we hired junior researchers and transitioned the program from the vendor, increasing subject matter expertise & operational efficiency.
  

How To Build A Rapid Research Program

The following steps outline a process for getting started with building your own rapid research program in your organization. Exactly which steps you choose to follow, or if you decide to add more or less to your process, will be entirely up to you and the unique needs of your team. Follow the proceeding steps while considering the above guidelines regarding scope, timing, compartmentalization, and consistency.

Determine If You Even Need A Rapid Research Program

While seemingly counter-intuitive, the first step in building a rapid research program is considering whether you even need one in the first place. Every new initiative or tactic intended to mature user research practice should consider the available talent and capabilities of the team and the needs or opportunities of the organization it sits within. It would be unfortunate to invest time to build a robust, rapid research program only to find that nobody uses or needs it.

Reflection On Current Needs

Start by documenting the needs of your experience teams or the organization you support by the different types of requests you receive.

• Are you often asked to deliver research faster?
• What are the types of research which are most often requested?
• Does your team have the capability or operational rigor required to deliver at a faster pace?
• Are you staffed enough to support a more rapid pace, even if you could deliver one?
• Is delivering faster, rigidly-scoped research in service to your long-term goals as a research team, or might it sacrifice them?

Gather More Information

Answering these questions should be your first step before any meaningful work is done to build a rapid research program. In addition, you might consider the following information-gathering activities:

• Audit previous research you or your team have done to determine their average scope, timeline, and method.
• Conduct a series of internal stakeholder interviews to identify what potential value a rapid research program might hold.
• Look for signals for where the organization is going. If leadership is hiring or training teams on agile methods or demanding teams to take a step back to focus on discovery can help you decide when and where to invest your time.

These additional inputs will either help you refine your approach to building a program or to steer away from doing so.

Limitations Of Rapid Research

Finally, when considering if you should build a rapid research program in the first place, you should consider what the program cannot do.

• What a rapid research program might save on time, it cannot necessarily save on effort. You will still need researchers to deliver this work, which means you may need to restructure your team or hire more people.
• If you decide to make your rapid research program self-service, you likely will still need ResOps support for recruitment and managing the intake process effectively.
• It is also possible to hire a research vendor partner to lead this program, though that will require a budget that not every team may have.
• As mentioned above, a good rapid research program is tight and focused in its scope, which limits the type of projects it can accommodate.

Identify Your Starting Scope, Timing & Cadence

Once you’ve decided to pursue a rapid research program, you’ll need to understand what form your program should take in order to deliver the highest value to your team and those you support. As mentioned above, a right-sized scope should consider the research methods, requirements, session quantity & duration, and participant profiles, which you can confidently accommodate. And you will need to determine the end-to-end timing and program cadence that differentiates from current work while providing just enough time to still deliver sustainable quality.

Determine Participant Profiles

Start building your scope backwards from the needs gaps you’re filling within your team based on the answers to the discovery questions above. You’ll want to identify the primary type(s) of end-users this program will research.

1. Audit the past 6–12 months of research you or your team has done, looking at the most common customer type with whom you do research.
2. Then, couple that with any knowledge you may have of where the business or your experience teams will be focused for the following 6–12 months.

For example, if your audit revealed that your team had focused most frequently on current customers over the past year, and you also know that your business will soon focus on the acquisition of new customers, consider including both current customers and prospective customers in your rapid research scope.

Remember the important note about consistency above? Once you’ve identified potential participant profiles, make sure you can consistently recruit them. For example, if you use a research panel to source participants for research studies, test the incidence of your participant profiles. If you find they don’t have many panelists with the attributes you need, you might spend too much time in recruitment and jeopardize the speed of the program.

A balance should be struck between participant profiles that are specific enough to be useful for most projects and those broad enough to reach easily.

“

Determine Research Methods

You can conduct the same audit and rough forecasting when determining the research methods your program ought to support but with two additional considerations:

1. Team strategy,
2. Individual career development.

User researchers tend to focus their work further upstream, where they’re driving product roadmaps or influencing business strategy. This can bode well for your rapid research program if it is focused on evaluative research projects, which are often quicker and cheaper to conduct.

The ultimate goal is for the rapid research program to be a complement to what your team provides or as an enabler for freeing up their bandwidth so that they can focus on the type of work they want to do more of.

Right-size Research Methods

Once you’ve determined which research methods you want to include in your rapid research program, consider the level of rigor you need to balance effort and complexity.

• Consider shorter sessions and testing with fewer participants to keep things manageable within each study.
• While there are a number of ways to analyze qualitative data, consider a lite thematic analysis to keep things manageable.
• Instead of building a robust presentation of findings, you may only have time to develop an effective executive summary of key insights.

Determining Timelines

Project timelines within a rapid research cadence are directly affected by the above scope decisions for participant profiles and research methodology. Timelines can also compound in highly regulated industries such as healthcare or banking, where you may be required to gather legal & compliance approval on every moderation guide. In order to call this a rapid research program, the end-to-end project timelines need to be shorter than a typical project of a similar scope, or at least feel that way.

1. Scope current minimum effort
   Start by jotting down the minimum amount of time it takes a researcher on your team to do each sub-step in your current non-rapid research process. Do this for the same participant profiles and methods you want to include in your rapid research program.
2. Dependencies
   Now, identify which sub-steps are dependent on others and think of ways to program them in order to build efficiency. For example, if you need legal approval on every moderation guide before data collection, which takes 2–3 days, see if Legal will commit to a change to a 24-hour SLA for rapid research-specific projects. Another example is if you typically give stakeholders a few days to provide feedback on moderation guides, change this for rapid research projects to cut down dependency time.
3. Identify compartmentalization
   In addition to programming project dependencies, consider the above guidance for compartmentalizing some of the programs in order to remove dependencies entirely, such as with recruitment. Identify what parts of the process don’t have the same dependencies in your rapid research program and can be started earlier. By removing dependencies entirely, you may be able to do several things simultaneously to speed up project timelines.
   

Once you’ve documented your current research process (steps, dependencies, timing) and the changes you need to make to build efficiencies or remove dependencies, document what ‘must be true’ in order to consistently deliver identified changes. Create a table to document all of these details, then sum up the total timelines to compare your typical end-to-end research project timeline with your potential new ‘rapid’ timeline.

Ask yourself if this seems ‘rapid’ when stacked against your average study duration.

• If not, look back at the guidance above. Ask yourself if there are other customer types that may be easier to get in front of that you haven’t considered. Consider whether you need to create a new process, expedite existing processes, or create new relationships in order to make your timelines even more rapid.
• If so, congratulations! You might have just landed on the right scope for your rapid research program. Consider whether this new rapid timeline is something that you can deliver consistently and reliably over time and whether or not you have enough access to participants, and enough budget, to carry out this cadence long-term.

Build Infrastructure, Standards & Rules

It’s time to set the foundation. Return back to the tables you made above and create an action plan with the following steps and a timeline to build the infrastructure required to bring your program to life. As part of this, you’ll need to establish the rules and standards for communicating with partners. You might consider a playbook and formal scope document to inform others of the ins/outs of the program.

Gather Buy-in

Prioritize any work that requires buy-in, generating understanding, or acquiring budget first before spending your time and energy building templates or documentation. You wouldn’t want to create a 20-page scope document outlining the bandwidth for two researchers, a limit to 1 round of stakeholder feedback, and a 24hr SLA for legal approval, only to find out others cannot commit to that.

Create Templates

You’ll need plenty of templates, tools, and processes specific to the scope of your program.

• If you’re limiting moderation guides to a maximum of 10 questions, then create a specific discussion guide template reflecting that.
• If your data analysis will be sped up by using structured note-taking templates, create those.
• If you’ve determined that all rapid research projects only require an executive summary one-pager, make that too.

Staffing

As mentioned above, even a drastically reduced version of your typical research processes still requires effort to support. You’ll need to determine, based on the expected scope and cadence of each rapid research project, how many researchers and/or research operations coordinators you’ll need to support the program. While all rapid research programs will require dedicated effort, there are creative ways of staffing the program, such as:

• A dedicated team of 1–2 researchers and 1–2 Ops coordinators to deliver projects with the greatest efficiency and quality.
• A dedicated team of 1–2 researchers who also handle the operations of running the program itself.
• A self-service program, with 1–2 Ops coordinators for supporting anyone doing the research work.
• Outsourcing the entire program to a vendor.

Work with your leadership, HR, and TA professionals on securing approval for any team restructure, needed headcount budget, or to onboard a new vendor. Then, take the appropriate steps to hire your next researcher or secure the staffing help you need to support your program.

Coaching And Guidance

Consider training, coaching, and check-in meetings as part of your infrastructure.

• If you are staffing new researchers to this rapid research program, make sure they understand the expectations and have what they need to succeed.
• If you’re implementing a self-service model, provide brown-bag sessions to partners to explain the program do’s and don’ts.
• Schedule quarterly check-ins with partners and leadership to discuss the program accomplishments and any needed adjustments to ensure it stays relevant.

Pilot, Get Feedback, And Iterate Over Time

No matter how much preparation you do or how much time and effort you spend building the alliances, infrastructure, training, and support required to run your rapid research program effectively, you will learn that there are improvements you should make once you put it into practice.

There are many benefits to piloting a new program in an organization. One benefit is that it can mitigate risks and allow teams to learn quickly and early enough to make positive enhancements.

“Piloting offers a realistic preview experience for users at the earliest stages of development. It allows the organization and design team to gather real-time insights that can be used to shape and refine the product and prepare it for commercialization.”

— Entrepreneur, “Tasting As You Go: The 5 Benefits of ‘Piloting’”

This means setting expectations early. Consider your first few projects as pilots and expect them to be rocky and imperfect. Use this to your advantage by asking stakeholders you’re closest with to be your trial projects and let them know how important their honest feedback is throughout the process. Ensure that you have clear mechanisms to gather feedback at each project milestone so that you can track progress. It is especially important to capture what might be slowing you down along the way or putting your ‘rapid’ timelines at risk.

Program Evolutions, Impacts & Considerations

Potential Evolutions & Variations

While I’ve outlined a process for getting started, there are many ways in which your rapid research program may evolve over time to meet the needs of your organization better.

• After a few periods, you might identify volume isn’t as high as you anticipated, so you extend the 1-week timeline to every two weeks.
• After a few months, your business might launch a new product line, requiring you to consider a new set of customer profiles in recruitment.
• You may decide to leverage your rapid cadence for individual segments of a longitudinal diary study to accommodate new methods.
• You might use rapid research projects to exclusively evaluate in-market products while others on the team focus on in-progress / new products.
• Rapid research projects could be a stage-gate for larger projects — proving a customer need before larger time investments are made.

However your rapid research program takes shape, revisit its goals, scope, and operations often in relation to your organizational needs and context so that it remains relevant and delivers the highest impact.

“

Solid Impacts From Rapid Research

Building a rapid research program can have a big impact and can contribute positively toward your team’s long-term strategy. One impact of instituting a rapid research program could be that now your team is freed up to focus on more generative research, which unlocks your ability to deliver deep customer insights that pave the way for innovation or strategy. And due to your new rapid pace, you may be able to keep pace with agile development and conduct end-to-end research within 2-week sprints. Another impact is that you may catch more usability issues further upstream, saving you over 100x in overhead business cost. A final impact of a rapid research program is that it can double your team’s throughput, allowing your team to deliver more research, more frequently, to accommodate more organizational needs.

Be sure to track these impacts over time so that you not only get credit for the hard work you put into building the program but so that you can sustain and grow the program over time.

Considerations When Building A Rapid Research Program

As mentioned in this article, there are many benefits to building a rapid research program. That being said, there are limitations to rapid research in regard to its pros and cons when it should be used, and if you have the available time to stand up a program yourself.

Pros And Cons

As with building any new program, one should consider both its benefits as well as drawbacks. Here are a few for rapid research programs:

Pros:

• Can free time for foundational work;
• Rapid studies may keep a better pace with development cycles;
• Can create meaningful opportunities for junior staff;
• Can double project throughput, increasing output volume.

Cons:

• Still requires work and dedicated bandwidth;
• Another thing to diligently track and manage;
• Not great for all types of research studies;
• May cost more money or resources you don’t have.

Guidance For Using The Program

Rapid Research programs are best for specific types of research which do not take a long time to complete or require rigorous expertise. You may want to educate your partners on when they should expect to use a rapid research program and when they should not.

• Use rapid research when:
  • Agility or quick turnaround is needed;
  • You need simple iterative research;
  • Stakeholder groups are easier to rally;
  • Participants are easy to reach.
• Do not use rapid research when:
  • The study method cannot be done quickly without risking quality;
  • A highly complex or mixed-methods study is needed;
  • A project requires high visibility or stakeholder alignment;
  • You have specific, hard-to-reach participants.

Ramp Up Time

While the exact timeline of building a rapid research program varies from team to team, it does take time to do it right. Make sure to plan out enough time to do the upfront work of identifying the appropriate scope, timing, and cadence, as well as gathering consensus from leadership and appropriate stakeholder groups. Standing up a Rapid Research program can take anywhere from 3 months to 1 year, depending on the following:

• Legal and compliance limitations or requirements.
• The number of stakeholder groups you need buy-in from.
• Approval of budget for outside vendors or for hiring an in-house team.
• Time it takes to build templates, guidelines, and materials.
• Onboarding, training, and iteration when starting out.

Conclusion

A rapid research program can be a fundamental part of your team’s UX Research strategy, enabling your team to take on new insight challenges and deliver efficient research at an unprecedented pace. Building a rapid research program with high intention by determining the goals, appropriate scope, and necessary infrastructure will set your team up for success and enable you to deliver more value for your organization as you scale your user research practice.

Don’t be afraid to try a rapid research program today!

Further Reading On SmashingMag

• “How To Build An Ethical User Research Practice At Any Organization”, Devin Harold
• “Eye-Tracking In Mobile UX Research”, Mariana Macedo
• “A Pragmatist’s Guide To Lean User Research”, Paul Boag
• “Better ROI For Your Digital Products: Why Continuous Research Is Key”, Suzanne Scacca
• “Ethical Considerations In UX Research: The Need For Training And Review”, Victor Yocco

(cc, yk, il)

We don’t live in an ideal world. Most of us have too much work, too little time, and too small a budget. When it comes to digital projects, it seems like our clients or bosses always prioritize speed over quality.

To make matters worse, we read countless articles telling us how we should do things. These articles emphasize research and testing but do nothing more than leave us disillusioned and add to our imposter syndrome.

In this article, I want to try a different approach. Instead of telling you what the best practice is, I’ll explore some practical approaches to user research that we might be able to fit into our existing projects.

I know what you’re thinking:

“I won’t be allowed to do research. I’ll be told there’s no time.”

So let’s start there.

Lean User Research Saves Time Rather Than Costs It

The notion that all user research must take away from the available time for a project is flawed. Lean user research has the potential to save you time, especially on projects with multiple stakeholders.

Consider how much time is wasted on calls debating the best approach or in Figma endlessly revising the design because the client can’t make up their mind. Then there is the time of the other stakeholders, all of whom have to attend those meetings and provide feedback.

A small amount of user research can solve much of that. It can replace endless opinions, discussions, and revisions with data.

We don’t need to ask for extra time for research. Instead, we can replace some of those meetings with a quick survey or test and cut through all the discussion.

But what about the discovery you are supposed to do upfront? What about the research into your audience before you begin? Isn’t that best practice, and shouldn’t you be doing that?

Well, yes and no.

What About Upfront Research?

Yes, a discovery phase is best practice. It is our chance to challenge our assumptions about the users and their needs. However, we don’t always get to do what we should, and not every discovery phase needs to take a lot of work.

If you’re not careful, discovery phases can be a little wasteful. General research into your audience and needs may not always provide applicable insights. That’s because it’s only once we start work that we learn what questions to ask upfront. Of course, by that point, you have already used time on the discovery phase, and stakeholders may be reluctant to do any more research.

Simply carrying out exercises like customer journey mapping because you’ve read that you should do it upfront is not a good enough reason when time and money are tight.

So, if time is tight, don’t feel like you have to do a full-blown discovery phase just because articles like this tell you to. Instead, start by collating what the organization already knows about the user and their needs. Most organizations know more than you think about their audience. Whether it’s personas produced by marketing, surveys run in the past, or analytics data, it can often just be a matter of gathering together what already exists.

Once you have done that, you will have a clearer picture of what is missing. If there are some significant and obvious gaps in your knowledge, then some upfront research is worthwhile. However, it might be that you have enough to start, leaving more time for user research as issues arise.

Either way,

Your focus should be on answering specific questions, not improving your general understanding of the user.

“

Focus On Answering Specific Questions

User research can quickly become a time sink if not managed carefully. Adding more and more questions to surveys because “it would be interesting to know” will slow down the surveying process. Equally, you can waste hours simply watching user sessions back. While this context is helpful, it is better to conduct user research only when there is a specific question that needs answering.

For example, if you want to know why people aren’t buying on your website, run a one-question survey that asks why when people go to leave the site. Or, if stakeholders are concerned that users will miss a critical call to action, do a quick 5-second test to reassure them.

Focusing user research on answering these kinds of questions not only ensures a better result but also ensures that user research saves time. Without user research, discussions and debates around these topics can drag out and slow momentum. Additionally, by focusing user research on addressing a single question, it keeps it small and easy to incorporate into an existing project.

Many little bits of user research are easier to insert than a single significant discovery phase.

“

Of course, this is only true if the types of user research you do are lightweight.

Keep Your User Research Lightweight

When trying to keep our user research lean, tough decisions must be made. One of these is to move away from facilitated research, such as user interviews or usability testing, as they are too time-consuming.

Instead, we should focus on research that can be set up in minutes, provides results quickly, and can be understood at a glance. This leaves us primarily with surveys and unfacilitated testing.

Run Quick And Dirty Surveys

Personally, I love quick surveys to resolve areas of disagreement or uncertainty. If in doubt, I argue, it’s best to ask the user. Just a few examples of surveys I have run recently include:

• Comparing two labels for a second on a website.
• Identifying tasks users wanted to complete on a website.
• Discovering why people weren’t signing up for a free trial.
• Assessing whether people understood an infographic.

I could go on, but you get the idea. Short, focused surveys can help answer questions quickly.

Surveys are easy to create and depending on how you approach them, you can get results quickly. If time is more of a barrier than money, you can use an app like Pollfish to recruit the exact demographic of people you need for a few dollars per submission. You can usually get results in less than a day with only a few minutes of work to set up the survey.

If money is an obstacle, consider sharing your survey on social media, a mailing list, or your website. You could even share it with random people who aren’t involved in the project if you’re desperate. At least you’d get an outside perspective.

When your questions are about a design approach you’ve produced, you can turn to unfacilitated testing.

Try Some Unfaciliated Tests

Stakeholders often spend days debating and revising design concepts when quick tests could provide the answers they need. Generally, these design discussions revolve around four questions:

• Did users see it?
• Did users understand it?
• Can people use it?
• Will they like it?

Fortunately, there are quick tests that can help answer each of these questions.

Did Users See It?

If stakeholders are concerned that someone might miss a call to action or critical messaging, you can run a 5-Second Test. This test presents users with a digital product, such as a website or app, for five seconds before asking what they saw. Tools like Usability Hub and Maze provide a URL for the test that you can share with participants, similar to how you would distribute a survey. If users recall seeing the element in question, you know everything is good.

Did Users Understand It?

A slight variation of the test can also be used to answer the second question: did users understand it? Show the user your design for 5 seconds, then ask them to describe what they saw in their own words. If they accurately describe the concept, you can be sure of your approach.

Can People Use It?

When it comes to the “can people use it?” question, you have two options.

If you have a prototype, you can run unfacilitated usability testing with a tool like Maze:

1. Define the task you need to see people complete;
2. Provide Maze with the most direct route to complete the task;
3. Give participants the URL Maze provides.

Maze will give you aggregated data on how long it took people to complete the task and the number of mistakes they made.

If you don’t have a prototype, the alternative is to do a first-click test:

1. Show users a mockup of your website or app;
2. Ask where they would click to complete a specific task.

According to a usability study by Bob Bailey and Cari Wolfson, if the first click is correct, users have an 87% chance of completing the action correctly, compared to just 46% if the first click is wrong. So, if people get their first-click correct, you can be reasonably confident they can successfully complete the task.

Usability Hub can help you run your first-click test. They will provide a heat map showing the aggregated results of where everyone clicked, so you don’t need to analyze the results manually. This allows you to get answers almost immediately.

Will People Like It?

The final question is, “Will people like it?” This is not easy to answer, as different stakeholders may have different opinions about what works.

To resolve this, I usually conduct a preference test or, ideally, a semantic differential survey.

First, I agree with stakeholders on the associations we want users to have with the design. These may include words like professional, friendly, inspiring, or serious.

In a semantic differential survey, users can then rate the design against those words. If the design scores well, we can be confident it will generate the desired response.

A Pragmatic Approach

I know this post will make user researchers uncomfortable, and I can fully understand why. The results you get back will be far from perfect and could possibly lead to false conclusions. However, it is better than the alternative. Resolving design decisions through internal discussion is always going to be inferior to getting user feedback.

This kind of lean user research can also be a great starting point for bigger things. If you can add even some user research to the process, stakeholders can start to see its benefits, and it can lead to bigger things.

Some may choose to pick holes in your approach, suggesting that you aren’t testing with the right people or with a big enough audience. They are, of course, correct. However, this provides you with an opportunity to point out you would happily do more research if only the time and budget were made available!

Further Reading On SmashingMag

• “How To Build Strong Customer Relationships For User Research”, Renaissance Rachel
• “How To Build An Ethical User Research Practice At Any Organization”, Devin Harold
• “Eye-Tracking In Mobile UX Research”, Mariana Macedo
• “A Comprehensive Guide To UX Research”, Christopher Murphy

(yk, il)

Nurturing connections with your customers is one of the most effective ways to gain valuable insight into their experiences with your product and make informed decisions that fuel growth and success.

A really great way to build customer relationships is by regularly involving them in user research. However, getting customers involved in user research in the first place can be challenging because they might not understand what you want to do with them, and they’re busy.

Building customer relationships is not always straightforward. It takes work and investment.

In this article, we will go through various ways that product teams can utilize to build relationships with customers.

The Importance Of Customer Relationships

If the growth in the customer relationship management (CRM) software market is any indication, businesses see the value in relationship-building with their customers. Gartner estimates that the CRM market will grow over 14% through 2025. In 2021, Gartner reported that the CRM market share had grown to $69 billion worldwide.

A CRM tool enables businesses to communicate with their customers in a scalable way. However, true customer relationship management goes beyond a software solution. It involves person-to-person interaction and building trust. Business experts recommend viewing customer relationships as long-term relationships. While it’s important to gain new customers, it’s vital to nurture relationships with your current customers. CRM tools have robust functions to engage customers, from sales and marketing to customer service.

As Forbes describes, customer loyalty and retention affect a company’s revenue. A 5% increase in customer retention can produce more than a 25% increase in profit.

While it is important to engage with customers in the sales and service stages (e.g., when they need help), engaging with them while they’re using the product is also essential.

More specifically, after a customer has bought your product or service, do you only want to talk to them when they’re having problems?

One way to show that you’re invested in your customers is to bring them into the product development process through user research, which enables them to provide their input and shape the future of the products that they’re using.

By taking advantage of user research techniques such as user interviews, surveys, usability testing, and focus groups, you can uncover valuable data that will help you refine and improve your product while still keeping it aligned with customer expectations. You can ask your customers for feedback on specific aspects of the product and learn how to make it better.

However, in my experience as a user researcher in business-to-business (B2B) companies, I’ve seen that it can be challenging to involve customers in user research if you haven’t done the work upfront to build a relationship with them. Email invitations from CRM tools to participate in user research often go unopened if additional work hasn’t been done to nurture that relationship.

A user research session typically involves a participant having a 1:1 meeting with the researcher. It’s challenging enough to get people to respond to a quick survey, much less take the time to meet with a person face-to-face. Studies have been done on why responses to surveys have declined over the years. One reason cited is trust: participants distrust the organization or feel that there’s a lack of empathy.

I’d like to share some tips and tricks from my experience that have helped me nurture customer relationships and build better products that go beyond using the software.

How To Start Building Customer Relationships

While CRM software does an excellent job of managing customer relationships, it lacks personality because it takes away that personal approach we’ve relied on for decades before the advent of the software.

However, there are many benefits of using CRM software that help make sure the customers have a better experience with your company. For instance, CRM software can streamline conversations so the marketing and product teams don’t ask the same person the same questions.

I think there’s a middle ground that we can find somewhere. The personal touch still matters because it puts a face to your business, creates emotional bonds, gives you an edge, encourages loyalty, and fosters trust.

To enable a personal touch and be able to elicit product feedback from customers, there are a few different techniques you can try out. Let’s discuss them below.

Technique #1: Find Champions Within Your Company

One way to start building a customer relationship is to find other people at your company who already work with the type of customer you are interested in learning from. Their role will likely be different than yours. They might work in sales or customer success. They might not have heard about user research or understand why it’s actually an important way to deepen a customer relationship. Thus, it might seem daunting to find people who are supportive of user research, especially if your job is remote, where there are no casual water cooler conversations.

I recommend starting with your colleagues who work closely with you and helping them understand what you’re trying to accomplish. They might have people and resources they can identify for you that would be a good person to connect with. Don’t be afraid to ask for a personal introduction.

When I first started in my first researcher role, I found it difficult to advocate for the value of the work I did. I was the first researcher my team had ever had before. We had an old-school product manager who thought that the client conversations he had meant that we didn’t need to do additional research. I sought advice from my teammates and read various case studies to look for ways to communicate the value of my work.

Tomer Sharon, the author of It’s Our Research: Getting Stakeholder Buy-in for User Experience Research Projects, writes that “when you work with stakeholders, not on them, everyone stands to gain.” In user research, stakeholders are those that have a stake in the outcomes of your research. They’re the ones who can act upon your research findings.

A common issue, Sharon writes in his book, is the following:

“Researchers plan a study with their stakeholders, then desert them and run the study by themselves, only to come back to stakeholders a month later expecting them to fix usability problems, change the product roadmaps, or stop a release of a major redesign.”

As much as we want it to, the world does not get put on hold while we conduct our research. The other people on our teams have their own work they’re more worried about than yours. That’s why it’s important to regularly engage your stakeholders throughout the user research process.

I started trying out some of the tips from Sharon’s book along with advice from senior researchers at my company and found a huge change in attitude with my teammates.

When planning a research project, involve the right people in the planning of it. A good best practice is to have people from product management, software development, design, and sales there.

• Product management knows the business goals and priorities. They can help identify the right audience for the product. Product managers also lead development timelines.
• Software developers are the ones most influenced by the research results because they have to make code changes based on the findings.
• Design is a user researcher’s natural partner in getting things right, and they tend to be great at capturing critical observations.
• Sales are in close relationships with customers and might be critical to recruitment.

Finding out what your stakeholders want to know AND what they plan to do with the results are crucial aspects of creating an effective research strategy because then your research is more likely to provide value and be implemented. At the time, one of my senior research colleagues told me to “get three questions from your stakeholders that they want to be answered.” I’ve found that to be a good rule of thumb when trying to gather effective research questions.

In general, don’t expect that your stakeholders will initiate research planning meetings. As the researcher, that’s something you should own. It’s all about working together as a team. Sharon suggests having high-level research planning meetings with a senior product and/or engineering manager once a month or quarter. He suggests you should have more tactical research planning and updates with a product manager or engineering team lead every week or two. In short, high-level meetings with senior leadership, tactical with the people doing the day-to-day work.

Just as you initiate meetings with them, you should attend the meetings they hold, such as weekly product management and engineering leadership meetings, so that you have a better idea of where your work can fit in.

I learned that continuous involvement in research = continuous buy-in. Bring in your teammates at the beginning of your research planning. Involve them in all aspects of it. As Sharon wrote, don’t just involve them in planning. I like to bring my stakeholders into my actual research sessions. I put them to work, too. I have them take notes, and then we discuss what we observed afterward. That way, nothing is a surprise to them.

Another effective tactic I first read in Sharon’s book is to keep research studies small. The larger and more ambitious the study, the longer it’s going to take. By the time you’ve completed the research project, the team might have moved on to something else. Thus, it’s a good idea to conduct research in a smaller, more agile way.

When you present your research findings to your stakeholders, don’t give them a data dump. As tempting as it might be to show everything you did, they don’t care. They want to know what you found, and if you answered the questions they wanted to know. Your presentation is just the tip of the iceberg. My advisor in college told me that as I was preparing my senior thesis, it crushed my soul at the time. But she was right. To be an effective storyteller, you have to know what to leave out.

By bringing your colleagues into the research process, you’re giving them firsthand experience of what you’re doing, and you’re giving them ownership of the research itself.

When they’re invested in the outcomes of the research, they understand the need to involve customers in the research. They’ll start to see the value of research. A product manager I worked with started thinking that doing user research would help sell the product to new and existing customers because of how we were able to showcase our redesigned product.

All in all, building customer relationships also involves building internal company relationships. One way to do that is to bring your colleagues directly into the research process.

Once you identify the people who can help you build customer relationships, you’ll want to get specific about your request from them.

Technique #2: Get Specific On What You Want To Request

A general request of, “I want to talk to this customer about this,” without the details of what you’re planning to do, will confuse the person you’re asking to assist you and thus might appear reluctant to introduce their contact to you especially if you haven’t involved them in your research because you’re still new or they’re not on your immediate team.

Sometimes you have to go outside of your immediate network to find colleagues who can connect you with the right participants.

I don’t advise trying to reach out to someone at your company without first developing your research plan. A research plan should include the title of the research project, who’s involved with the project, the date last updated, the background of how the study came about, what goals the study should address, the research questions the project should answer, details about the research method, the target user and the number of participants needed, the schedule, and the script.

Once you have a research plan, you should have a clear understanding of what kind of participants you need for your research project. If you need to talk to someone outside of your immediate team that you don’t know very well, you probably don’t want to show them your whole research plan since that’s going to be a lot for someone who isn’t familiar with research to take in. I suggest pulling out the following details from your plan:

• Target user: Who we’re looking to connect with;
• Goal: What’s the goal of the research study;
• Research questions: We plan to ask them these sorts of questions;
• Time commitment: How much time you are asking for.

Getting clear on who you want to talk to and why will help your colleague know whether they can help you. Knowing details about what you’re doing with the customer and how long the time commitment is will help them understand how to introduce you to the customer.

Once you’ve gotten clear on those details, you can also try to share them on internal messaging boards such as Slack. For instance, I’ve made a “one-pager,” or a one-page flier that includes these details and posted them on Slack channels where I can find people with the customer connections I need. For example, our customer success managers have a special Slack channel, so I’ve posted it there.

When making a one-pager, think about making it as actionable and consumable as possible. I made the following sample one-pager in Google Slides. I used the basic “title and body” layout to write the title and three bullet points. Then I used the “Explore” feature to make the layout look more engaging.

One-pagers like this can be used in other avenues besides Slack. If you’re meeting directly with someone, you can show them the one-pager so they can read it and have a visual to go along with what you’re asking. If a product manager is presenting to customers, you could ask them to include the one-pager at the end of their presentation as a call to action.

Getting specific on what you’re requesting from your colleagues also has the benefit of helping you hone in on your search for customers because it forces you to articulate what you’re looking for.

These first two techniques have looked at how to find internal advocates and assistance to connect with customers. The other tactics involve ways to look externally and find customers directly.

Technique #3: Discover Where Users Spend Their Time

You want to try to understand where your customers spend their time. If you don’t know that, how can you expect to reach them? This might mean taking a look at your web analytics to see where people are coming from before they land on your website or product. It might also mean conducting some user research to ask people where they get their information or what kinds of websites and apps they use on a daily basis. This might be hard to do if you have no one to talk to, but once you do, it’s a strategy you can implement to build out your contact list even more.

One of the most reliable places to connect to your product users is to find a user group or online community specific to your industry or product. For example, if you make accounting software, there’s a good chance that there’s an online community for people who use QuickBooks. You can go to the forums dedicated to QuickBooks and see what conversations people are having. You might try to directly message people who’ve made helpful comments or try posting on the forum itself. You can also get insights into what these users are thinking and feeling about your product or industry and keep a pulse on what product users are talking about. In addition, use the community forums as a method to do some light research.

The benefit of user groups and online communities is that you can get very specific with who you’re talking.

I took advantage of the IBM Community forum to engage with our customers in the hopes of getting them to sign up for user research sessions. In the example post below, I included a screenshot of the homepage for one of our products and asked what the forum members thought about it. You can view the post here and see other questions we asked here.

I got a relatively high response rate on this post because it had a visual for them to respond to and was related to a commonly used feature. In certain posts, I included a link to a sign-up page or a survey that participants could take to get extra raffle tickets.

The raffle tickets were part of how we created an entire event experience on the forum itself to encourage engagement. If you’d like to read more about how to run an entire event on a forum, check out my blog post about it.

You could try doing something similar by posting an image from your product, asking for feedback, and giving a link to sign up to participate in research. If applicable, you could mention the monetary incentive they might receive by participating.

Engaging with customers on a forum is also a nice way to do some continuous user research and get feedback outside of individual research projects.

It’s hard to write content in forums that doesn’t sound spammy, so it might take some trial and error to figure out what kind of messaging will work for your particular target user group.

To strengthen the personal touch, there are other strategies you can use. We will discuss them below.

Technique #4: Attend Trade Shows And Conferences

Trade shows and conferences are great places to meet potential customers. They give you that face-to-face interaction you miss with online communications. A list of upcoming trade shows and conferences can be found by searching online, on social media, or by finding out what events your marketing team plans to go to.

Once you’re at the trade show or conference, be sure to stop by the booths of companies that might be potential customers. Introduce yourself and let them know that you do user research. It’s also a good idea to wear a name tag that has your name and the company you work for. These name tags make it easy for people to find you and start a conversation.

Attending talks and sessions that are relevant to your product or industry is also extremely helpful. I’ve had amazing luck just walking up to a conference speaker after they give a presentation and asking them if they’d like to join our research program. After watching someone give a talk, you know if they might be an articulate participant.

For instance, I was at a user group conference and saw someone give a really helpful presentation on the product I worked on. I knew I’d love to work with him to improve our product, so I introduced myself and asked if he’d be interested in giving product feedback. We exchanged contact information, and he became very engaged in future research activities.

The next time I attended the conference, he told his colleagues about how much he enjoyed participating in research activities and how much that meant to him. It encouraged more interest in our research program.

As you start attracting people to do research with you, you’ll want to find a way to organize and manage your relationship with them.

Technique #5: Build Your Own Participant Panel

It might sound daunting, but let’s start small. When you think of participant panels, you might think you need at least a thousand people. While that is a great goal to shoot for, having that many is not necessary to start running research studies.

Believe it or not, you have a participant panel when you have connected with just one customer. That customer can connect you with other people at their company or in your network.

Be sure to check your company’s policies around storing personal information and what tools you can use to do that. You don’t want to be in violation of any internal policies.

“

You might consider using your company’s CRM tool to manage your participant panel, or you might use panel management software, that’s more specific for user research use cases. Panel management software differs from CRM software in that there are dedicated functions to help you screen participants and schedule research sessions. Evaluate what tools you have available and see what your needs are.

For example, your company might use Salesforce as its CRM to manage customer relationships. Salesforce works really well as a marketing, sales, and support tool. It aggregates data from various sources, allowing businesses to have a central source of truth about their customers, giving a complete picture of what’s happening with a customer.

That said, Salesforce does not fully meet the needs of user researchers. While it’s very helpful to see information such as customer browsing behavior, demographic information, and support ticket history, it doesn’t necessarily help a user researcher connect with that participant for a focused research session.

In order to properly run a research study, you have to find the right participants, schedule sessions with them, and, if applicable, have them sign non-disclosure agreements (NDAs) and compensate them for their time. Technically, something like this could be done in a CRM tool, but it would involve more steps than a tool designed to run research. It might be challenging to get your research team to adopt such a complex workflow.

A dedicated panel management tool like User Interviews or Great Question works with CRM software by pulling the relevant data (such as name, job title, demographic information, employment type, and so on) into its platform and allows you to focus on running your research. These tools are designed to have a database of participants you can invite to participate in a study. If you need to send them a screener survey, that capability is built into the tool. You’re able to set a calendar with your availability, and the participant can book a time with you. Document signing and consent forms are built into the process, so you don’t have to worry about data compliance. After a session has been completed, you can distribute monetary incentives to the participants to thank them for their time.

While using a panel management tool might be simpler than a CRM tool for conducting research, it does add another tool to the mix, so that’s where it’s important to evaluate your team’s needs. When there’s another tool with customer data, you’ll have to update two datasets. Does the easier user experience outweigh the possible difficulties of having another dataset? If you’re able to integrate the CRM software with the panel management tool, then you don’t have to worry about keeping data up to date in multiple places.

Once you’ve got a panel, you’ll start having access to potential participants without having to go through gatekeepers or spend time recruiting people.

However, the access to participants only lasts if you maintain those relationships. You’re not done after you recruit them the first time.

Technique #6: Take The Time To Build Relationships

Getting to the point where you have easy access to customers can take a lot of work. But just as it is with any relationship, you still need to put in work to maintain customer relationships.

When you get busy with your job, it can be easy to forget the personal touches. You might start reaching out to your customers only when you want to get their feedback on the product. That can start to feel very one-sided. It’s about taking the time to connect and learn about them as people.

Be sure to ask them how they’re doing, what they’re working on, and what their weekend was like. Share something about yourself. People love to connect with others around common interests and values. Also, send thank you notes, too! It’s just as crucial in user research as in any other field.

In my practice, I like to send out newsletters to the customers I work with to update them on how we used their feedback to improve the product. Since the value to the customer is seeing the product improve, I tell them about the takeaways from research projects and how we plan to address them.

Remember that you’re playing the long game here. Relationships take time to build, so it requires perseverance and patience.

Technique #7: Be Patient And Track Your Impact

Building relationships with customers is key to success in user research. However, it takes time. You might not get immediate results, but if you’re patient and keep at it, you’ll eventually find the people you need to talk to.

You might be surprised that the work you’ve put in now will pay off in huge ways you might not have anticipated six months down the line or even a year or more.

It can take time to get buy-in. The important thing is to keep going!

Be sure to recognize your wins: big or small. Keep track of them and remind yourself of them when you get discouraged.

• Did someone invite you to talk about your customer feedback program at one of their meetings? Yay! That’s huge!
• Has one of your stakeholders who’s been difficult to work with in the past asked you for your opinion on something? Congrats!
• Did someone sign up for a research study from a community forum post? Amazing work!

It may not seem like much at this moment, but over time it truly adds up to a meaningful impact.

Final Thoughts

Building customer relationships is vital to user research. These steps will help you get started, but you’ll have to adapt the method to your specific needs.

The most important thing is to be patient and to take the time to connect with people. User research is all about relationships!

(yk, il)