How to Navigate UX Research for GenAI

Now, let’s consider some topics and example questions that you’ll need to answer relating to each topic—at least at a high level:

• Model and training:
• Is this an LLM? Is it open source? Has your team fine-tuned it? On what data was it trained? Have you considered biases in the training data?
• What GenAI capability are you using for the feature or product—for example, summarization or a Q&A?
• Is the algorithm using retrieval-augmented generation (RAG) to optimize the output of the large language model by referring to an authoritative knowledge base outside its training-data sources before generating a response?
• Do users need to input prompts? Is prompt engineering necessary?
• General performance of the model:
• How accurate is the model? Specifically, what is the risk of hallucinations for the model—that is, the generation of inaccurate, made up, missing, or misinterpreted information?
• What is the model’s latency—that is, the time necessary to load the answers?
• How often does the model provide an answer? What happens if it cannot determine an answer?
• Are there certain parameters in place that you should convey to users?
• Output variance:
• Given users’ increased awareness of and experience using consumer GenAI products—for example, ChatGPT, Gemini, or Perplexity—how does your experience fit users’ expectations?
• Does the user have control? Can the user modify the tone? Length? Can the user select and verify sources?
• Onboarding and guidance:
• How do you introduce users to the experience? How do you teach them the boundaries and limits of the experience?
• Does the product provide examples or templates?
• Does the product offer follow-up recommendations?
• User data privacy:
• Does the product require personally identifiable information (PII) or any other personal user information? Would the user be aware of this and have control over it?
• What data would the product collect from the user to train the model? Would the user be aware of and have control over this? (Check out the recent publicity regarding LinkedIn choosing to automatically opt-in users to use their personal data in training their GenAI.)
• User transparency and control:
• How much control does the user have? Must they use an AI feature? Are they aware that they are using an AI feature? Does this align with their expectation? Can they opt in or out of using it?
• Can they modify the output? Edit it? Select sources to use or verify them afterward?

These are just some of the topics that might be relevant to you as a UX researcher. Your cross-functional team should be able to help you answer these questions. Never feel bad about asking for help, especially if you have only a rudimentary grasp of the overall technology. Convey why you are asking the questions. If your team does not know, your asking them helps identify areas that need further investigation by the team. Of course, the answers to these questions are purely informational, but matter because all of them can hugely impact the desirability, usability, and ultimately users’ trust and value for a GenAI experience. You must increase your knowledge enough to communicate how the team’s choices and the risks of certain approaches would impact the user.

Changing Your UX Research Methods and Approach

Beyond learning about GenAI technology, successfully navigating GenAI UX research (UXR) requires a different mindset regarding how we approach and execute our work. Although traditional UX research methods are absolutely foundational and helpful, we have found that tweaking these methods provides more meaningful insights and informs the creation of a better user experience. Some helpful approaches include the following:

• Being flexible with timelines:
• GenAI development moves at breakneck speeds, so product teams must move fast to adapt to this environment. Therefore, we need to adapt our UX research approaches to meet the needs of our teams.
• Rather than relying on traditional, fixed-timeline studies, incorporate iterative, agile research methods that can provide quick pulse checks to learn how users adapt to AI features over time.
• Incorporating longitudinal research approaches:
• Users’ perceptions and beliefs about AI change over time with exposure to the technology. Do not make the mistake of thinking their needs won’t be different in even just three months.
• Ensure that you’re checking in with representative users regularly and finding ways to incorporate longitudinal research methods to track the changing relationships between humans and AI.
• If you can’t conduct diary studies or capture within-subjects repeated measures, consider simpler pulse surveys or benchmarking.
• Not relying only on populations that have not been exposed to AI for concept testing:
• To mitigate any bias and expose any differences in acceptance or understanding across user groups, focus on recruiting users with various levels of AI comprehension.
• Be aware that it is incredibly hard for users to tell you whether a concept makes sense to them when they’ve never experienced GenAI or AI before or are experiencing GenAI in a new context of use.
• Owning responsibility for experience outcomes—that is, responsible AI:
• You must own your role as the voice of the user within the context of these experiences.
• By clearly communicating the human risks of AI experiences, you can uphold the tenets of responsible AI. Document these risks for your team.
• Your research read-outs, reports, and presentations should reflect this level of responsibility and focus on actionable steps that let you foresee and mitigate poor experience outcomes.
• In general, everyone wants to do the right thing. If you have concerns about an AI experience, don’t be timid—communicate them.
• Understanding cognitive aspects of trust and how trust varies by the user’s role and context of use:
• With AI experiences, a key piece of the UX puzzle is how users perceive, gain, and lose trust through interactions with an AI.
• As a UX researcher, you are best positioned to not only bring awareness of the role that trust plays in these experiences but also to understand, measure, and track users’ trust in AI and your GenAI experience over time.
• Acknowledging that users might not even exist or could emerge suddenly:
• The hardest part of managing UX research for an AI experience might be that users of these experiences don’t yet actually exist. You might be researching a brand new experience or be the first person exposing users to AI within their context of use. This presents unique challenges in interviewing participants and introducing them to GenAI experiences.
• Plus, new personas or job profiles could emerge suddenly and without much warning, especially if your experiences serve other businesses or providers. While we often think of technology’s replacing tasks, in the case of AI, we’re also seeing new tasks, new jobs, and new roles.

Shifting Your Communication Strategy

We’ve found that the way in which we communicate insights has evolved and changed and expect that is the case for most UX researchers working within the context of AI. The audience for our research and the impacts of our research insights are different. Ensuring that our presentations and insights meet the needs of our stakeholders is more important than ever, especially with extremely cross-functional teams that have varying degrees of AI experience and ownership of different parts of the product’s experience.

• Cross-functional partners must make decisions based on your UX research, often in real time. Plus, these decisions are often riskier in terms of their impact on the user experience, your UX strategy, the overall business strategy, and cost.
• The pace at which you deliver insights and notify your team of changes over time requires more rapid cycles of delivery with an emphasis on actionability.
• Conducting research while supporting the pace of innovation can test your limits as a UX researcher. You might need to make sacrifices or face burnout if you try to do it all. Instead, find ways to incorporate bigger questions and strategic initiatives into quicker cycles. Regularly link your findings back to the bigger picture for your team.
• Different messages and levels of insights are necessary for different teams—for example, AI Applied Research versus Design versus Product versus UX Research. Understand what each team needs to make decisions or take their next step and tailor your findings accordingly for maximum impact.

Applying Your Knowledge and Elevating the User

As you navigate this new frontier, please know that you likely possess all the necessary skills and abilities to thrive, grow your knowledge, and positively impact users. Although many of the tools in the Gen AI UXR toolbox are familiar, you might find that the ways you use them and the impacts of your work are very different.

When working on a cross-functional product team, you might be the only person present who can reflect the actual user’s needs and ensure that the experience you create adds real value and is worthy of their trust. Do not be afraid to ask questions and speak up and share your findings. More than ever, we need to leverage users’ feedback to make important decisions and influence adoption and investment across the entire organization. Represent users responsibly and with confidence! 

Resources for Growing Your AI Expertise

NN/g’s “AI Glossary.”

Coursera course from DeepLearning AI, “Generative AI for Everyone.”

Nvidia’s “What Is Generative AI?”

“Principles of Generative AI, A Technical Introduction,” (PDF) by Karen Singh, Assistant Professor of Operations Research at Carnegie Mellon Tepper School of Business

McKinsey & Company, “What Is Generative AI?”