The Biggest Mistakes in User Research, Part 1

Relying Too Much or Too Little on What Participants Say

The goal of user research is to understand what your users want and need in the product you’re designing for them, so you can get this information simply by asking them what they want, right?

Wrong! The goal of user research is to understand the users, their tasks, their tools and technology, and their environment. The best way to get that understanding is to observe and interview them in their natural context as they perform their typical tasks.

Methods of research in which users are outside the natural context of their tasks and that rely too much on asking people what they want are least reliable. That’s because it’s very difficult for people to know or define what they want and need in a product. It’s also difficult for them to describe their tasks and the problems they face accurately when they’re not in the context within which they usually perform their tasks.

However, some people take the fact that people have a hard time describing what they need too far and completely discount everything people say. I’ve seen UX articles that advise, “Don’t listen to users,” all over the Web. While that idea might make an unexpected, attention-grabbing title for an article, such statements can mislead people, causing them to devalue the importance of user interviews. Plus, while observing users’ tasks is key, you can’t fully understand what people are doing and why without listening to what they tell you.

Solutions

To gather all the information you’ll need when designing a product, do the following:

• Focus your research on understanding users, their tasks, their tools, and their environment by visiting them at their workplace or in their home and observing them perform their typical tasks that relate to what they’ll do using your product.
• When you can’t visit and observe people in their natural context, ask them to recreate their typical tasks, as you’d normally do during usability testing. It’s much easier for people to talk about their tasks and the problems they face when they have something to demonstrate or something concrete to which they can react.
• Use a combination of observation and interviewing to understand what users are doing.
• Instead of asking users what they want and need directly, infer what they need by observing their tasks and discussing the goals they are trying to achieve and the problems they face as they carry out their tasks.

Separating Research from Design

Treating research and design as separate activities that different people perform is a mistake. While it’s fine to have specialist UX researchers and designers, it’s very important for designers to develop a deep understanding of the users and their tasks. When designers aren’t directly involved in the research, they can learn about the users only through a findings presentation or deliverable. It’s difficult to absorb a deep understanding of the users in this way.

Solutions

Research is part of the design process. Keep research and design together by doing the following:

• If your project team consists of generalist UX designers, have them conduct the user research themselves.
• If your project team consists of user-research specialists and design specialists, have them work together throughout the project’s research and design phases. The research specialist may be primarily in charge of the research, but should partner with the designer in planning the research, observing the sessions, capturing notes, and analyzing the results. By participating directly in the research, both researchers and designers gain a much deeper understanding of the users and their tasks, enabling the designers to make much better design decisions.

Leaving Others Out of the Research

Even when researchers and designers are conducting the research together, it’s a mistake to leave everyone else on the project or client team out of the research altogether. When everyone else must wait to hear the results of the research, they’ll probably wonder: What’s going on? Why is it taking so long? When are we going to start designing something? Plus, they won’t feel as invested in the research findings so, by the time you give your findings presentation, they’ll already be eager to move on to design.

Solutions

Involve the project team and your clients in the research in the following ways:

• Plan the research around what they want to learn about the users. What questions have they had in the past? What have they always wanted to know? What do they assume about the users, but don’t know for sure?
• Have them review and provide feedback on the research plan and discussion guide.
• Involve them in recruiting and approving the research participants. This prevents them from later feeling that the participants weren’t representative of their users.
• Encourage them to observe the research sessions whenever possible.
• Provide updates about what you’re learning along the way. While you won’t be communicating your formal conclusions, you can tell interesting anecdotes about the types of people who have participated in the research and things you’ve witnessed during recent sessions.
• Involve them in analyzing the results. While you may want to do your own analysis of the data independently, you could also involve them in affinity diagramming or group discussions about the things they observed when they attended research sessions.

Bringing Too Many People Along on Field Studies

While it’s great to allow clients and project-team members to observe the research sessions firsthand, there’s a practical limit to how many people can comfortably attend field studies without intimidating the participants. Usability labs, focus-group facilities, and remote sessions make it easier for many people to observe research sessions discreetly, without affecting the participants physically or psychologically. However, bringing too many people along on field studies can make participants feel uncomfortable, which defeats the purpose of trying to observe their natural behavior.

The primary advantage of field studies is that you can observe people in their natural environment, performing their typical tasks. Of course, you can’t avoid the reality that your presence will affect their behavior somewhat, but you should minimize that impact as much as possible. The more observers you bring along, the more likely it is that participants might feel uncomfortable.

The ideal number of people conducting field studies seems to be just two people—to ensure both participant comfort and that you’ll fit comfortably into the physical space. Most homes and workspaces can accommodate a few people, but not a crowd of people. Plus, we often need to observe users’ tasks in detail, which requires sitting very close to a participant. It’s often physically difficult for more than one or two people to get close enough to see what’s happening on a computer screen, tablet, phone, desk, or counter top.

Solutions

You can allow extra people to observe field research, while avoiding crowding and intimidating participants, by doing the following:

• Ideally, limit the number of people attending field studies to two, including the primary researcher and an observer. Do not bring along more than three people at the most.
• Enable more people to observe the research by having them take turns being the observer. Observing two or three sessions is enough to give clients and project-team members a better understanding of the value of user research and to help them empathize with users.

Being Too Inflexible

No matter how well you plan a study, user research can be extremely unpredictable. You’ll often find that much of what you assumed about the participants and their tasks is very different from the reality that you encounter. Unlike usability testing, field studies are often very unpredictable. You may begin with a set of tasks to observe and questions to ask, but you’ll usually have to adjust those to take into account the reality of each participant’s experience. Being too inflexible and trying to stick with your preplanned list of questions and tasks can make a session feel too formal—more like a traditional interview instead of a natural demonstration of the user’s tasks.

Solutions

Do the following to avoid making user-research sessions feel too formal and inflexible:

• Create a discussion guide that lists the questions you’ll ideally ask and the tasks you expect to observe, but emphasize to your client and project team that user research is very unpredictable, so it’s only a preliminary guide. Your actual sessions can vary widely from this script. Make it clear that it’s important to be flexible and adapt the actual questions and tasks to each participant and his or her situation.
• Use the initial version of the discussion guide to organize your thinking. Provide this document to the client and project team for review, then revise it, incorporating their feedback.
• Once you’ve finalized the discussion guide, save another version, then cut it down to a very brief document that you can easily read at a brief glance during research sessions. Because sessions can be so unpredictable, jumping around from topic to topic, it’s very difficult to follow a long, detailed discussion guide. So cut it down to just one or two pages at most, with bullet points comprising topics of discussion or brief questions. Ideally, you should be able to glance down briefly to read a topic or question. For example, a brief bullet point such as, “Learning the system,” is easier to read quickly than the question, “Can you tell me about how you learned to use the system?” Check off those topics or questions you’ve already covered.
• Despite your having planned topics and tasks, be flexible about letting participants lead the session in whatever direction makes sense for their experience. Give them some latitude. But sometimes, if participants drift too far away from what you need to learn about, you’ll need to lead them back to a relevant topic.
• If you find that most participants are leading you in a different direction, away from the topics and tasks you’ve planned, perhaps you’ll have to change the focus of your research to better fit their reality. Alternatively, you may need to re-evaluate whether you’re doing research with the right people.

Not Providing Enough Time for User Research

What’s the one thing that user researchers never have enough of? Time! Although we can always use more time, these days we usually feel lucky if user research is included on projects at all. While there are ways to save time—such as conducting remote research, doing fewer sessions, or creating lighter deliverables, we can’t get around the fact that user research does require a significant amount of time. Recruiting and scheduling participants, traveling to their locations, conducting the sessions, analyzing large amounts of data, then creating a deliverable that effectively communicates the findings to your audience—all of this takes a lot of time.

Solutions

Consider doing the following to ensure that you provide enough time for user research:

• Before scoping a project, gather as much information as possible about the user groups and their tasks, so you’ll have enough information to provide a realistic estimate of how much time you’ll need for research.
• Include other user researchers or UX professionals in the project scoping.
• If there isn’t enough time to conduct user research during a project, conduct it before the project begins, as part of the initial requirements gathering and planning process.
• If time is tight, narrow down the research goals to understand a smaller set of issues or focus on fewer user groups.
• Eliminate the need for formal deliverables by including the designers and other project-team members in observing the research and analyzing the findings.

Scheduling Sessions That Are Too Short

It takes a lot of time and effort to recruit and schedule participants, plan their sessions, travel to them, and conduct the sessions. So it’s very frustrating when sessions are too short and you run out of time for the tasks you want to observe and the topics you want to discuss. If the sessions are too short, you’ll often end up skipping some tasks and rushing through others, and you won’t be able to ask as many questions.

In an ideal situation, you’ll know which tasks you need to observe with each user group and how long those tasks normally take, so you can determine how long the sessions need to be. However, it can be difficult to get accurate information up front and, because of the unpredictability of user research, you can’t know exactly how long each participant will take.

Solutions

To avoid running out of time during research sessions, do the following:

• During the planning phase, identify the user groups and the tasks they perform. Then find out how long it typically takes to perform those tasks. Consider the fact that each task will take longer to perform when participants are demonstrating them and talking about what they’re doing. Your questions and their answers take up additional time. Plus, you need to consider the time it takes to introduce yourself and explain your research method to participants.
• Err on the side of scheduling sessions that are longer than necessary. It’s much better to have extra time and end early because you’ve covered all the tasks and all your questions. No one minds ending early.
• Also, balance session lengths with the amount of time participants are willing and able to give you. You don’t want to ask someone for a four-hour session if that would discourage them from participating. Higher-level executives may be too busy or consider their time too valuable to give you more than 30 minutes, while lower-level employees may be able to give you much more time. 
• If the number of tasks you need to observe and the lengths of those tasks is too much for a single session, you can divide the tasks between different participants in the same user group. For example, if you want to see five nurses perform tasks A, B, C, and D, you could ask participants 1 through 5 to perform tasks A and B; then, ask participants 6 through 10 to perform tasks C and D. 
• If it is important for you to see the same participants perform all of the tasks, but the tasks collectively take too long to perform them during a single session, schedule multiple sessions with each participant.
• Session length also depends on your research method. In a contextual inquiry, you require participants to take an active role in demonstrating their tasks, talking about what they’re doing and answering your questions. This can be both mentally and physically taxing, so there is a practical limit on how much time a person can give you. However, if your method is simply to observe participants, without interruption, you could spend an entire day or longer observing them, without requiring much additional effort on their part.

There Are Too Many Mistakes to Fit in One Column

As you can see by now, user research is actually very difficult, so it’s easy to make mistakes. In fact, there are so many possible mistakes, I can’t fit all of them in just one column. So, in Part 2, I’ll discuss the following user-research mistakes:

• poor scheduling
• failing to prepare participants in advance
• getting distracted by notetaking
• failing to record the research sessions
• interrupting participants too often
• failing to manage observers effectively
• rushing through the analysis and deliverable phase
• doing nothing about the findings

While, in this column, I haven’t presented a comprehensive list of all possible user-research mistakes, I have covered many of the most common mistakes. However, if there are other mistakes you’d like to share, please feel free to join in the conversation in the comments.