17 Usability Testing Myths and Misconceptions

Myth 4: Usability Testing Is Artificial

People often think that usability testing brings people into a lab to perform preset tasks that are not what they would normally do. And that, since the artificial environment and unrealistic tasks do not show you what people do in real life, you need to get out of the lab and conduct more user research in the field.

The Truth: Usability Testing Doesn’t Have to Be Artificial

Yes, some aspects of usability testing can be artificial, but you can make usability testing more natural. Usability testing can include participant-defined tasks, which makes the tasks more personal and meaningful. You can also conduct usability testing in the field, going to the users and observing them perform tasks in their natural environment. Field studies are better for gathering information about people’s behavior, real-life tasks, and contexts of use; while usability testing lets you observe people trying to use a product, which is the best way to evaluate a product’s current user experience.

Myth 5: You Must Have a Usability Lab

Many people think that usability testing must be performed in a usability lab to get valid results.

The Truth: Usability Testing Is Often Better Outside a Lab

While testing in a usability lab offers many advantages, you can conduct tests almost anywhere. In fact, it’s often more natural and comfortable for participants when you go to their location. Remote usability testing is another way to include participants to which you might not be able to travel. This can make it easier for you to reach the right participants, which is much more important than testing in a usability lab.

Myth 6: You Must Have a High-fidelity Prototype to Test

Some believe that it’s best to wait until you have a fully interactive prototype to test. They’re concerned about your not being able to test all of the interactions until you have a realistic prototype with which people can interact. They’ll say, “If we’re going to do testing, we should wait until we have something more final, so we can get realistic reactions from the participants.

The Truth: It’s Better to Test Early and Often

By the time a team can build a fully interactive prototype, they’ve already set the design direction. Sure, they can still make changes, but unless testing reveals that a design is an utter failure, rarely do they make major changes. Testing can and should occur at many stages throughout the design process—whether in the form of card sorting, tree testing to determine the organization of content, or usability testing with paper prototypes, low-fidelity prototypes, and finally, high-fidelity prototypes.

Myth 7: You Should Test Just Before Launch

Some think that usability testing should occur once a Web site or application has been developed to find any usability problems before going live.

The Truth: Testing Before Launch Reveals Problems, but It’s Too Late to Fix Them

Testing before launch will either reveal that you’ve designed an effective user interface or identify problems that users will face, complaints that you’ll get, support problems you’ll have to deal with, and training that will be necessary. If you wait to test until just before launch, it’s too late to fix anything but the most minor problems. You’ll have to fix the big problems in the next release. The only good thing about testing before launch is that it highlights the value of doing iterative usability testing during the design process.

Myth 8: You Need X Participants

I’ve heard people say, “You only need to test with five participants.” “You need to test with 12 participants.” “You need to test with at least 30 participants.”

The Truth: The Number of Participants That You Need Depends on the Type of Testing

There’s no magic formula to determine the right number of participants. The number of participants you need depends on the type of testing you’re doing. If your goal is to test a design to find problems, then fix them through an iterative design process, it’s better to test with five or six participants at a time, over several rounds of testing. If your goal is to assess the usability of a user interface and you’re gathering metrics, you’ll need a larger number of participants.

Myth 9: You Should Use the Same Participants You Used for Earlier User Research

Some think that you should use the same people who participated in earlier user research for later rounds of usability testing on a project. They think that the purpose of usability testing is to validate what you previously learned from those participants and give them an opportunity to let you know whether the system now meets their needs.

The Truth: It’s Better to Use New Participants

While it’s possible to use the same participants again, there’s no real need for it; and it’s usually better to use people who haven’t previously been involved in your research. Involvement in earlier research can cause participants to think more about a user interface and their tasks during a later study, causing them to have different reactions from participants without any preconceived notions.

Myth 10: You Should Include as Much as Possible in a Test

Some people assume that, to make the most of your time and learn the most, you should try to fit as many tasks and questions into your study as possible.

The Truth: Less Is More

When you unrealistically cram too many tasks or questions into each session, you’ll often run out of time before covering everything. The facilitator may have to skip tasks or rush through each session to fit in as much as possible, which doesn’t leave much time to ask probing questions about problems or gather more meaningful information. It’s better to focus on fewer tasks and questions, leaving you enough time to explore the reasons behind participants’ actions. If you think you may be trying to test too much, review your test objectives, then focus only on what is most important for you to learn.

Myth 11: You Should Test Several Different Design Solutions

Some think that it’s best to test several different versions of a design to see which one works best.

The Truth: It’s Often Better to Test Fewer Versions

Comparative testing to evaluate different design alternatives can be helpful, but some people get carried away, trying to test too many different versions of a design solution. This increases the complexity of the sessions for both the facilitator and the participants. Testing multiple versions requires either testing with a large number of participants—each using a different version—or having each participant experience multiple versions. The more versions that each participant experiences, the harder it becomes for them to distinguish between the versions. Testing multiple versions also requires more time, which limits the number of tasks that participants can perform during a session.

While comparative testing can be useful and has its place, too often teams use this approach when they’re unable to make a design decision. Instead of relying on testing to make design decisions, choose a design direction, test it, then iterate the design as necessary. When you do need to compare alternative designs, try to limit testing to two versions.

Myth 12: You Should Make Design Changes During Testing

People sometimes believe that, once you’ve seen a few participants having the same difficulty, you can conclude that you’ve identified a problem and don’t need to test that task anymore. They might think that you should eliminate that task from the test and use a different task for the remaining participants. Or they might want to design a quick solution to the problem, then test it out with the remaining participants.

The Truth: Making Changes During a Test Is Often Premature

While facilitators may adjust their questions depending on participants’ actions and responses, they know that they should avoid making definitive conclusions until the testing has concluded. Observers, however, sometimes jump to conclusions after seeing only a few participants experience a problem. Changing tasks or changing the design based on the results of just a few participants is often a bad idea. It’s usually too soon to determine whether something is a problem, and analysis becomes difficult when too few participants have experienced the same tasks using the same user interface because you’ve made changes midstream.

Unless there’s a problem with the prototype or the tasks, it’s better to keep things the same throughout testing and see what all of the participants do. Instead of making changes in the middle of a test cycle, divide testing into two or more rounds with fewer participants. You can draw conclusions from the first round of testing, then incorporate task and design changes for the next round of testing.

Myth 13: Usability Testing Is Like User Acceptance Testing

Some believe that usability testing is just like user acceptance testing. They think you’ll find out what the users think about the user interface design. Does it match their requirements? Does it meet their needs? Do they like it?

The Truth: Usability Testing Is Nothing Like User Acceptance Testing

UX professionals know that usability testing is very different from user acceptance testing. People who are representative of users participate in the testing, not project stakeholders or subject-matter experts. Usability testing focuses on the user experience. It’s not about whether the user interface meets business requirements.

Myth 14: We Must Do What Users Tell Us to Do

Some people think that, during usability testing, users tell you what you need to change in the design. They assume that you have to do what the users’ feedback and test results tell you to do.

The Truth: The Project Team Decides What Changes to Make

I’ve had clients who have said, “We’re not just going to let the users tell us what to do, are we?” They’d assumed that we might have to do whatever participants told us to do during testing. They worried that testing would take away their control over design decisions. Usability testing provides information about problems and possible solutions, but it’s up to the project team to decide whether to implement the recommendations. They’re not forced to do anything based on the results of testing.

Myth 15: Testing Once Is Enough

Some people believe that you can test to find problems, make changes to the design accordingly, and all of the problems will be solved.

The Truth: It’s Best to Design and Test Iteratively

While one round of usability testing is better than no testing at all, you’ll find only the problems that exist in the user interface at that one point in time. The recommended design changes are well-educated guesses about what should solve the problems, but there’s no guarantee that they’ll result in an optimal design solution or that your recommendations will be implemented correctly. Iterative design and testing is the best way to ensure that your recommendations have solved the problems and didn’t add any new problems.

Myth 16: We Don’t Need to Test What We’re Redesigning

Some people think that you that, if you’re completely redesigning a product’s user interface, it doesn’t make sense to test the current user interface.

The Truth: Testing the Current Version Provides Valuable Insights

When you’re doing a redesign, testing the current user interface is a great way to understand the strengths and weaknesses of the current version. This provides valuable insights into what the future design needs to provide. The current version gives participants something concrete to react to. You can also learn a lot about users’ characteristics and tasks, which is helpful in designing the new version.

Myth 17: Usability Testing Alone Is Enough Research

Many people believe that, if you do usability testing, that’s enough to understand your users.

The Truth: Usability Testing Is Just One of Many User Research Methods

Usability testing is a great method of identifying specific problems with an existing design, but it doesn’t give you much depth of understanding of your users, their contexts of use, or their needs. It’s necessary to conduct observational user research within the users’ own context to understand their needs.

Conclusion

Despite the fact that more people know about usability testing than ever before, you may still encounter these myths and misconceptions on projects. Some of these misconceptions may sound out of date—like something you haven’t heard in years—and you may think, “Oh, people can’t still believe that.” But I often find that, just when I’ve started assuming that everyone must know what I’m talking about, another old myth or misconception arises.

Even if your clients and project team members do seem knowledgeable about usability testing, it’s best not to assume that they know about and understand user research methods. Explain everything clearly and ask for questions to ensure that you can avoid any misconceptions.