Moderated Versus Unmoderated Usability Testing

Best Practices for Moderated Usability Testing

To ensure the success of your moderated usability-testing sessions, employ the following best practices:

• Define goals and the scope of testing. You’ll need to figure out which components of your product you want to test, as well as your main testing goals and the tasks participants should complete to meet each goal.
• Clearly define the selection criteria for test participants. To acquire the most meaningful insights, make sure you get test participants belonging to the correct demographic. If you’re wondering where to find these people, check out this article on recruiting usability-test participants for free.
• Conduct moderated usability testing according to the right process. Sessions should follow this sequence:
  
  • pre-session questions—Inquiring about participants’ past and general behavior will aid in the identification of common habits and other personal facts about your target audience.
  • actual tasks—The test participant performs a list of tasks using the product. All tasks should be selected according to the goals of the testing.
  • post-session questions—Ask participants general questions about their experience and gather feedback on the product. Post-session questions give test participants the opportunity to share their thoughts and ask questions regarding the product.
• During the testing, urge test participants to think aloud. It’s a good idea to have them talk through what they’re doing as they’re doing it. When test participants think aloud while completing tasks, the moderator can better understand the reasoning behind each decision, revealing the participant’s motivations and opinions.
• Take notes during the test sessions. You can learn a lot from your participants, and you won’t be able to remember everything that happens. Therefore, it’s a good idea to jot down any intriguing observations or data you come across so you can refer to your notes later while analyzing the test results.
• Analyze the findings and present them in a report. Write a report that contains the most significant findings and assists the product team and stakeholders in prioritizing design decisions. Use simple, easy-to-understand language and images to explain the facts in your report.

Tips for the Moderator

Following these tips can help the moderator achieve better outcomes:

• With the approval of your participants, record the sessions.
• Allow participants to deliberate and complete their tasks without any undue influence such as leading questions or suggestions.
• Don’t draw any conclusions during the sessions. Wait until you’ve gone over all the data from all the sessions.
• Keep an eye out for participants’ gestures and expressions, which provide much of the input you need.

Unmoderated Usability Testing

This is a usability-testing technique in which the test sessions are not monitored and no one is present when the participants answer questions or complete tasks. Unmoderated testing allows participants to complete the tasks or provide their answers at their own pace—and at a time and location of their choosing. Unlike moderated usability testing, participants cannot receive real-time assistance. Because of the lack of interaction with a moderator and the inability of participants to raise inquiries, this testing approach is recommended for less complex prototypes.

Pros of Unmoderated Usability Testing

Let’s consider some pros of unmoderated usability testing, as follows:

• These test sessions can be completed at any time and anywhere—whether the participant is at home or on a mobile device—making it a very convenient usability-testing method.
• This test method provides access to a larger geographical sample and, therefore, enables you to test a product with a wider audience.
• Unmoderated testing is less expensive and time consuming to conduct and requires fewer resources.
• There is less risk of bias, and the observer effect—individuals’ acting differently when they are aware that they are being watched—does not apply.

Cons of Unmoderated Usability Testing

Now let’s look at some cons of unmoderated usability testing, as follows:

• During the sessions, no researcher is present to guide the participants. Therefore, if participants get stuck, run into technological difficulties, or don’t understand the questions, no one can help them. This could have negative impacts on the test results.
• Another disadvantage of the lack of real-time interactions with participants is the risk of their skipping through the early tasks and moving straight to the follow-up tasks. Participants may diverge from the testing flow that you had planned because they are receiving no guidance.
• Data collection is limited. Only questions that a UX researcher has prepared in advance get answered. There is no opportunity for a researcher to ask any follow-up questions during a session. Therefore, before the start of the sessions, you must thoroughly prepare all of your questions.
• You’ll miss out on observing the participants’ subtle responses to certain test questions—such as hesitancy when answering specific questions.

Best Practices for Unmoderated Usability Testing

To ensure the success of your unmoderated usability-testing sessions, employ the following best practices:

• Because there won’t be any moderator to guide the participants, make sure the test activities are straightforward and easy to understand, with clear instructions on how to approach them. This eliminates the possibility of a participant’s misunderstanding a task and, therefore, performing it incorrectly, which could distort the test findings.
• Increase the number of test participants to avoid the risk that some of them might not show up, which sometimes happens in unmoderated usability testing.
• Thoroughly analyze the data and present them in a report that includes your most significant insights. Use simple, easy-to-understand language and images to explain the facts in your report. Your report can assist the product team and stakeholders in prioritizing design decisions.

Moderated Versus Unmoderated Usability Testing

When to conduct moderated usability testing:

• Use this approach when you have the necessary time and resources.
• You need in-depth feedback from your users.
• During the earlier stages of design, when your team is working on a product concept, this method can help you explore various solutions.

When to conduct unmoderated usability testing:

• Use this approach when tasks are straightforward and participants would be unlikely to require any guidance during their test session.
• You’re testing an already functional Web site or high-fidelity prototype.
• You’re on a tight deadline, because you could conduct unmoderated tests with hundreds of participants, within a short period of time.

Which Testing Approach Should You Use?

There are both advantages and disadvantages to both moderated and unmoderated usability testing. Moderated testing enables UX researchers to communicate directly with their test participants, allowing them to gain a better understanding of users’ problems and, hopefully, address them.

Unmoderated usability testing is a rapid, effective way of gathering data from your participants. Because of the quick turnaround, firms can attract a higher number of participants, providing a larger test sample.

In specific situations, you can conduct either or both moderated and unmoderated usability testing. You could begin with an unmoderated session, during which the participant answers questions whose answers would enable you to address the most important usability concerns. Then you could switch to a moderated session, asking questions to home in on specific issues the participant raised during the first session and gain more insights into them.