Unmoderated Remote Usability Testing
Unmoderated remote testing is one of the most popular methods of testing Web applications. Many confuse such testing with the use of Web analytics tools such as Google Analytics and Clicky, but unmoderated remote testing focuses on obtaining data about the usability of individual workflows and experiences rather than capturing broad, overarching metrics about general usage.
In unmoderated remote tests, participants complete predetermined tasks using an application. Services like Loop11 or UserTesting let us collect data about how long it took participants to complete a task, where they made mistakes or referred to the Help system, whether they were able to complete the task, and how they navigated the application to complete the task. Researchers can analyze the results and provide specific feedback on the product design.
Pros
Unmoderated remote testing lets you recruit and test using large numbers of participants in a relatively short timeframe. Because remote testing does not rely on a moderator’s conducting the test sessions, participants can work through your test tasks simultaneously. Large amounts of data can be very powerful when you’re making a case for a significant and potentially costly change to existing software. If you’re working in a Lean UX environment, unmoderated testing gives usability a foot in the door, providing immediate value at a low cost. An unmoderated remote–testing service generally handles participant recruitment—although you can take this on yourself—and scheduling is unnecessary. In addition, you can easily test competitors’ products, creating an opportunity to present side-by-side results. You can easily fit unmoderated testing into an agile sprint schedule and provide timely feedback for the next sprint.
Cons
Unmoderated remote tests require a fully functional product or prototype, so you cannot use this approach when testing concepts through a cognitive walkthrough. Unmoderated testing often leaves the researcher to wonder why? With no means of obtaining participants’ feedback aside from comment fields or a questionnaire that you administer at the end of a test session, it can be difficult to accurately interpret the reason why a participant has made a mistake or misstep.
For example, an unmoderated test might show that several participants were unable to complete a shopping cart workflow, abandoning the task at the last step before committing the purchase. Was it difficult to see how to complete the purchase? Did participants think they had submitted their order, so were finished with the task? Did a cat sit on a participant’s keyboard and cancel the transaction? Did the process take too long, so participants got bored and quit? In unmoderated tests, these questions remain unanswered. A participant’s voice is silent for the developer and executive as well. Hearing and seeing someone’s frustration and confusion during a test is invaluable. Non-researchers’ hearing such visceral feedback can make the difference between fixing a problem and merely accepting it as it is.
From an enterprise perspective, unmoderated remote testing may be inaccessible to large companies because of legal considerations, the scale and complexity of their software, or the need to test with a limited and highly specialized user base. Many large B2B (Business-to-Business) companies have significant legal exposure when testing their software with users, requiring lengthy non-disclosure agreements and assurances of confidentiality. Remote testing services work on the assumption that the applications to be tested are Web based and consumer oriented. Neither of these assumptions is valid for most large B2B companies—many of which still maintain thick-client, complex software that is geared toward an elite user base such as security administrators or systems administrators.
Is Unmoderated Remote Testing for You?
Unmoderated remote testing is generally best for assessing targeted areas of a user interface that have well-known issues, A-B testing, simple competitive analyses, collecting data in large numbers, and initial problem discovery—assuming you follow up with moderated testing to uncover the why. Unmoderated testing is not recommended for solving complex or poorly understood issues or as a substitute for discovery work such as user interviews or cognitive walkthroughs.
