Usability Testing with Time Constraints | Remote Usability Testing

Prioritizing Usability Testing

“In my experience, there are almost never enough time or resources to usability test everything that you’d like to do. So you always end up prioritizing—and essentially betting the odds,” declares George.

Start with What Is Most Important

“Conduct an expert review first to separate the obvious usability problems from those that are more serious or require validation by users to determine whether they really are problems,” suggests Jim. “Obviously, if your time is limited, you should test the most important tasks. You can determine the importance of tasks by understanding which tasks users perform most often, which are performed by the most users, and which would result in the most serious errors if users could not complete them correctly.”

To choose what to test, Mike recommends, “Test the riskiest parts of the user interface, including

• anything users would do early—which, therefore, affects their initial perception of an application’s usability or trustworthiness
• anything that is critical to your business model—for example, registrations or completion of financial transactions
• any cool, new features”

“What are the most critical and frequently used parts of the system? Test those areas,” agrees Paul.

George suggests, “To identify the areas of an application whose usability has the most consequences—that is, the most critical parts of its design—you can ask the following types of questions:

• What are the most mission-critical areas—to your users and to your company or your client?
• What functionality or content would users use most frequently?
• What functionality or content would the most people use?
• What functionality or content would the users who are most important to your company or your client use?

To test a Web site, Todd recommends, “Once you’ve confirmed that a site complies with usability guidelines, first impressions on the home page or landing pages are very important, so I would start testing there. Dig deeply into users’ emotional responses to Web pages. Ensure that your site is conveying your value proposition effectively, appears professional, encourages the first click—without expecting users to expend too much effort or time—and transparently demonstrates access to information that confirms vendor credibility.”

Test Users’ Paths Through a Product

“Check users’ expectations for navigation bar labels and discover how easily users can find what they want, ensuring your information architecture is clear and understandable,” suggests Todd. “Once clear navigation is in place, I would test the primary conversion path.”

When you’re doing usability testing, Whitney and Caroline recommend that you “follow these paths through your product:

• getting-started path—What is the very first thing people do?
• critical opening path—The path from users’ point of entry for any natural beginning of an interaction.
• critical exit path—Users’ final transaction. There is no point in having a great user experience for finding things if you can’t check out or use them in some way.
• most popular path—Where would most people go? If you’re testing an existing site, you have looked at the analytics, right?
• parachuters’ path—What happens when someone lands in the middle?
• highest value path—What makes the most money or meets your company’s goals the best?
• simplest path—What is the most anticipated, most straightforward thing people should be able to do?
• most embarrassing path—What would give you the reddest face if people couldn’t do it easily?”

“Be sure to test the ambiguous or controversial paths,” recommends Whitney. “The areas of a design you are least sure of or where there is still disagreement on your team. Similarly, test paths that represent key design decisions. For example, let’s say you have a feature that seems cool, but would cost a lot to build. Wouldn’t you like to be sure?”

In summary, Caroline says, “And then, finally, to help you choose among all these paths, ask: What do you need a decision on right now, to keep the team working as effectively as possible? This brings you back to defining the scope for change.”

Resolve Your Teams’ Uncertainties About a Design

“Where do you have the most uncertainties about whether a design is effective?” asks George. “The answer to this question is one of the key ways I prioritize which parts of a design to test under time constraints. There are a few ways of answering this question:

• Are you following known best practices? If so, testing that part probably isn’t worth it. However, the exception would be if that part is something that’s critical to both your users and your company or client—with high consequences to one or both groups if there are problems.
• Are you following common design conventions? For example, if look at a dozen retail sites, you’ll see common ways of designing a product information page. Of course, you don’t necessarily know whether their designs are proven, best-practice designs. I’ve seen companies copy their competitors’ designs verbatim—even copying things that were compromises due to technical constraints or things that were just plain bugs. Plus, someone else’s users may or—more likely—may not be like your users. So something that’s a good design for someone else, may not be appropriate for your users. That said, if you’re following a commonly used design convention, while it may not be the optimal design, the odds are that it’s not going to be a horrible one. Plus, a design’s being in common use in itself makes the design more familiar to users.
• How innovative is your design? Really? Much as UX designers—and their employers—like to feel they’re doing innovative work, the instances in which one is designing something that’s truly new to the world or new to a market are the exception. If you’re truly innovating, such designs are often hard to usability test, as well, because it can be hard to tell whether users just haven’t had time to digest the concept, or there truly are design problems. Imagine trying to do a usability test of today’s Web sites back in the pre-Internet era. This isn’t to say one shouldn’t attempt to do usability testing in these cases, but rather recognizing that testing might be of limited value in such situations. Plus, most of the time, we’re not doing cutting-edge design, so the designers’ gut feel is often a good indicator of what’s probably solid design and where uncertainties exist.
• Does the design team have confidence in the design? Too often, I’ve seen teams use usability testing as a crutch. If you’re confident in your design, be willing to stand by it—and accept the consequences if you’re wrong. Obviously, one’s instincts can be wrong. Mine have been. Designers can have blind spots—so, you discover the hard way that you didn’t know what you thought you knew. Been there, done that. But those are the things that build experience, and with experience—hopefully—comes a gut sense of where a design is solid and where there are uncertainties. Obviously, the talent and experience of your design team is a factor, so you need to be mindful of that.”

“Test the things that you, your clients, and the people on the project team are not sure about,” agrees Jim. “Designers usually have had to make design decisions between different ways of doing something, so wonder whether the choices they have made will work. Ask the people on a project team to tell you the most important things they want to know or the things they want to validate in a design.

“If there are parts of a design that are innovative or use an unconventional or unfamiliar interaction model, those are the parts you should test—rather than the more conventional aspects of your design that follow best practices and conventions that have already been proven in other user interfaces.”

“If you’re short on time or resources for testing, you’ll still probably have a list of questions that’s longer than your team can address,” adds George. “So it’s critical to make sure all of the relevant stakeholders are aware this is the case. Indicate your confidence level in the areas you’ve identified as potentially having issues or being highly important ones you won’t be able to test—as well as why you’ve either got or lack confidence in different parts of a design. If you’ve flagged enough areas as being of high concern, it’s possible you might be able to get additional time or resources for testing. But if not, you’ve at least made it clear where you’re betting the odds and why—and ideally, your stakeholders agree that those are bets they’re willing to accept.”

Take a Goal-Oriented Approach

As I recommended in Part II of this series, “Usability Testing on a Budget,” make a list of your product’s top-three goals and top-three measures of success, plus the top-three personas it should please. Then, do everything possible to reach those goals, attain those measures of success, and please those personas. If you’re tempted to cut corners, remember to ask yourself this: If you don’t have time to do a project right, what makes you think you will have time to do it over? Be sure your organization is thinking about how well your product will succeed in the long term.

“If you have time to test only certain key areas of a user interface, you need to ask yourself two questions,” suggests Kyle:

1. What are the main reasons people use the product?
2. What are the main business goals for the product?

“Focus your testing so it covers the tasks and areas of the user interface that you’ve identified in answer to these questions. You don’t want to waste your time testing a cool, new widget if users won’t use it to accomplish their key goals for using the product and it won’t have a direct impact on the business objectives for the product.”

How to Conduct Remote Usability Testing

Q: What is your process for remote usability testing, and what are the best software tools to use for remote usability testing?—from a UXmatters reader

The following experts have contributed answers to this question:

• Caroline Jarrett—Owner and Director of Effortmark Limited
• Jim Ross—Principal of Design Research at Electronic Ink
• Whitney Quesenbery—Principal Consultant at Whitney Interactive Design; Past-President, Usability Professionals’ Association (UPA); Fellow, Society for Technical Communications (STC); and UXmatters columnist
• Stephanie Rosenbaum—CEO of pioneering UX consultancy Tec-Ed, Inc., and a charter member of UPA
• Paul Sherman—Principal of Sherman Group User Experience; Vice President, Usability Professionals’ Association; and UXmatters Columnist
• Kyle Soucy—Founding Principal of Usable Interface

Tools for Remote, Moderated Usability Testing

“We use WebEx or TechSmith’s UserVue, combined with telephone conferencing,” answers Stephanie. “So the researcher and members of the product team can observe the participant’s screen and hear the participant’s voice, while the researcher facilitates the session by telephone. The researcher, the participant, and observers on the product team can all be in different locations.

“When I asked Tec-Ed’s VP of Client Services, Laurie Kantner, about when we choose to use each of these tools, she explained that WebEx lets users perform tasks on our lab computer, without exposing their computer to us. UserVue handles high-bandwidth Web applications more smoothly, but allows the testing team to see the participant’s desktop. We’re open to using other tools that provide similar capabilities. We’ve just settled on these particular tools for the time being.”

“When I’m doing remote usability testing, I try to keep the technology as simple as possible,” explains Whitney. “I have found that GoToMeeting works well for screen sharing. It seems to be easy enough for people to use and doesn’t get caught in firewalls. I do a lot of work in healthcare settings, where security issues can be a real concern, especially if we have to download and install any software. Just add a phone line or the GoToMeeting Voice over IP (VoIP), and it’s all you need.

“I’ve also found GoToMeeting useful when people who want to observe a usability test are at remote locations. If I want to record a session, I can still use Morae—or any other screen recorder. By the way, one thing I really like about GoToMeeting is how easy it is to let the participant be the presenter or to run the software or site from my computer. Being able to make that decision on the fly can be really helpful in adjusting to each situation.” Whitney and Caroline use the same process for remote, moderated testing.

“All you really need is a speakerphone and a Web conferencing application like WebEx, which lets you and the participant share views and control each others’ computers,” offers Jim. “WebEx is fairly common in many workplaces and requires minimal effort to download and set up for participants. The speed of a participant’s Internet connection can slow down a session, but that is less of an issue nowadays, because more people have broadband connections.

“It is helpful to send an email message to participants several days before a test session, asking them to download, set up, and connect to the Web conferencing software they’ll need for the session to ensure that it works. If you explain this in advance, it also makes participants feel more comfortable about downloading the software. Still, plan for some extra time at the beginning of a test session for participants to connect to the conferencing application. That can add five minutes or more to a session.”

Paul has described his “process for on-the-cheap usability testing with remote observers” in his UsabilityBlog post, “Low-Rent Usability Tech Tip: Remote Usability Testing with Observers Watching.” Paul told us, “The gist of what I said there is this: You can use GoToMyPC and regular conferencing software to allow a user to access a testing machine, while observers are watching via the conferencing service.”

“The majority of the testing I do nowadays is remote, moderated testing,” answers Kyle. “I prefer to use GoToMeeting or UserVue for my screen-sharing software. GoToMeeting has a few advantages over UserVue, as follows:

• You can’t use UserVue for international testing.
• There can be firewall issues with UserVue that may make it hard for your observers to connect in certain secure office environments.
• The UserVue servers tend to be down more often than those for GoToMeeting.
• I can record both sides of the conversation with GoToMeeting, using a phone recording adaptor that costs approximately $30 at RadioShack.”

Testing Paper Prototypes Remotely

“A lot of people think it’s not possible to do remote testing of paper prototypes,” observes Kyle. “But you can easily scan your paper prototypes, then use Axure to turn the images into a clickable, coded prototype, or use Acrobat to create an interactive, clickable PDF. The possibilities are really endless!”

Observing and Interacting with Users During a Test Session

“Remote usability testing can be either completely automated, requiring no facilitator, or facilitated,” explains Stephanie. “Automated remote usability testing records users’ onscreen behaviors and collects users’ opinions, but it doesn’t provide insights into the reasons why people behave as they do. The benefits of questionnaires and other self-reporting by users can be limited, because we don’t know the rationale behind their answers.

“In contrast, a skilled user researcher conducting in-person, qualitative research observes and explores people’s overall behavior—onscreen, verbal, and body language—to gain a deeper understanding of user needs, goals, and expectations. That’s why my colleagues at Tec-Ed do remote facilitated usability studies, which give us many of the benefits of the researcher’s moderating, interviewing, and observational skills.”

“One benefit of participants not being in the same room with you is that you can focus more on what they are doing on the screen and take more detailed notes,” observes Jim. “Unlike an in-person interaction, you don’t have to maintain eye contact and follow other conventions of polite interpersonal behavior. Instead, you can focus more attention on the screen, take notes on paper or a laptop, or even log events in a software tool like Morae.

“Some people mail a Webcam to each participant to use during a test session, so they can see participants’ facial expressions, but I find this is not necessary. Doing this can be expensive, and it’s difficult for some participants to set up a Webcam, install the software, and troubleshoot any compatibility problems. The small value that you get from seeing their delayed facial expressions during a session is not worth the technological hassles for either you or the participants.

“Since you cannot directly observe the participants,” continues Jim, “it is especially important to ensure that they think aloud. When you are in the same room with participants, you can usually observe why they have stopped thinking aloud—for example, because they are confused or reading something on the screen. When doing remote testing, you cannot be sure why participants have suddenly stopped thinking aloud. On the other hand, fortunately, it is often easier to get participants to think aloud during remote testing, because the need to think aloud makes more sense to them when you are not in the same room with them. They feel more of a need to tell you what they are doing and thinking.”

Dealing with the Unexpected

“No matter what tools you use, you always want to have a back up!” exclaims Kyle. “For instance, if I’m using UserVue, I’ll still set up a session in GoToMeeting and have it ready on standby—just in case a user has trouble connecting. I can’t tell you how many times this has saved me in the past. The last thing you want to have to do is scramble to create another meeting session, using a different service or product, during a testing session.”

“Be prepared for interruptions during remote testing,” advises Jim. “Participants who are at work often get interrupted by phone calls, instant messages, email alerts, or colleagues coming to their desks with questions. Participants who are at home get interrupted by family members, pets, and people at the door. You should plan additional time for a remote test session to allow for interruptions. However, you can also view such interruptions as an opportunity to get a more realistic sense of the participants’ environments and the situations they actually experience as they use a user interface you are testing. The ability to observe people using their computers in their own environments is a definite advantage of remote testing over lab testing.”

“We plan the majority of our remote facilitated studies to be remote from the outset,” says Stephanie, “but we can also add remote sessions mid-stream. For a recent project, we initially planned to hold in-person sessions in the U.S. Midwest and in Silicon Valley. Once the project was underway, the client identified a key target audience segment to be found only in India. With our remote facilitated methodology, we were able to include a participant from India within the project budget and schedule.”

Doing Remote, Unmoderated Usability Testing

“There’s also remote, unmoderated usability testing, where you get users to try something, then report on it,” says Caroline. “In this case, you don’t have to use any software tools at all if you don’t want to. You can just send users a packet of instructions about what to do and how you want them to report on it. I strongly recommend that you test the instructions themselves, in a face-to-face usability test, because the success of your whole testing process would revolve around the quality of those instructions.

“If you do want to use a tool for remote, unmoderated usability testing, there are several good ones out there. I use Usability Exchange, because I’m based in the UK and want UK users and, in particular, because the main reason I’m doing remote, unmoderated testing is that I want users with disabilities to try tasks in their own time, using their own assistive technologies. It also helps that the charges for Usability Exchange are modest and the service quality excellent. I’m not as familiar with the tools that are available in the USA and other parts of the world.

“Look out for a book that’s coming out in January 2010, Beyond the Usability Lab: Conducting Large-Scale User Experience Studies, by William Albert, Thomas Tullis, and Donna Tedesco. I’ve been lucky enough to read this in manuscript form, and I found it really impressive. The authors have extensive practical experience doing a wide range of remote, unmoderated tests, and they explain exactly what to do and what to look out for.” 

References

Albert, William, Thomas Tullis, and Donna Tedesco. Beyond the Usability Lab: Conducting Large-Scale User Experience Studies. San Francisco: Morgan Kaufmann. Forthcoming on January 29, 2010.

Rosenbaum, Stephanie, and Laurie Kantner. “Learning About Users When You Can’t Go There: Remote Attended Usability Studies.”PDF 2008 IPCC Proceedings, Montreal, Canada. IEEE Professional Communication Society, 2008. Retrieved November 22, 2009.

Rosenbaum, Stephanie. “Not Just a Hammer: When and How to Employ Multiple Methods in Usability Programs.PDF” UPA 2000 Proceedings, Asheville, NC. Usability Professionals’ Association, Inc., 2000. Retrieved November 22, 2009.

Sherman, Paul. “Low-Rent Usability Tech Tip: Remote Usability Testing With Observers Watching.” UsabilityBlog, August 29th, 2009. Retrieved November 22, 2009.