Should Your Entire Product Team Observe Usability Testing?

Make the Business Case

“The right answer depends on who your client is, as well as the nature of the project,” responds Jordan. “It isn’t necessary to observe all usability testing first hand—although it may be desirable to do so. There are many cases where an expert analysis is all the rest of the team really needs. When you hear your clients saying things like this, you usually need to make a business case for spending the extra man-hours observing test sessions. There are many ways of doing this, but the most persuasive arguments require an understanding of how your particular team is structured and how the organization will use the findings from the usability tests.”

Make Sure the Right People Observe

“I actually take this type of statement to heart because I do believe that not everyone does need to observe usability testing,” replies Baruch. “We first have to examine who exactly comprises the rest of the team. I would rather have the right people observe sessions than all of the people. You can pick your champions and let them spread the word about observing sessions, so it will be easier to get others to observe the next time. Chances are that team members who did not observe usability testing will get excited about doing it and tell your client that they want to observe the next round of testing. I have had exactly this situation happen—where the Project Lead wanted to see the usability testing, but did not want anyone else to observe. But once he experienced it, he saw the value of observing and wanted to involve the whole team. He even took credit for making it all happen. That was okay, because, in that case, usability testing grew organically from within the project team. An external consultant did not force usability testing upon them.”

“I tell them that’s fine,” answers Cory. “While, in an ideal world, all stakeholders would observe all test sessions, I know that this isn’t always feasible. In my consulting experience, it’s actually pretty rare for everyone to attend all test sessions. More typically, some stakeholders watch some sessions, but not every session has observers. As a standard part of my usability testing process, I make video recordings of the sessions and send the team links to download and watch the sessions whenever they want. Often, I’ll also include a note about which sessions I think they’ll find most interesting. Depending on the budget for a project, I may also create video clips that highlight key findings from the study.”

“We digitally record everything so people can watch sessions on their own time,” says Gavin. “But we know people often don’t watch them. When we get a negative response to a request for other team members to observe usability testing, we must communicate that it is important—not just to identify usability problems, but also to identify the source of the problems.

“For example, a problem might be that a user moved on before completing a task. The source of the problem may be the call to action for the concluding step or a lack of feedback that the user has not yet completed the task. Merely stating that a user didn’t complete a task is insufficient to get to the root of the problem. If your other team members are not present during test sessions, you have only your own descriptions of the problems and insights about solutions to convince the team and provide a clear rationale for necessary changes. So you’ll need to be more descriptive if many of your team members aren’t going to be there to observe the pain that led to a finding. And you’ll need to be crystal clear on the source of the problem if you’re to be able to fix it.”

Some Alternative Responses

“In a perfect world, everyone on a product team would want to observe usability test sessions and team leadership would support their doing so,” replies Carol. “But the reality, in many situations, is that tight development schedules and other commitments can make that difficult, if not impossible. So, I’ll suggest many different possible responses that you could use in such situations—for example:

1. Could team members take turns observing sessions? At least one team member should observe each session. Each team member could observe just one session, but it would be better for each team member to observe at least two sessions to get a broader sense of users’ experiences.
2. Could we set up remote viewing via GoToMeeting, WebEx, or similar collaboration tools, so team members could observe test sessions while multitasking and without having to leave their workspace?
3. If the response to either of the first two options is no, ask whether the decision maker would be the team member observing. If the decision maker can guide the rest of the team to understanding what you’ve learned about the user experience and what the team needs to do to improve it, one observer is enough.
4. If the answer to all of the above options is no, your next question should be whether you or, better yet, the team member who did observe, could present the findings to the full team. If time permits, you could support this presentation with video highlights to let the users speak about the issues for themselves. Alternatively, if the findings meeting has to happen right after testing concludes, you could do a joint presentation with the team member who observed the sessions to inform the rest of the team of your findings.
5. If the answer to all of the above options is no, suggest that everyone on the team go through the session recordings to understand what you’ve observed and learned from users. However, I have to say that, at least in my experience, teams rarely follow up on this option—even when a client or team says they want to review the sessions. There just never seems to be enough time for this effort—in part because it occurs after the fact and the development timeline is already moving ahead.”