Conducting Large-Scale User Research

Coordinate the Efforts Between Teams

If you have multiple research teams, it’s very important to coordinate their efforts. Conduct daily debriefing sessions to keep the research teams in sync. Even brief, 30-minute phone calls can be helpful in keeping physically separate teams close. These sessions allow teams to share and compare their findings, observe trends, propose changes to the research process, and feel connected to one another.

Keep Up with the Data

A typical user research project generates and collects a lot of notes, recordings, photos, files, and physical artifacts. In large-scale research, this flood of data multiplies to a level that can be overwhelming. How do you keep track of all this data and keep what you’ve learned in mind?

Record Sessions, but Don’t Depend Too Much on Your Recordings

You can’t capture everything in written notes, and you can’t possibly remember every detail, so recording audio and video is essential. When you have time to review the recordings and take additional notes from them, this deeply instills the information in your mind. Doing that results in your experiencing every session at least twice—first in person, then through the recording—and taking notes lets you think more deeply about what you’re observing. However, it’s extremely time consuming to review each recording in full—typically taking at least two to three hours to view every hour of the recording.

An alternative is to have one person take very detailed notes during the sessions and review only the parts of a recording where your notes need clarification. Another alternative is to send the recordings to a transcription service, which will type up notes for you. However, this can be expensive, and the information won’t be as deeply ingrained into your mind as if you type notes yourself, while reviewing the recordings.

Regularly Type Up Your Notes

Regardless of whether you’re going to review the recordings, on a large-scale research project, it’s extremely important to type up your written notes at the end of each day. The last thing you should do is wait until the end of all your research, then finally deal with a huge stack of notebooks from sessions that took place weeks ago.

I don’t know about you, but my paper notes from each research session are a disorganized list of paraphrased comments, participant actions, and observations. When I read them shortly after the session, I can understand what I meant, and I can remember what I saw. Then, as I type up my notes, I can add more detail and context than I captured in the words I wrote down during the session. But as days pass and I see more and more participants, my memories begin to fade. This problem compounds when there are tens or hundreds of participants and the research takes place over many weeks or months. If you don’t stay on top of typing up your notes, you’ll forget a lot of details and quickly become overwhelmed.

Use a Survey Tool to Unify Everyone’s Notes

Using an online survey tool is a good way to organize and unify notes from multiple research teams. Last year, when I conducted a large-scale research project with over 150 contextual inquiries, we used Wufoo, an online survey tool, to coordinate the notes from two research teams. This gave us a central and safe location for entering our notes instead of risking losing our physical notebooks. Although each team had its own style of note taking, the standard questions on the form gave our notes consistency and reminded us of things that we needed to capture.

Because we used an online survey tool, we could look at the other team’s notes to see what kinds of things they were finding in their research. Everyone could see how often we were updating the survey with our notes, which encouraged us to update them at the end of each day. As the data accumulated, the standard format made it easy to compare specific topics or issues across participants. We could even run reports to find specific issues.

Manage the Data

Large-scale research generates a ton of paper notes, electronic documents, photos, audio recordings, videos, and paper documents that you’ve gathered from participants. These initially reside on multiple computers, tablets, phones, cameras, and other devices belonging to various people. If you don’t have a well-organized method for naming and storing all of this data, you can end up with a confusing mess. Determine where you’re going to store these files, how you’re going to organize them—by participant, by topic, by issue, or something else—and use a file-naming convention.

Analyze the Data

The voluminous amount of data that multiple teams have gathered obviously poses a challenge when it comes to analysis. If you’ve used a survey tool to capture everyone’s notes throughout the project and all of your files are well organized, you’ll be in a good position to start your analysis.

Provide Time for Individual Analysis First

Because multiple people have conducted the research, it makes sense to analyze the data as a group, but first, let people do their own initial analysis of the information on their own. Many people first need time alone to think about what they’ve observed before discussing it with others.

Analyze the Data as a Group

Next, conduct group analysis sessions to organize and discuss the information that you’ve gathered. Doing affinity diagrams is a good way for multiple groups to combine their findings, putting each finding on an individual Post-it note, then organizing the notes into groups to reveal common themes. Mapping out processes and tasks using Post-it notes is also a good group activity because it lets the group visualize the tasks and come to a common understanding of them. Along with the tasks, group members can use different colored Post-it notes to denote problems in the process, the use of specific tools or technologies, and environmental factors.

Keep the Client and Project Team Updated

Obviously, you need a lot of time to analyze large-scale research data. The problem is that, by the time analysis begins, the research itself has already taken up a lot of time, so your client and project team may be getting antsy about having the findings. They may not understand what analysis involves or why it takes so long. To keep them satisfied, frequently communicate status during research and analysis. Provide previews of your findings such as anecdotes about what you’ve observed. Just make it clear that you’re not providing fully analyzed or final information.

Deliver the Findings

Once your analysis is complete, you’re left with a mountain of information and the quandary of how to most effectively deliver it. Should you provide a high-level summary or all of the details? If you provide only the details, you’ll likely bore and overwhelm them. If you provide only a high-level summary, they may wonder why they spent all that time and money to get only vague generalities. So you need to provide both a summary and details, but this is usually difficult to achieve in one deliverable.

Create Several Deliverables

The solution is to create several deliverables—a report with the details and a presentation with just the summary information. Your report can provide great detail and serve as a long-lasting archive that people can refer to later to better understand the findings—even if they didn’t attend or don’t remember your presentation. Once you’ve created a detailed report, this frees you up to create an engaging, high-level presentation that can focus just on the most important points.

If your research covers several different topics or has different audiences, it’s often better to create several audience-specific presentations. For example, once I gave an extremely long presentation to representatives of several different departments in a company. Each group was bored and perked up only when I got to the findings and recommendations that pertained to their department. It would have been better to create and deliver several smaller presentations that I’d targeted to each department.

Remain Calm

Conducting large-scale research with hundreds of participants, in an unfamiliar subject area, can be very intimidating. Once, when I was slightly freaked out about undertaking such a complicated project, my manager encouraged me by making an analogy to a marathon. You run a marathon one step at a time. Instead of focusing on the whole thing at once, focus on one step at a time, and eventually, you’ll get there.