In May 2020, User Fountain conducted a study of the usability-testing industry to research digital professionals’ attitudes and opinions and learn how companies are managing usability testing and user research, as well as the challenges within the space.
For the study, User Fountain recruited 55 digital professionals, who work across a range of industries. The majority of respondents were working in a UX role (58%), but other roles included digital, conversion optimization, user research, design, product, sales, and insights. In this article, I’ll summarize the key findings of the study.
Champion Advertisement
Continue Reading…
Moderated Usability Testing: The Preferred Testing Method
For 80% of respondents, moderated studies were the preferred method, with 60% preferring in-person, moderated studies and 20% preferring remote, moderated studies. Only 10% of respondents preferred in-person, unmoderated studies and 6% preferred remote, unmoderated studies. Figure 1 depicts this data.
Figure 1—Preferred usability-testing methods
It’s clear that the greater depth of insights from in-person studies outweighs the benefits of ease of execution and logistics that remote studies offer. Because moderated studies enable further questioning and on-the-fly conversations, respondents strongly favored moderated studies over unmoderated studies—for which the researcher is not present, so there is no opportunity for further questioning.
Usability Testing: The Most Useful Method for Discovering Insights
We asked respondents to rate the usefulness of a selection of methods for discovering user insights on a scale of 1–10. They rated usability testing most highly, with an average score of 8.7 out of 10; digital analytics followed closely, with an average score of 7.7 out of 10, as you can see in Figure 2. Eyetracking methods and biometric analysis received the lowest average ratings for usefulness.
Figure 2—Usefulness of different methods for discovering user insights
How Often Organizaations Conduct Usability Studies
Most organizations run usability studies quarterly, with 31% of respondents stating that their organization conducts studies that frequently. Running studies monthly followed, with 20% of organizations doing monthly studies; while 22% of organizations do studies significantly more often, running weekly studies (11%) or biweekly studies (11%). At the other end of the spectrum, 16% of organizations rarely run studies and 9% never did any usability testing. Figure 3 shows the frequency of organizations’ studies.
Figure 3—The frequency of organizations’ usability studies
What Organizations Are Testing
Organizations conduct usability testing predominantly to evaluate desktop Web sites—with 82% of respondents stating that they’re currently testing sites. The testing of prototypes is the next most common type of evaluation—with 70% of organizations doing usability testing of prototypes.
Usability testing of mobile Web sites and apps appears to lag behind, with only 42% of organizations testing mobile Web sites and 38% testing mobile apps. Plus, only 16% of respondents stated that their company invests in the testing of competitors’ sites to understand their competitors’ experiences or conduct benchmarking exercises. Figure 4 shows how frequently organizations test different types of user experiences.
Figure 4—Types of experiences organizations test most frequently
The Number of Participants
According to 54% of our respondents, organizations typically recruit five to ten participants for a usability study. Just 16% stated that they usually recruit fewer than five participants, while 14% recruit 15 to 20 participants, as shown in Figure 5.
Figure 5—Typical number of participants for usability studies
This finding aligns closely with Jakob Nielsen’s guidelines for participant recruitment. His research shows that, with five users, you usually achieve approximately the maximum cost-benefit ratio for usability testing. However, if multiple personas with different needs represent your users’ roles, you should recruit five participants for each persona. Plus, for very low-overhead projects, just two users could be enough to gather some valid insights.
The Lack of Process in Many Organizations
A significant proportion of organizations lack both a formal usability-testing process and documentation for sharing test results. Among respondents, 46% said that their organization does not have a formal process for usability testing, 20% don’t document test results, and 10% do not share the results of usability studies across their organization.
As I mentioned earlier, usability testing is one of the most useful sources of insights for digital professionals, and most organizations are completing studies on at least a quarterly basis. So it seems odd that such a high percentage of respondents state that there is no formal process for usability testing.
This may, in part, be because there is a lack of clear ownership for usability testing in many organizations. Only 40% of organizations have a specific team that is responsible for usability testing; while in 20% of organizations, usability testing is nobody’s primary responsibility; and in 24% of organizations, usability testing is the responsibility of people across different teams, as Figure 6 shows. It appears that a lack of clear ownership for usability testing is responsible for the lack of a defined process. As a result, these organizations are neglecting important phases of the testing process such as documentation and the sharing of results.
Figure 6—Ownership of usability testing in organizations
The Biggest Challenges of Usability Testing
Of the respondents, 18% indicated that the biggest challenge their organization faces when conducting usability testing is the recruitment of usability-test participants. Plus, 11% of respondents stated that the quality of participants was their biggest challenge. Anecdotal evidence suggests that both of these challenges stem from the need to recruit niche audiences that they could not find on pre-existing usability-testing panels—such as participants with high salaries, specific job titles, or unusual interests. There is also frustration with the professional test participants who are on pre-existing panels, are too familiar with usability studies, and whose user experience doesn’t accurately represent that of the true user.
The next biggest challenge that participants reported was a lack of internal resources or expertise, which might also be responsible for some of the process challenges respondents described. No budget or a limited budget for usability testing was another common challenge, which likely inhibits the frequency with which organizations conduct studies and the internal resources they can assign to this work.
Figure 7 shows more of the challenges that organizations encounter when conducting usability studies.
Figure 7—Challenges of conducting usability studies
Conclusion
The results of our study highlighted a number of key themes. While digital professionals value usability testing highly as a source of insights and many organizations are conducting usability testing on at least a quarterly basis, challenges exist with organizational processes and the successful execution of usability studies with high-quality participants.
Cofounder and Lead Usability Consultant at User Fountain
London, UK
Lydia’s background is in conversion-rate optimization (CRO), and she has managed CRO teams working on some of the world’s biggest brands. She has been running usability studies for more than seven years. Since Lydia joined User Fountain, a managed service for usability testing, since has noticed the power of the insights that usability testing delivers and the positive impact they have on CRO programs. Read More
