As UX researchers, we have a great variety of tools and methods that are available to us. However, at times, a project’s scope, timeline, and budget limit our choices. Therefore, we may need to get creative and come up with a new method that both fits the business criteria and helps us discover usability issues.
In this article, I’ll describe how I combined a software-development technique—pair programming—with usability testing on a recent, exciting project at a smart-lighting company for which I was the usability engineer. The project had a very tight deadline. One of my tasks was to test the user experience of a mobile app called Casambi, a remote-control app for smart lighting, which is shown in Figure 1.
Champion Advertisement
Continue Reading…
Figure 1—Casambi app
This app has both technical and nontechnical user groups, as follows:
technical staff, or installers—These users rely on using the app during the installation of smart-lighting systems and when administering the settings for wall switches and scene programming.
customers—These users purchase smart-lighting systems and use Casambi as a remote control.
Trained technical staff have domain knowledge and understand all of the features of Casambi. The breadth of their knowledge helps them to understand the needs of smart-lighting customers, who must communicate their needs for customizations to the technical staff. The technical staff then make the necessary customizations to the app during lighting installations.
The Project’s Goals
The goals of the project were as follows:
Reduce the number of redundant features.
Simplify the app for nontechnical users, enabling them to program the wall-switch settings without relying on technical staff.
With respect to reducing the number of features, it was essential to consider the points of view of both technical and nontechnical users. This, in turn, could give us an idea of which features were duplicative or not used at all.
After running several rounds of usability testing and analyzing our data, we identified 14 design recommendations—seven of which we classified as high priority. However, the more tests we conducted, the more we came to realize that usability testing does not guarantee the discovery of every potential area for improvement.
Borrowing a Software-Development Technique for Usability Testing
At that point, I realized that perhaps the most fruitful approach for investigating the usability of what was inherently a multiuser product might be a pair testing—originally, a software-development technique in which two team members work together at one computer to test a software application.
So I decided to try pair testing during a subsequent usability study, as shown in Figure 2. For our pair testing, we encouraged two different user groups—comprising nontechnical and technical users—to complete the test scenarios together while discussing what to do. This naturally resulted in their thinking aloud, helping us to understand why they took particular actions.
Figure 2—Pair testing with technical and nontechnical users
Because one of the most important goals was to enable the nontechnical users to program and administer the settings on their own, without a technical user’s assistance, pair testing revealed a lot of insights on the knowledge gap between our user groups and what caused user frustrations and misunderstandings.
We created scenarios that included tasks that were originally targeted toward technical users and encouraged pairs of participants to have a dialogue between themselves as they completed the tasks. This, in turn, helped us to discover insights that might have been impossible to capture during conventional usability testing. Ideas for additional features usually arise only when those features are necessary to complete a task, so you’re unlikely to stumble upon them during usability testing. But, with pair testing, we were successful in investigating each user’s point of view, as well as their misunderstandings and difficulties in figuring out certain application features.
Our pair-testing approach gave us a clear understanding of the knowledge gap between nontechnical and technical users, as well as what vocabulary was unfamiliar to nontechnical users. In turn, our pair-testing process helped us to simplify the application features and make it easy for nontechnical users to program the settings for wall switches and scenes without relying on technical staff. Pair testing enabled us to verify the efficaciousness of Casambi’s primary method of settings administration.
One disadvantage of our pair-testing approach that we noticed during two usability-test sessions was that one of the participants dominated a session, so we missed getting the inputs and insights from the second participant.
Benefits of Pair Usability Testing
I have since decided that it would be beneficial to continue to combine this software-development technique with usability testing. Pair usability testing, an infrequently used method that combines usability testing and pair testing, can be beneficial for user experience research—especially for products that have different types of user groups.
Here are a few of the impacts that pair usability testing had on our project:
We found more usability issues at a lower cost, in a shorter timeframe.
We were able to analyze the root cause of users’ misunderstandings—especially those of nontechnical users—more easily.
Technical and nontechnical users could exchange knowledge about the app. This, in turn, helped us to identify the knowledge gap and the differences between the two different user groups’ points of view more effectively and quickly than when we had conducted conventional usability tests with individual users.
The UX designer gained insights on redesigning features and how to simplify them for nontechnical users.
There was no need for a think-aloud protocol. As pairs of users completed their tasks together, there was a natural flow of shared thoughts between them. Through participants’ conversations with one another, they exposed their viewpoints more completely.
Conclusion
Pair testing can be a great approach for testing the usability of complex products that different user groups use for different purposes. As I mentioned earlier in regard to our project, nontechnical users used the Casambi mobile app as a remote control for smart lights. In contrast, installers used the app during lighting setups for scene programming and other features.
Since this was a project with a very tight deadline, we had to be creative, yet follow a scientific approach that would give us the best results in a shorter timeframe. As a result of running usability tests with pairs of users, we were able to utilize their different skillsets and knowledge to reveal the product’s flaws and issues, in an efficient and effective manner.
Reference
Beizer, Boris. Software Testing Techniques. New Delhi: Dreamtech Press, 2003.
In this case you only had 2 user groups, so it seems that this kind of testing fits very well to this kind of situation, but what would you have done if you had encounters more than 2 user groups?
Approaching UX research with mixed methods and innovative techniques are what is indeed needed for finding great insights while adapting to project limitations and business goals.
Multi-user products such as Casambi are in fact not that common. Products like this usually encounter major usability issues because of their focus on two or more different segment markets. This is quite on contrary with the philosophy of design thinking and human-centered design which is focusing on individuals (rather than segment markets).
Approaching a product with more than 1 user groups also depends on the underlying differences between those user groups. Here our interest was the knowledge gap and specialty language and jargon used by different user groups.
Now getting back to your question on what would I do if it was more than 2 user groups. I might have approached this by usability focus groups. This way each participant could start with a one-on-one session testing. Following the test sessions, participants could then congregate to discuss the experience and how it relates to their everyday needs.
Hossein’s interest in human behavior can be traced back well before his pre-university years. Even as a little boy, when he still had no idea what adults meant by behavior, he was curious about the fundamental causes of human behavior. He tested the threshold of other people’s patience—including that of his parents—by being the naughtiest kid in his family. Since studying applied psychology and human-computer interaction, Hossein has pursued a career as a user experience researcher at a software and game-development company. When not conducting UX research, he cooks Persian food, writes, and gives public speeches. Read More
