This brief highlighted for me how online, qualitative research techniques can generate valuable insights. But matching the proper qualitative research technique to a brief has become quite an art. As a qualitative researcher, I deal with lots of different research topics. But user experience is a specialist area that, perhaps more than any other area of qualitative research, really requires some thought about the best approach for each brief. Why is this? Because doing face-to-face UX research has some disadvantages that can really impact the reliability of the research findings.
I love doing UX research and specialize in it, but even so, I am still learning about the best techniques for tackling specific objectives. What have I learned so far? For the remainder of this article, I’ll share the benefit of my experience in UX research with you, highlighting the challenges that you might face in certain scenarios and the solutions that you should adopt to overcome them.
Face-to-Face Qualitative Research Methods
Face-to-face qualitative research usually involves individual, in-depth interviews, with a moderator sitting next to and shadowing a participant. Such interviews are generally useful when a site design is in a wireframe format—so the moderator may need to intervene if a participant encounters parts of the site are not yet designed—or a Web site is already built or even up and running.
Another qualitative approach is the Web clinic, which is a test involving banks of personal computers and eight or so participants using a Web site simultaneously. Web clinics are generally useful in evaluating navigation rather than Web site design and are usually conducted during the final stages of Web site development.
Qualitative research sessions often occur in a viewing studio, which uses video technology to enable observers behind a mirror or just in another room to see participants’ face and mouse movements on a screen. Sometimes, organizations set up such a studio to support video streaming across the world, allowing observers to watch and listen in their own offices.
Disadvantages of Face-to-Face Methods and How to Overcome Them
There are some specific research objectives for which face-to-face methods are most appropriate—and they usually fall under the broad areas of navigation and search. For this type of research brief, it is important to be able to observe participants’ routes through a Web site. Testing a Web site’s navigation is quite often a goal of a research brief, so we rely on face-to-face methods of research a great deal, but there are some distinct disadvantages to them.
The first disadvantage is that participants always feel like they’re being tested, even if it doesn’t show. Some participants are visibly affected by the presence of technology, cameras, and a viewing facility—if you’re using one—or even just by the presence of a moderator and her recording equipment in their home.
Plus, it is common to instruct participants to “Speak your mind as you go through the Web site,” making them feel that they are under even more pressure to perform. Unfortunately, this often results in their starting to click links fairly randomly and unnaturally. At worst, this might manifest as a series of very unusual clicks that take a participant into a very unusual area of a site that he would never visit intentionally! On occasion, I’ve stopped an interview because a participant became flustered and asked me, “Am I doing it properly?”
This is, I am sure, a natural human reaction. I’d love to be able to say that we just need to warm participants up properly. But even with the best warmup in the world, participants’ test mentality persists. The consequence is that a test’s resulting insights regarding navigation or search may be unreliable. But you can do three things to mitigate this problem to some extent:
- Ignore participants’ early clicks. At the start of an interview, participants may be quite nervous. So, ignore their clicks early in an interview, when they’re still nervous. Once participants get into the swing of things, you’ll observe that they move around a site more naturally.
- Assign tasks that replicate real-life scenarios. Later in an interview, assign participants tasks—which are a kind of test, I know!—that use real-life scenarios. Some example tasks might be searching for product details, comparing products, choosing a product, or going through an ecommerce payment system. Such tasks reflect real life to some extent, so relax participants.
- Give participants control of the mouse. Ensure that participants can click whenever they need to. The mouse should be readily at hand at all times. Even extremely experienced moderators can lose track of a session if a participant feels he does not have control over the mouse.
But, if none of these approaches works for you, there is one more trick you can keep up your sleeve when moderating a session: You can ask participants to have a look around a Web site on their own for a few minutes, then leave them alone in the room for a while and watch them from behind the viewing mirror. On their own, without the presence of a moderator, participants click links quite naturally. You can use this approach very successfully at the start of an interview. It works so well that I am increasingly using it as my preferred method for testing navigation. It is especially useful if the objective of your research is to evaluate whether users’ entry into a Web site is intuitive for them.
The Importance of Observation
What I’ve described so far highlights how much the test environment and the presence of a moderator affect participants. Nevertheless, when you’re testing navigation and search, you need to monitor what’s going on, so face-to-face methods are the most reliable way of evaluating a site. But here’s the thing: moderating is more about observation than anything else, so you should be watching participants’ hands and face as much as what is happening on the screen.
Why? A participant’s eyes may sometimes follow the mouse, but not always. Occasionally, participants follow their eye movements with the mouse, pointing where they are looking, but this is not that common. People usually look at the screen while their hand remains motionless on the mouse, moving the mouse only to click a link. Therefore, observing where participants look is important. To help during analysis, you can use PiP (Picture in Picture) technology to watch video playback of both the screen and a participant simultaneously.
Closely follow participants’ route through a Web site. Note the patterns: where the user journey begins and where it falters. When a participant hesitates or falters, those observations are gold dust. It’s your responsibility to capture whether the participant is confused, irritated, or just rereading some text. People naturally read from the left, so as a rule, participants’ eyes initially focus fleetingly on the left. When people visit a previously unknown Web site, I find that their clicking links on the left is more common.
Testing systems’ processes is another good application of a face-to-face method. For example, I think it’s best to tackle testing of ecommerce systems face to face, so you can observe how intuitive a Web site’s purchasing and payment processes are. Companies lose a huge number of purchases because of poorly designed ecommerce systems. Therefore, it is important to capture information about participants’ expectations before testing proceeds. Amazon has a very well-regarded payment system that establishes the benchmark by which people measure other ecommerce systems, so don’t be surprised if many participants tell you this! Your client’s ecommerce process may well have restrictions, so capturing participants’ expectations before test sessions and observing how they manifest during the sessions lets you report the corresponding issues as category 1, 2, or 3 barriers to purchase.
