User Experience and Scientific Methods, Part 1: Eyetracking

By James Coston

Published: August 5, 2013

“In this series, I’ll describe the application of some of the scientific techniques that user experience and market research have adopted, including eyetracking, EEG, and fMRI.”

In this series, I’ll describe the application of some of the scientific techniques that user experience and market research have adopted, including eyetracking, EEG, and fMRI. Based on my experience with these techniques, I’ll debunk some myths surrounding their use. This series will help you to understand

  • their technical limitations
  • fundamental limitations attending their use within the UX industry
  • implications of introducing a new scientific method to our industry

I want to ask you to do two things while reading these articles. The first is to maintain an open mind. The second is to take a step back and really think about what happens when a sensitive and specialized scientific tool becomes commoditized.

Ruthless Profiteering Breeds Bad User Experiences

“One element of remaining competitive is to develop a unique selling proposition (USP)—for example, adopting a scientific technique in the industry of user experience.”

One element of remaining competitive is to develop a unique selling proposition (USP)—for example, adopting a scientific technique in the industry of user experience. For this to work, adopters must be experienced in using the tool. Their knowledge has to be strong enough to enable them to adapt its function and apply it effectively to a new area.

Naturally, this forces others to either catch up or differentiate in other ways. Those playing catch-up may have little experience with the technique, so are unable to make an informed decision about how to apply it. As a consequence, they are under pressure to sell and use something that

  • they don’t understand
  • has ambiguous value
  • has hidden limitations

The more obscure an approach seems, the harder it is to understand when you should use it, when you could use it, and when it’s a complete waste of time.

Those who don’t really understand a technique often try to sell it at every opportunity, believing that it always adds value. This devalues the technique’s strengths and overcomplicates what should often involve a very simple form of descriptive testing and discourse.

The more we use something, the less special it becomes. As we begin to accept its use as providing a minimum level of value, this undermines a technique’s actual added value. We can often attribute this to a lack of transparency around how the technique works. Often, the pioneers who make use of a service seek only to distance themselves from the competition. This promotes short-term company growth, but hinders industry development. Many who could clarify the situation do not want to set themselves apart in their field as leaders or educators.

All of this forces other UX professionals to guess when to use a tool or technique, with the result that its real value and applicability become obscured, and many come to perceive it as being right for any job.

Eyetracking and User Experience

“The most obvious scientific technique to achieve a high rate of adoption in the UX community is eyetracking.”

The most obvious scientific technique to achieve a high rate of adoption in the UX community is eyetracking. However, the average UX professional’s knowledge of its scientific applications has remained focused on how and when to apply eyetracking—not why.

As a result, eyetracking guides are big business, and many great, practical application guides have been created. However, few have taken the time to step back and look at the impact that false claims and the forceful introduction of the technique have had on the UX industry.

We No Longer Sell Clients on Eyetracking—They Demand It

One of the hardest parts of our job can be justifying a new or novel approach. This is especially difficult when we begin to work with a new client who has experience working with one or more agencies—especially when that agency views eyetracking as a minimum standard for research.

Clients are now requesting eyetracking without any consideration or awareness of the true implications of including eyetracking in user research. And why should they? The majority of our clients have little or no knowledge of the technique’s scientific heritage. They request it because agencies have sold eyetracking to them as the gold standard in research rather than educating them about the situations in which its use is appropriate. They perceive it as a bar that their research must meet and as being ubiquitous in usability testing—not as an ancillary service.

Profiteering on Eyetracking

Clients are now requesting eyetracking without any consideration or awareness of the true implications of including eyetracking in user research.

Agencies frequently drive the hard sell by pushing eyetracking as an essential addition to user research. Many of them don’t fully understand what the tool can and can’t do, so include it in order to cover all the bases. They don’t offer their clients a baseline measure for a true standard in usability testing. Nor do they provide an informed comparison of the quality versus the quantity of their included services.

Once clients have been sold on using eyetracking, we must persuade them that it’s not always necessary to use eyetracking. We must reassure them that it’s okay to go back to the old-fashioned style of usability testing—without the bells, whistles, and tinsel. We must show greater reliance on something in which we are experts: discourse.

Using Eyetracking: Science Versus User Experience

To fully appreciate why it is okay to rely on more natural forms of investigation, it is important to understand how scientists actually use eyetracking—and why the UX industry is physically incapable of using eyetracking in the most appropriate way.

Reason 1: User experience uses the least invasive, but also the least accurate eyetracking hardware on the market.

“To fully appreciate why it is okay to rely on more natural forms of investigation, it is important to understand how scientists actually use eyetracking—and why the UX industry is physically incapable of using eyetracking in the most appropriate way.”

At the University of Kent where I studied, the eyetracking hardware that they use varies, depending on participants’ needs. Most agencies use a Tobii T60/T120. The uninvasive nature of the Tobii means that it’s incredibly forgiving of head movements. As a result, it provides the only way to conduct eyetracking experiments using children. However, in a scientific setting, researchers would not consider using this tool for testing with adults. It is simply not accurate enough. Instead, they would use a head-mounted device. But this would never work for a usability study because it would make participants feel uncomfortable—both physically and socially.

Reason 2: Eyetracking wasn’t designed for usability studies!

The reason that scientific studies use a head-mounted device is because, within reason, it doesn’t matter how comfortable or uncomfortable the participants are. Eyetracking was originally designed to evaluate subconscious biases—something over which people have little or no control. In science, eyetracking operates in a context-free environment. Scientists reduce all external influences to measure innate, invisible, and uncontrollable cognitive processes. Science relies on refutation, removing as many variances as possible and working from the bottom up.

Reason 3: Eyetracking doesn’t track what we see.

“Tracking a person’s gaze can’t inform us about their conceptions or true perceptions of information.”

The key issue with eyetracking is that the average UX professional has little knowledge of how people process information. Tracking a person’s gaze can’t inform us about their conceptions or true perceptions of information. The brain handles that unconsciously. Before we start paying attention to something, we have to work out whether it’s worth our time.

The brain has an internal moderator that manages where we look and the information that we take in. The amount of information available to us at any given point in time is overwhelming. We simply cannot attend to all of it. Therefore, we use a set of strategies that ensure we attend only to important information. To work out what to take in, we have to work out what we can take in. That’s where discourse steps in—exploring what participants actually read, what they understood, and what they now know.

Reason 4: Eyetracking is a numbers game.

“When using eyetracking to test Web sites, the opportunities for participants to change their gaze shoot up exponentially. It becomes harder for us to understand what they have truly looked at and where they have merely glanced.”

The more elements that you add to a stimulus, the more likely it is that someone will look at them. When using eyetracking to test Web sites, the opportunities for participants to change their gaze shoot up exponentially. It becomes harder for us to understand what they have truly looked at and where they have merely glanced. To get reliable results, we have to use more participants.

Eyetracking is a quantitative technique. It relies on high numbers of participants to establish an effect. The more people who attend to an area, the more certain we can be of its stealing their gaze. Usability testing is a qualitative technique. It pays less attention to numbers and instead focuses on rich, informative data that is based on opinion. This forces us to shoehorn eyetracking into a qualitative context. It can also shift the focus of a study. It’s easy to fall into the trap of basing your entire analysis on the eyetracking results, while losing track of the strengths that come from a dominantly discursive approach.

When combining qualitative and quantitative approaches, your must sacrifice one of them. You can’t easily and cost effectively get rich, qualitative data from hoards of participants, and you can’t get reliable, valid quantitative data from just a few participants.

User Experience Appears Immature as an Industry

“ False promises around what eyetracking can and cannot show, coupled with a lack of training and expertise on the part of researchers, may sour clients on the usefulness of the eyetracking to user experience.”

Because we can’t easily give eyetracking the numbers of participants that it requires, the results that we get from eyetracking are fundamentally limited. They cannot establish a direct link to cause, but they can

  • act as a strong focal point for discussion
  • provide a way for good facilitators to focus the dialogue
  • highlight where a design is letting users down

Eyetracking’s introduction to user experience has forced many agencies to offer the service—regardless of whether they believe in it—so many clients now view an eyetracking service as a measure of competency. False promises around what eyetracking can and cannot show, coupled with a lack of training and expertise on the part of researchers, may sour clients on the usefulness of the eyetracking to user experience.

Whenever we introduce a new technique without justifying its introduction and supporting its application, we reduce the quality of the results that the UX industry produces. We have always prided ourselves on being an industry that collaborates, so our communication about the use of complex scientific methods like eyetracking should reflect this.

My next article in this series will focus on EEG (Electroencephalography). I will explain what it is, describe how UX professionals use it, and discuss whether it has a true place in user experience.

4 Comments

Interesting points, James, however all 4 of your reasons seem unfounded and generally very biased against eyetracking. To me this illustrates the importance of more education on the methodology of eyetracking for UX research, to clear up these misconceptions.

Let me point out some counter-arguments for all your reasons against eyetracking, to give this article some more balance:

  1. High-end, unobtrusive remote eyetrackers (Tobii) have, in fact, higher accuracy than most, if not all, headmounted eyetrackers; down to 0.4 degree, which is more than sufficient for UX testing or even testing on small mobile phones. Scientific researchers use Tobii’s remote eyetrackers mainly, not headmounted systems. Just Google Scholar for Tobii. Only tower or dual purkinje eyetrackers can achieve better accuracy, but these are quite rarely used, usually only for high-speed gaze contingency or linguistics studies for example.
  2. Your argument is a sophism, it does not state why the subconscious behavior is not useful for UX research and pretends this is the only use for the method. Furthermore, eyetrackers from Tobii, for example, were initially designed for, as a main goal, measuring conscious gaze for interaction (to help disabled people communicate) in an unobtrusive way; eyetracking for research has mostly focused on distinguishing conscious versus subconscious behavior. This can be done by looking at fixation data or simply asking participants to elaborate on their behavior and determine which decisions were made consciously or not.
  3. It is a misunderstanding to say that because eyetracking captures subconscious behavior as well, we can’t trust the results. Fixation filters easily help define what is true attention and what is not, eyetracking is usually also combined with an interview to establish the reasons behind certain gaze behavior. Often participants explain their search/scan strategy, for example, and can give you insight in why they missed a certain element or button. We can’t look at something without moving our gaze over it, so eyetracking will make it measurable which elements were seen or not seen and in which order, often already providing insights to UX practitioners with live viewing during in a study.
  4. Most UX companies using eyetracking use it just as an extra qualitative tool, in fact, that gives them unbiased feedback from the user about their thinking and actions during a study. Often just the live viewing functionality of eyetracking is used for this. Many studies have been done on sample size in eyetracking for usability. Although it can vary depending on the stimulus and complexity of the task, even Nielsen Norman states that often just testing 5 participants will be sufficient to find the large majority of usability issues. Eyetracking does allow quantitative results as well, which does require higher samples, of course, as for all quantitative research, which can be crucial in, for example, A-B testing to establish which design is performing better and perhaps back up some of your qualitative findings.

Clients are asking for eyetracking because they are aware of the value they get from them. Clients are also more open to innovative methods, unlike many UX practitioners who feel eyetracking is a threat to their traditional methods or too complex to learn. Instead, I would argue that eyetracking is a great addition to traditional UX research methods. It just requires some proper education on the method to get the full potential out of it, which is something manufacturers and many others offer.

How interesting, Laurens, that while you are quite happy to directly accuse the author of bias, you neglect to mention that you work for the eyetracking company Tobii. See Laurens on LinkedIn.

Of course, working for an eyetracking company does not preclude you from entering a discussion about the usefulness or otherwise of the technique. It’s just a shame that none of your “counter-arguments” actually contradict the author’s four reasons.

For example, in your argument against Reason 3—the one I feel most strongly about (see my article—you say, broadly, that people have to look at something to pay attention to it. Indeed they do. But the author never disputed this. Rather, his point is that it is not a one-to-one relationship: looking at something doesn’t mean a person is paying attention to it.

Every tool can be used and abused, and eyetracking is no different. But it galls me to hear you say, “clients are asking for eyetracking because they are aware of the value they get from them [sic].” If clients are asking for eyetracking because it has been sold hard, and its constraints and cost have been significantly downplayed, I applaud James for helping to actually correct the imbalance.

Thank you for your reaction to my comment, Jessica. I never intended to hide the fact that I work for Tobii. I used my own name and Tobii email address, which anyone can Google and mentioned Tobii several times. We all are biased in some way, I believe, so I just see this as giving a little more balance to the arguments raised by the author.

May I also point out that I work as a Trainer and Product Specialist at Tobii and have a background in usability testing from my education. My job is to educate people on the proper use and value of eyetracking, and I actually critique improper use of the method as well—such as the practice of just “selling heatmaps.”

Eyetracking should, in the ideal case, be just one of many tools at the disposal of UX professionals and does not need to impact study design or cost greatly, eyetrackers are getting more robust and much lower priced every year, which all helps to democratize eyetracking, so it is not just an exclusive sales tool for some, but a helpful addition to the UX toolbox of many.

Regarding your point that a person’s looking at something doesn’t mean they payed attention is in some cases correct. As I mentioned also in my arguments: metrics such as fixation duration, but also fixation or visit count can provide more insight into whether a user really looked at or just glanced at something. But above all, a retrospective interview is often recommended to find out whether people actually remembered or noticed something or ignored it. We can easily ignore things we are really not interested in, such as banners, by using our pheripheral vision to identify these. So when we really move our foveal vision—the part measured by eyetrackers—over something for a longer time, there is a pretty big chance we do pay attention to it.

It is a big misconception that eyetracking alone will give you the answers to attention to certain elements. Combining the method with traditional methods such as post interviews and recall is the way I would recommend to use eyetracking.

My comments on the value of eyetracking come from numerous customers and their clients who I have met over the years, who have explained to me how eyetracking adds value to their research or has helped improve clients’ Webs ites and conversions, for example. This is not the result of a hard sell, but rather tangible insights provided by eyetracking to those clients.

I appreciate hearing and discussing arguments against eyetracking and giving my opinion about it. I see in my work that there are still very much two camps amongst UX professionals: you are either for or against eyetracking. I welcome anyone who is against eyetracking to try it out with an open mind and look at some successful, but critical users of the method in your own field—such as Aga Bojko from User Centric—to see how and if the method would apply to your work in the same way.

Seems I am arriving late to an interesting discussion.

At the start, I should probably note that I work for myGaze, the new $500 eyetracking technology from Germany; yet my comment reflects my personal opinion and observations.

My thesis is: eyetracking is too valuable and of too high potential to just be excluded.

Facts support that: Amazon, Google, Zappos, Zalando, Apple— the masters of UI and UX—have big eyetracking labs in house. The tools are an integral part of their methodology. I don’t think they are mistaken.

The problem in the past was: for a small company, a technology that costs >$20k carries the promise to answer all questions, which of course is not the case. Hence the disappointments at time. Now imagine having the tool for 1/10th that price - it comfortably moves to a more complimentary, yet still very important, role in your process. And that is what I see with our early customers who, due now to its affordability, start integrating eye tracking into their work. It provides another level of depth of understanding, when of course used with other methods like Morae, live observation and interviews. That is one way to do away with the looking Vs. attention discrepancy. Another would be scale and statistical significance of data, which, given my bias to data I prefer.

And the good news is, with price no longer a concern, the first UX labs w/ eye tracking are starting to happen and I feel statistical gaze data manipulation will bring a whole new world of insights to UX soon. Now that is very exciting, isn’t it?!

Join the Discussion

Asterisks (*) indicate required information.