Eyetracking: Is It Worth It?

Is Eyetracking Really Necessary?

Do you really learn anything from eyetracking that you could not learn through regular usability testing? Can a good designer foresee how users would view a design and guide their eyes through the design?

Johansen and Hansen sought to answer these questions in their 2006 study, “Do We Need Eye Trackers to Tell Where People Look?” [1] For this study, which compared conventional usability testing with eyetracking, they conducted a usability test in which participants reported from memory what they saw. They found participants remembered 70% of the elements they had actually seen, but were not very good at remembering the order in which they had looked at those elements.

They also asked experienced Web designers to predict the eye movements of a typical user and found the designers could predict only 46% of the elements users typically saw. Plus, only 36% were successful in predicting the first three areas of the screen participants looked at. The study concluded that designers cannot accurately predict where people will look, and participants cannot reliably report where they looked.

Of course, you can conduct usability testing without eyetracking, but eyetracking provides valuable additional data about where participants looked during tasks. This is data a facilitator cannot otherwise observe and participants cannot report accurately.

What Are the Limitations of Eyetracking?

Although seeing where participants fixate is valuable, there are limitations to what eyetracking can reveal.

Eyetracking Cannot Track Our Peripheral Vision

Eyetracking can be misleading, because it does not capture peripheral vision. Eyetracking records and displays foveal fixations, in the small part of our visual field that produces the sharpest vision. It does not record peripheral vision, which makes up 98% of our visual field. This is significant, because we use peripheral vision to choose where to fixate our fovea next.

So we can see things on a screen without directly fixating on them. For example, we can see banner ads on the right side of a page, using our peripheral vision. We know enough from their position on the right side of the page, their visual appearance, and from our previous Web experience to identify them as ads, and we often choose not to fixate on them.

Because eyetracking visualizations do not show peripheral vision, they can be misleading. For example, in a heat map, some assume users did not see areas without hot spots at all. It is important to recognize that the eyetracking fixations, or hot spots, do not represent everything participants saw. They are just the areas participants fixated on, viewing them in detail.

For example, Figure 7 shows two versions of the same heat map. The gaze opacity heat map on the right shows only the areas of the page with hot spots. The rest of the page is black. Obviously, this is not how we really see Web pages. We see entire pages through both our foveal and peripheral vision. The hot spots are the areas on which we fixate our fovea to see them in more detail. It is important to explain this distinction to the audience for your testing results.

Fixations Don’t Represent Attention or Understanding

Participants’ fixations do not necessarily equal attention or understanding. Eyetracking shows only fixation points—that is, points where the fovea stop to take in information—and saccades—eye movements between fixations. It does not reveal anything about the higher-level processes of attention and comprehension. Just because a person’s eyes fixate on something, that does not mean he or she is consciously paying attention to that element—or understands it.

This is why eyetracking data has little meaning by itself. Careful observation of tasks and discussion with participants is necessary to understand their mental processes. Eyetracking provides complementary data to aid in this understanding. It is not sufficient alone.

Fixations Don’t Communicate Meaning

Eyetracking shows only where participants fixated, not the meaning of their fixations—that is, it shows where, but not why. For example, if an element received many fixations or longer fixations, it could indicate either attraction to that element or difficulty in understanding it. Again, this is why you should not use eyetracking data alone. It requires interpretation. Careful observation and discussion with participants let you understand the context for your eyetracking data.

How Does Eyetracking Affect the Testing Process?

Eyetracking is not just software and hardware that you turn on to record usability test sessions. Using it effectively requires making changes to your usability testing process.

Interactions Between Facilitator and Participant Change

Eyetracking changes the interactions between the facilitator and participants. Getting accurate eyetracking data that shows what participants look at during tasks requires letting participants perform the tasks without interruption. Asking participants to think aloud, asking probing questions, or even allowing participants to ask questions interferes with capturing accurate eyetracking data.

Instead of using a concurrent think-aloud technique, you can use a retrospective think-aloud technique, in which you ask participants to describe what they were doing while you play back a recording of the tasks they performed. However, when using this technique, participants’ comments lose some of their immediacy. You should consider the trade-off between having participants think aloud and getting accurate eyetracking data, depending on which is more appropriate for the goals of your test.

Eyetracking Can Be Intrusive

Eyetracking can interfere with the naturalness of a test session. The ideal eyetracking system is completely unobtrusive to participants. Eyetracking systems have evolved from very intrusive systems that involved participants’ wearing headgear and having their heads immobilized to almost completely unobtrusive systems that look like a regular computer monitor. With these modern systems, most participants have no idea they are sitting in front of an eyetracker until they are told so at the beginning of a test session.

Yet, even with the most unobtrusive eyetracking systems, participants can experience a sense of self-consciousness—an uncomfortable awareness of everything they look at—because they know they are being eyetracked. Eyetracking self-consciousness tends to fade over time as participants get involved in the test tasks.

Another limitation of eyetracking systems is that participants need to remain within range of the eyetracker. Though modern systems can track participants’ eyes over a wide range, participants often move out of range over time—for example, they slump, lean back, lean forward, or rest their chin on a hand. When this occurs, the facilitator must ask a participant to move back into range. Each time this happens, it interrupts the participant’s task and disrupts the naturalness of the situation. If a participant repeatedly moves out of range, the facilitator has to decide whether it is more important to get the eyetracking data, or it is better to sacrifice the eyetracking data to see the participant’s natural, uninterrupted behavior.

Eyetracking Tests Take More Time

Planning and setting up an eyetracking study can add some extra time to the usability testing process. Both the need to explain eyetracking to participants and the need to have participants retrospectively think aloud to the playback of the eyetracking recording either add time to test sessions or reduce the number of tasks you can test. Eyetracking also generates additional data, adding significant time to the data analysis phase. With all of these considerations, eyetracking is probably not the best choice for testing with a low budget or when you need a quick turnaround.

Eyetracking Is Expensive

Although the cost of eyetracking hardware and software has come down, it is still somewhat expensive. The cost of a new eyetracking system is $40,000 to $60,000. Rental prices can be up to $3000 per day.

Eyetracking Can Be Difficult to Learn

Eyetracking is not something a usability professional can start using effectively right away. More important than learning how to use the hardware and software is learning how to effectively plan and conduct eyetracking studies and analyze eyetracking results. This is not easy. Beginners often learn through trial and error that, without careful planning, eyetracking results can be meaningless—and even when everything goes right, they can be difficult to interpret. Like anything else, analysis of eyetracking results is a skill that you must develop through experience.

Eyetracking Is Subject to Technical Problems

The more technology you have, the more can go wrong. Although eyetracking technology is continually improving in its reliability, anyone who had been involved in eyetracking has horror stories of technical problems that threatened to derail their testing. Experiencing such problems during a test session—especially with clients observing—is not fun. It can make you yearn for the good old days of old-fashioned usability testing.

Eyetracking Can Become a Gimmick

Eyetracking can be very impressive and exciting for clients, which makes them eager to do more usability testing and eyetracking. But the downside is that they may want to use eyetracking for everything, even when it’s not the most appropriate method for your testing goals. It is important to be sure eyetracking does not become a sales gimmick. Use it purposefully, where it provides value.

So Is Eyetracking Worth It?

Considering the advantages and disadvantages of eyetracking, is it worth it? Some have concluded that the benefits of eyetracking are not worth the high cost, effort, and complexity it adds to usability testing. On the other hand, some eyetracking vendors and consultants have promoted the idea that you cannot conduct usability testing effectively without eyetracking.

The truth lies somewhere between these extremes. If you know how to use eyetracking effectively, it can provide additional insights to usability testing that can help you find problems and answer questions about user behavior. Eyetracking is not essential to usability testing, but if you can afford it and have the time to learn how to use it effectively, it is definitely worth it. Ultimately, you have to make the decision—based on your testing goals and the considerations I’ve outlined in this article—whether eyetracking would provide value and its benefits would outweigh its limitations. 

Reference

[1] Johansen, Sune A., and John P. Hansen. “Do We Need Eye Trackers to Tell Where People Look?” Conference on Human Factors in Computing Systems: CHI ’06 Extended Abstracts on Human Factors in Computing Systems, 2006. Retrieved October 11, 2009.