When Observing Users Is Not Enough: 10 Guidelines for Getting More Out of Users’ Verbal Comments

By Isabelle Peyrichoux

Published: April 9, 2007

“Observing a user perform a task provides more reliable information than simply asking the user how easy it would be to perform the task.”

One of the principles underlying usability testing is that observing a user perform a task provides more reliable information than simply asking the user how easy it would be to perform the task. By observing users, you can assess whether they are actually able to use a product. By asking them, you simply cannot.

However, as you try to derive valid conclusions about how to design a user interface, relying only on—or even mostly on—observation can be

  • misleading—because often user behaviors that you observe can have many different interpretations. For example, if a user did not click a link, perhaps the user did not see the link or did not understand it. You cannot know the reason with certainty without asking the user. Your assumptions might be biased.
  • limiting—because you lose the opportunity to gather valuable verbal data by relying only on observational data.

While some usability professionals might claim that you cannot rely on what users say—and there are some risks in relying on users’ comments—there are means of avoiding or minimizing those risks. To understand these means, we must leave the realm of objective science and enter the realm of human relationships and empathy.

A user interview—including one that occurs during usability testing or user observation—is a relationship between two people—the interviewer and the interviewee—in which emotions, fears, and judgments come into play. Thus, my training and practice in psychotherapy have greatly enriched my technique in doing user interviews, because they have helped me avoid or minimize certain biases when eliciting and interpreting users’ verbal comments.

Inspirations From Psychology

The following psychotherapeutic and psychological approaches have inspired some of the ideas in this article:

Carl Rogers’s humanist approaches:

  • person-centered approach of Carl Rogers—Developed in the 1940s and 1950s, this approach belongs to the humanistic school of psychotherapy. Its core concepts include empathy with patients’ emotions and perspectives, genuineness, and unconditional positive regard.
  • Colette Portelance’s creative nondirective approach to psychotherapy—Developed in the 1980s, this approach was inspired by both Carl Rogers’s humanist approaches and Lozanov’s suggestology. Its core concepts include empathy, genuineness, and acceptance of our own emotions, needs, and defence mechanisms.

Carl Jung’s theories:

  • psychological types and the Myers-Briggs Type Indicator (MBTI)—Jung’s psychological types correspond to the MBTI functions: introvert (I) versus extravert (E), intuitive (N) versus sensing (S), thinking (T) versus feeling (F), and judging (J) versus perceiving (P). The dominant orientations in an individual define his personality type—for example, ENTP. The MBTI is one of the most widely used personality tests.
  • shadow of the personality—According to Jung, the shadow of the personality represents unconscious parts of our personalities that we have repressed—because we either don’t accept them or pass judgment on them—and tend to project onto others. For example, a person who doesn’t accept the emotion of anger tends to judge himself each time he feels angry and might judge other people who express their anger easily.

To help you get more out of users’ verbal comments, this article will provide ten guidelines and various interviewing techniques I’ve learned from experience. These techniques work best if they are used with genuine empathy for users. If users feel that you are not genuine—even if you are not aware of it or try to hide it—these techniques won’t work. I’ve described most of these techniques within the context of usability testing, but some techniques are also applicable to other user research activities—such as field studies and task analyses—and to stakeholder interviews.

1. Be aware of your own judgments and projections.

“If you want your interventions to be effective and users to be comfortable speaking as freely and honestly as possible, you must actually be nonjudgmental.”

It’s easy to say you’re not judgmental, but it’s not so easy to achieve that in reality. And if you want your interventions to be effective and users to be comfortable speaking as freely and honestly as possible, you must actually be nonjudgmental. It is useless for you to say there are no good or bad answers to your questions if your behavior says otherwise.

So, for example, be careful about saying “Excellent” or “Good” or any word that implies a positive judgment following a user’s answer. Saying an answer is excellent might imply that a user’s answer could be good or bad and that you are judging the user’s performance. Instead, depending on the context, you might say something like “Got it” or “I understand.”

In user interviews, as in all relationships, you’ll meet all kinds of people—both people with whom you feel comfortable and those you don’t, whom you might tend to judge. Unless you are careful, you might let your first reaction to a person color an entire user interview. It’s natural for you to be uncomfortable with the personalities of some users, but you must be conscious of your feelings and overcome them if you want to get the most out of an interview.

Observe your feelings about each user. Take note of any fear you feel or judgments you make. We often negatively judge others because they remind us of aspects of our own personalities that we do not accept. This phenomenon is called projection, according to Jung’s shadow theory. Observing your own feelings will help you to become less judgmental, which will, in turn, make users feel more comfortable with you, letting them speak more openly. This will enable you to get more useful information from your user interviews.

A Real-World Example

During a usability study, I met a woman whose manner was harsh. I felt uncomfortable and intimidated. My first tendency was to judge her: “She is rude.” Her frankness made me fear her judgment. Unaware of my own feelings, I thought she was actually judging me, but she was not. She was simply a direct person. I was projecting my own fear of judgment onto her and also my prejudices against harsh people. To compensate for my discomfort, I was overly nice to her during the interview. I was also very subtly judging and undervaluing her comments. After a while, I realized my own feelings were biasing my perceptions of her. I was imagining things that were not real. This helped me to stop judging her, and our interactions became easier.

We all tend to judge others. It’s human. By becoming aware of and taking responsibility for your judgments about users and the feelings that you project onto them, you can go beyond these and become more empathic.

Of all the guidelines I’ve given in this article, this one is actually the most difficult to apply. Doing so requires self-observation and a willingness to overcome your biases and defences. However, being nonjudgmental has a huge positive impact on your relationships with users.

2. Be genuine and transparent.

“The more your behavior aligns with your words, the more users will feel comfortable with you.”

The more your behavior aligns with your words, the more users will feel comfortable with you. Being truly transparent about your interview process or anything unusual that happens during an interview helps build users’ confidence in their relationship with you. If you are genuine and open, it will encourage users to be the same with you. Don’t pretend that everything is okay when users can sense that something is not. Any disconnect between what you say and what you do will make users feel insecure, and they’ll be less open with you. Here are a couple of scenarios to show you how this works.

Scenario 1

Problem: A user tells you something, but you were distracted or were thinking of something else and lost some important information that you need.

Solution: Let the user know that you were mentally absent. Say “I missed what you said. Would you please repeat it?”

Scenario 2

Problem: You want to follow a specific process during the interview or need to move quickly from one question to another and want only a user’s first impressions.

Solution: Let the user know before you start that you will move very quickly from one question to another.

3. Adapt to each user. Do not ask users to adapt to you.

“It is easy to fall unconsciously into the trap of expecting a user to adapt to your way of communicating rather than trying to adapt to the user’s.”

It is easy to fall unconsciously into the trap of expecting a user to adapt to your way of communicating rather than trying to adapt to the user’s.

After a usability test session, you might find yourself saying, “Oh, this person wasn’t a good test subject.” He was too something—perhaps too shy or too talkative. It’s possible that the comments a particular user made were not very helpful—no matter how hard you tried to get valuable information from him. However, to make the most of each user interview, you must ensure that you are doing your best to adapt to the user’s rhythm and personality. Otherwise, you risk losing important data.

A Real-World Example

In a usability test session, a user was answering one of my questions. Once he finished his sentence, he did not say anything for a little while. I thought he had finished speaking, so I went on to my next question. He suddenly interrupted me, giving me a very interesting and thoughtful response to my previous question. At that moment, I realized that I had misinterpreted his silence. He had not actually finished answering. He was thinking about his answer. After this, I gave him more time to answer my questions, and I received very relevant comments I would have missed if I had not respected his rhythm.

This example reflects the differences between introverts and extraverts, as defined by Jung’s psychological types and the Myers-Briggs Type Indicator (MBTI). Extraverts usually tend to think and speak at the same time, whereas introverts usually tend to speak only once they have thought through what they want to say. You should give people enough time to think before answering your questions—especially introverts.

It also shows how easily we can misinterpret users’ behavior. You must stay objective. If a user is not talking and there is an extended silence, don’t assume you know the reason for the user’s silence. Instead, observe how quickly he answers your first few questions and adapt to his rhythm. If he takes some time before answering, but gives detailed and thoughtful answers, be sure to give him enough time to answer your questions.

This example illustrates how different people can be and how important it is to be aware of their differences to make the most of user interviews. Learning about Jung’s psychological types can help you become aware of the diversity of personality types and how they can affect your relationships with users. This understanding will also help you to be less judgmental when confronted with a user whose personality is very different from yours.

4. Be conscious of the way users are interacting with you.

Even though you’ve carefully explained to users that they are not being tested, you’ll often encounter users who feel they are being tested and are afraid of giving a wrong answer. If a person is nervous throughout a test session, even though you’re being empathetic and nonjudgmental, it is useless to try and change his or her feelings. Regardless of how hard you try, it won’t change anything. Even worse, a user might become irritated by your mothering behavior.

Observe carefully how users interact with you, and take these observations into account when interpreting your findings.

A Real-World Example

During a usability test, a user continually asked me whether his answers were good. After observing him for about thirty minutes, I realized that this user was very concerned about the quality of his answers and wanted to make a good impression on me. Sometimes he was even showing off. At one point, when I asked him whether he had seen a link, he very quickly answered “Yes” in an overly confident tone that made me feel very uncomfortable. I had difficulty believing him. My previous observation of his behavior backed up my intuition that he might be lying and eyetracking confirmed that the user, in fact, had not seen the link. Based on these observations, I was very careful when interpreting the results of this session.

5. Get users to speak about their own experiences.

In nearly all usability test sessions, at some point, you’ll hear a user say something like one of these remarks:

    • “For me, it’s okay, but the average person might find it difficult.”
    •  “For my mother, it would be hard.”
    •  “Older people would have difficulty with it.”
    • “For someone who is looking for something like that, it’s good.”
“It places users in a less compromising position to speak for someone else rather than to speak for themselves and say what they really think.”

It is very common for users to speak for someone else during a test session. It often happens when users feel uncomfortable stating their own point of view. For example, they might fear being judged or want to please the interviewer. It places users in a less compromising position to speak for someone else rather than to speak for themselves and say what they really think—for example, “I find it very difficult,” “I think it’s really bad,” or “It’s useless to me.” This is something people do unconsciously every day, but do not let yourself be fooled by this. Users really know only their own experiences, abilities, and opinions. Gathering information about what users think the user experience would be for other people has no value.

To make sure users speak from their own points of view, don’t reformulate what the user said about a product’s user experience for other people. Instead, just restate the part of the user’s answer that represents his own opinion. When you do this, users will stop talking about other people’s opinions and speak for themselves—for example:

User: For me, it’s okay, but for the average person, it might be difficult.

Interviewer: For you, it’s okay.

User: Yes. It’s okay, because….

Alternatively, you can ask a question about a user’s opinion like this:

User: For my mother, it would be hard.

Interviewer: And what about for you? What do you think?

These examples show ways you can smoothly get a user to come back to his own opinions. If you do this with genuine empathy, the user will feel comfortable speaking more freely and honestly about himself and his personal opinions. Doing this acknowledges the user’s true opinion, indicates that his opinion is important to you, and shows that you are not judging him. Reformulating a user’s answers conveys empathy and acceptance.

While this generally works very well, in the rare case that a user keeps talking about other people’s viewpoints, do not push too hard and insist that the user talk about his own opinions. Otherwise, the user may become defensive.

6. Notice when users are censoring their own comments.

“If you have carefully observed a user’s behavior throughout a test session, you can probably judge whether the user will try to please you by self-censoring his real impressions or really has mixed impressions.”

You’ll often see users self-censoring their opinions. This often happens when users fear their opinions are too critical. For example, at the end of an interview, you might ask a user about his general impressions of your Web site. Perhaps the first words that come to his mind are “very complicated,” but he hesitates to express this negative judgment, fearing he might offend you. So, he tones down his original thought and says, “very complicated, but when you get used to it, it’s okay,” or “but for people who know the field, it might be easy.” In some cases, users really have mixed opinions about a product, but in other cases, they are just trying to be nice. If you have carefully observed a user’s behavior throughout a test session, you can probably judge whether the user will try to please you by self-censoring his real impressions or really has mixed impressions.

To ensure you capture a user’s real opinion, reflect back the user’s initial opinion like this:

Interviewer: What are your impressions of this Web site?

User: Oh, it’s very complicated, but I guess, for people who know the field, it’s okay. Yes, I think it’s okay.

Interviewer: You said it was very complicated.

User: Yes, it’s very complicated because….

7. Get users to speak in terms of problems, not solutions.

“You can help the user to provide more precise information by asking follow-up questions that are appropriate to the context.”

Often, during usability testing, users offer solutions to problems. For example, after failing to find a link on a Web page, a user might say, “I did not see that link. It should be in bold, or it should be bigger.”

The user is not a designer, so the solution the user suggests—that the link should be in bold—might not actually work. What will help you find the right solution is to investigate why the user did not see the link. So, if you can, get the user to tell you why he couldn’t see the link. Sometimes, the user won’t know, so don’t push too hard, but he might give you very interesting information that will help you identify why he didn’t see the link and, ultimately, help you find a solution. For example, he might say, “I was concentrating on another part of the screen and didn’t notice there were links in this area,” or “I thought it was just text.”

And you can help the user to provide more precise information by asking follow-up questions that are appropriate to the context—like this one, “Were you expecting to find the link on another part of the screen?” Each piece of information you glean will help you better understand the reason why the user did not see the link and help you find a solution to the problem that you identify. Only when you have accurately identified the problem can you come up with the right solution.

Here are two examples of how you can help a user to clarify a problem:

User: This label isn’t right.

Interviewer: Why isn’t it right?

Don’t initially ask, “What would be a better label?” That would be asking the user to solve rather than identify the problem.

Once you understand the problem, you can ask follow-up questions that are appropriate to the context—like “What were you expecting?” or “Did you have a word in mind?”

User: “This page is dull. I don’t like it much.”

Interviewer: “Why you don’t like it?”

Don’t ask, “How would you improve it?”

It’s actually easier for users to first explore a problem rather than thinking right away about a solution. Plus, you’ll avoid losing important data about the problem, which in the end will help you to devise the right solution. Though, once you and a user have explored a problem together, the user might come up with a very good solution.

8. Ask “Why?” and dig deeper.

“When interviewing a user during usability testing, asking “Why?” and exploring users’ statements in depth is essential.”

When interviewing a user during usability testing, asking “Why?” and exploring users’ statements in depth is essential. If you don’t dig deeply enough in trying to understand a user’s point of view, you won’t get enough information to make the proper recommendations to improve a user interface. Statements like the following won’t provide sufficient information to your product team:

“Participants preferred the previous version of the Web site.”

“Participants did not understand the label.”

“Participants did not click the link.”

You must understand and explain why. Without your providing the reasons behind such statements, it will be hard for designers to know how to improve the design of a product’s user interface. To come up with a good design solution, they must have an in-depth understanding of the problem they are trying to solve. Thus, when interviewing users during usability testing, always keep in mind what you want to do with the findings and ensure that you gather all necessary pieces of information to help you reach your  goal—generally, helping your team to redesign a user interface.

This guideline pertains to many user research activities. For example, Indi Young points out how important it is to ask “Why?” when doing a task analysis and to “dig into the background of a topic until the interview participant has no more to say about it, or takes you on another tangent.” For a task analysis, the ultimate goal of user interviews is to clearly identify users’ tasks and build a complete mental model of their work. To succeed, you must keep your final goal in mind during the interviews.

Do not be afraid of digging too deeply or getting into too much detail. You are better off having too much detail than having an incomplete explanation of a problem when redesigning a user interface. Sometimes, when first interviewing users, it’s hard to know what specific pieces of information you need. You’ll learn what to explore by trial and error. If you find some of the details you’ve gathered aren’t relevant, you can avoid exploring them further in your next interviews. 

9. Make objective and precise observations.

“Objective and precise observation… is a simple, but very powerful tool for avoiding misinterpretations of user behaviors and getting users to talk to you.”

During my training in the creative nondirective approach to psychotherapy, I learned something that helps me a lot in usability testing: objective and precise observation. This is a simple, but very powerful tool for avoiding misinterpretations of user behaviors and getting users to talk to you.

For example, if a user is looking at a part of the screen without doing anything, don’t interpret what the user is experiencing by saying, “You are hesitating.” You can’t really judge whether the user is hesitating. Instead, as a result of objective and precise observation, say, “I notice that you have been looking at this part of the screen for a while.” If you make an objective observation, the user will generally explain what he was thinking.

If a user smiles when looking at a Web page, but does not speak, you might wonder why he is smiling. A smile can have many different meanings, but there is no way to know the exact reason why a user is smiling without asking. If you don’t ask, you won’t learn why and might lose an interesting bit of information, so try this:

Interviewer: You are smiling.

User: Yes, because I like the image on the page.

This technique can help with any user behavior that you observe and want to understand better—whether silence, nonverbal expressions, or a user’s pattern of navigation through a user interface. It provides a lot of rich information you would not have without asking the user, and if you don’t ask, you risk misinterpreting the user’s behavior.

10. Allow users to be spontaneous and follow their flow.

In usability testing, the more spontaneous a user’s answers are, the more reliable they are. Here are a few techniques for getting more spontaneous responses from users:

Let users talk without interruption unless they go outside the scope of a usability test. Also, let users remain silent or pause for a while if they need time to think.

This is often hard to do, because you might become impatient or have difficulty bearing the silence, but you should avoid interrupting a user’s thought process. An introverted user might still be composing what she wants to say in her mind. If you interrupt, you might lose some very interesting information the user was about to tell you.

For example, if a user is scanning a page of search results, and still in the process of thinking about them, starts saying, “Ah, the search results are highlighted…,” you should not interrupt the user by asking, “What is it?” Instead, give the user time to gather her thoughts.

Always go along with a user’s flow—regardless of the sequence of questions you’ve planned for a user interview.

For example, perhaps a user starts talking about a topic you intended to address at the end of your interview. While much depends on the particular situation, I generally recommend letting users talk rather than telling them you’d prefer to go back to some point later on. If a user spontaneously raises a point you wanted to know about, it is golden.

Let users speak about their spontaneous reactions rather than asking them questions right away.

For example, once a user lands on a Web page, first wait a bit for his spontaneous comments. Don’t immediately start asking the user questions.

If you do inadvertently interrupt a user, try returning to the user’s spontaneous comments.

Fortunately, if you miss something a user says or cut a user off, it’s usually possible to go back to what the user was saying. Even when you’re careful, it’s all too easy to cut off a user’s remarks. To help get a user back on track, you might say, “A moment ago, you were saying…” and repeat the words the user was saying when you interrupted him. The user will generally go back to his previous situation and explain it to you as though it has just happened.

This technique also works if a test session is interrupted for any reason—for example, if a computer breaks down or someone comes into the room—and you want to return to what the user was saying before the interruption.

Conclusion

“The way an interviewer interacts with users influences the outcome of test sessions greatly.”

A usability test implies, among other things, a relationship between two people—an interviewer and a user. The way an interviewer interacts with users influences the outcome of test sessions greatly. Drawing conclusions from only observation is risky. You must elicit verbal comments from users in a way that enriches your observations and helps you avoid biases. To make the most of your user interviews, convey confidence and empathy, adapt to users’ personalities and rhythms, get users to talk about their own experiences and the reasons behind their comments, explore users’ comments in depth, and follow users’ flow.

When doing eyetracking studies, you should always elicit verbal comments to ensure that you interpret users’ behaviors correctly. For example, a hot spot on a word might have different explanations—such as interest, confusion, or surprise. However, relying too much on users’ verbal comments can be just as risky as relying too much on observational data. For example, a user might say he likes a Web site after failing all the tasks during a test session. A successful usability test session results from the right combination of observation and verbal comments. Observational and verbal data are more reliable in combination than when used separately.

Bibliography

Briggs Myers, Isabel. Introduction to Type: A Guide to Understanding Your Results on the Myers-Briggs Type Indicator. Palo Alto, California: CPP, Inc., 1998. Revised by Linda K. Kirby and Katharine D. Myers.

Briggs Myers, Isabel, and Peter B. Myers. Gifts Differing: Understanding Personality Type. Palo Alto, California: Davies-Black Publishing, 1995.

Monbourquette, John. How to Befriend Your Shadow: Welcoming Your Unloved Side. Ottawa, Canada: Novalis, 2001.

Portelance, Colette. Helping Relationship Through Self-Love: A Creative Nondirective Approach to Psychotherapy and Education. Montréal: CRAM Editions, 1995.

Rogers, Carl. On Becoming a Person: A Therapist’s View of Psychotherapy. Boston, New York: Houghton Mifflin Company, 1961.

Young, Indi. “Six Steps to Better Interviews and Simplified Task Analysis.” Adaptive Path: February 16, 2004. Retrieved April 7, 2007.

25 Comments

Sounds very much more like counseling than Web design! Some great advice though.—Thanks!

Very nice. I’ll definitely have to practice asking users to speak in terms of the problem instead of how to fix the problem.

Isabelle, it is a good article. All the advice given in the article can be used for other types of interviews as well. As the previous comment said, it sounds like counseling advice sometimes.

I wonder, if through your research, you have come across a set of common words or phrases used by users that are particular to describing specific areas or functionalities of a Web site. Maybe if we identify these common words or phrases, we can then create appropriate questions for an interview. A proactive approach for the interview. Some food for thought.

Thanks all for your comments! I’m glad you like the article and find the advice useful.

Sam—I’m glad you have noticed the guideline about asking users to speak in terms of the problem instead of how to fix the problem. I’m using it as much as I can, and I’ve gotten very useful and deep information from it.

Juan—I do agree. All the advice I’ve given in the article can definitely be used for other types of interviews as well. And I would add, not only user interviews, but also stakeholder interviews as well. A colleague of mine who is a strategic planner and who read the article told me she had used some of this advice in her stakeholder interviews. As a consultant, when dealing with clients, I also use some of these guidelines. I have to deal with all kinds of clients who have very different ways of communicating—some very direct and fact oriented, some very talkative, some more reserved, and so on. Knowing more about the types of personalities—through MBTI, for example—helps me a lot in adapting to each of them and finding out how to better interact with them. I’m not sure I understand what you mean by “a set of common words or phrases used by users that are particular to describing specific areas or functionalities”. Could you give an example?

Nathanael and Juan—The content of my article is clearly influenced a lot by my training and practice in counseling. Many things I’ve learned in counseling help me so much in my everyday practice as a user researcher and as a consultant as well. I do think we can learn so much in our interview and user research practice by including techniques from many fields, not only from ethnography, sociology, and cognitive psychology, but also from counseling techniques.

Very nice article. I have posted a link to my blog also. Thanks.

Very good advice and pointers.

Specifically, not asking users to tell you what they would do differently in the user interface or what they would like to see is great advice. This is a common mistake made.

Remember, users are not designers.

Thanks Sahil and Jim for your good comments!

Jim—I’m glad you pointed out this sentence: “Users are not designers.” I agree it’s a common mistake made by interviewers and also by people who are not expert in our field. As a consultant working with many different clients, I’m often asked by them to get solutions from users in interviews or usability testing. Users are not expert in design, but they are expert in what they do and in the objectives and tasks they want to accomplish that relate to the scope of the Web site. We must gather information about what users are expert in and not ask them to be designers.

Nice job Isa!! Keep up the good work!!

Thanks Nick!! I’m so glad to see your comments! Thanks for your support.

Isabelle,

Thanks for your prompt response to our comments, great work!

Like you said, we have to adapt to each of our users to find out how to better interact with them, and I would like to see if you have come across common words or phrases used by different user groups to describe a Web site or Web application?

For example, during audience research, a focus group of middle-aged teachers describes a Web application as “up to standards.” In their vocabulary, this phrase means that the text was properly organized, there are no spelling mistakes, and proper grammar and punctuation.

A different focus group of young IT professionals describes the application to be “up to standards,” because of its Ajax functionality, CSS design, and seamless workflow.

If there is a way we can identify a common Web vocabulary for particular user groups, we can adapt our questions and responses before and during the interview rather than fully adapting to our users only during the interview.

I like your article a lot. I am just curious if there is a common vocabulary used by different user groups.

Juan,

Thanks for your question. It contributes to the discussion. I’m still a bit confused about what you mean exactly, but here are a few of my thoughts.

What I find interesting in your example is that the phrase “up to standards” doesn’t always mean the same thing. It means something different depending on your audience. It shows how important it is to help people say precisely what they mean in their own words rather than assuming right away we know what they mean. The phrase “up to standards” can be interpreted in so many different ways, depending on the person and maybe on the type of audience.

In usability testing or focus groups, I generally have a set of questions or tasks for each user or group, which I’ve prepared in advance. But I try to avoid any bias in the questions and tasks to leave the answers as open as possible. So, to avoid bias, I wouldn’t introduce any too specific words of my own in the questions or answers. Each user or group of users can have a different view of the problem and a way to express it. We have to leave all the options open as much as we can to avoid any bias. After a usability study, we can see if there any patterns in the words or phrases used. But I wouldn’t use these words or patterns for the next study with a new group of users, even if the group has the same profile. I would instead leave all options open and see whether I observe the same patterns in the answers and in the words used.

Hope it helps!

Sounds good!

I think this article can raise and solve the important question regarding usability.

Keep up the good work!

Thanks Muckoda! I’m glad you like the article.

Great article, Isabelle. Thank you. I did an introductory course in counselling a while back and was amazed at the number of people who thought they could ‘tell more about people by their behaviour than by what they said’.

Such an arrogant and mistaken view needs a timely reminder that behaviours can have all sorts of explanations and that there are as many interpretations as there are interpreters.

Thank you for reminding us that we have to ask as well as look.

Regards, Simon

Thanks, Simon, for your comment!

I’m really glad you share my opinion about the risk of interpreting people based on how they behave. This is true not only for user research activities, but for all human relationships in every context. So many misunderstandings—and shall I say conflicts—come from this conscious or unconscious belief that we can interpret people based on how they behave without asking them questions—as if we were able to read people’s minds. In user research, we really have to be careful if we want to base our recommendations on the right observations. As human beings involved in relationships with other human beings, we can misinterpret users in the same way we misinterpret people in our everyday lives.

And I do agree with you when you write: “there are as many interpretations as there are interpreters.” And I would add: Our interpretations tell more about ourselves than about the user or the other person. Unconsciously, we tend to interpret what we see based on our own beliefs, hypotheses, or states of mind. If we are not careful, we can project them onto others without even being conscious of it.

That’s why it’s so important to distinguish what we observe objectively—what I call objective and precise observations—from what we think our observations mean and find a way to verify with the person in reality what our observations actually mean.

This guideline helps me so much as a consultant and user researcher, not only in user interviews, but also in all my contacts with colleagues, clients, stakeholders, and so on. I do hope it can help others as well.

Thanks, Simon, for sharing your experience and opinion.

Isabelle

I found some interesting comments about my article on a public Internet group called UCDChina, which is dedicated to promoting user experience in China. I’d like to respond some of the comments here.

About asking ‘Why?,’ Ian Liu wrote: “Are you sure it’s a good way to always ask ‘why’ to testers? As I know, it will make them feel upset if we frequently ask questions. They may feel this test is testing them, not software. Also, it will cause them to ignore some problems to avoid answering any ‘why’ questions, especially some Chinese users. To avoid more trouble, they may pretend and say, ‘It’s okay. No problem for me.”

My thoughts in response: This is a very interesting point. It’s true that we have to be careful when asking Why? This question might seem rude to some users. What is important here is to dig deeper into a user’s response, whatever means we use to do it. A few things I’ve tried include

  • letting a user talk—Sometimes the user will naturally give you the reason if you wait a bit.
  • reformulating a user’s statements—Many users will naturally give you more information, and it’s a very smooth way of helping the user to dig deeper. The user won’t even notice.
  • replacing ‘Why?’ with a more conversational question such as ‘How so?’

I also find the aspect of cultural differences very interesting. Like personality types, cultural differences can play a big role in interviews. It does help when you belong to the same culture as users. If that’s not the case, you must try to adapt to the users and their ways of communicating. In Montreal, we meet users with many different cultural origins. When I’m not familiar with a user’s culture, I observe the user more carefully to see how he reacts to my questions. It helps me adapt to him.

About how to be less judgmental, Ifan Chou wrote: “Therefore, how to dig into those pretty, but not useful comments without [getting] our own judgments involved is a lesson we need [learn] along the journey. It takes time and effort to make it happen.”

I’d like to make a few more suggestions about this: f83a040076ea99bdc372d5e29448a52b

About ‘pre-practice,’ Angela Fan wrote: “Before you actually have an interview with a user, you can talk with your family or friends beforehand, and they can tell you something like ‘You should make a joke here,’ ‘a picture will [make a bigger impression on a] user. … I regard this as great advice to me.”

I’d like to offer another suggestion: Do a pre-test with a colleague you know can give you insightful comments on how he felt about your questions. After the test, check with him about how he felt about your questions and how easy it was for him to answer them. This can help you to refine your questions.

Usability testing is also used in technical writing. We observe people using the document in a realistic situation in order to discover errors and areas for improvement. It is very good work Isabelle! Good article !

Thanks Valerie for your comment!

I’ll be very interested in discussing with you how you apply usability testing to technical writing. I think usability testing can be used with any type of user interface, product, or document—electronic or paper.

Very nice! Clear and informative. The dialogues help to make the point very much.

These guidelines will be very useful for my team, especially number eight; I’ll be sending this article to them.

Thank you!

Thanks, Shuan, for your comment! I’m glad the guidelines are useful for you and your team. I’m also glad the dialogues help to make the point. It’s very important to me that these guidelines can be clearly understood and easily applied by as many readers as possible.

Actually, it’s fine to ask people how they would fix the problem. But you mustn’t take those responses literally. The types of solutions you collect are simply another form of data that you must interpret. Are there default expectations being established by other applications—for example, Google—that your design is competing with? Or other cultural ideas—from science fiction say—that are setting expectation levels? These answers reveal barriers to adoption of your solutions.

Ideas for fixes also reveal other frames of reference. Ask what they’d like to see different, then ask why. You don’t need to solve it their way, but understanding—from many different perspectives —what they believe the problem to be is crucial.

A simple example we always use is the difference between “It needs to have a handle” and “I need a way to carry it.” There are other solutions to the carrying problem besides a handle, but it’s the obvious one for a non-designer. Asking questions that will generate the handle solution—as long as you are willing to interpret the response—are excellent.

Thanks, Steve, for your insightful comment. It helps make more precise what I meant when I recommended getting users to speak in terms of problems, not solutions. I agree with you. The problem is not specifically in asking people how they would fix a problem, but more in taking those responses too literally and not investigating enough to generate a good solution. I’ve seen many interviewers ask users only about how they would fix a problem and taking users’ responses literally, without exploring more deeply what the problem and the root need were—for example, as you mentioned, “I need a way to carry it.” This is why I think it’s important to warn people about taking that approach.

As a consultant, I also want to ensure that the client in the back room is not taking the solution a participant gives literally, which is often their first temptation. Quick solutions expressed directly by users generally catch clients’ attention, and it can be hard to explain afterward why they are not relevant. It’s part of the client’s education process though.

My main point is really that we have to ask questions that will generate the root need or the root problem. If we only ask questions that generate solutions from users—for example, “It needs to have a handle,” it may be hard to get enough data to brainstorm a proper solution.

I find it interesting when you write, “The types of solutions you collect are simply another form of data that you must interpret” and “Ideas for fixes also reveal other frames of reference”. I’ll think more about this when listening to users to see if I can get insights from the solutions they give. However, in some cases, I’m not sure the solutions users give are very helpful or easy to interpret. I don’t have specific examples that come to mind though. I need to think more about it and find concrete examples. Thanks for sharing your thoughts and ideas! It helps go deeper into the topic.

Hi Isabelle,

I’m joining the discussion slightly late, but I still wanted to comment on your excellent post. I have just posted an article on key business benefits of user testing, and within a section titled “How to maximise the potential of think out loud user testing,” I have provided a link into this article as I certainly didn’t want to re-invent the wheel. (Plus my article would have doubled in size!)

The primary reason for posting on this subject is that, through my experiences over the past few years, with both SMEs and more surprisingly with larger blue chips and multi-nationals, user testing is a practice almost always overlooked.

As the amount of marketing spend on recruitment with the likes of Google AdWords constantly increases, if businesses were to allocate some of that budget to carry out professionally planned and executed user testing, the conversion improvements would speak—and pay!—for themselves.

Once again, a great article.

Paul

Hi Paul,

Thanks for your comment and for linking to my article from your site. I do appreciate it. Sorry for the delay in answering your comment.

As you do, I think usability testing and having users think aloud during usability testing are necessary to find out the why behind users’ behaviors. For example, as you mention, finding out “why checkout conversion rates are low and drop-out (abandonment) rates for a particular stage in the checkout process are high.” Analytics, which are very useful tools, can’t answer the “Why?” question and without answering that question, there is nothing we can do—or at least very little—to fix the problem.

Thanks again for your post and good luck in convincing your clients of the value of usability testing!

Isabelle

Thanks for the great article. There’s a real art to facilitating a think-aloud. And unless you’ve actually been doing it for a little while, it’s hard to imagine how many things can go wrong and how hard that can be. Here’s another article that touches on some of these topics:

How Much Interaction is Too Much?

Join the Discussion

Asterisks (*) indicate required information.