User Research Is Unnatural (But That’s Okay), Part I

By Jim Ross

Published: April 5, 2011

“From the perspective of a participant, user research is not very natural.”

Thanks for coming in today. I appreciate your taking time away from work in the middle of the day to drive downtown, park, and find your way to our office. Most people wouldn’t do that, but the recruiting company said you’ll be getting a big check, and who can’t use some extra money these days?

Let’s go into this room here. Oh, the big mirror? Yes, as you can imagine, there is a room full of people behind it who are watching and listening to us, but try not to think about that. We also have some people who couldn’t be here who are listening in through this speakerphone, and we’ll be recording the entire session through these cameras and microphones. But don’t worry; no one outside the project will see the videos. Ha ha! No, they won’t end up on YouTube.

Let’s sit down at this computer. I’ll show you a Web site and give you some things to find and do. They may not be things you would normally do, but try to imagine that you really want to accomplish these tasks.

By the way, this is just a prototype. It doesn’t have the images or text in it yet. No, that’s not Spanish, that’s just placeholder text. Since it’s a prototype, not all the links and functionality work, so at times, I’ll just describe what would normally happen, and you’ll just have to imagine what the site would really do.

Just try to do these tasks as you normally would if you were doing this at home. Except, try to think aloud, telling me, step by step, what you’re trying to do and what you’re thinking. That might be kind of awkward, but if you stop talking, I’ll just keep pestering you to think aloud, or I’ll start asking questions. At some point, it will probably become difficult to perform these tasks, think aloud, and answer my questions all at the same time, but try your best anyway.

Meanwhile, I—and all of those unseen observers that I mentioned—will be watching and listening to you and taking notes. I’ll ask you a lot of questions, but I won’t be able to answer many of your questions. You may not know whether you’ve correctly accomplished the tasks, but I’ll know, and I’ll make note of that. You’ll probably know when you’ve done or said something really interesting, because you’ll notice me furiously jotting down notes.

But remember, this isn’t a test of you. You’re helping us test out the Web site. So just act naturally and show me how you would really do these things, if this were a real Web site, these were tasks you would normally want to do, and you were at home in your normal environment. Any questions?

From the perspective of a participant, user research is not very natural. We ask participants to try to act naturally in the artificial environment of a lab, or we impose ourselves on their environment and hope our presence doesn’t affect their behavior. We often forget how unnatural user research can be and what effect it can have on participants.

Why Does It Matter That User Research Isn’t Natural?

“We want to observe natural behavior, and we want to get participants’ honest feedback. If they feel uncomfortable or as if they are doing something artificial, it affects their behavior and responses.”

It matters because we want to observe natural behavior, and we want to get participants’ honest feedback. If they feel uncomfortable or as if they are doing something artificial, it affects their behavior and responses.

Experiencing user research from the participant’s perspective can be eye opening. I had my first bad experience as a participant about 20 years ago, long before I became involved in usability. I was traveling to France for the first time and had a layover of a few hours at Heathrow Airport in London. I was young, jetlagged, confused, alone, and very self-conscious about being in Europe for the first time. After I had cleared customs, a woman asked me whether I would participate in a study of what people do while waiting in the terminal. It sounded interesting, so I agreed—not realizing they would give me a very large number to hang around my neck, so they could observe where I went. Of course, I was completely embarrassed and sat, as inconspicuously as possible, reading a book for three hours. The poor design of this observational study completely affected my behavior. I still wonder whether any of the other numbered participants were brave enough to act naturally that day.

What’s Unnatural About User Research?

“As long as we understand and acknowledge the ways in which user research is unnatural, we can minimize the impact this has on participants’ behavior.”

User research studies aren’t completely natural, but that’s okay. As long as we understand and acknowledge the ways in which user research is unnatural, we can minimize the impact this has on participants’ behavior. In this first part of a two-part Practical Usability series, I’ll examine ways in which user research can be unnatural. Part II will recommend ways of minimizing or eliminating these unnatural aspects of user research, allowing you to observe more natural user behavior.

Recruiting

The most natural user research would take place without participants’ even knowing we were studying them. However, in most cases, this would be highly illegal and unethical unless you were working for an espionage or law enforcement agency. The principles of informed consent require that participants understand your study and consent to participate.

The People We Recruit

“The first place unnaturalness creeps into user research is in how well the people we recruit for a study represent the typical user.”

The first place unnaturalness creeps into user research is in how well the people we recruit for a study represent the typical user. Unfortunately, typical people don’t usually volunteer for user research. Those who do volunteer are sometimes in it only for the money, or they just enjoy being a participant. Such professional research participants may lie to get through a screener and may be either uninterested or overly eager to please, in the hope that they’ll get invited back for future studies.

The Incentives We Provide

In addition to attracting professional research participants who want to supplement their income by participating in research, incentives influence behavior. Research that has evaluated the effects of providing incentives on user research has shown that incentives cause participants to feel obligated to put more effort into their participation.

Rewarding Good Participants

“People who are natural when thinking aloud and can eloquently express themselves … provide great sound bites, feature heavily in highlight videos, and get invited back for future research.”

Well, what’s wrong with getting people to put more effort into a study? People who are natural when thinking aloud and can eloquently express themselves are considered great participants. They provide great sound bites, feature heavily in highlight videos, and get invited back for future research. Observers are more likely to remember them and place more weight on their comments, while dismissing quieter participants as bad or unrepresentative participants.

By screening for outgoing, articulate people, we reward people who we think would be good participants and punish those who would make bad participants. Professional research participants learn this and adapt their behavior, so they’ll get invited back. Because of this, introverted and quiet people are often underrepresented in research—even though they may represent a significant part of the user population.

Where the Research Takes Place

User research can take place either in the lab or in the field.

Lab Studies

“We often ask participants to leave the place where they would normally do something we’re studying and come to our artificial usability lab or focus group facility.”

We often ask participants to leave the place where they would normally do something we’re studying and come to our artificial usability lab or focus group facility. Why? Usually, we do this for the convenience of the researchers and observers. That’s where we’ve got the recording and observation equipment set up; we can save time and money on travel; everyone can come to one place to observe the research; and we can get the study done more quickly. Meanwhile, the participants go through the inconvenience of taking time off from work or their personal lives to come to us and participate in the study.

Field Studies

Most people feel much more natural and comfortable when being observed at work or at home. They are on their own turf, using their own technology, with their usual tools and artifacts available. Because the purpose of field research is to observe natural behavior, this eliminates most of the artificiality of lab-based research.

However, even field research is not completely natural. It’s not an everyday occurrence to have several people come into your home or workplace who want to observe you, interview you, record what you do and say, and take photos. Your home is a much more personal space than your workplace, and you have a high expectation of privacy there.�With several strangers in their home, we can’t expect people to act the same they would when they’re alone.

Explaining the Research

Researchers work hard to create a casual rapport with participants. We strive to seem like relatable, regular people taking participants through a casual conversation, instead of cold scientists examining subjects in a lab.

The Explanation of the Research

“Unfortunately, our warm, friendly tone changes when we explain the research. The rules of informed consent sound like a legal disclaimer or the reading of a suspect’s Miranda rights.”

Unfortunately, our warm, friendly tone changes when we explain the research. The rules of informed consent sound like a legal disclaimer or the reading of a suspect’s Miranda rights: “The session should last about an hour, but you have the right to stop the session at any time for any reason. With your consent, we will record the session, but you have the right to request that we stop recording at any time…” If this is handled improperly, an abrupt change from a friendly to a formal tone can result in a very awkward transition that undoes your careful work to put a participant at ease.

In graduate school, I was advised to read exactly the same instructions verbatim from a cue card to every participant to avoid bias. I agree that doing this with every participant would help avoid bias, but does that consideration outweigh the impact of our greeting participants with a robotic reading of what sounds like a legal disclaimer?

The Consent Form

A consent form is integral to informed consent, but the need to read and sign a formal document sets a formal tone for the session. When we ask people to sign what looks like a legal document or waiver, it puts them on edge. They might think, What am I giving away in this session? So, what we do with the intention of empowering and informing participants often seems as if we’re asking them to waive their rights and protect our organization.

The Things We Ask Participants to Do

“In usability testing, we sometimes ask participants to do things they wouldn’t normally do, using software or Web sites they wouldn’t normally use.”

In usability testing, we sometimes ask participants to do things they wouldn’t normally do, using software or Web sites they wouldn’t normally use. We ask them to imagine a scenario and show us what they would do in that situation. Ideally, if we have recruited the right participants, the things we ask them to do aren’t that far from reality, but sometimes they’re a stretch.

If you ask participants to do things they aren’t comfortable doing, you won’t get realistic behavior. If you do manage to find people who aren’t embarrassed by doing things that would embarrass most people, you have to wonder how representative those participants are. For example, a marketing person once described some in-home market research he did for a shaving products company in which five people crowded into bathrooms to observe how women shave their legs. I don’t know how they found women who were willing to volunteer for that kind of study, but I have to wonder how representative those participants were.

Prototypes

“The lower the fidelity of ”

Although there are great benefits to early usability testing, low-fidelity prototypes are not what is familiar to the average person who uses software or Web sites. The lower the fidelity of a prototype, the more we have to ask participants to pretend and imagine.

We sometimes forget that wireframes and low-fidelity prototypes can look very foreign to those who are not used to seeing them, especially those that still contain lorem ipsum text. Inevitably, there’s at least one participant who asks, “Why is this in Spanish?” Even worse, when a designer or developer has used humorous names or unbelievable data, participants tend to latch onto these kinds of details, taking them out of the reality of the situation.

Low-fidelity prototypes require more interaction between a facilitator and participants, which interrupts the normal flow of tasks. The facilitator often has to explain how the actual software would respond to participants’ actions. Because many screens and actions are missing, the facilitator has to ask participants what they think would happen in response to their actions. This can yield some good insights, but it’s far from the experience people would have with a fully functional user interface. Some participants have no problem imagining and working with a low-fidelity prototype, while others have difficulty getting past the unfinished aspects.

Thinking Aloud

“Having participants think aloud while performing tasks is a great way to get insight into their experience, but it’s not what people normally do when they are alone.”

Having participants think aloud while performing tasks is a great way to get insight into their experience, but it’s not what people normally do when they are alone. (If it is, they may need professional help.) Some participants are able to express their thoughts and experiences quite naturally, while others find it difficult and uncomfortable. To deal with those who have difficulty thinking aloud, our training tells us to continually remind participants to think aloud or ask “What are you thinking now?” However, too many prompts can cause further anxiety and disrupt a task flow. Even those who are naturals at thinking aloud have difficulty doing it while performing difficult, cognitively complex tasks.

Another problem is that thinking aloud affects task performance. Because this makes people think more deliberately about what they are doing with a product or service, it can change how people perform tasks, what they notice in a user interface, and how long it takes them to perform a task. For example, it’s common for participants to stop mid-task and discuss something at length with a facilitator.

The Facilitator

“To build rapport with participants and make them feel comfortable, facilitators must be friendly, reassuring, and empathetic.”

Facilitators play dual roles. To build rapport with participants and make them feel comfortable, facilitators must be friendly, reassuring, and empathetic. On the other hand, to avoid biasing participants, they have to maintain a neutral demeanor and be very careful about what they say. Balancing these two roles is very tricky.

The Facilitator’s Neutral Demeanor

In maintaining the required neutral demeanor, facilitators can sometimes seem distant and cold. It’s very difficult to be reassuring and put participants at ease when you cannot even tell them whether they have performed tasks correctly. Facilitators ask participants many questions, but cannot answer very many of of a participants’ questions. It’s no wonder they often feel as though we’re testing them, despite our reassurances to the contrary.

Being Observed

“People change their behavior when they are aware that they are being studied.”

A well-known principle in social psychology is that people change their behavior when they are aware that they are being studied. This unavoidable bias is known as the Hawthorne Effect, from the early 20th century studies of Elton Mayo. [1] Observation affects participants’ performance, and they may change their behavior to fit what they think we expect. We can only do our best to take these biases into account and minimize their effects whenever possible.

The Presence of the Facilitator

Having a facilitator looking over your shoulder, noting your every action, is a constant reminder that you are under observation. Plus, our notetaking can inadvertently give a perceptive participant a sense of what we value. Furious notetaking indicates a participant has just said or done something that is very important to us. No notetaking means a participant has not said or done anything of value to us. This feedback can subtly affect participant behavior, especially for those who want to please a facilitator by providing useful comments.

Because the presence of a facilitator can be disruptive, participants in some usability tests complete their tasks alone in a lab, with the facilitator observing from another room. This removes the discomfort of a facilitator’s observing them directly, as well as unintended feedback from notetaking. But it can be even more impersonal and intimidating when a facilitator communicates with a participant from another room through a speaker—seemingly as the voice of God.

The Observers

“For most participants, being observed is the most anxiety-producing and unnatural aspect of user research. While observers in the same room can be disruptive, the idea of unseen observers can produce even more anxiety.”

For most participants, being observed is the most anxiety-producing and unnatural aspect of user research. While observers in the same room can be disruptive, the idea of unseen observers can produce even more anxiety. The two-way mirrors that are typical in traditional usability labs and focus-group facilities immediately create an intimidating environment and are a constant reminder of the presence of unseen observers. These mirrors can create self-consciousness in people who don’t like to think about how they look and are a distraction for those who enjoy looking at themselves in mirrors.

Who the observers are can also affect participant behavior and comments. Some people are reluctant to provide negative comments when they know the people who created a product or service are observing them. Employees may wonder whether their peers or managers are observing them and may fear that you’re evaluating their job performance and competence.

Recording Sessions

The presence of audio, video, eyetracking, and other types of recording technology can add another layer of anxiety to user research sessions. Unless recording devices are very unobtrusive, their presence is a constant reminder to participants that everything they’re saying and doing is being recorded.

What Will Happen to the Recordings?

“Despite our reassurances about security and confidentiality, participants often wonder who exactly might see and hear the recordings.”

Despite our reassurances about security and confidentiality, participants often wonder who exactly might see and hear the recordings. Twenty years ago, when recordings were on video and audio tape, there was a physical limitation on who could view them. These days, with digital video and the Internet, there are no longer any physical limitations to where a video could turn up or who could see it. Might it end up on YouTube? Is a researcher streaming it live to observers around the world? Sure, the recordings may be secure now, but where could they end up in the future?

Could User Research Be More Natural?

With all of these issues, are there ways to make user research seem more natural? Of course, there are. In Part II, I’ll discuss some ways to minimize or eliminate many of the unnatural aspects of user research, including the following:

  • Be a participant yourself.
  • Recruit real people.
  • Minimize the formality.
  • Set the right expectations.
  • Use think aloud when appropriate.
  • Use realistic tasks.
  • Properly explain prototypes.
  • Minimize the effects of observers.
  • Minimize the effects of technology.
  • Conduct more field studies.
  • Use remote research techniques.
  • Do natural observation.

Note

[1] Shuttleworth, Martyn. “Hawthorne Effect.” Experiment Resources, 2009. Retrieved March 22, 2011.

2 Comments

Excellent article. I could relate to everything from my own experience as a Usability Analyst sometime back! Another unnatural factor is the “non-leading” nature of cues, prompts, and questions that the facilitator is expected to adopt…. “How easy or difficult was that task?” “How useful or not useful did you find that feature?” and so on. I doubt anyone talks like that in normal conditions.

Looking forward to Part 2 of the article.

Thank you for your wonderful points. They have got me thinking: How can I make things more natural the next time I spend time observing?

I have also experienced thinking aloud and doing being very difficult for some people at the moment they have to switch from doing to articulation. I have seen it result in disorientation, with people actually forgetting where they were and even what they were doing, and that’s a shame for everyone. Sometimes I’d rather record activity and recall afterward.

In my opinion, the recruitment and payment of helpers creates an obligation on behalf of the participant. The payment aspect is a transaction that most people need to know they are honouring.

It is my experience that people generally want to please and give you the mystery answers or information they believe you are looking for.

The hidden agenda and feelings of obligation can result in an insecurity that has participants saying: “Is that what you’re looking for?” during a session.

Feelings of potential failure in observation situations are high for some people, even if we try to negate those feelings by saying, “There are no rights or wrongs” at the start of a session.

I have observed people who appear to constantly look for clues as to how well they are performing during a session. People can interpret a consultant’s behaviour as feedback and re-examine or backtrack on their actions.

I have experienced such extremes when a consultant sniffs or makes a movement to get more comfortable in a chair and a participant interprets that action as feedback to their last action, a clue to what’s right or wrong. (I wonder if people possibly look for micro-clues as to how well they are doing, because of the unnaturally neutral disposition most consultants maintain in sessions. Does neutrality raise suspicion in participants?)

About observers: Be they directly with the participant or behind glass, I feel they are very intimidating. I can’t remember the last time my behaviour didn’t change in that instance.

As a consultant, I also remember my fear of testing for the first time with an audience I could not see behind the glass!

The more natural we can make these observation opportunities, the better for everyone.

I’m looking forward to the next installment.

Join the Discussion

Asterisks (*) indicate required information.