How to Ask About User Satisfaction in a Survey

By Caroline Jarrett

Published: November 1, 2012

“The state of satisfaction may include a variety of emotions and … their intensity may vary according to how much you care….”

Recently, Janet Six devoted the October edition of her Ask UXmatters column to customer feedback surveys. That column has inspired me to have a go at one particular aspect of customer feedback in more detail: asking about user satisfaction. That and the excerpt from an email message, shown in Figure 1, which I received after an encounter with a customer support facility, complete with its odd repetition of the question.

Figure 1—An unsatisfying excerpt from a customer satisfaction survey

An unsatisfying excerpt from a customer satisfaction survey

The message in Figure 1 equated good support with satisfied; bad support with unsatisfied. What if a polite and charming support person were forced to explain a deeply annoying business rule to me? I’d be a mixture of conflicting emotions and would probably want to answer what: perhaps good, but unsatisfied?

Satisfaction Is an Emotional Response

Richard L. Oliver delves into human emotions—and every other aspect of satisfaction—in his classic text Satisfaction: A Behavioral Perspective on the Consumer. He points out that the state of satisfaction may include a variety of emotions and that their intensity may vary according to how much you care—or putting that into psychological language, on a person’s level of engagement (arousal) or disengagement (quietude). This can make it tricky to understand what satisfaction means:

“It appears that emotional extremes including delight, excitement, and distress are naturally occurring emotions in satisfaction…. If a consumer responds that he or she is satisfied, does it mean that this consumer is in a state of contentment, or of pleasure, or of delight?”—Richard L. Oliver

Or putting this more simply: if you’re really interested in something, being satisfied with it might fall somewhere between being pleased and delighted; but if you’re not, being satisfied probably puts you somewhere between being pleased and relaxed.

Satisfaction Is All About Comparisons

“To be satisfied, you’ve got to do some mental processing of an experience—to compare it with some sort of mental idea or standard of what the experience ought to have been like.”

When making a comparison, it’s not just how much you care; it’s what you’re comparing something with. To be satisfied, you’ve got to do some mental processing of an experience—to compare it with some sort of mental idea or standard of what the experience ought to have been like.

For example, bronze medal winners tend to be happier than silver medal winners, as depicted in Figure 2 and as described by Victoria Husted Medvec, Scott F. Madey, and Thomas V. H. Gilovich in their 1995 study, “When Less Is More: Counterfactual Thinking and Satisfaction Among Olympic Medalists.”

Figure 2—It’s all relative

It’s all relative

A 2006 replication of the study explained it like this:

“Winning any medal in Olympic competition is an amazing accomplishment. Folk beliefs suggest a linear decrease in positive emotions, with the gold medalists experiencing the most, followed by the silver medalists, and then bronze. However, in reality, [in Judo] the silver medalist loses the gold medal match, and the bronze medalists win their last match, capturing the bronze and avoiding going home without a medal…. Bronze medalists from the 1992 Barcelona Games appeared happier than silver medalists at the end of the match and on the podium, and … silver medalists’ comments were much more characterized by counterfactual thinking about ‘what might have been’ and were associated with greater regret and less enjoyable emotion.”—From
D. Matsumoto and B. Willingham’s “The Thrill of Victory and the Agony of Defeat: Spontaneous Expressions of Medal Winners of the 35 2004 Athens Olympic Games,” from the Journal of Personality and Social Psychology, September 2006.

Richard L. Oliver delves into the varieties of factual and cognitive comparisons that can contribute to satisfaction or dissatisfaction, which I have summarized drastically in Table 1. He points out that, if you don’t process the experience at all, the result is indifference—not dissatisfaction.

Table 1—Comparisons that contribute to satisfaction

An experience compared to… A person’s resulting thoughts…

Nothing

Indifference

Expectations

Better / worse / different

Needs

Met / not met / a mixture of the two

Excellence / the ideal product Good / poor quality / good enough
Fairness Privileged / equal / picked on / disadvantaged
Events that might have been Vindication / regret

Memorable Experiences Are Also Complex

Memorable experiences tend to be complex ones.

If the emotions in satisfaction are complicated, what about the experience that gives rise to them?

Let me tell you a story about a rather simple experience. Today, I’ve used a whole bunch of Web sites: I bought some stuff. I browsed some stuff. I checked the news. But the experience that stands out as memorable: signing up for a Web site to which I’m going to log my routes and training for a half-marathon.

Was the experience satisfying? Well, yes and no. Looking at it through the lens of the different comparisons that affect satisfaction from Table 1, where did I start?

  • I expected it to be straightforward. Registering for a Web site should be simple these days.
  • I needed it to be reassuring. This half-marathon is double the distance I’ve ever attempted.
  • Ideally, I wanted it to be inspiring—to suggest ways of conquering this training challenge in ways that I hadn’t yet thought of.
  • I feared that it would be too US-centric and wouldn’t understand that I need to be treated a bit differently because I’m a UK-based person.
  • If I hadn’t signed up for this site, I’d have had to go hunting for another one. Plus, over a few months, I’d already invested effort getting familiar with this site as an unregistered guest, so looking elsewhere seemed like a lot of work.

And where did I finish? With mixed results. It was mostly easy, but part-way through, there was a bit where I ended up paying a very modest fee to remove an ad. I had thought I was paying to use the site ad free, but then there appeared to be a fee to remove a particular ad rather than remove all advertising. Grumpily, I thought about spending time to find a different option. Later, I realized that I needed to sign in, so the site would know that I’d paid. Just a few bumps along the road, but they tangled me up and undermined the initial experience. Looking back, I had

  • multiple goals—signing up, intentions about long-term use, meeting a personal challenge
  • multiple levels of effectiveness—completing the signup, paying the fee, my initial use of the service, my long-term use
  • a variety of types of resources—the actual payment, my time, my emotional reaction to the mix-up
  • multiple contexts of use—while it was just me at my desk, there are so many different contexts of who I am as a person: beginning runner, UK person, UX consultant, supporter of a particular charity

So, on a scale of 1 to 10, how satisfied am I? You answer for me! Your guess will be at least as accurate as mine. The point here: memorable experiences tend to be complex ones.

Figure 3—The nature of experiences

The nature of experiences

The Opportunity for Satisfaction in a Survey: Focus

“The key is to pick one small aspect of satisfaction, and ask about that.”

With all this bad news about complexities, is there any point trying to ask about satisfaction in a survey? My view: yes, definitely. The key is to pick one small aspect of satisfaction, and ask about that. Briefly. And provide space for a free-form comment.

Find Out About Users’ Contexts, a Bit at a Time

When I was first collecting stories about how UX experts use surveys in their work, Suzanne Boyd of Anthro-Tech told me about asking one simple question: “Why did you come to this Web site today?”

They’ve achieved great results from asking this question, and that makes a lot of sense to me. It’s an easy question, and it explores one of the most important aspects of user experience: the user’s goals.

Of course, it’s tempting to go on to ask about all the other aspects of a visitor’s experience at the same time. Many of the worst surveys that I see do that—apparently taking the view that, if they’ve managed to grab my attention “to answer this short survey,” I’ll be prepared to answer screen after screen of questions about all sorts of details.

Don’t fall into that trap. Instead, keep your respondents happy by asking just a bit at a time—and not every single time they visit your site either. You’ll gradually build up the details that you need.

Ask About Their Recent, Vivid Experiences

“People are much better at reporting how they feel right now and what they’re doing right now than on their experiences from days or weeks ago.”

People are much better at reporting how they feel right now and what they’re doing right now than on their experiences from days or weeks ago. Try to answer these two questions:

  • “Did you print anything in the last hour?”
  • “How many pages did you print last month?”

Most of us find it much easier to answer the first question and are much more likely to give an accurate answer. But an easier question than either of these might be:

Did you buy a printer this week?

For most of us, printing is a rather frequent and somewhat trivial task, whereas buying a printer is a rather unusual, more important, and therefore, more vivid task. The more trivial the experience, the less memorable—so the less accuracy you’ll get in people’s answers about it. Try to ask about recent, vivid experiences.

Ask a Sample, Not Everyone

If you ask every customer to provide feedback, every time they visit your site, that’s a feedback form, not a survey. You know that, your respondents know that, and this contributes to declining response rates and poorer quality data—for both your feedback forms and your surveys.

Ask fewer people, and make it clear that you’re deliberately asking only a specific few. They’ll feel more special and be more likely to respond.

For Any Survey: Interview First

“A successful survey should always include an interview stage, during which you explore the topics of the survey in detailed interviews with typical respondents. But the interview stage should always happen early in the survey process.”

Often, stakeholders consider that it’s quicker to put together a quick survey than to go out to find users and meet them face to face. Or even worse, they might say, “Let’s do a survey, then do some follow-up interviews.”

Survey methodologists do agree that a successful survey should always include an interview stage, during which you explore the topics of the survey in detailed interviews with typical respondents. But the interview stage should always happen early in the survey process. Interviewing first saves so much time because:

  • It helps you avoid problems like asking questions that users don’t understand or don’t have answers for.
  • It gives you preliminary data from the interviews that may be enough to move your project forward without the expense of an actual survey

Summary

Satisfaction is a complex emotion, and memorable experiences are also complex.

Don’t try the patience of your survey respondents with long questionnaires that probe every aspect of those complexities. Instead, you should:

  • Find out about users’ contexts—a bit at a time.
  • Ask about their recent, vivid experiences.
  • Ask a sample, not everyone.
  • And as with any survey: interview first.

Acknowledgement—Thanks to Francis Rowland for the sketches.

Resource

On SlideShare: “How to Ask About Satisfaction on a Survey

2 Comments

What are your thoughts on NPS (Net Promoter Score)?

Jason:

Sorry I missed your comment somehow.

Net Promoter Score can be a useful and valuable tool when it’s used appropriately—that is, in the way for which it was designed. To find out more about that, it is definitely worth reading either the original book on NPS:

Reichheld, Fred. The Ultimate Question: Driving Good Profits and True Growth, 2006. (It’s widely available, including second hand.)

Or the updated version:

Reichheld, Fred, and Rob Markey. The Ultimate Question 2.0: How Net Promoter Companies Thrive in a Customer-Driven World, 2011. (It’s also widely available.)

The problems arise when organizations try to use this question in ways it was never meant to be used.

Best, Caroline

Join the Discussion

Asterisks (*) indicate required information.