Talking Out Loud Is Not the Same as Thinking Aloud

Operant Conditioning

During a usability test, I rely on two principles from B.F. Skinner’s operant conditioning: reinforcement and extinction. Most of us are pretty familiar with reinforcement, but extinction might be a new concept—or at least one you haven’t thought of including in your participant management toolkit.

Reinforcement

Reinforcement is the use of praise or a reward to encourage someone to continue a certain behavior. For example, when a facilitator says, “You’re doing a great job,” a user is likely to try to do more of what the facilitator just praised.

Reinforce participants’ behavior only when they give you data, and praise them for exactly that: giving you data. Do not reinforce their giving you design suggestions.

For example, if a user tells me, “This instruction here is really confusing,” I first try to clarify where the confusion lies—that is, is it a word he doesn’t understand, some ambiguity, and so forth. Then, I usually say, “Thanks, that’s useful to know—that’s just the kind of data we are looking for.”

I do the same if a user wants a product to do something it doesn’t. “So you expected the application to automatically correct the word manger, making it manager, because manger wouldn’t make sense here. It’s helpful to know what you would like it to do.” Notice that I did not say, “That’s a good suggestion.” Why not? For one thing, I’ve got a room full of developers watching this test who know it’s an unreasonable expectation for a spelling checker—or at least not for one that wouldn’t put the product back into a lengthy development phase. If I said, “That’s a good suggestion,’ I’d start to lose credibility with them. Also, they might start discounting all of the input that I get from this user as a way of discrediting that one suggestion.

And most important, if I were to praise a participant for making a design suggestion, what behavior would I be reinforcing? Making design suggestions! But I didn’t bring this person in to make design suggestions. I wanted to see how real people make sense of a user interface within the context of doing authentic tasks.

Extinction

The next technique I’ll discuss is extinction. How do I deal with unwanted feedback such as style preferences and design suggestions? I do nothing at all. That is what extinction is: the absence of feedback that would reinforce an undesired behavior. A common application of the technique of extinction is to take a child who is acting out and give him quiet time. The intent is not to punish the child, but to get the child out of the environment that is reinforcing the unwanted behavior—typically, the extra attention the child is getting from his teacher or his peers.

While reinforced behavior typically continues, behavior that does not get reinforced eventually goes away, or becomes extinct. So a typical exchange would go something like this:

Participant: I didn’t see the Submit button.

Me: So you didn’t know what to do when you finished the form because you didn’t see the Submit button. Thanks, that’s useful information.

Participant: Yeah, I think you should make it red.

Me: In this next task, we are going to ask you to…

My first response was positive reinforcement for the insight that the button had not captured the user’s attention.

However, my second response moved right along without acknowledging the design suggestion. Thanks to the first response, we know we may have a problem with users’ not noticing the Submit button. I’ll let our visual designer who has the degree from the well-respected design institution run with that.

I know there are some people who might want participants’ design ideas. But once you start down that path, participants stop sharing their thoughts, which would give you insights into a problem space, and instead, start giving you their ideas for design solutions—because they begin to believe that’s why you brought them in. Why? Because you keep reinforcing that belief. When, in my earlier example, that participant said, “12 windows,” I knew his answer, but had no insight into how he got there. Again, I usually have no shortage of experts in a solution’s domain space; what I lack is a clear vision of how the problem looks from a user’s perspective.

Conclusion

Getting participants to think aloud turns their tacit behaviors—such as interpretation, planning, and reacting—into explicit behaviors that you can observe and analyze. For this reason, thinking aloud is a valuable tool in doing usability testing. But if participants start to believe that you are interested in their design suggestions, they’ll stop sharing those internal behaviors you want to know about to get insights. Instead, they’ll start to give you insights into how they would have designed something—not into how they are using it. They’ll become like the participant who merely said “12” during the count-the-windows exercise.

Clients who are eager to jump on any input they get from real users sometimes place too much emphasis on such design suggestions. It is the qualitative equivalent of a similar quantitative problem I have with clients’ placing too much importance on descriptive statistics from small samples. “Oooooo, the average time to install went down 12% between the first version and this version—we’ve made it easier to install.” Maybe; maybe not. This makes me just want to wave my hand and say, “These aren’t the droids you’re looking for.” So the more I can do as a moderator to keep participants focused on sharing how they are counting the windows, so to speak, and less on what number they come up with, the easier it is for me to keep my clients focused on obtaining truly valuable data.

So, by all means, get your participants to think aloud, but encourage comments that illuminate the problem space—because that’s what usability testing is all about.