Can Good Design Be Measured?

Five Do’s and One Don’t of Measuring

In summary:

• Do think holistically.
• Do strive for balance.
• Do seek meaning.
• Do use multiple data sources.
• Do consider using a framework.
• Don’t make it all or nothing.

Do Think Holistically

Notice that I’ve reframed the original question to be about experience rather than design. That’s not an accident. People experience technologies holistically. If you ask people to describe their favorite site or app—which I have by the hundreds—they’ll remember very little. If you ask people to draw a site that they use daily, they’ll usually draw the logo or, if they think really hard, a few elements that they use all the time. But people remember the feeling they have when using their favorite technologies. They remember the people who populate that space. They remember what they do regularly. But they don’t experience design as a green Sign Up button, a sidebar on the right with links, or a feed of images. Astonishingly, they do remember many sites as “a lot of blue,” but I think that is beside the point.

Because people experience technologies holistically, we must measure them holistically—at least as a start. Sometimes when you are searching, Google will ask, “How satisfied are you with the search results?” As far as it goes, I think this is actually the right idea, but I would go even further. Having tracked almost 20 measures across 250 sites for a couple of years now, I’ve learned that happiness correlates strongly with likelihood to recommend, to returning, and to purchasing. Positive emotions prompt positive actions.

As a way to understand experiences holistically, I recommend asking people how they feel about the experience when they leave the site, then using this data as a barometer for the experience. You can just use faces, as in a physician’s pain chart, but with a range of positive and negative feelings. This makes it easy for people to decide rather than thinking, “Am I at a 4 or a 5 on a happiness scale?” Plus, this works for a variety of literacy levels—and generally, across cultures. You can use happiness as a kind of proxy measure to understand the experience as a whole.

I know what you are thinking: Ask people? We can’t ask people, we must observe. That is our way. But emotions are not so easy to observe and categorize, whether by a trained affective psychology researcher or using biometrics—at least not yet. Most emotional research relies on people reporting their emotions and not getting too granular about it—because that is tricky, too, and requires some sensitization exercises. For now, asking people a single question about how they feel after an experience is a simple way of getting an initial read on the experience.

Do Strive for Balance

The holistic measure I’ve just described is subjective. That’s fine and, as is the case with subjective measures, the more the merrier. It’s subjective, so there is going to be variation. More responses will give you greater confidence in the results. This is true of asking how people feel, how they rate their satisfaction with a customer-service call, or any question that asks people to reflect on an experience.

You also want to get objective measures, which can often get at the in-the-moment experience. These are all of the behavioral traces like clicks and scrolling, from your analytics package, eyetracking data, or GPS data from an app. The things that you’re measuring as people experience a site, app, or other technology—when they’re just doing whatever they would normally do. Pull all of these together in an event stream, and you can get at activities like reading, looking attentively, or maybe thinking. These are the types of measures that machines are really great at tracking—although you can also gather in situ observational data through ethnographic study.

I use Daniel Kahneman’s model of experiencing self and reflecting self. We need to understand and balance both aspects of the experience to come up with meaningful measures.

Do Seek Meaning

If you want a measure that will actually move your team toward creating a better experience, you may have to get creative with measurement. For example, if you wanted to measure audience engagement at a concert, you probably wouldn’t look at ticket sales. Instead, you might look at the number of standing ovations or surreptitious mobile-phone use. In an online seminar, you might look at when people raise their hands. The same holds true for technology of any kind. If you want to measure the true impact of an experience, you probably won’t look just at sales or signups.

Just because a measure is popular, that doesn’t necessarily make it a good choice. Net Promoter Score (NPS) is widely used, but is a useful measure only if recommendations play a dominant role in a purchase decision or if you’re measuring something that people feel comfortable recommending at all. So NPS may be more relevant for a brand of pet food than for an investment product.

When someone in your organization says they want to measure the success of the user experience or customer satisfaction, the best way to get started with measuring is to ask questions. Start by asking, “What do you mean by X?” and “Why does it matter?” That will point the conversation toward meaningful measures.

Do Use Multiple Data Sources

Measuring experiences is not a matter of just taking a metric that’s built into your analytics package—no matter how sophisticated—and using that as the only measure. Likewise, it’s not about just looking at the data that we happen to have on hand or that is easy to collect and extracting a metric. For instance, A/B tests are a great way to inform design decisions by testing alternatives that come out of usability testing. However, A/B testing data is not a great way to measure experience or good design, because A/B testing is too granular.

We often tend to favor one data source over another because it’s easy, well accepted in our organization or the UX community, or we just think it’s the best. That’s okay, we all do it. Nevertheless, diversifying your data will create a richer understanding of the experience. Here are a few ways to accomplish that goal.

First, vary the sources. Think about a performance review. A better performance review would include several points of view: colleagues, supervisors, and customers. The same holds true for Web sites, apps, and devices. For example, using analytics in combination with interviews lets you get at how people might integrate a smartwatch into their daily routine, with measures combining messages that a user answered or dismissed with the time saved—actual, perceived, or both.

Next, vary the timeframes. This means creating measures that reflect both short-term and long-term goals. Business metrics might be customers’ first time-purchase totals and return-purchase totals. When measuring an experience, this could translate to either the time to complete a task—if it’s a task that is best completed quickly such as paying a bill—or the level of engagement—perhaps, how many things people try out when they return. I use metrics for problems solved and possibilities created. These are simple categories that reflect different timeframes.

Diversifying measures also makes it harder to game the system. Not that anyone here would do that deliberately, of course.

Do Consider Using a Framework

Following these do’s, so far, we have one measure that we can use as a barometer, plus several other measures that are diverse enough to be sensitive to nuance and compelling enough to tell you something new about the experience. That’s great, but you may need to go bigger depending on your organization.

Bigger organizations typically have little pockets of measures that are siloed by line of business or department. The user experience and the organization can benefit by bringing these measures together. A framework is a good way of doing just that. Many organizations are using Google’s HEART framework to combine business and experience metrics. Avinash Kaushik’s See-Think-Do framework brings together marketing, business, and content. Lately, in my consulting work, I’ve been using metrics relating to five principles of positive design: ease of use, trust, creativity, connection, and meaning.

The core idea of a framework is to tie together measures across the organization, while remaining flexible enough to accommodate new metrics as goals change and new ideas take hold.

Don’t Make It All or Nothing

Measuring an experience is not about reducing your data to a single value or an exact value or achieving absolute certainty. If that were true, very few things would be measurable. And data certainly wouldn’t be very interesting or meaningful.

Measuring an experience does not require reams of data or perfect data. Nor does it require an awesome, new tool. The things you care about leave tracks, and you may not need as much data as you think to learn something new.

Yes, you can measure good experiences in a meaningful way. And, if you are doing it right, you will end up with both new insights and even more questions.