Data-Informed Design: Three Data Stories

Why Bother with Data?

For any Web site or application, it’s typical to do analytics, A/B tests, social-media sentiment, surveys, intercepts, benchmarks, and scores of usability tests, ethnographic studies, and user interviews. We end up with a lot of data. There is much debate over which data is most meaningful and hand-wringing over how our organizations can connect all the dots. But at its core, data about Web sites and applications is data about the people who use our products.

So, for me, the traces that the people who use technology leave behind—no matter their source—are data that we can use to inform UX design. Just as we wouldn’t look at an archaeological dig and expect to create a complete picture of life in Ancient Rome, we can’t look at any dataset and create a complete picture of the user experience. Data represents an approximation of the user experience, not a matter-of-fact truth.

A big part of using data to inform UX design is figuring out what we care about most. When UX designers use data, I’ve found that there are really three main stories:

• Proving a point
• Improving an experience
• Discovering something new or, at least, something that’s not obvious

Proving

For better or worse, UX professionals often use data to prove which choice is right or wrong—whether to solve battles with stakeholders or counter a gut-feel approach. For example, proving return on investment (ROI), or which variant of a design would be more profitable, is a valid and common use of data. Proving a point, at its best, is really about determining the right path to success.

Let’s say your team wants to show that a design change is responsible for an increase in signups. You could simply look at how many people visited a site in relation to how many people signed up since your team redesigned the site and make that claim. Of course, there are reasons for people to be a little suspicious of data and how it connects to design decisions. First, there are many variables that factor into signups. Second, this metric is too high level to tie back to specific design improvements.

Perhaps, instead, if there were specific data that mapped to a particular design change, that would make a better story. For example, designing an A/B test to see which of two sign-up pages produces the better result is another way to use data to prove a point. A/B tests are great at showing which design decisions have the most impact. On the flip side, A/B tests are very granular and, at a certain point, you have to step back to see the big picture.

In the end, proving a point with data is not clear cut. You can use one dataset to prove that cats are more popular than dogs on the Internet and another to prove the opposite. Typically, the data that UX designers use to prove a point is data that comes with ready-made metrics such as those from analytics tools. This is not always the most relevant or meaningful data.

The bigger the idea, the harder it is to tie it back to one data point—or even one dataset, or collection of data. But if you don’t understand how to use data to prove a point or you’re not comfortable working with multiple data sources relating to a user experience, you’re at a disadvantage.

Improving

Another way you can use data is to show improvements in a user experience—typically, by quantifying a data point that you want to improve and tracking it. There are plenty of examples that have nothing to do with online user experiences. For instance, the Chief Analytics Officer of New York City—yes, there is one—has used data science to reduce ambulance response times there.

Now, let’s look at our signup example again. If you wanted to see how much you’ve reduced the amount of time it takes to sign up, you could look at analytics to determine that. Yet you would still have a big question: why? So you would also need to conduct usability tests to determine why that had changed and what you could improve further.

You don’t have to start with analytics data to improve a user experience though. Let’s say that, instead, you wanted to understand how people perceive the ease of use of the signup process. You would need to develop a questionnaire to learn about that—preferably using something like the System Usability Scale to see how much it has improved over time or how that perception matches up across all touchpoints.

The key to using data to improve a user experience is a lot of consistency in how you measure things, plus a little experimentation. Experimenting can help Web sites like Etsy to make incremental improvements that add up to a better user experience. Measuring in the same way—over time or across sites—can help you to see changes and, hopefully, improvements in a user experience. So, whether we want to see how dogs show steady improvement in their quest to beat out cats as rulers of the Internet or to understand how to improve key aspects of a Web site’s design, data from a variety of sources can help us to make the user experience better.

Discovering

Discovering is the most interesting story, but it is also the most complicated data story. For example, whether we look at Target’s ability to predict that a woman is pregnant or Facebook’s questionable social experiments on unwitting users, there are certainly questions about the user experience that can be answered by looking at relationships between data points or among datasets. That is what discovery is all about.

Going back to our signup example one more time, maybe we want to see whether there is a relationship between the actions users took before signing up on a site. This could be an analytics question about where they came from or even how much they read, depending on what analytics tool you are using. Social-media data analysis could also answer this question by looking at conversations around signing up on the site.

In the field of user experience, we are most familiar with using qualitative research studies to discover new insights. We could answer this same question about signups through an ethnographic study examining the journey to sign up and the context around it. Qualitative studies are data, too. Smaller datasets and convenience samples—whatever participants happened to be available to a researcher—are typical of behavioral economics research and have led to major breakthroughs in that field, but that data is doubly useful once it’s coded and tallied.

Algorithms can answer some questions, but generally, data does not speak for itself. For example, to understand the subtleties of the Internet cat and dog conflict or discover what behaviors precede a sign-up process, you need to be constantly asking questions—the most important one being: So what?

Defining the Thing

There is a lot of buzz about data-driven or data-informed design, but very little agreement about what that really means. The data that we use to inform design goes beyond analytics and A/B tests, spanning behavioral tallies, textual analytics, and data from studies of all kinds.

One thing we do know is that the more closely connected data is to the experiences of people and how they perceive them, the more meaningful and actionable that data is. The challenge is asking the right questions of the data to deliver insights that are relevant to the user experience.

What’s Next?

In future installments of my column, I’ll cover topics such as the following:

• What counts as data for user experience?
• Thinking with data
• Aligning business, marketing, and UX data goals
• Metrics, KPIs, and scores
• Qualitative versus quantitative data
• Where to start with analytics
• A/B testing for UX professionals
• Social-media data and user experience
• Measuring engagement
• All about conversion metrics
• Usability scores and measures

But this is by no means an exhaustive list, so please comment if you have other topics or questions that you’d like to explore.