Over the past decade, more and more organizations have been doing their best to become data driven. This is a huge and much-awaited mindset leap—especially given corporate dinosaurs’ typical way of thinking: “I’m the HiPPO (Highest Paid Person in the Office) and my gut tells me this is what the customer needs.”
User research is one of the most powerful antidotes to this outdated mindset and is becoming part of the vocabulary and practice of an increasing number of organizations. It also signifies an increase in UX maturity, so is a very promising trend!
The Problem with Data Being in the Driver’s Seat
Making decisions based on data reduces risk and increases your potential for creating useful outcomes. However, if you make decisions based on bad data, you risk making a similar—or perhaps an even worse—kind of gamble or bet as when making decisions based on untested assumptions.
Champion Advertisement
Continue Reading…
The way you gather and treat your data can make or break the thing you create, whether it’s a product, service, or any other kind of system that people design—such as an organization or a healthcare system. If your organization is data driven, but your data is bad, you’ll just make poor decisions with greater confidence!
According to the Harvard Business Review, the cost of bad data is $3 trillion per year in the U.S. alone.
Let’s consider an example: What if I told you that there is data that shows—with statistical significance (p≤0,05)—that eating ice cream every day is better for your health than eating vegetables? That this data is from a study with more than 20,000 participants, across three continents?
Would you make any decisions based on this data? If you did, would they be data-driven decisions? Who would be to blame when all hell broke loose?
In the aforementioned study, researchers had asked: “What would you like to eat more of so you would be healthier? A) Vegetables or B) Ice cream?” They ostensibly conducted this experiment with over 20,000 kindergarten children. Of course, this is actually a fictional study.
I’ve obviously exaggerated to make a point. But, unfortunately, there are many real examples of bad research design and execution. Problems include asking leading or biased questions during surveys, interviews, or usability tests or cherry-picking data. Introducing bias in user research can generate lots of bad data. In fact, sloppy user research could exponentially increase the chances of gathering bad data. So, as UX researchers, we need to be mindful, vigilant, and rigorous about our research. Unfortunately, it’s not that hard to make mistakes that diminish the quality of UX research. The devil is in the details. Tomer Sharon has shared some potent advice on how to ask questions when interviewing users and stakeholders by avoiding asking leading questions, biasing, or priming interviewees.
What are the chances of your creating something useful on the basis of bad data? At best, you might hit the bullseye, but on the wrong target. You could create something great that no one needs or wants, and you’d be burning a lot of money in the process.
“Crap in, crap out.”—Darren Hood
Not All Research Data Is Created Equal
The way you plan and conduct your UX research and gather quantitative or qualitative data plays a huge role in the quality of that data and determines how much you can rely on your data without taking huge risks.
You should focus on two things: First, what are your learning objectives and what data do you need to achieve them? Second, how can you gather the necessary data in a way that ensures it’s valid and reliable for use in decision making?
What You Need to Learn
What do you want to learn? What data do you need? What kind of data do you need to be confident in making your decisions? Define the key questions you need to answer. Skip all the vanity metrics that tell you almost nothing and, instead, focus on things that matter.
How to Gather Data
Once you’ve defined what you need to learn, you can plan your research. This is where both internal and external validity become important factors. Doing research in a scientific way can help you to avoid most research pitfalls and ensure that you gather valid data.
Data is one of your organization’s most valuable assets. Treat your data with respect!
Conduct testing to learn, not to prove you are right. Avoid jumping to conclusions—for example, if you get some encouraging results from a single data source. While this would be a good indication, you should not be hasty in drawing conclusions. Don’t forget to triangulate your data. Triangulation helps you reduce risk and achieve your desired confidence level. Calculate a confidence interval that shows how confident you can be that a certain result would occur.
If you want to know more about how you can use multiple data sources to improve your decision making, you should definitely take a look at Tricia Wang’s presentation about how relying on just big data and ignoring qualitative insights killed one of the most successful companies—Nokia. If you focus only on collecting quantitative data and don’t speak with people to gather qualitative data, you’ll get a skewed view of the world.
Never forget that correlation does not equal causation!
UX Research Is Not Novel
User Experience did not invent conducting research with people, about people. Social scientists paved the way with battle-tested methods that meet most, if not all of the needs of UX professionals. We can adapt their methods as necessary.
Be Scientific When Conducting UX Research
The scientific method, shown in Figure 1, is a powerful approach. You should use it for your UX research. This would enable you to overcome your own cognitive biases, improve the quality of your research, and collect clean, useful data. Dr. Nick Fine has given a spot-on talk about how to practice scientific design. Check it out!
Figure 1—Using the scientific method in User Experience
Treat Requirements as Hypotheses
Change your mindset and handle requirements as assumptions. Then turn them into hypotheses that you can test. To do so, you must determine your organization’s desired outcomes. John Pagonis, in his talk “Evidence-Based Product Backlogs,” explains how to use user research to prioritize and cleanup your backlogs. Laura Klein’s talk “Identify and Validate Your Riskiest Assumptions” describes how to identify and test your riskiest assumptions early on in your projects. Both are spot-on. I definitely recommend your watching these videos
Test whether your hypotheses are valid. The sooner you can gather enough evidence, the better. If your hypotheses prove to be valid, great! You can start building your product. If your early testing indicates that you should reject a hypothesis, you’ve just saved your organization a lot of money! Think about it: You didn’t have to throw away any code. Your team’s morale didn’t take a hit. You’ve spent no money on marketing or promotion. No one was screaming about negative business outcomes in your office—or your Zoom meetings.
The Importance of Documentation
When you document your UX research—from research planning to execution, including data collection and analysis—you and anyone else in your organization can check the validity of your findings. If you make this common practice, it becomes way easier to find and exclude bad data from your decision-making process. Plus, this makes it much harder for other people to come to the table with so-called evidence—data they’ve made up or that is inconclusive. In their article “How to Handle Bad Data,” James Lewis and Jeff Sauro describe how to identify and handle bad data, including information about how you might have ended up with bad data and what you can do to mitigate this problem.
In Conclusion
We need to help our companies become data-driven organizations, then keep up the momentum. But we also need to take a scientific approach and be vigilant about how we collect and treat our data. UX professionals must avoid making mistakes that would diminish our credibility and create barriers to the adoption of evidence-based, data-driven ways of working. We need to educate ourselves, our teammates, and our stakeholders about correct research practices and their proven advantages. As Sotiris Sotiropoulos has said, “You gotta fight for your right to research.”
Of course, I understand that not everyone can conduct rigorous UX research from the outset. It’s a learning process. But take one step at a time, sustain your momentum, and keep moving forward. You must adapt your UX research practices to your specific context, needs, and constraints. Do your best to provide useful research results to your team.
If you’re just getting started with UX research, educate yourself so you’ll understand what sloppy research is, be able to identify it, and avoid practicing it. You can’t beware of potential problems with research practices if you don’t know they exist. Integrate your new knowledge into your work. Start with your next project. Once you try conducting UX research, make the necessary adjustments. Design solutions based on the data that you’ve gathered. Do not get discouraged. Stay on track and keep moving forward.
Share your knowledge about UX research with your team and stakeholders. Creating a shared understanding should always be one of your goals.
Please share your thoughts on this topic in the comments. I’d love to start a conversation in the UX community and raise awareness of risks, opportunities, and best practices.
Dimitris is a product and systems-design engineer by training and a human-centered, interdisciplinary designer by choice. He has worked for startups, small-to-medium businesses (SMBs), and multinational enterprises, designing business-to-business (B2B) and business-to-consumer (B2C) platforms and applications. At ANIXE, Dimitris is working on the digital transformation of one of the biggest banks in the Middle East and Europe, where he is responsible for UX research and user-centered design. At Seque, Dimitris created a productivity tool for iOS. He takes a systems approach to UX and product design and is strong proponent of human-centered, evidence-based innovation. He holds an integrated Master’s in product and systems-design engineering from University of the Aegean. Read More
