Testing Content Concepts
Published: December 21, 2009
As UX professionals, we’re all familiar with the need to test user experience designs. Testing content, however, might be a different story. Most companies haven’t given testing content the attention it deserves—partly because it’s challenging. One challenge is that time and budget usually do not allow us to test every single piece of content. Another challenge is that gathering too much unfocused feedback can freeze our projects in analysis paralysis. To meet these challenges, try testing your content concepts—and start testing them early in your projects.
I have found surprisingly little advice about testing content that is integral to rather than supportive of the user experience. Also scarce is advice about testing content for more than usability. A good starting point for understanding the need to test content is a blog post by Ginny Redish, “Usability Testing: Be Sure to Test Content as Well as Navigation.” According to Redish:
“Too many usability tests focus only on finding information—not on how the information itself works for people.”—Ginny Redish
This column explains the value of testing content with real people and offers tips on evaluating content concepts.
The Value of Testing Content
Similar to testing UX designs, testing content at important points throughout a project lets you get feedback from people and ensure your content works. If your content isn’t working, you can adjust the content and test again until it does. By testing throughout your project, you save yourself the pain, frustration, and cost of reaching the end of a project only to find the content does not work for people.
For content, usability is whether someone can find, read, and understand the content. To accomplish our business goals, we often need content to be more than findable, readable, and understandable. Content might need to engage, influence, or support decisions.
Balancing What with Why
Perhaps the greatest value of testing content is that it lets you gather more qualitative data to complement your quantitative data such as analytics. It is difficult, if not impossible, to measure the effectiveness of content using quantitative data alone.  If you test content throughout a project, you assimilate a more complete understanding of whether it’s achieving your goals. While analytics can tell you plenty about what is happening with your content, qualitative testing helps you understand why it is happening. For example, analytics can tell you that a certain page has a high bounce rate. Interviews with people can tell you that the content doesn’t cover the topics they need or the content’s tone is lifeless or the images look unprofessional—and give you insights that can help you fix the problems.
Testing content with people throughout a project lets you triangulate qualitative feedback with expert review and quantitative data—from analytics, call metrics, or multivariate testing. Where you see similarities, patterns, or connections between all three, you’ll find the strongest insights. Triangulation is a widely recognized research principle in the social sciences, where strict, empirical tests that follow the scientific method—complete with a control group, variable group, and statistically significant results—are often not possible. Instead of trying to answer a question through one scientific experiment, triangulation answers the question through the combined results of several methods. Figure 1 shows how triangulation can apply to content.
Figure 1—Triangulating data about content
What Is a Content Concept?
A content concept is a mockup or draft of your content. I find testing at three different levels of fidelity useful, depending on what qualities of the content I want to test. The higher the fidelity, the more variables you can test. However, it can be harder to identify which variable is causing your results. (See “How Do I Test a Content Concept” later in this column.)
Table 1—Content concept fidelities
|Fidelity of Content Concept||Most Helpful for Testing These Qualities…|
Content independent of the design and, possibly, formatting
|Content in a low-fidelity design
Content in rough wireframes or prototypes, including basic formatting
|Content in a high-fidelity design
Content in polished wireframes or prototypes, including polished page formatting
What Content Should I Test?
You probably can’t test every single piece of content that might be a part of your project, so you’ll need to test a sample. I find the best approaches to sampling content are critical case sampling and common case sampling.
Critical Case Sampling
Select the instances of content that are most important to your project goals. For instance, if your project involves convincing people to sign up for something, you should test the content for the sign-up module or landing page.
Common Case Sampling
Choose the types of content that comprise the majority of the content for your Web site or interactive user experience. For example, if the goal of your Web site is to provide credible healthcare information to improve people’s healthcare decisions, test the content for a typical healthcare topic.
When Should I Test Content Concepts?
Test content concepts at the beginning of your project. Rachel Lovinger, Content Strategy Lead at Razorfish, offers a helpful visual showing the evolution of content strategy throughout the project process in Figure 2.  Concept is part of the early discovery phase.
Figure 2—Concept during the first stage of the project process
I find testing a concept works best once you have a clear understanding of your project goals and have gained enough insights from other data sources—such as analytics, call metrics, best practices, content analysis, market research, or competitive analysis—to form a reasonable concept.
How Do I Test Content Concepts?
The specific approach you should take to testing concepts might vary, depending on your project, but I find these basic steps always apply. If you have solid experience in testing designs, you simply need to make a few adjustments to focus your testing on content. However, if you have not conducted such testing before, I recommend that you read a book, attend a conference, or take a class about usability or similar types of testing.
- Identify your testing goals.
The overall purpose of testing a content concept is to figure out what content strategy you should follow throughout your project. Consequently, make sure your specific testing goals align with the project goals. For instance, if the goal of your project is to help people make better health decisions as they travel, make sure your test covers related content and explores whether it influences people’s decisions.
- Choose critical or common content cases to test.
When choosing the samples of content you’ll test, use the sampling method that makes the most sense for your project and testing goals. (See “What Content Should I Test?”)
- Create concepts that explore the right qualities.
Whether your team is crafting the content from scratch or adapting it from existing source content, you need to ensure the concepts of the sample content reflect the qualities you want to test. For example, if you want to test the tone and style of the content, ensure the content concept actually has a distinct tone and style.
For comparison, you could test more than one concept, with different tones and styles. I like using comparison to test less tangible qualities, because it elicits more and better-quality feedback from people.  Usually, however, testing two or more concepts does not result in a winner, so do not position a comparison test as a contest when communicating with your stakeholders, client, or boss. Rather, testing multiple concepts reveals the best and worst of each concept. Most likely, you’ll try to combine the best—while avoiding the worst—into your final content approach.
- Plan the right method to test the content qualities.
Usually, you’ll ask people to complete some tasks and participate in an interview. Table 2 offers a breakdown of what methods are best for testing what quality.
Table 2—Methods for testing content qualities
|Method||Content Quality to Test||Tips|
Observe and, optionally, ask participants to think aloud
Regarding other logistics, you can conduct test sessions in person or remotely. I prefer testing with eight or nine people to guarantee that some patterns emerge. For a good, detailed resource about testing logistics, see Handbook of Usability Testing, by Jeffrey Rubin and Dana Chisnell.
- Measure the appropriate results.
Evaluating the test results in the right way is important. Table 3 offers some basics. Handbook of Usability Testing offers helpful insights, as well.
Table 3—Basic measurements for testing content
A Special Challenge: Measuring Influence and Engagement
We need to figure out the right way to assess intangible qualities such as engagement and influence. The difficulty is that, when trying to describe such qualities, people cannot necessarily articulate why. What’s more, if we press people to articulate why something does or does not engage or influence them, they are likely to rationalize—and perhaps even talk themselves into a completely different response from their real reaction. 
Some have used measurements from the gaming industry to help assess engagement. My concern about using such measurements is that games are designed to be uniquely immersive, making someone feel lost or completely involved in an experience. Most content-focused experiences strive for engagement, not complete immersion.
Demetrius Madrigal and Bryan McClain’s UXmatters article, “Testing the User Experience: Consumer Emotions and Brand Success,” explores subjective and objective measures for emotion—a close cousin of influence and engagement. The objective measures involve highly specialized techniques such as face reading. In a similar vein, some marketers actually scan people’s brains to assess their emotional reactions and preferences for products or marketing materials.  While I find these approaches fascinating, they are currently impractical for most user experience projects.
So, what approach to assessing content’s influence and engagement is practical now? We should capture people’s first reactions to content. Then, through some expertly moderated discussions with participants, we can try to understand their reactions, ensuring we don’t press too hard and, thus, end up with misleading rationalizations. One way of avoiding such rationalizations is to discuss two different concepts in comparison. Finally, we need to understand whether content affects people’s decisions or perspectives. For example, an opening questionnaire could ask participants about what their decision might typically be in a particular case and why. A closing questionnaire could ask them whether they would now make a decision that differs from their typical decision and to rate how well the content informed their decision.
The UX community has amassed extensive knowledge about testing interactive designs. Applying that knowledge to content has exciting potential to help companies and organizations answer the question Does our content work for people? One valuable approach is to test content concepts early in a project lifecycle, enabling you to form a clear content direction.
Both a tough challenge and a big opportunity is testing how content influences or engages people. We need practical, yet reliable approaches to help us understand whether content has impact on people’s perceptions, decisions, and more. I am personally exploring ways of addressing this need, and I look forward to the rest of the UX community doing so, too. When I think about our accomplishments in the realm of testing, I’m confident our collective brainpower can meet the challenge.
 Halvorson, Kristina. Content Strategy for the Web. Berkeley, CA: New Riders, 2009.
 Lovinger, Rachel. “Content Gone Wild.” MIMA Summit, October 5, 2009. Retrieved from SlideShare on December 7, 2009.
 Lehrer, Jonah. How We Decide. Boston: Houghton Mifflin, 2009.
 Park, Alice. “The Brain: Marketing to Your Mind.” Time Magazine, January 19, 2007. Retrieved December 10, 2009.