Challenges of Card Sorting
Part of the appeal of card sorting today is that researchers have the option of conducting studies either online or in person. When using an online tool, large numbers of participants can complete the exercise, lending additional statistical weight to the findings. In some organizations, the large sample size and statistical basis of online card sorting is helpful in dealing with decision makers. In-person card sorts let researchers interact with participants and ask probing questions to determine their organizational strategies, as well as other follow-up questions. A number of resources are available online that provide detailed steps on running and analyzing studies in both contexts.
Sounds easy, right? Just load your content into a card sort application or create actual cards, have participants complete the exercise, and the right organization for your site will be revealed! Unfortunately, it is not usually quite that easy. As Sam Ng wrote in an article on UXmatters, there are a number of issues to consider when conducting a study, from the timing of a card sort to setting proper expectations with stakeholders. Two of the points Sam mentioned in his article ring true and concur with my experience:
- Run multiple studies. Designers can expect participants to sort a maximum of 80–100 cards during any one study. For sites with a large volume of content—such as intranets, retail stores, or research portals—such a limited number of cards may not accurately reflect the full breadth of content on the site. For such a site, you’ll need multiple studies to understand each individual level or section of the site. Running multiple studies adds significantly to the time and effort completing the process requires, and you should account for this when planning.
- Count on doing some subjective analysis. Although you can apply statistical methods to results, you’ll still need a certain level of subjective analysis to interpret the intent of card sort participants, as well as the consequences of content relationships. As a result of this necessary subjectivity, designers may have different interpretations of the same results, depending on their depth of domain knowledge and previous experience with the content. User researchers have proposed several analysis templates, involving spreadsheets and visualizations in place of dendrograms, but their analysis still involves a level of subjectivity.
In addition to the points Sam Ng raised, I’ve found some other challenges to conducting successful card sort studies.
- selecting and naming cards—Creating the cards for an online sort is very difficult in some domains. For physical objects, the process is easier. Choosing objects that have universally recognized names—and, potentially, using images—helps ensure respondents will understand the cards. But on an informational Web site, complex pieces of content are more difficult to describe, so you must make significant effort to ensure respondents will interpret the names of cards consistently, without introducing bias to the groupings participants will create.
- labeling groups—One of the most important elements of a site hierarchy is the labeling for each category or menu item. While open card sorting can provide insights to users’ view of content relationships, designers should not necessarily expect open card sorting to provide useful solutions for the names or labels of the resulting content groups. During a card-sorting exercise, participants give names to the groups they create, but each participant may create groups of varying sizes, with different intents. As a result, looking for trends among group names from different participants is difficult and may not yield helpful insights. While closed card sorting lets you test group labels, you can test only one set of labels at a time, and it does not allow any interplay between the labels for multiple levels of the hierarchy.
Further Considerations for Card Sorting
While there are certainly challenges to conducting effective card sort studies, we can overcome those challenges. For example, user researchers can run multiple studies, pilot test their studies to uncover biases in card names, and collaborate with others to gain different perspectives on the subjective analysis of results. However, my experience with card sorting has led me to question a few primary assumptions about the method. To my knowledge, there isn’t any specific research that validates these assumptions, but it may be important to be aware of them when conducting a study.
Sorting Versus Finding
When participants complete an open card-sorting exercise, they sort groups of related items. It is not clear whether the results would be inherently different if the research method were based on a finding task.
Designing Outside the Design
Researchers intentionally conduct card sorts outside the context of a particular site design. The idea is to get a pure view into participants’ understanding of the relationships between content items. However, for a Web site hierarchy, a number of factors impact users’ ability to find content, including the paths they followed to get to the site and contextual design clues that orient them to the site and related content. Studying content relationships outside the context of the site design may not accurately reflect users’ perception of the site hierarchy as they use it.
Pointing to a Single Answer
At the outset of a site reorganization process, project teams are generally excited about card sorting’s potential for settling questions about the labeling of menu items or the placement of particular content on the site. They hope that card sorting can provide the answers they need. Sam Ng mentioned this phenomenon in his article on UXmatters. The problem is that there is not always one answer. Certain users might look for a piece of content in one category, while others would look for it in a different category. Or perhaps users might look for content differently, depending on the context of their visit to the site. By focusing on the aggregate results that show the average content relationships from a card sort study, teams may miss the variety of perspectives the site should accommodate.
Focusing Only on a Categorical Hierarchy Solution
Additionally, card sorting may not give insights into alternative navigational paradigms. For example, perhaps the best navigation system for the site is listing content alphabetically, by task, or by audience type, or is metaphor driven. Focusing on the results of a categorical card sort can blind a design team to these other possibilities.
