Learning from Closed Card Sorts with Different Inputs

Card Sort 1: Categories, but Not Items

For this workshop, we provided categories, but no items to place in them. I’d based the categories primarily on the existing system, but with a few modifications resulting from some other research that my colleague and I had already done. We left it to participants to come up with items that they would expect to find in each category, as shown in Figure 1.

This exercise took the longest. You generally need to allow more time for participants to come up with ideas from scratch than you do for them to comment on or work with something that you give to them. Plus, it’s important to be able to give a few examples from your own experience, in case you need something to start the ideas flowing. Once the brainstorming opens up and participants hear what other people are saying, they’ll usually understand what you’re asking them to do, and this exercise will go faster.

As we expected, although we had told the participants that they could modify the categories that we had given them, they didn’t make many changes. They created only two new categories—basically, because there was some debate about where several particular items should go. Still, this provided useful information because it not only gave us another potential section for the site architecture, but also revealed a discrepancy in our thinking about how people viewed certain items.

Benefits

The most important result of doing this type of card sort is learning what items participants come up with. Without our feeding participants pieces of content or functionality, the items they think of and write down end up being those that are most important to them. In this way, a researcher can get a sense of priorities that is absent from the results of a card sort in which all items are present with equal weight from the start. Plus, participants inevitably come up with items that you would never have thought of.

During this session, several different participants added pieces of functionality that do not currently exist in the system. Since we have not used the system extensively ourselves, we never would have known how important this missing functionality is. Thus, this session provided insights into key content and the prioritization of functionality, as well as making us aware of the need for some new items.

Drawbacks

The drawbacks of this approach—beyond those resulting from doing a closed instead of an open card sort—are twofold. First, while the aforementioned time requirement may not be an issue, depending on your situation and testing setup, if your time is limited or you have a large group of participants, the open-ended nature of this exercise may prove unwieldy.

The other drawback is that, lacking significant preparation, participants are likely to miss a lot. If you’re not dealing with an existing system, as we were, or one that your participants are very familiar with, they’ll almost certainly forget about many items that are nevertheless important. There’s usually just too much for anyone to remember everything. So you’ll have to expect that there will be items that participants don’t categorize at all and figure out how to balance the relative importance of those items with those that participants mentioned during the session.

Card Sort 2:  Categories and Items

This is probably the type of card sort that most people who haven’t run a card sort before would naturally think of. For this card sort, we provided a set of categories that were consistent with the existing system, as well as a large pool of existing items of content and functionality, as shown in Figure 2. We asked participants to sort the items and encouraged them to add or remove either items or categories as necessary.

When running a card sort in this way, the general concern is that participants who are familiar with an existing system will tend to put items into the buckets they’re already used to. Since we expected that this might happen, we intentionally added some items that didn’t already exist, and we were also sometimes slightly vague in describing existing items on the cards.

We were hoping that, at the very least, participants would sort the new elements that we had added—and perhaps reclassify a few of the existing items that they thought were awkwardly placed. What we actually found was that the card sort produced a very different information architecture from that of the current system.

In part, this occurred because—as we already knew—some things in the existing system were confusing and participants were able to identify them. But more importantly, this happened because of the way my colleague and I worded the items that we provided. For instance, an item that we had labeled Payments, which we intended to indicate the place where a user enters how they would like to get paid, participants alternatively thought meant one of the following:

• the place where they see when they will get paid
• a list of payments they have already received
• a list of other people who have purchased things on their site

Clearly, our use of the term payments—which is prevalent throughout the existing system—was an inherent source of confusion. We might not have learned this had we not asked participants to define the term for us.

Benefits

As the Payments example illustrates, the chief benefit of this kind of closed card sort was the clarification of our existing content and terminology. This type of learning is most important when you are redesigning an existing system—as we were. Even when you’re creating a site or application from scratch, you may be able to discover some of the organizational predispositions that your business owners have.

The varied interpretations of Payments also told us something about prioritization We had assumed that, to some degree, people would naturally base their classifications of items on what they thought was most important—or at least, what came first to mind. This, in turn, taught us a little about how different people in the company saw the business requirements for the system, because there were correlations between how participants defined items and the roles and positions of the people who were defining them.

I was actually pleasantly surprised by the variety in the participants’ categorization of the items. It was further from the current system than we had expected, plus participants actually added several categories that hadn’t previously existed to enable them to adequately bucket their items.

Drawbacks

The major drawback of providing this set of inputs was, as you would expect, the inherent limitation in the potential knowledge that we might gain because we gave participants a set of items to categorize at the outset, and they tended not to add many more items of their own creation. So this is not the best method to use if you’re trying to find out what might be missing from an existing information architecture.

Despite participants’ creating some new categories, they split the only really new items off from existing items. In the case of Payments, for example, we realized that we would need several different items to cover all of the things that term had previously comprehended.

You should expect to do a lot of explaining during a card sort like this, because participants are likely to ask what you mean by certain terms. Whether you should answer them depends, as always, on what you’re looking for. If you’re interested strictly in categorization, you might want to answer them and describe any confusing items. However, I have found that I often gain more interesting knowledge by not providing answers or by directing the questions to other participants, asking them what they think, to see how they interpret items.

Card Sort 3: Categories and Presorted Items

Our strictest set of starting inputs once again included both predefined categories and items, but also grouped the items into categories using the current system’s information architecture. We asked participants to think about their experience with the current system and identify painpoints, then to rearrange the classification of items in a way that they thought would reduce its complexity and remedy known problems. For instance, if an existing workflow was inconvenient because two items of information that a user needed to complete a task were on separate pages, maybe those two things should instead be grouped together.

As we expected, this workshop took a little bit of time to get going. Regardless of the clarity or confusion of an existing system, once an audience becomes familiar with it, it’s difficult for them to imagine anything else. As with the other methods of card sorting that we’ve described, as a moderator, you must sometimes give an example or two from your own experience to start the ball rolling. Once participants realize that this is their chance to remedy some known painpoints and complexities, they’ll start thinking about their day-to-day routine and will be better able to propose ideas.

Nevertheless, we found that this setup was a wonderful conversation starter. In fact, this approach led to the most in-depth discussions and analyses of the existing system. When we presented participants with what were essentially a set of visual aids that mapped out the system that they used every day, they thought about its familiar patterns more than is typical for more open card sorts. This seemed to make talking about those patterns and recalling the issues that they have with them easier.

We came out of this session with a host of insights that we had gained largely from very detailed conversations that got sparked by participants’ commenting on a single item’s placement. For example, a participant’s moving just one card from one category to another started an informative, ten-minute long conversation about why he had moved it. Overall, participants may have done less sorting of the cards than with open card sorts, but we learned something important just the same.

Benefits

The most prominent benefit of going into an exercise having already provided this many inputs to participants is familiarity. We found that, although it took a while for them to start making changes, it was very easy to get them to think about the current system and start discussing it.

There is also the added benefit of specificity. Card sort 1, as I described earlier, had the downside of missing a lot of the smaller items that participants wouldn’t necessarily think of. For card sort 2, that was less of a drawback, but participants still tended to focus mostly on more major elements. Since card sort 3 kind of represented the warts-and-all current system, it led to the most granular and specific discussions and feedback. If one of your business requirements is the preservation of many pieces of functionality in a complicated system, exercises like this one can give you insights into some things that are less obvious, but still important, which may cause problems if you do not address them in your redesign.

Drawbacks

As expected, the primary drawback with such a closed card sort was an exaggeration of the same problem that we saw with predefined categories and items in cart sort 2. When you proscribe the inputs by presenting participants with cards that conform to the existing system, they tend to make few changes. So your new information architecture may end up looking very similar to the existing one—even when there are actually many changes that you should make.

At the end of our session, the buckets of items were very similar to those that we had started with, because we had not forced participants to think up a new categorization scheme from scratch. Furthermore, our placement of items within existing categories eliminated most of the terminology confusion that had appeared during card sort 2, when we had only predefined categories and items. So participants did not create new categories as a result of any reinterpretation of their meanings. As a consequence, I do not recommend this method of card sorting when your goal is to restructure an information architecture—unless you use it in combination with other more open card sorts.

The other major drawback of this approach is the time that the discussions take. This was one of the longest workshop that we ran. To gain significant knowledge from this method, you need to let the sorting process jump start discussions among participants, and those discussions can take a long time. Of course, those discussions can be well worth the time, and you can gain valuable information from them, but you need to give them time to breathe. Of all the methods that we used, this is the one that might benefit most from doing multiple sessions. It might even be better not to approach this discussion through a card sort at all and, instead, get participants to talk about just a few sections or categories at a time. Of course, you should allow participants to point out items that seem to be misplaced.

Guidelines for Selecting a Closed Card-Sort Method

Ultimately, each workshop that we ran provided useful information. The question, then, is how should a prospective workshop facilitator decide which inputs or combinations of inputs to use? The answer depends on what you want to get out of a session.

Based on the outcomes of our workshops, here are some guidelines:

• Provide categories, but not items. Use this method if your primary goal is to find out what is missing from an existing system or you need to understand participants’ priorities.
• Provide both categories and items. Use this method if you primarily need to learn about the clarity of an existing system’s terminology and areas of confusion—particularly as different groups of stakeholders perceive them.
• Provide items that are already categorized. Use this method if your goal is not to completely change an information architecture, but to get detailed feedback about specific categories, items, or parts of the system.

Since I’ve based these suggestions on the findings from this particular workshop series, they are primarily applicable to redesigns of existing systems or projects within established organizations. Open card sorts are more useful when you’re designing a system from scratch or doing an overall redesign. My purpose in writing this article was simply to describe a few ways in which a more closed card sort can be very helpful, depending on where you are in a project and what you’re trying to learn.