Review of Information Architecture Evaluation Tools: Chalkmark and Treejack

Testing Information Architectures with Treejack

Treejack is a tool for testing information architectures. Participants’ tasks involve finding something in a clickable tree structure, as shown in Figure 9.

Each branch of a tree structure expands with a click, showing the subcategories within that branch. To choose the final category in which participants would expect to find an item, they click an I’d find it here button, shown in Figure 10. The test results show which categories participants chose during each task, the percentage of correct choices, the paths taken, and other related data.

Treejack is ideal for testing a proposed information hierarchy. In many ways, it is superior to other information architecture evaluation techniques such as closed card sorting and reverse card sorting. Like reverse card sorting, Treejack asks participants to complete finding tasks. Thus, Treejack is not like closed card sorting, which asks participants to organize content into categories. But unlike either closed or reverse card sorting, which limit evaluation to the first level of categories, Treejack evaluates the entire depth of an information hierarchy.

Setting Up a Treejack Study

Similar to Chalkmark, the first steps in creating a new study in Treejack are specifying the participant login information, as shown in Figure 11, and creating the welcome message, instructions, and concluding message, which appears once participants complete a study.

Setting up your tree structure in Treejack can be somewhat complicated. To facilitate the process, create a spreadsheet containing the data first, then copy and paste it into Treejack, as shown in Figure 12. It is worth noting that indentation must be correct for Treejack to render a tree correctly. Helpfully, a Preview tree button lets you make sure Treejack created it correctly.

The next step is creating the tasks—for example, Find acne treatment cream. For each task, you select the correct answer—that is, the subcategory or subcategories in which an item resides, as shown in Figure 13. However, when setting up tasks, the need to select the correct answers and the way to do this are not obvious at first, so it is easy to overlook this until you unsuccessfully try to launch a study.

There are several helpful settings you can select, such as randomizing tasks, revealing only a certain number of tasks to each participant—so you can have a large list of tasks, but each participant does only a reasonable subset—and ignoring results from participants who skip more than a certain number of tasks.

Testing with the Participant User Interface

Treejack suffers from some of the same problems as Chalkmark—a limited instructions page that contains only text, with no images or visual examples, and no way to view the instructions once the tasks begin. Fortunately, the task pages are fairly intuitive.

Unlike Chalkmark, which displays a task and a screenshot at the same time, Treejack shows a task first, then waits for a participant to click a Start this task button, shown in Figure 14, before showing a tree structure like that shown in Figure 9. This helps ensure participants are ready, lets participants take a break between tasks, and probably results in more accurate task times.

Clicking through a tree structure is fairly intuitive, because it looks like a navigation menu and expands to reveal subcategories. Once a participant clicks a subcategory, a Top link appears at the top of the tree, as shown in Figure 15, allowing participants to go back to the top of the hierarchy. I have found that some participants do not notice this link or do not understand its purpose. As a result, they did not realize they could go back up in the hierarchy. Both the label Top and the concept of the top of a hierarchy may be unfamiliar to most people. While a Back link might better match the familiar concept of Web navigation, it would go back up a hierarchy only one level at a time.

Participants complete each task by clicking an item in the tree, then clicking an I’d find it here button, as shown in Figure 16. Participants have said they prefer this interaction over Chalkmark’s, because it lets them choose their final answer in two steps, giving them a greater sense of control. The I’d find it here button gives them a chance to go back and choose something else, while in Chalkmark, their first click unexpectedly takes them directly to the next task. Thus, Treejack provided a more satisfying experience.

Working with the Test Results for a Treejack Study

Treejack provides test results both in an online summary, shown in Figure 17, and as a downloadable Microsoft Excel file. The online results page provides very little information describing what it is showing and, as throughout the rest of the application, there is no link to online Help that would explain how to analyze the results.

From the online summary page alone, one might conclude that Treejack provides little value. Each task and the study overall receives a score. Bar charts show percentages for success, speed, and directness. While the meaning of success is obvious, it is unclear how Treejack can represent speed as a percentage, and the meaning of directness is unclear. Plus, there are pie charts with no labels at all. Again, some additional information about how to interpret the results would be helpful, but Treejack does not provide it.

Fortunately, there is also a link that lets you download an Excel file, which provides far more comprehensive and useful information. In the Excel file, the first tab, shown in Figure 18, shows the overall success rate and participants’ choices for each task. It also shows the average success rate across all tasks, the direct success average—showing the frequency with which participants chose the correct category right away—the indirect success average—showing the frequency with which participants eventually made the correct choice, after backtracking through other categories—the failure average, and the average percentage of those who skipped tasks.

The second tab in the Excel file, shown in Figure 19, shows participants’ first clicks for each task, including the percentage of first clicks for each top-level category and the percentage of participants who clicked each top-level category during the tasks.

The third tab, shown in Figure 20, shows the path each participant took through the hierarchy during each task. Paths are color coded, with green for direct success, yellow for indirect success, orange for failure, and gray for skipped tasks, making it easy to find and focus on specific results such as failures.

My Overall Assessment of Treejack

Treejack is a great tool for testing a proposed information hierarchy before incorporating it into a page design. The ability for participants to drill down through multiple levels in a hierarchy makes Treejack far more useful than either closed card sorting or reverse card sorting. Using Treejack more closely matches people’s actual behavior on Web sites, because it asks participants to find things within an information hierarchy rather than asking them to organize content into categories, as with closed card sorting.

Trying and Buying Chalkmark and Treejack

Both Chalkmark and Treejack let you create a free account with a limit of three test tasks, allowing you to run test studies with friends and family and evaluate these tools for yourself. It is especially important to view the results and determine whether they will give you the type of information you need to make design decisions.

The pricing of these tools is quite reasonable when compared to other online research and evaluation tools. Both Chalkmark and Treejack are available individually, at the following prices:

An annual plan that includes Treejack, Chalkmark, and Optimal Sort—an online, card-sorting tool—is available at a reduced price of $1,350.

Like most usability professionals, those of us at my company, Electronic Ink, are usually on the lookout for new ways of evaluating information architectures and other user experiences quickly and inexpensively. Chalkmark and Treejack fit the bill, with easy-to-use and inexpensive tools that let you gather extra user feedback and evaluate information architectures during your design process.