Testing Your Own Designs: Bad Idea?

Envision the Future

The role UX professionals play

September 21, 2009

This column was spurred by a simple question I posted to Twitter in mid-August: Can designers effectively usability test their own designs?

This isn’t just an academic question. With the current state of the economy and many UX teams downsizing, it’s entirely probable that your company will call upon you to both create a UX design and do usability testing to validate it. In the future, as the field of user experience progresses, agile UX becomes more common, and functional disciplines become more blended, I think this will occur more and more.

People have often likened doing both design and usability testing on the same project to defendants serving as their own counsel in a court of law. How does that saying go? Something like this: A lawyer who defends himself has a fool for a client. Is testing one’s own design a similarly bad idea? What are the pitfalls? Are there any advantages? And most important, if you must do it, what pitfalls should you beware of?

Champion Advertisement

In this column, I’ll answer these two questions:

Is it possible to do both design and usability testing effectively?
If so, how can we test our own designs well?

UX literature is rife with cautionary tales about designers testing their own designs. The objections to doing so typically follow this line of reasoning: Designers are emotionally invested in their designs, they believe in their rightness, and are loath to change them or bear criticism of their baby.

I think this is a specious argument. Certainly, some designers—particularly those of the genius design bent—are willing to brook little criticism of their designs. However, most designers I’ve encountered are more interested in actually solving users’ problems than in maintaining the typically illusory artistic integrity of their designs. So I don’t think most designers are naturally resistant to criticism of their designs, particularly when the criticism originates from the people who are the intended users of the products they’re designing.

When I posted this question on Twitter, Facebook, and my own UsabilityBlog, I received some interesting responses. Here’s a sampling of some the best, most pithy opinions. (Keep in mind that folks who replied via Twitter were limited to 140 characters, while those who replied via Facebook or my own site had a bit more room to expound.)

“Designers can test [their] own stuff. Everyone has bias whether [they’re a] designer or not. [You] just need to be aware of your biases.”—an interaction designer/usability analyst, via Twitter

“[Designers] can effectively perform [usability] testing to gather additional insights and new ideas, but to be fully unbiased and seek out real UX issues, and to do it effectively…it’s a struggle. If you are playing the role of both researcher/analyst and designer, you have to be fully aware, at all times, of how you are forming your conclusions. For example, am I just seeking insights that prove my design solutions? It’s best to partner with an unbiased—yet collaborative—researcher.”—a user experience manager, via Facebook

An experienced user researcher who works for a large developer of both desktop and Web applications had this to say:

“Yes [designers can test their own designs], but they have to be actively trying to ‘dis’ [them].”—via Twitter

A hardware designer and usability analyst expanded on this theme:

“I think it is very challenging for UX [professionals] to objectively test designs they’ve created. While [some] designers are accustomed to having a number of ideas and [going through a] critique process, [others] tend to select one option and focus on it. This generally results in significant bias. I know some companies separate the design folks from the test folks. I suspect many organizations see that as too cost prohibitive these days, though.”—via Facebook

You may have noticed that I’ve been holding back my opinion so far. Well, here it is: I generally agree with the consensus these comments have demonstrated. Regardless of whether we like it or think it’s a good idea, designers will increasingly be testing their own designs.

Potential Pitfalls of Designers’ Testing Their Own Designs

I do think there’s a more subtle argument to be made about the potential pitfalls of designers’ testing their own designs. My own take is that it’s entirely possible for designers to test their own designs effectively. However, there’s one catch: the designs they’re testing have to be close to the right solution, because in testing their own designs, it’s likely designers would concentrate more on fixing the design as it exists rather than being open to the possibility that their design is not an appropriate solution. In other words, my hypothesis is that designers would be less willing than a third party to throw out their faulty design concepts and, instead, more likely to try to patch their flaws. Why do I say this?

Think back to your Psych 101 class. In its social cognition unit, you probably learned a bit about a psychological phenomenon called confirmatory bias. I won’t go into the foundational research behind confirmatory bias, because the Wikipedia definition is serviceable:

“Confirmation bias is an irrational tendency to search for, interpret, or remember information in a way that confirms one’s preconceptions or working hypotheses…. The bias appears, in particular, for issues that are emotionally significant—such as personal health or relationships—and for established beliefs [that] shape the individual’s expectations.”
—Wikipedia

What this means in a design context is this: The fact that you’ve taken a stand and created a design in the first place means you’ve articulated your design hypothesis and instantiated your hypothesis in the form and function of your design. It is going to be difficult to keep yourself from wanting to confirm your design hypothesis, because you’re hardwired to preferentially seek out confirming rather contradictory evidence. Even though you’re able to criticize your own designs and recognize that a fundamentally sound design needs some adjustment, confirmatory bias makes it hard for you to realize that your design is the wrong approach entirely.

Guidelines for Testing Your Own Designs

So, enough opinion. Here are some guidelines for testing your own designs:

When testing your own designs, always concentrate on the negative.

Yes, it’s normally part of good, balanced testing practice to look for both what works and what doesn’t. However, knowing what we know about confirmatory bias, it’s probably a bad idea for designers who are testing their own designs to try to be balanced. Instead, you should always focus on failing your design, because once you start looking for the good in your design, you’re likely to weight that information more heavily than the negative.

This, of course, begs the question of how you can put yourself in a frame of mind to focus on what’s wrong. So, here’s guideline number 2.

To focus on the negative aspects of your design, try to keep yourself focused on your long-term goal, which is to solve your users’ problems.

The real outcomes you’re aiming for are to make your client happy—if you’re a consultant—and make your target users happy by successfully providing them with a tool that helps them solve a problem. It might help you to be objective if you visualize what would happen if you gave your design a pass, your client launched it, then users found it lacking. Keeping this in mind has often helped me criticize my own designs—knowing that, if I don’t get it right, both the users and my client will be unhappy.

Guidelines 1 and 2 have been about motivation and focus. Guideline 3 is all about helping you recognize when you should scrap your design.

If your users are unable to grasp the task at hand or are experiencing repeated failures and missteps when navigating your design, you should consider rethinking the entire design.

When you see users endlessly hunting for functional access points or repeatedly struggling to map the task you’ve asked them to do to anything in your user interface, you should take this as a sign that you should redesign your information architecture, navigation, or interactions from the ground up. In this case, don’t patch, scrap.

A final thought—as I write these words, it occurs to me that these guidelines are equally applicable to third-party usability professionals. What do you think? I welcome you to add your thoughts, affirmations, or disagreements in the comments.

In UX Strategy | Usability Testing

12 Comments

Geoff Mazeroff

September 21, 2009 7:04 AM

Excellent post, Paul. The comment about confirmation bias was especially apt.

I have one small grammar correction. The statement “begs the question” should be “raises the question.” Begging the question refers to a circular argument—see “Begging the Question, Again.”

Jim K

September 22, 2009 6:30 AM

The question for me is whether a designer can adequatley perform the role of a usability engineer. A good usability test shouldn’t have anything to do with who did the design. A design is a design, and a well-constructed usability test will uncover objective usability flaws. The subjective aspects of the design shouldn’t matter.

Designers wearing both hats should probably get training in how to do good usability testing.

Whitney

September 22, 2009 8:57 PM

Great article, Paul.

You could even go beyond this and start to think about usability testing not just as a way of seeing if your design works, but as part of the design process. That’s one of the notions behind iterative design. It’s not just that you create design, then test it, then change it, and so on.

Instead, you could start with usability sessions that move gently from user research—information gathering—to explorations of tentative implementations—I mean sketches or rough prototypes—of a design concept, proceeding through ever more substantive prototypes.

This is especially useful when there are a lot of possible directions. For example, if you learn that users would value a way to interact and communicate on a site, does that mean: comments, allowing users to post new articles, a reader forum, an associated email list? None of these is hard, but which ones match the style, content, and intensity of interaction they want? Are they willing to sign in? Insist on doing so? This sort of testing cannot only help you pick the right features, but understand how to design them right.

Paul Sherman

September 23, 2009 7:37 AM

@Whitney—Great comments, Whitney, and I, of course, completely agree. I do just what you say in the design processes I’ve specified and implemented. Incorporating various types and levels of usability testing into the design process is pretty much the way to do iterative and truly user-centered design.

@Jim K—I’m not sure I’m following your argument, but I do respectfully disagree with your point about a good usability test not having anything to do with who did the design. No matter how well designed a usability test is, there is still lots of subjectivity in how the test facilitator behaves while running the test sessions. Even in situations when the facilitator is behind glass and communicating only sporadically with the participant—a protocol I don’t recommend, by the way if your goal is to really discover the design’s flaws and advantages—the facilitator will still be susceptible to what the experimental literature refers to as experimenter effects. In short: There is no objective usability test protocol.

Daniel Szuc

September 23, 2009 9:44 PM

Hi Paul,

What helps designers guide themselves through their own design to ensure they keep their eye on the ball and complement their own thinking? What has worked for you?

One could be overly negative about just about anything, but the feedback does not necessarily help the design. Just as one could be overly positive too.

Regards, Dan

Robert Skrobe

September 25, 2009 4:06 PM

Hi Paul,

Interesting article. I wonder if your recommendations are conditional to either the designer working solo or as part of a project team.

If the former, I can picture going through the motions. Do you think it would be better to do what Dan suggests and look at the pros/cons of each approach, rather than putting on the Black Hat only?

If the latter, it’s definitely about showing others and getting feedback via discount usability testing. If there’s a time crunch to hit a deadline—which happens at our agency quite a bit—it’s usually the default approach.

Thanks for contributing and sharing your point of view.

Yours, Robert

Drdeb

September 28, 2009 5:56 AM

This is one of my favorite questions, as I do test my own designs. I like all the comments in this thread. :-) It also occurs to me that all scientific researchers in effect “test their own design,” in that they formulate a hypothesis, which reflects their own current beliefs, or at least their pet theories, and then they design an experiment to test their hypotheses. Scientists do not generally formulate a hyphothesis, and then go out and hire an uninvolved other scientist to objectively carry out the experiment. So as testers of our own designs, I don’t think we bring any more bias to our testing than any researchers do. The established experimental methods are designed to minimize bias. As long as we follow those methods, I think we are on pretty secure ground.

Personally, I believe there are distinct advantages to testing our own designs that probably outweigh the potential biases we may bring. As designers, we have concerns, questions, or reservations about specific aspects of our designs, and we design our tests around those. We are looking with a very focused eye to learn about those aspects. An independent tester will not be so sensitive to those aspects. I realize this can introduce bias or at least risk missing other things we are not looking for. But I agree with the above comments suggesting that for many of us, we really are pursuing the truth and looking for the problems in our designs rather than simply trying to confirm them. I guess, as with many things, it depends on the integrity of the practitioner, but personally, I would always rather test my own designs than leave it in the hands of others who are not intimately familiar with what I am trying to find out about my design.

Paul Sherman

September 28, 2009 7:56 AM

@Dano—Good question. Since about 60-70% of my work now is design, I can tell you what has worked for me regarding “keeping my eye on the ball.” It’s walking through the design as the target persona, but really putting on that persona.

A fair question here is What do you mean, Paul?…and darned if I’m having trouble answering it. I guess what I mean is that I try my best to not just walk through the interaction hurriedly, but actually say to myself Okay, I’m Harriet the HR manager, and I’m trying to set up a new personnel record. At times I’ll even put the persona’s related work information in front of me or try to do the interaction under the conditions Harriet would. For example, is she on the phone while filling out the form? Then, I cradle the phone on one shoulder and try to complete the interaction.

If you have the luxury of working with other UX or design people on a project, I’ve also found that tandem walkthroughs with another interaction designer or a visual designer also work well. This is similar to what Robert suggests in his comment.

Regarding “putting on the black hat,” I realize I probably was a bit too unilateral in recommending that designers concentrate on the negative, but I’m trying to come up with a way to defend against pathological positivity, knowing how easy it is to both seek and overvalue confirmatory evidence. Any ideas?

tobogranyte

October 1, 2009 1:38 PM

I’m a user experience designer who has been called upon in several circumstances to do testing on my own designs. This is fertile territory, and there is a lot I would have liked to see discussed regarding such a fertile topic that was missed.

First, I would have liked to see some of the many, many opinions here backed up with something other than more opinions.

Second, this article almost entirely glosses over the various steps in the usability testing process, all of which have different pitfalls and challenges if the designer is in charge of the testing.

And lastly, it largely ignores the social and political context in which all of this takes place.

Concerning the second point, take, for instance, the preparation of the testing plan and the writing of the script. Do the plan and script cover all of the required use cases, or is it possible that the designer may well bias the plan and the script toward things that the designer has focused more heavily on, and therefore, things that the design is more likely to perform well on?

Then there is the administration of the test. Is a designer who isn’t as experienced in usability testing going to faithfully execute the testing script, assuming he or she even prepares one? And if not, where are the opportunities mid-test that the designer might actually be steering the results of the test as they are being generated by the users?

Also there is the interpretation of the results and the preparation of the report. This, likely, will include recommendations for modifications to the design. Again, there is ample opportunity here, as a result of both inexperience and bias, to generate a final deliverable that falls far short of the mark.

The fact is that problems with the process can be the result not just of bias, but of a designer with otherwise good intentions being in a situation for which he or she is not trained, and therefore, far less capable of keeping a ready sense of objectivity amid the swirl of the testing process.

On the third point, it is easy to chalk up bias as being the result of a designer loving his or her design and being unable to be objective as a result. But this is a very simplistic notion of the origin of bias. All designers work in both a social and, yes, a political environment. Just like anywhere else, a designer likely has a career path. The designer also has other team members—likely designers as well—with potentially competing career paths within the organization. The designer also has managers, supervisors, and other senior team members who are stakeholders in the design process, who evaluate the results of any design effort, and who, yet, are not experienced in the discipline of design. Some of these people may also have a role in evaluating the performance—formally or otherwise—of the designer. All of these things can exert a potentially strong force on the short-term motivations of a designer cast in the role of usability tester, regardless of their otherwise good intentions.

Paul Sherman

October 2, 2009 6:19 AM

Looks like I tapped a rich vein here. Just goes to show that you can never predict your own hit single…

I’d like to address Deborah’s comments—and also bask in a bit of reflected glory, seeing as this is the first time I’ve gotten a comment from one of my early UX heroes…. She drew an apt analogy with experimenters. In fact, there’s a phenomenon that’s described in literature that’s known as the file drawer problem. When researchers run an experiment that is either methodologically suspect or that doesn’t support their hypothesis, they simply file it away and don’t publish it.

I really like Deborah’s discussion of the advantages of testing one’s own designs, but I’d like to point out an exception to her statement: “An independent tester will not be so sensitive to those aspects [of the design that we as designers have specific concerns about].”

This may hold for the situation where the usability analyst is a consultant or is being drawn in from another group. However, when I have worked internally, I have always configured my teams so that the usability analyst is working closely as an advisor to the designer throughout the ideation and design phases. This makes it much more likely that the analyst will indeed be sensitive to the designer’s concerns and interests.

Just a couple more responses to this excellent feedback, and I’ll get off the soapbox—and I’ve got some card sort sessions to run this morning, so I’ll make it quick… :-)

@Jim K—You said, “Designers wearing both hats should probably get training in how to do good usability testing.” I think the reverse is even more true—most usability analysts I’ve encountered really, really need more training and experience in design. This was actually one of my prime motivations to learn the craft of design. I started as a usability analyst in the late ’90s—after completing an advanced degree in experimental and observation human factors psychology—and realized fairly quickly that, although I could run a good test, I wasn’t yet qualified to make design recommendations.

@tobogranyte—You make some good points, and I plan to address them in a followup column that Pabini suggested I write. But I have to say—and please take this not as a personal indictment, but more as a comment on experience designers as a whole—that initially your comments came across as an example of the—I don’t know, is it arrogance?—that experience designers sometimes demonstrate. Later on, your comments became quite insightful. But the beginning just strikes me as experience designer smugness. Eh, maybe I’m just being too defensive.

One my mentors always tried to instill in me the “what and how” rule: There’s what you do and say and how you do and say it. Both are necessary for effective communication and teamwork.

Susan Hura

October 2, 2009 6:46 AM

This discussion hits home for me, because I’ve been a usability analyst, a designer, and both. As a consultant, I’ve often been in the position that Whitney described, where usability testing is not a separate activity, but a part of an iterative design process. As a consultant, the choice comes down to do-your-own-usability-testing or don’t-do-any-usability, and I’ve always opted for the former, but it hasn’t been without challenges.

I agree with Paul about confirmation bias. Once you’re in the weeds of a design problem, it’s really difficult to be open to design possibilities that you never considered—or possibilities that you considered and rejected. When a user does something that points to one of my previously rejected design hypotheses, I have the overwhelming urge to explain why I chose the alternate design and get the user’s buy-in.

I like Paul’s suggestion of orienting yourself toward negative feedback when you test your own designs, but I think it closes you off to some valuable information in usability testing. Focusing on negative feedback doesn’t help you get to a better design solution, especially if the better solution is one you already considered and rejected.

One of the classic usability books—can’t remember which one now—discusses the importance of coming into usability testing with a “beginner’s mind,” open to many possibilities without ego or bias. In practice, this is impossible, because we all tend to favor our own solutions—precisely because we believe we’ve chosen the best solution to the design problem at hand. But beginner’s mind is what we all need to cultivate to more effectively test our own designs.

cliff anderson

October 27, 2010 10:42 AM

This seems like an awfully academic discussion to me. Over the years, I’ve been able to watch a number of designers test their own designs. In general, it wasn’t very pretty. (Probably my favorite was the designer who spun in his chair when the user finally clicked the right link!)

To tell you the truth, I’m not sure everyone can do testing properly, let alone test their own stuff. Would it be possible to have a colleague—not a usability engineer—do the testing? Maybe someone who seems to have a knack for it or some actual experience doing it? (I’ve found the few exceptions to the “not pretty” rule are those designers who have had some background as UEs.)

One more thing to consider—a crit. This is where you present your design to your team—other UX professionals, not management, developers, or others. We’ve used it at several places I’ve worked, and it seems to work pretty well.

Paul J. Sherman

Founder and Principal Consultant at ShermanUX

Assistant Professor and Coordinator for the Masters of Science in User Experience Design Program at Kent State University

Cleveland, Ohio, USA

ShermanUX provides a range of services, including research, design, evaluation, UX strategy, training, and rapid contextual innovation. Paul has worked in the field of usability and user-centered design for the past 13 years. He was most recently Senior Director of User-Centered Design at Sage Software in Atlanta, Georgia, where he led efforts to redesign the user interface and improve the overall customer experience of Peachtree Accounting and several other business management applications. While at Sage, Paul designed and implemented a customer-centric contextual innovation program that sought to identify new product and service opportunities by observing small businesses in the wild. Paul also led his team’s effort to modernize and bring consistency to Sage North America product user interfaces on both the desktop and the Web. In the 1990s, Paul was a Member of Technical Staff at Lucent Technologies in New Jersey, where he led the development of cross-product user interface standards for telecommunications management applications. As a consultant, Paul has conducted usability testing and user interface design for banking, accounting, and tax preparation applications, Web applications for financial planning and portfolio management, and ecommerce Web sites. In 1997, Paul received his PhD from the University of Texas at Austin. His research focused on how pilots’ use of computers and automated systems on the flight deck affects their individual and team performance. Paul is Past President of the Usability Professionals’ Association, was the founding President of the UPA Dallas/Fort Worth chapter, and currently serves on the UPA Board of Directors and Executive Committee. Paul was Editor and contributed several chapters for the book Usability Success Stories: How Organizations Improve by Making Easier-to-Use Software and Web Sites, which Gower published in October 2006. He has presented at conferences in North America, Asia, Europe, and South America. Read More

Testing Your Own Designs: Bad Idea?

Envision the Future

The role UX professionals play

Potential Pitfalls of Designers’ Testing Their Own Designs

Guidelines for Testing Your Own Designs

12 Comments

Join the Discussion

Paul J. Sherman

Other Columns by Paul J. Sherman

Other Articles by Paul J. Sherman

Other Articles on Usability Testing

New on UXmatters