On Sunday, April 29, 2007, when flying from Montréal to San Jose to attend my first CHI conference, I worried I might have made the wrong decision by choosing CHI as my conference for this year. When reviewing the program a few days before, I had found myself thinking “I really don’t think these academic-centric and very specific papers will interest me and help me in my practice. Oh! I shouldn’t have chosen this conference!” I even thought of changing my return ticket to go back to Montréal earlier, so I could get back to work on my mental model project, which I’d been enjoying so much. My first morning at CHI seemed to confirm my feelings and even made me feel worse. I wasn’t really turned on by the opening plenary, and listening to the day’s program during CHI Madness—a great concept other conference organizers should use—confirmed my impressions. To give you a bit of context, I’m a practitioner who thrives on user research and on learning new insights and methods from others in a practical, didactic, and clear way, including many concrete examples I can relate to and learn from.
Surprisingly, just four days later, after the Silicon Valley tour I’d signed up for, my mood was exactly the opposite: I just didn’t want to leave San Jose and my fellow CHI attendees, with whom I’d had so much fun. I was ecstatic about my CHI experience.
Champion Advertisement
Continue Reading…
What happened in those four days that made me go from one extreme feeling to its opposite? It’s all about international usability, peanut butter, candies, and field studies.
Despite my passion for international challenges in user experience, I went to this SIG with some apprehension. At the IA Summit in 2005, I had attended my first panel on global UX issues and was actually disappointed. The moderators, who had few personal examples to bring to the discussion, had relied mostly on the experience of the audience, who did not have a wealth experience to bring to the table either. The SIG at CHI, on the other hand, was very well organized, and the moderators, who have extensive experience in international studies—both in usability and user research—presented us with concrete scenarios that stimulated the discussion. These scenarios “exemplified many of the key challenges of doing user research and usability evaluation internationally.” All of the scenarios were similar to this one:
“Renee has finally convinced her colleagues of the importance of exploratory evaluations of prototypes, in order to provide early input into design. So, now they want her to go to Japan to learn about the mental models of Japanese users. She has never been there before, but she has heard that it is really hard to get Japanese users to think aloud.”
What do you think are the key challenges in this situation? What would you do if you were Renee? These are the kinds of questions they asked the audience to engage us in the discussion. After the discussion, they shared their own tips and recommendations.
The interaction and discussion were really inspiring and stimulating—mostly because of the richness of the moderators’ experience with and knowledge of international studies, but also because of the interest and very active involvement of the audience, who came from many places around the world, including Asia, Europe, and North America.
Here are just a very few among the many interesting insights and recommendations we discussed during the sessions:
Don’t simply translate your screener questionnaire, localize it—that is, adapt it to the context of specific countries. The cultural assumptions on which you’ve based your criteria could vary a lot between different countries, even between countries from the Western hemisphere. To give just one example, the notion of income might vary a lot between countries, as well as the notions of technology ownership and technology usage. You should translate the assumptions behind the words, not just the words themselves.
“Collaborate closely with a local recruiter on the translation and adaptation of your screener. Assume there will be a need for adaptation to local conditions, and get recruiter’s input on this before contracting with them.”
Localize timing and scheduling, too. The interview schedule that works in one country may not work in another. You may have to adapt to a country’s ways to determine when it is best for your targeted audience to take time to participate.
Don’t assume you can understand cultural differences by yourself—by reading a book, for example—especially if a culture is very different from yours. “Work very collaboratively with a facilitator and translator, and spend plenty of time in debriefing them. They are likely to pick up subtle cues that will help you to interpret users’ behavior.”
“Adapt your data-collection approach to each culture, including your style of questioning”. The way you express yourself in your own country might not be culturally appropriate in another country. For example, “encouragement may not work and repeated encouragement may imply criticism.” Explore new ways of gathering data such as “having the user teach a younger person” or “having the person verbally direct someone else to copy their actions. The way they describe what they see on screen and what they are doing will imply how they think about it.”
As a Francophone who has interviewed Anglophone users, I was very interested in the moderators’ opinion about facilitating interviews in a language different from your mother tongue. The moderators had different views on this question:
Rolf Molich recommended that the facilitator should have the same mother tongue as the interviewee and belong to the same culture. Otherwise, he thought you might lose some cultural subtleties, even if you are fluent in the participants’ language.
On the contrary, David Siegel thought interviewing someone who has a different mother tongue from yours can be an asset—as long as you are fluent in the other language, of course. You probably won’t assume you understand what a participant says, and you put more effort into ensuring that you understand each participant than you normally do in your own language. He also thought that having a different culture from the participants you observe in field studies can be an asset as well. You might notice things a person belonging to the same culture won’t notice, because they seem obvious.
Personal Thoughts
The debate about whether an interviewer should have the same mother tongue as participants is less applicable in multicultural cities like Montréal, where there are people from all over the world, who typically speak English or French, but for whom neither is their mother tongue. From my experience interviewing participants in English in Canada, I was able to connect with them very well and get as many insights as I would have in French. I was even able to connect better with some Anglophone participants than with some Francophone participants. Without minimizing the cultural and language differences, which often require a local facilitator and translator, culture and language are not the only barriers in communicating with someone. You can have a hard time communicating and understanding someone who shares your language and culture, because you have very different personality types or ways of expressing yourself, for example. Personality types and ways of communicating tend to transcend cultural differences in some cases.
Peanut Butter
I finished my first conference day with Juan Pablo Hourcade’s brief, fun experience report entitled “Learning Observation Skills by Making Peanut Butter and Jelly Sandwiches.” Hourcade, who is a professor at the Department of Computer Science at the University of Iowa, explained how he teaches his students to practice observation skills by asking them to observe their fellow students making peanut butter and jelly sandwiches. Based on their observations, the students sketch designs for a peanut butter and jelly sandwich maker.
I found this activity was a great way of teaching how to conduct user observations—from both the perspective of the observer and the person observed. Here are some insights Hourcade imparted through this exercise:
It’s valuable to observe people—even when they are doing an activity with which you are already very familiar—for example, making a peanut butter and jelly sandwich. While observing each other, the students realized people make peanut butter and jelly sandwiches in so many different ways they’d never thought of. “Most students used two slices of bread; others went for a low carbohydrate version, using only one slice. Some students put the peanut butter on one slice and the jelly on the other, while others put them on the same slice. One student made a triple-decker sandwich. Another liked her sandwich cut into two triangles. Some students spread the jelly with knives, others used spoons. One student liked to lick the spoon after using it.”
Thinking aloud is not natural and easy in practice. Students “realized it is not something that comes naturally to everyone.”
Cultural differences can begin with something as trivial as making sandwiches. Many non-US students in the room had never made a peanut butter and jelly sandwich. So you’d need to localize such an activity if you were doing it outside North America, where people don’t know what a peanut butter and jelly sandwich is.
It’s uncomfortable being observed when you are a novice at doing something. Several international students in the class were making their peanut butter and jelly sandwiches for the first time and were intimidated by being observed while doing this.
There are challenges in observing an expert user conduct a task with which the observer is unfamiliar. International students who were not familiar with the making of a peanut butter and jelly sandwich realized “understanding what actions are important and which are unimportant, and whether order is important” is difficult.
Peanut Butter and Nutella: Localization Starts Here!
Before immigrating to Montréal and North America, I had no idea what a peanut butter and jelly sandwich was. I even didn’t know such a thing existed! As a French native, I would have made a Nutella sandwich, as a fellow attendee suggested to me when he found out I was French! Nutella is a very popular, sweet spread made of chocolate and hazelnuts that many French people—especially children and teens—eat on slices of bread for breakfast or afternoon snacks. In some countries, you might even have to localize the act of making a sandwich, because the sandwich isn’t a food that’s part of their culture.
Candies and Field Studies
On the following day, Tuesday, I had signed up in advance to attend a course on field studies. What a great surprise to find out that two of the moderators of the enlightening session about international studies I’d attended the previous day were actually teaching the course—Susan Dray and David Siegel. I was really looking forward to the course, and I was not disappointed. It was really inspiring—so inspiring that I’ve decided to emphasize field studies in my practice.
The course, entitled “Understanding Users in Context: An In-Depth Introduction to Fieldwork,”PDF combined lecture and hands-on exercises. Its aim was to teach us how to plan and carry out user studies in the field. After providing an introduction to field work, the speakers presented four methods that, in real practice, user researchers often use simultaneously:
naturalistic observation—observing users while they carry out their activities without engaging with them. Also referred to as shadowing, this activity comes from the field of ethnography.
contextual inquiry—“engaging in conversation with users while they carry out their activities.”
artifact walkthroughs—“exploring artifacts’ histories and uses as clues to user goals, intentions, and needs that created them; process of generating them; functioning of the human systems and processes they are part of.” For example, in an office, artifacts might be memos, drawings, user annotations, Post-it® notes, email messages, and voice mail messages. Exploring such artifacts gives us information about the goals and processes underlying users’ behavior.
naturalistic usability evaluation—“usability evaluations done in the user’s real world, with user’s own equipment, own configuration, with own content, and using scenarios generated by or tailored to this specific user.”
The instructors finished their presentation by giving ways “to provide valid and useful input to influence design,” which in my view is one of the key challenges in doing field studies. I felt engaged throughout the session. Several aspects of their presentation made the experience compelling:
the presenters’ vast experience of field research, both in the US and internationally
their ability to convey the material in a clear and interesting way
the concrete examples they provided, including videos
their attentiveness in answering all of the questions the audience raised
Among the many interesting ideas the presenters communicated, a few especially caught my attention. Field studies can benefit not only the actual design of a planned product, but also product planning and strategy, which “is bigger than design.” Field studies can actually feed into product planning and make it “deeply user-centered,” which is rare. Without field studies, requirements are “often based on untested input from sales reps, marketing’s sense of what will differentiate the product in market, executive opinions, developers’ intuitions.”
Gathering data about users’ behavior only through self-report—when people describe what they do—is limiting and risky. What users say they do and what they actually do is often different. Field studies help ground self-reports in direct evidence. Artifact walkthroughs, for example, can help in this regard. In a study the instructors had conducted, they explored the artifacts families leave on their refrigerators, such as Post-it® notes, calendars, and so on. One person actually told them she put theater tickets on the refrigerator so she could find them easily. When she actually tried to show them, she had a hard time finding them, because the tickets were hidden beneath a calendar. It was not that easy in reality.
Personal Thoughts
Though I’ve observed this phenomenon myself in user studies I’ve conducted, their observations powerfully illuminated the limitations of self-reports as the sole method of gathering user data—for example, conducting only in-depth interviews to capture users’ goals and tasks, as in the mental-model approach. While such methods are very valuable, I think it’s necessary to complement them with observation techniques to ground self-report data in users’ actual behavior.
However, when observing user behavior, you often “realize you may need self-report to understand what you observe.”
Personal Thoughts
My take on this is that we really need to mix observation techniques with self-report. Deriving our conclusions based only on observation can be nearly as dangerous as using only self-report. Some observations just don’t speak for themselves, and you need to elicit verbal comments from users in a way that enriches your observations and helps you avoid biases. Otherwise, you risk misinterpreting them. See my UXmatters article on interviewing techniques for more of my thoughts on this subject.
Initially, having a granular focus—that is, a list of specific questions and hypotheses you want to explore in the field—avoids biases that stem from the observer’s preconceptions and “provides a conceptual baseline for data gathering and analysis”.
The instructors illustrated this point through a very enlightening hands-on exercise. They distributed a small basket of candies to each row of participants in the classroom, and each student had to select some candies while the others in the same row observed them. They asked the participants in the rows on the left side of the classroom to observe freely, without any specific area to focus on—exploratory observation without any initial focus. They gave the participants in the rows on the right side of the classroom very specific questions to answer by observing the others—a granular initial focus—for example, observe whether students actually pick a candy and eat it or leave it in the basket; or observe how many candies they take.
When they asked us to share our observations with the class, we noticed that the observations from those on the right side—granular initial focus—were much more specific than the observations from those on the left side—no focus—which concerned many different aspects of their behavior. As one of the participants who were asked to do a free observation, I felt that the number of possibilities for things to observe were overwhelming—I could observe anything—and it would be hard to gather comprehensive data on a subject and determine what was actually worth observing. I also realized that I unconsciously focused my attention based on my own interests. I concentrated on the candy-selection process, because I was fascinated by this aspect, rather than on what was relevant to the study—of which I had no idea. My own preconceptions influenced my data-gathering process.
Personal Thoughts
I think it’s nearly impossible to be free of any preconceptions, so having a clear initial focus can help you recognize any preconceptions you have and “helps ensure adequate granularity of findings.” Avoiding having any focus, with the intention of being as free of preconceptions as possible, seems like an unrealistic approach to me. Even when we are unconscious of our preconceptions and interests, they determine the focus of our observations. Having a granular initial focus helps us to become aware of our preconceptions. We need not stick with our initial focus. If we find our observations contradict our hypothesis, we must go beyond our initial focus. But at least we’ll be able to recognize this if it happens.
I think having a granular initial focus would be helpful in various types of qualitative research, including usability testing.
One of the advantages of field studies, among many others, is the usage scenarios they generate—scenarios that are tailored to specific users and are grounded in their actual needs and usage.
Personal Thoughts
I find this aspect of field studies quite interesting, especially because I’ve seen so many test scenarios—including some I’ve done myself—made up without their being grounded in what users actually do. It’s useless to test users’ success in performing a task they would never do in their day-to-day lives.
However, there are ways of introducing some naturalistic methods in the lab to ground them in users’ real lives. For example, when conducting an e-commerce study, you could use a “forced shopping” methodology—that is, give users some money and ask them to spend it during the session. Another way of using “customized, naturalistic scenarios in the lab,” is to tailor scenarios to users’ specific needs and habits.
Personal Thoughts
These were great ideas to hear, because, for many reasons, including budget or time constraints, it’s sometimes impossible to conduct field studies. These examples show how you can limit some of the biases of usability studies in the lab. An article in the same vein by Jared Spool, “Interview-Based Tasks: Learning from Leonardo DiCaprio,” describes a very interesting approach.
The aim of this course was to address the key challenges we face when analyzing large masses of qualitative data from field studies:
how to make sense of a mountain of field notes, artifacts, photos, and audio and video recordings
how to ensure your conclusions are not impressionistic or anecdotal, with vague or misleading implications for design
which analysis techniques to use to improve the credibility and validity of your findings and keep them focused on design
how to apply scientific rigor to qualitative data
While the high-quality content Siegel presented was very rich, reflecting his deep knowledge and experience with the subject, I found this course more difficult to follow than the one I’d enjoyed so much the previous day, for the following reasons:
There was too much content to assimilate through a one-day course, especially for those who were unfamiliar with the many theories and techniques he presented. Each slide presented such rich content that I would have needed more time to reflect on them and really assimilate them. The course content was so dense that we had little time to spend on the hands-on exercises, which were very interesting.
Some of the instructor’s explanations were so abstract and theoretical, it was difficult for me to comprehend them fully and what they meant in actual practice. The course would have gained in clarity if Siegel had provided concrete examples to really support his theoretical explanations from the outset. He presented many different approaches to exploring data patterns across cases—such as clustering, dimensions, matrices, and networks. I was somewhat frustrated, because these are topics that really interest me with my background in information science and my experience analyzing and grouping all types of information. Because of the abstract way in which Siegel presented the information, I had a hard time making connections with my own experience grouping data from non-directive interviews or doing content analysis, even though I’d followed a similar pattern.
However, I really don’t regret attending this very high-quality course, in which I gained some very interesting insights that I can also apply to other types of qualitative studies, including usability testing and open-ended interviews. Siegel shared this insight with us: There are three major sources of errors in qualitative research that all user researchers and usability specialists should be aware of. These come from Kahneman, Slovic, and Tversky’s “Judgment Under Uncertainty.”
anchoring—“calibration based on first few instances”—After observing the first two or three sessions of a qualitative study, we may tend to look for a pattern and see the subsequent sessions through the lenses of this pattern. We unconsciously look for this pattern throughout the remaining sessions, and this influences the way we interpret the other data, with the risk of our overlooking important new facts. In the two courses, two key points were:
We really have to be careful to avoid premature closure.
We have to tolerate confusion when we don’t see clear pattern.
availability—“greater weight given to easily retrievable instances”—We tend to remember and put more weight on specific situations that strike us during individual sessions and are particularly memorable. For example, if a participant told us a very unusual story, we might tend to remember this participant better, with the risk of giving too much weight to one individual instance.
representativeness—With a small sample, it’s impossible to have a sample that is representative of the entire population.
It’s really important how you take notes and train others to take notes for you:
Capture all information—what users say they do, their intent, and what they actually do. For example—“Says puts theater tickets on fridge, so can find them. Tries to show us, but can’t find them. Put them there just last week. Sees edge of ticket sticking out from under calendar.”
Clearly differentiate between what users say they do and what they actually do. Also, don’t simply write “very easy to find phone numbers on fridge” because you won’t know afterward whether the note referred to what the user said or your opinion about what you observed.
Personal Thoughts
Even though I was already aware of many of these cautions and apply them myself, I realized I needed to be more systematic in the way I take notes during usability test sessions and spend more time training the observers who take notes for me. In a very fast-paced environment like the one I work in, where I have to quickly deliver my recommendations to clients, it’s hard to be so systematic. Sometimes we tend to sacrifice some of the rigor of our process for quickness.
We did a very interesting hands-on exercise in which we had to tag some notes—that is, narrative snippets—from a field study. Then, we explored the tag clusters. The exercise concerned a family’s artifacts on their fridge: Post-it notes, calendars, contact information, and so on. The clustering exercise revealed how many different ways there actually are to group tags and artifacts. For example, the tag post-it might have referred to the type of artifact, but, it might also have referred to the type of content it contained—a phone number, a note, and so on. Depending on a study’s goal, you might choose one way of tagging it or another. What I found interesting was avoiding the premature closure that would result from assuming right away that a type of tagging and clustering was not relevant.
Chatting, Partying, and Dancing
By this point, I was really ecstatic about my CHI experience and thought I had gotten value for my investment of time and money. In these three days, I had learned so much. So, it didn’t really matter to me that I didn’t enjoy the presentations I attended the last day, including the closing plenary. Attending two demanding full-day courses in a row had been exhausting, especially because, on top of that, I spent all my breaks and evenings meeting new people and talking. I just needed some time to relax. The day was quite fun in this respect. I had a chance to do what I needed to do after my fully packed days—chat, relax, eat ice cream, party and go dancing! It actually provided very good balance! My brain just needed some time off.
After the closing plenary, there was a presentation about CHI 2008, which will take place in Florence, Italy. This was quite enjoyable, with fun moments when Ben Shneiderman came out in costume as Leonardo da Vinci and the organizers pitched small soccer balls into the audience at the end.
Afterward, at the 25th Anniversary Party, I ate some good ice cream and said goodbye to my fellow attendees. By then, the only thing I was missing was some music and dancing. My idea of an anniversary party comes with music and dance! Fortunately, while chatting with some friends and other attendees at the IxDA Face to Face, I found out that there would be a party for the student volunteers that night, with a DJ and dancing. I enjoyed dancing there for awhile, then we finished the evening at a nightclub, which to my great surprise and disappointment closed at 1:30 am. That’s so early compared to my French and Montreal habits! In the end, it was again all about cultural differences and localization!
Campus, Cell Phones, and Usability Labs
But my CHI 2007 didn’t end there! I was one of 250 lucky attendees to go on a full-day tour of Silicon Valley companies and universities. My tour included visits to IBM, Stanford University, Nokia, and Yahoo! The experience was great—even though I was less interested in the two first visits, which were too academic or technology centered for my taste.
The visit to Nokia was fun, with a prize raffle and demonstrations of their newest technologies. These included a cell phone that lets you get more information about any item in the street by pointing its camera at it—for example, a car you want to buy or a poster for a movie you want to get more information about.
The visit I enjoyed the most was at Yahoo! in Sunnyvale. As someone who is not based in Silicon Valley, for me, it was a great opportunity to visit one of the best-known Web companies in the world. We visited one of their seven usability labs and I discovered, with great surprise, interest, and puzzlement that each lab consisted of three rooms, separated by one-way mirrors: one for the participant, one for the interviewer, and one for the observers. One of the Yahoo! user researchers explained to me that some user researchers prefer to be in a different room from the participant to avoid their nonverbal language influencing the participant’s behavior. This arrangement was quite new to me, because in all the labs I’ve been using, there are just two rooms, and the interviewer always sits in the same room as the participant. I guess there are pros and cons to each approach, but I’m really wondering whether interviewing a participant from another room would create any discomfort that might inhibit the participant. I’d be very interested in discussing this issue with anyone who has had experience using that kind of setup.
Journeying Through Pakistan, New Zealand, China, France, and More…
One of the great things about CHI was that I met people from all over the world: China, India, New Zealand, Scotland, France, Germany, Lebanon, Pakistan, and the US—to name just a few. I think it’s one of the strengths of CHI in comparison to the IA Summit and the UPA Conference, where the audience is less broadly international and more North American and US centered. It was so great to talk about Pakistani culture, Scotland’s history and relationship with England, and the Indian Silicon Valley in Bangalore. I felt at home in this very international and multicultural environment, with people talking in so many different accents and with so many rich multicultural backgrounds—people who have work and life experience in two or three countries like myself. I really felt welcomed with my French accent! It wasn’t a barrier at all. On the contrary, it was a great way to connect with people. I was also quite surprised to meet many people who can actually speak a bit of French and were interested in talking about France! It was great to reconnect with my native language and country for a while. A bit of fresh air in a busy, fast-paced, and English-speaking environment! Thanks CHI!
From San Jose to Florence?
After these five fully packed and exciting days, I flew back to Montréal in so different a state of mind than I had when I arrived. CHI had really opened new horizons for me. I actually had made the right decision in attending CHI 2007. This experience helped me grow professionally and clarified the next steps I want to take in my career, which will definitely include more emphasis on field research.
Do I want to go to CHI in Florence next year? I don’t actually know. It’s still too early to decide. I need to assimilate and take action on all the things I learned this year at CHI. What really drew my attention and interest this year was the Special Interest Groups and the courses. I’m not sure I would have had so much fun attending the paper presentations or that many of them would actually address my specific needs and interests. We’ll see, but I will definitely have a closer and more interested look at the program when they announce it.
Isabelle has 20 years of experience applying her research and analysis skills to User Experience and career exploration. With a multifaceted education and background in information science, psychology, coaching, personality differences, and relational mastery, she has contributed to innovative approaches in the fields of UX research and career exploration. Isabelle has created a step-by-step approach to career reinvention that goes beyond the limitations of traditional career counseling and has helped hundreds of people find fulfilling careers. Currently based in the San Francisco Bay Area, she has lived and worked in France, Switzerland, the United Kingdom, and Canada and has conducted several international user-research studies. As a user researcher, she has worked for companies such as Yahoo!, Bell Canada, and the French Speaking University Agency. An engaging speaker, Isabelle has given presentations and workshops for professional associations and conferences such as the IA Summit and at General Assembly. She will be speaking at the Association for Psychological Type International in 2021. Read More
David A. Siegel, Susan M. Dray, and Rolf Molich moderated the SIG.
