In Participants’ Own Voices: Using Quotations from User Research Ethically

Example Descriptions and Quotations

Let’s look at the possible consequences of the transformation process that happens as we turn raw data into print-ready material. To demonstrate these consequences, I’ll provide three examples that were inspired by actual events.

Example 1: The Clubber

This example is from an on-site study. The participant was a 23-year-old male. We asked him to view a series of videos on a mobile device, then respond verbally to an online questionnaire asking about his impressions. A moderator, who was present during the latter part of the study, asked him to comment generally about his experience and impressions. She recorded the interview on videotape and typed some notes on her notebook computer during the interview.

Here is an example of a representative participant quotation from the interview, which the moderator transcribed into her typewritten notes:

“I felt like the video gave you the feeling you were in a club. You had all your senses stimulated. The feature enhances the enjoyability of watching video on the phone. Because the screen is small, the feature helps evoke the feeling that you’re there. This phone had better resolution. It helped give a better feeling of being there. Like the music was louder.”

The following quotation derived from the transcribed data, and the researcher added it to a PowerPoint presentation that served as a report to stakeholders:

“I felt like the video gave you the feeling you were in a club. You had all your senses stimulated. The feature enhances the enjoyability of watching video on the phone.”

And finally, here is the quotation that appeared in a published whitepaper on the research results:

“The video gave you the feeling that you were in a club. All of your senses were stimulated.” (An image of a cheering crowd in a nightclub accompanied the quotation.)

In the transformation that occurred between the transcript and the presentation, we see a trimming down of the quotation; between the presentation and the whitepaper, further trimming and refinement of the verbiage. Furthermore, we see the addition of an image that provides a visual context for the quotation. Should we be ethically concerned about the changes that transpired between the first and third iterations of this quotation?

In my view, these changes are in line with the original intent of the quotation, and a reasonable smoothing of the remarks into something more readable has occurred. Moreover, although the image accompanying the quotation was not from the video that the moderator recorded during the session, the context is similar to what the participant experienced during the study.

Example 2: The Dancer

This example is from on-site user research. The participant was a 43-year-old, stay-at-home mom. We asked her to view a series of videos on a mobile device, then answer questions about her impressions in a print survey. No moderator was present when the participant recorded her responses to the questions in the survey. Following the study, the researchers transcribed the survey responses into a spreadsheet.

Here is an example of a written response from the survey sheet:

“I remember this video from last time. I liked it more before—which really made me feel more involved and like I wanted to dance. Not so much this time though. It was a big difference in the music. It’s like the difference of making you want to dance and not dance.”

The following quotation appeared as a bullet point on a PowerPoint slide in a research report:

“It was a big difference in the music. It’s like the difference of making you want to dance and not dance.”

Finally, here is how the quotation appeared on a printed sheet of collateral for a customer event:

“It’s like the difference between making you want to dance and not dance.”—Kimberly, 19 (An image of two young girls looking at a tablet device appeared beside the quotation.)

In the transformation that occurred between the survey responses and the report, we see a paring down of the quotation; between the report and the print collateral, further trimming of the text.

More important, we see an image in which the subjects belong to quite a different age demographic from the participant, and the text identifies the participant as a teenager. Both misrepresent the potential appeal of the technology to a younger audience. Also, as a researcher, I found it quite striking that this technology compelled a more mature, stay-at-home mom to dance—a sentiment that gets lost in the final version of the quotation.

Example 3: The Skydiver

This example came from an on-site study. The participant was a male, 37-year-old operations director. We asked him to view a series of videos on a mobile device, then tell us his impressions by verbally responding to an online questionnaire. A moderator, who was present during the latter part of the study, asked him to comment generally about his experience and impressions. The moderator recorded the interview on videotape and typed some notes on her notebook computer during the interview.

This is a participant quotation that the moderator captured in her typewritten notes during the interview:

“It was like supplementing or helped me recall a true experience. It kind of brought me back to the memory of something—like driving a cool car or like skydiving. Yeah, in that last video, the—the vibration of the wind, I remember that. So it brings you back to that true life experience.”

Here is how that quotation appeared in a bullet point on a PowerPoint slide in a research report:

“It was like supplementing or helping me recall a true experience. It brought me back to the memory of something—like driving a cool car or skydiving.” (A still image from a video of a skydiver appears on the slide.)

Finally, here is how that quotation was represented in printed collateral for a customer event:

“It feels like I’m there again!”—Arthur, 22 (An image of a snowboarder jumping appears on the sheet.)

In the transformation that took place between the moderator’s notes and the report, we see a smoothing process that removed a stutter and some redundant information and made the quotation more concise. Stuttering is common in human speech, but in print, text is easier to read when we remove it. However, in the final transformation to print collateral, the text has lost its authenticity. A statement the participant expressed in past tense is now in present tense—as though the participant were describing what is currently happening. The collateral material identifies the participant as belonging to quite a different age demographic, and the imagery accompanying the quotation presents a different visual context from that the participant experienced during the study.

Best Practices

A useful approach to handling quotations from user research comes from another discipline that often faces the same dilemma: journalism. Every day, journalists must accurately represent the statements of their interview subjects while rapidly producing perfectly written representations of broader human opinion.

An article by Killenbery and Anderson [1] suggests a practical test for alleviating the multitude of ethical dilemmas facing journalists in the handling of quotations. They suggest adapting Sissela Bok’s test of publicity, [2] which asks journalists to willingly imagine themselves under scrutiny by asking: Is my behavior such that I’d be comfortable having all of my audiences—interview subject, editor, public, and self—simultaneously monitor my choices, knowing that someone might later ask me to justify my actions? 

Asking ourselves how research participants might feel upon seeing their quotation in print is quite useful. However, while a journalist is actually quite likely to have this happen, UX research participants rarely see the outcomes of studies in print. As a consequence, some might not feel compelled to observe the same level of professional rigor. But we should. What if a participant did see that presentation? How would that person feel? Would he respond with indignity—or tacit acquiescence? Would she feel empowered and glad to have her voice heard? As a researcher, when speaking for the user, I hope my work holds to the highest standard. 

References

[1] Killenberg, G. Michael, and Rob Anderson. “What Is a Quote? Practical, Rhetorical, and Ethical Concerns for Journalists.” Journal of Mass Media Ethics: Exploring Questions of Media Morality, Volume 8, Issue 1, 1993.

[2] Bok, Sissela. Lying: Moral Choice in Public and Private Life. New York: Vintage Books, 1979.