The Value of Education in Research and Human Factors

Formal Research Education

A formal education in research helps you to understand user research at a fundamental level. For example, research education ensures you understand the differences between

• objective and subjective data—User research data falls into one or the other of these categories. Researchers obtain objective data through direct observation or measurement of participant behavior. Subjective data, on the other hand, is data participants provide by self-report through interviews and surveys. While subjective data is easier to collect, user researchers generally consider objective data to be more reliable and accurate—mainly because of the well-documented confounds researchers often experience with subjective data, such as recall errors and biases.
• single-factor and factorial designs—These are types of experimental design. Single-factor designs have a single variable that a researcher controls—the independent variable. A factorial design includes multiple independent variables and testing encompasses all combinations of the various independent variables. For example, a concept test of a new mobile device with both men and women would be an example of a single-factor design with gender as the independent variable. Adding age group as a second independent variable, then testing all combinations of age group and gender would represent a factorial design.

Single-factor designs rarely provide an adequate understanding of the real world, because they do not reflect the complexity of reality. Factorial designs, on the other hand, can quickly get out of control as you add more factors and more variants of each factor. In our example, having two genders and two age groups generates four distinct groups of people; but if there were three age groups, there would then be six distinct groups. If you introduced another independent variable such as smartphone ownership, the number of groups would increase to 12.

The complexity of factorial designs can dramatically affect recruiting, because your participants must adequately represent each distinct group. A researcher could choose not to study all possible factors when devising a factorial design. However, the researcher should then be very careful when drawing conclusions based on the research. In our example, the researcher might specify that the study’s conclusions apply only to women between the ages of 25 and 35 who do not own a smartphone rather than making general statements about women or even women who do not own a smartphone. The most useful studies tend to be factorial designs that limit the number of independent variables to a reasonable range.

• between-subjects and within-subjects variables—These are both types of independent variables. Between-subjects variables are independent variables that tend to be representative of different groups and get measured only once—gender is a great example. Within-subjects variables are independent variables that differ within a group and are measured multiple times—a good example would be experience with a smartphone. You can treat within-subjects variables as between-subjects variables by separating the characteristic levels of an independent variable into distinct groups, then measuring them once. An example would be recruiting some people who have smartphone experience and some who don’t, then putting them through the same test. Researchers commonly use between-subjects designs to examine differences between groups, while they often employ within-subject designs to study how people might change over time.

Though understanding some of these aspects of research might not be that important in the typical, day-to-day work of a user researcher, when it comes to doing something a little off the beaten path, these are the sorts of things you need to know to innovate research methods, while still maintaining a solid study.

As user researchers, we work on a lot of projects that are somewhat different from the typical research project. For example, on a recent project, we did a usability study that required participants to drive a car while using a device. Our client came to us because they weren’t sure how to go about performing this kind of study. Luckily, we’ve had experience with methods of usability testing that enabled us to handle the additional challenges of performing a usability test in a moving vehicle.

For this study, safety was a major issue. We had to make sure participants could perform all necessary tasks, using the planned product when driving a vehicle on public roads, while still maintaining safety and gathering all the data we needed. To accomplish this, we worked with a licensed California driving instructor and used a dual-control vehicle. The driving instructor acted as a safety driver, monitoring the vehicle and traffic and ready to engage the secondary controls whenever necessary to avoid collisions. Additionally, we scouted and prepared a route that represented a variety of traffic conditions and identified locations that were well suited to performing specific tasks. Finally, we built specialized rigs that let us mount various pieces of audio and video recording equipment in a vehicle.

As we took all of these steps in preparation for performing a usability study in this specialized environment, we had to pay special attention to preventing potential problems that might get introduced into the study. For example, we had to ensure that the safety driver wouldn’t help a participant to complete tasks. We planned how we’d cover contingencies such as participants’ missing turns or making the wrong turns. Plus, we had to counterbalance different types of participants with different times of day to account for different traffic and ambient lighting conditions. Thanks to our understanding of research fundamentals, we were able to manage all of these factors. In thanks for the work we’d done, our client developed a short video documenting our study, which Figure 1 shows.

Having a fundamental understanding of research allows us to be flexible and innovative in meeting our clients’ user research goals and gives us the capability to answer just about any user research question. We can create custom approaches to user research—whether for innovative products or to answer unique research questions—and produce actionable data. Whether we are assessing visual scan patterns on a mobile device, observing the moment-by-moment emotional reaction of shoppers in a retail environment, or studying how people decide what they want to do in an amusement park, we can do user research that fits our clients’ specific needs.