Toward a More Human Interface Device: Integrating the Virtual and Physical

Is Touch the Future of Interaction?

The Korg Kaoss Pad and the Wacom Cintiq pen display both use touch interfaces to enhance and deepen their user experience, providing people with ways of physically connecting with the technology.

The HP TouchSmart desktop attempts to do the same for the home computer, as Figure 3 shows. The TouchSmart leverages the touch capabilities of the Windows Vista operating system, and its bundled software includes a photo editing application, text notes, voice notes, a Web browser, and a music player. Interacting with all of these applications is entirely gestural, letting users point, select, and manipulate digital objects with their fingers—no mouse is required. In the TouchSmart, we can see a movement toward unmediated interaction with digital objects, removing a level of abstraction from such interactions.

And, of course, no overview of touch capabilities would be complete without mentioning the revolutionary multi-touch capability of the Apple iPhone, which offers users a mobile experience unlike any other. Among the features of the iPhone is a soft QWERTY keyboard, which takes the place of the physical keyboard on most mobile devices like the BlackBerry. The advantage of a touch-screen keyboard, of course, is that it’s hidden when a user doesn’t need it, reducing the overall form factor of the device, increasing the size of the display, and increasing the device’s flexibility.

Author and interaction designer, Dan Saffer, has this to say about gestural interfaces in the first chapter of his forthcoming book from O’Reilly, Interactive Gestures: Designing Gestural Interfaces:

”It’s only going to get more complicated as time goes on. Users, especially sophisticated users, are being trained to expect that devices and appliances will have touchscreens and/or can be manipulated by gestures. But it’s not just early adopters: the general public is being exposed to more and more touchscreens via airport and retail kiosks and voting machines, and people are discovering the ease and pleasure these devices can give them. ...

Touchscreens and gestural interfaces take direct manipulation to another level. Users can simply touch the item they want to manipulate right on the screen itself, move it, make it bigger, scroll it, etc. This is the ultimate in direct manipulation: using the body itself to control digital (and sometimes even physical) space around us.”—Dan Saffer

Integrating the Virtual and Physical Experiences

Sophisticated human interface devices can enable users to complete their tasks—whether for work or play—quickly, efficiently, and naturally. Such input devices can also make interactions more interesting, exciting, and ultimately satisfying, which is an advantage from both a marketing and user experience perspective. But these devices may offer other significant benefits to users—in particular the ability to close the gap between physical actions and virtual responses.

Flow State and the Expert Interface

Much of our effort as UX professionals revolves around reducing the learning curve for novices. A touch screen, on which a user can simply touch something to select it, is an inherently intuitive user interface that makes it easy for first-time users to jump right in and start working. But what about expert users? It turns out that touchscreens offer something to those at the other end of the spectrum, too: flow.

Shortly after the birth of the GUI, software designers realized the necessity of including keyboard shortcuts for expert users, so people could avoid the time-consuming actions of digging through menu hierarchies, using only the point-and-click capabilities of the mouse.

Once expert users internalize these shortcuts, they can think less about how they are accomplishing a task and more about performing the actual task. Ideally, an expert user should be able to achieve a high level of focus while working, sometimes referred to as a flow state.

In a recent article on Boxes and Arrows, “Design for Emotion and Flow,” author Trevor van Gorp described how users perceive the flow state.

“In this state of consciousness, people often experience intense concentration and feelings of enjoyment, coupled with peak performance. Hours pass by in what seems like minutes. We tend to enter these states in environments with few interruptions, where our attention becomes focused by a challenge that we’re confident we can handle with our existing skills. Feedback is instantaneous, so we can always judge how close we are to accomplishing our task and reaching our goal. The importance of the task influences our level of motivation and perceptions of how difficult the task will be.”—Trevor van Gorp

Advanced human interface devices like the touch screen can aid this ability to work with technology as an extension of ourselves and our minds—to be able to imagine something, then do it instantly. For an example of how we might achieve this, we can look at some noncomputer technology that relies entirely on touch input—musical instruments.

For expert musicians—those who have mastered the techniques required to create the desired sounds using their instruments—the intense focus that comes during a performance is akin to the flow state expert software users experience. The performer no longer looks at keys or strings, but instead plays his instrument relying entirely on a combination of muscle and pattern memory. The physical connection the performer has with his musical instrument enables him to concentrate on musical expressiveness rather than technique.

What we can learn from examining human interaction with musical instruments is this: Users can connect with truly great physical interfaces so completely the technology can become an extension of the person. This situation is not entirely analogous to that of a computer user, of course. It may not be possible to achieve a flow state when working on certain mundane tasks. However, this focused state of engagement, in which we feel as if we are one with our actions, may be a worthy goal of interaction design.

Wearable Computing

While gestural interaction is almost certainly the way people will connect with computer technology in the immediate future, the integration between the physical and the virtual will not stop there.

The wearable computing project at the MIT Media Lab illustrates ways in which we can incorporate user interfaces even more closely into our day-to-day lives, as part of a user’s clothing.

“To date, personal computers have not lived up to their name. Most machines sit on the desk and interact with their owners for only a small fraction of the day. Smaller and faster notebook computers have made mobility less of an issue, but the same staid user paradigm persists. Wearable computing hopes to shatter this myth of how a computer should be used. A person’s computer should be worn, much as eyeglasses or clothing are worn, and interact with the user based on the context of the situation. With heads-up displays, unobtrusive input devices, personal wireless local area networks, and a host of other context sensing and communication tools, the wearable computer can act as an intelligent assistant, whether it be through a Remembrance Agent, augmented reality, or intellectual collectives.”—MIT Media Lab

Wearable computing may not be so far away. At the Boston Museum of Science, the Tangible Media Group at the MIT Media Lab recently sponsored the event SEAMLESS: Computational Couture.

“Fashionistas and techies unite at SEAMLESS, a fashion show and celebration showcasing emerging designers from around the globe and functional creations that push the boundaries of wearable technology. The Museum transforms into a catwalk for ‘computational couture’ as models strut groundbreaking clothing to live media performances by video artists sosolimited and DJs Eddie O. and Mike Uzzi of Zero G Sounds. The evening evolves into a living exhibition, where guests interact with the designers and their fashions.”—Museum of Science, Boston

New Opportunities for Interaction

As UX professionals, we often take for granted the fact that our users will be dealing with a keyboard, mouse or track pad, and monitor. We think about users’ physical relationship with their digital devices very selectively, if at all. But, as we explore new human interface devices and incorporate new interactions into our designs, we have the opportunity to create deep connections between users and their technology.