Show and Tell: Imagining the User Experience Beyond Point, Click, and Type

Audio Data

Just as big strides in video- and photo-recognition software are changing the landscape of visual data, developments in audio recognition and analysis are similarly reshaping the way computers enter the world of sound. In fact, in some ways, computers already know more about audio than we do.

A New Yorker article by Malcolm Gladwell from October 2006, “The Formula,” described the audio software of startup Platinum Blue:

In a small New York loft, just below Union Square...there is a tech startup called Platinum Blue that consults for companies in the music business. Record executives have tended to be Humean: though they can tell you how they feel when they listen to a song, they don’t believe anyone can know with confidence whether a song is going to be a hit, and, historically, fewer than twenty percent of the songs picked as hits by music executives have fulfilled those expectations. Platinum Blue thinks it can do better. It has a proprietary computer program that uses ‘spectral deconvolution software’ to measure the mathematical relationships among all of a song’s structural components: melody, harmony, beat, tempo, rhythm, octave, pitch, chord progression, cadence, sonic brilliance, frequency, and so on. On the basis of that analysis, the firm believes it can predict whether a song is likely to become a hit with eighty-percent accuracy.”

Platinum Blue’s software has successfully predicted such unexpected megahits as Norah Jones’s album “Come Away with Me” and “Crazy” by Gnarls Barkley. Based on their data, Platinum Blue’s consultants advise record-industry executives and producers on how they can improve their work, if a song doesn’t meet the criteria for a hit. Such algorithmic understanding of human experience means that, while such software may not be capable of truly learning our preferences, it may be able to anticipate them.

Voice recognition is another intriguing input technology now coming of age. There are numerous software products that are slowly, but surely improving the computer’s ability to take human speech as input. One such piece of software currently receiving plenty of attention—and advertising dollars—is Microsoft Sync, which recognizes the voice commands of drivers, allowing them to make mobile phone calls or play specific songs in their iPod music collections hands free.

Show and Tell Instead of Point and Click

The mainstream application of such visual and voice-recognition technologies in combination can support a much richer user experience by increasing the types of input we can use with our computer systems. Currently, the methods of interaction we encounter in many digital products, services, and devices are highly unnatural and artificial. We adapt our behavior to these methods of input and point, click, and type , because it’s necessary if we want to avail ourselves of the capabilities these products offer.

But when we interact with other people, we rely upon methods of human communication that have evolved throughout our history—such as our natural storytelling abilities. We show and tell. The rich information we convey through verbal modes of communication—for instance, through our body language, facial expressions, tones of voice, gestures, and so on—will become more important as we integrate visual and voice-recognition technologies into our user experiences.

Imagining the Future of User Experience

As computers become capable of recognizing non-text data input, we can imagine the potential for improvements and enhancements in user experience across a wide range of digital products and services. For example, perhaps you’d like to purchase a replacement for a malfunctioning auto part from eBay, but know neither the manufacturer’s name nor the model number. You could show the computer what you mean—holding the part in front of your video camera to input that data.

Or picture the glee of the Daily Show writers of tomorrow who need only ask their computers to find them relevant sound bites and video clips of the last dozen times a politician contradicted himself, without requiring any cataloguing or tagging by a human.

And, since social networks like Facebook, MySpace, and LinkedIn already connect our personal and professional information with our portrait photos, it’s easy to see the possibilities for facial-recognition software to enhance information exchange at networking events or other professional gatherings. Taking a snapshot of a speaker at a podium, a demonstrator in a company’s booth, or a convention attendee with your camera phone could immediately reveal to you the person’s name, resumé, portfolio, or other background information. A surreptitious snapshot would mean you’d never again embarrass yourself by failing to recall the name of someone you’d previously met. Of course, the potential for abuses and breaches of privacy with such facial-recognition technology is high. And, it’s likely that, as we work through such issues, the powers that be might impose severe limitations on the uses of such software.

As we slowly progress toward freedom from the keyboard and technology enables us to convey information to computers without requiring us to type, designers of user experiences will not only need to develop tools and techniques for managing these new kinds of interactions, but also begin to consider the social concerns and consequences these user experiences will bring to the fore. And while these fantastical possibilities might seem far off, their real-world analogues are, in fact, much closer to becoming part of everyone’s reality than you might think.