The User Interface Vanishes: How Smartware Will Change the User Experience

The Modern Personal Computing Ecosystem

To understand how the future we’ve described in our column will become possible, let’s first examine what a standard, modern computing setup looks like.

Smartphone / Mobile Use

For most of us, our multi-purpose smartphone is the device we use the most. It bridges our work and personal contexts. While a smartphone is a poor substitute for a notebook or desktop computer, it provides uniquely mobile services such real-time GPS navigation, a camera that lets us capture photographs and video, and access to just-in-time services such as ride-sharing apps.

We now get our greatest computing utility from mobile. By superimposing a software layer over our experience as we travel through the world, the smartphone enables us to enjoy the greatest benefit from a wide variety of technologies. But, while the smartphone is currently the enabling device for mobile experiences, other technologies may absorb its capabilities over time. What is most important is the ability to leverage just-in-time computing capabilities, using a layer of functionality that overlies our movement through the world.

Notebook / Creative Use

For those of us who need to accomplish work using a computing device, our notebook or desktop computer is our primary device for getting things done. It’s necessary for a wide range of work and personal activities—as varied as filing our taxes, word processing, and creating PowerPoint presentations. These computers provide a greatly enhanced experience over the smartphone for most things that don’t require just-in-time mobile services.

The only reason we still have notebook- and desktop-computing devices is to enable various types of creation—whether 3D animations, financial spreadsheets, or presentation slides. Creating these in an efficient way requires a robust computing environment that includes a large display, a full, sturdy keyboard, and a mouse, trackpad, or tablet rather than the gestural interactions that define small-screen, mobile computing. For the foreseeable future, we will continue to have creative computing needs that require a more powerful computer.

Tablet / Consumptive Use

Our tablet is a supplementary device that we generally use for consuming instead of creating—unless we augment it with input devices that turn it into a makeshift notebook. Tablets often have higher-resolution screens, and their larger screens provide a better computing experience than the smartphone. Plus, tablets are more ergonomic, convenient, and comfortable for consumptive use than notebook computers are.

We spend most of our computing time consuming content—watching videos, listening to music, browsing Facebook, or reading in a Web browser or app. These activities require minimal power and resolution in a computing device. Some are important functions that translate our past behavior into a more modern context. After all, we’ve been watching television for about 70 years and listening to music on mobile players for almost 40. Others are consequences of our current technologies. For example, while Facebook feeds may seem ubiquitous today, we’ll likely leave them behind in the future. The bottom line, though, is that we’re primarily consumers when using our computing devices. Thus, our computing needs for notebook computers are often similar those for our mobile devices—at lower levels of fidelity—or tablets for a richer experience.

Specialized Devices / Narrow Use Cases

There are so many of these devices and they are so varied that it is just about impossible to consider all of the possibilities. However, most of us have some devices that support specific, specialized tasks or extend our core computing ecosystem to other user interfaces around our home.

By breaking down our computing needs into these use contexts, we can begin to see patterns in our computing needs and why we need certain devices. The smartphone brought a revolution because it offers so much computing utility that is appropriate for mobile devices—in the form of just-in-time services and entertainment. But we still need a more robust environment in which to create and work. Paradoxically, our computing needs are much simpler than those that are supported by the complex, crowded ecosystem we currently inhabit, shown in Figure 1, but nuanced enough that our future of invisible interfaces must support diverse ways of computing.

Image source: Domenico Loia on Unsplash

Envisioning the Smartware Ecosystem

Considering all of this, how will user interfaces disappear in the future?

Input Devices

A substantial portion of our computing devices’ physical footprint relates to input—the keyboard and mouse or the smartphone screen that, while clumsy, is worlds better than the face of a smartwatch. These input devices are changing—thanks to advances in both hardware and software. On the hardware side, we’re moving toward integrated rather than external computing devices. We see this in prominent use with features of consumer products such as Apple’s Siri or Amazon’s Alexa that use voice control to replace typing, clicking, and tapping. There are also examples of using a person’s eye movements to communicate intent to a machine. While these technologies are still in their early days and far from stellar, they give us a glimpse into a future of hardware that blends into our environment and even ourselves, freeing us from the need to hold something in our hand or bend our head down unnaturally to see it.

Plus, future computers will simply need fewer inputs. They may know what we want and need better than we do. They’ll often know better ways to do things, giving us the eye of an artist or the judgment of an analyst without our needing to do a single, simple thing. The requirement to provide input to power on a machine will, in many cases, simply go away. Instead of machines’ requiring human input, software will power the computer’s ability to serve us, so the need for clunky, cumbersome inputs will disappear as well.

Output Devices

The monitor as an output technology predates computers, going back to the days of early television and even film—arguably, all the way back to cave paintings! While audio outputs also play a role, our eyes watching a display is viscerally core to the computing experience. Crucially, the monitor has always been an external device—one that sits on our desk or in our hand or on our lap. Thus, computing requires the presence of a device that is alien to the way we would otherwise go about our day. But this is now changing thanks to advances in virtual-reality technology.

Remember Google Glass, which looked like a futuristic pair of eyeglasses? In 2013, this computing device enjoyed a hype-fueled release. Many technologists purchased Glass and proudly tweeted using the device. Yet, it quickly faded away as just another over-hyped new technology product. The reason so many smart people were excited about Glass was the superior idea behind it: shifting device output from a large screen to a head-up display.

But the technology came nowhere near to being able to deliver on the concept. The experience of using Glass and the capabilities the device offered paled in comparison to that we foresee in future devices. Today, virtual-reality (VR) systems such as the Oculus Rift, HTC Vive, and Sony PlayStation VR are introducing us to a different take on integrated outputs—one in which the content is much richer, as well as more worthwhile. Titles include Eagle Flight, Arizona Sunshine, and Minecraft VR. The problem with VR today is that, unlike Google Glass, the systems are big, chunky wraparounds like those depicted in Figure 2. These headsets have more in common with a helmet than a pair of glasses. We’re heading toward a future now—in years, not decades—in which solutions will have more ergonomic, natural form factors, even as the quality of the experiences they provide continues on an upward trajectory.

Image source: Guido van Nispen on Flickr (CC BY 2.0)

Advanced Processors

Of course, nothing happens in computing without a processor. Thanks to perpetual advances in hardware and software, the footprint of the core engine that drives our computing devices has gotten ever smaller, while at the same time, more powerful. As much as we continue to muddle through with clumsy input and output devices, the engines of these wonderful machines are already great and will just keep getting better. Famously, the computing power inside a garden-variety smartphone today is significantly greater than the entire computing capability that ran the Apollo 11 mission, which put the first people on the moon. This is the very reason why our smartphones can be so small, yet do so many helpful things.

Historically, or at least in the years before the smartphone revolution, the processor and memory in a desktop computer were separate from the interface devices, enabling better ergonomics. The monitor, keyboard, and mouse could be positioned without concern for the often-bulky innards that powered the user experience. Today, each device in our more complicated computing ecosystem generally has its own integrated processor. The ability to store our files on portable, external storage devices or in the cloud mean we no longer need cavernous storage capacity on every device. This distributed system uses computing resources more sensibly and efficiently and, at the same time, decouples the extent of our machine’s capabilities from whatever its physical form factor is able to support.

This model is helping to define the smartware computing ecosystem. We may wear a processor on our wrist or something like a key fob that powers entirely distinct input and output devices. We already know that the processors of the future will be more and more powerful. The invisible interfaces they power will not necessarily be integrated into the same devices that accept our inputs and serve their outputs.

Image source: COMSALUD on Flickr (CC BY 2.0)

Conclusion

We believe the future of computing will be invisible—and integrated. Instead of having a notebook computer and a smartphone and a tablet and a smartwatch, we’ll have computing capabilities that fit organically and seamlessly into the contexts of our lives. For mobile and consumptive computing, we’ll communicate with our machines with our eyes and voice more than with our hands. We’ll still need a screen—perhaps in the form of an apparatus we wear like glasses similar to the fanciful user interface shown in Figure 3 or an accessory we wear like a watch. However, it will always be with us as we move through the world—more like clothing we wear than a pocketbook we carry.

Yes, there will also be computing environments in which we create, which require high-fidelity inputs and outputs, as well as more processing power. But even these will get better as their screens blend into our environments and improve in resolution, the combination of artificial intelligence and more natural input paradigms reduce the need for typing and clicking, and transitions from our creative platform to mobile devices happen seamlessly.

The digital world has already become nothing short of magical, but the years and decades ahead will usher in a world that—for all but the youngest of us—will feel like it’s from the pages of a science-fiction book rather the reality of our everyday lives. Let’s enjoy and make the most of the experience together!