From Manufacturing to Medical, Mixed Reality is Improving Complex Interactions

One of the most exciting user experience (UX) modalities to design for is that of Virtual Reality (VR). Far from being the gimmick that it was once thought to be, VR and related interaction styles are quickly becoming a viable solution for many complex interactions, ranging from retail display to life-saving surgical interventions.

But how did this technology evolve, and where is it going?

For many people, the 1999 movie The Matrix was a mind-blowing first introduction to the concept of VR. The idea that a computer could generate sounds and images equivalent to what could be experienced in real life was a novel concept to many. Even more amazing was the fact that engineers were already working on VR technology, and have made dramatic strides in the 20 years since the movie was released.

"If real is what you can feel, smell, taste and see, then real is simply electrical signals interpreted by your brain." 

– Morpheus, “The Matrix” 

The concept of VR, at least in science fiction literature, goes back to a book called “The Judas Mandala” published in 1982. The timing was such that computers were just becoming available for home use, and the videogame industry began to grow. In 1992, Neal Stephenson’s pivotal novel “Snow Crash” came out, introducing us to the concept of a Metaverse where people could work and live, interacting with others through a totally immersive computer-generated medium.  

Fast forward to today, when one of the biggest companies on the planet is betting big that the “Meta” will become the new normal on the web. On a broader scale, the concepts of Mixed Reality (MR) or Extended Reality (XR) cover a wide range of interaction possibilities along the Reality-Virtuality (RV) Continuum. Moving slightly from a real-world environment, we encounter Augmented Reality (AR), which overlays computer imagery on the real world to provide additional information and context. (Augmented Virtuality (AV) is a lesser-known concept where real-world objects may be brought into a virtual world.)

No Holodeck Means Hardware is Still Essential

Until we have a working Holodeck, interactions in the MR space will require some level of personal hardware to achieve. Socially speaking, AR applications provide the least amount of friction to be viable. As the Pokemon Go craze of 2016 demonstrated, a smartphone with a camera is all that’s needed to deliver an engaging AR-based experience. 

While it wasn’t ultimately successful, Alphabet’s Google Glass project was a very ambitious attempt to provide a viable consumer-grade wearable device for AR. Despite the misstep, sectors from retail to gaming have embraced AR as a very viable modality. For instance, Massachusetts-based OnPoint Surgical is demonstrating use of AR technology for surgical assistance.

VR requires much more “socially invasive” hardware to achieve its effects, typically in the form of a full eye-covering headset and hand-held controllers. Many people are comfortable with looking at their phones as they walk down a street or sit at a table in a coffee shop, but less so wearing a VR headset in public (besides in gaming circles). 

Still, use cases for VR technology abound. Agricultural equipment manufacturer John Deere has a room-sized VR “cave” solution used for training and simulation. Virtual tourism has gained popularity during the pandemic years, allowing people to visit places without the need to get on an airplane. Doctors have been using VR to help treat phobias and anxiety for many years.  Even the traditionally conservative medical field has begun seeing surgical training systems created in VR, to let surgeons become familiar with new techniques and equipment before interacting with patients.

Whatever the industry, AR and VR techniques are poised to be huge disruptors. Companies like Unity Technologies have released SDKs and built communities around their MR offerings, lowering the barrier to entry for individual developers and small companies alike. Designing for these new experiences will require very particular skill sets and techniques, which we’ll cover in a future blog.