What is the next step in virtual reality technology? It’s “mixed reality,” the merger of virtual reality with the real world, according to R. Benjamin Knapp, PhD, director of the Institute for Creativity, Arts, and Technology (ICAT) at Virginia Tech during the “How Close Can We Get to a Holodeck event at Moog fest Friday morning.

So, is a Star Trek holodeck environment really possible?

“We are at the edge of what virtual reality is going to bring us. We’ve been at it for a quarter of a century but haven’t gotten it.” But researchers are at the cusp of technologies that will make VR more likely holodeck experience.

They’re trying to get as close as possible in an environment called “The Cube” at Virginia Tech. The four-story environment has 24 motion capture cameras and 150 speakers. Knapp pointed out that immersive sound is as important as immersive video in creating a realistic environment.

Headset Blues

Currently VR environments require wearing some sort of headset, although the use of 360° screens could reduce the need to repair of IMAX glasses. Newer versions of headset technologies however, are getting much better, Knapp said. That’s important if you want to expose hundreds of schoolchildren to VR, sense headsets can cost $1000. Cheap 3-D glasses make that a lot easier.

Virginia Tech is one of only five institutions that received a Microsoft grant to test its Hololens, which he says is very impressive, and it still relies on more conventional although smaller headsets in its virtual-reality environments.

The problems VR researchers have to face are daunting. You need he said, light that doesn’t reflect off of anything, movement sensors that react instantly, and mind-reading software run on extremely fast processors. You also need to be able to interact with other people in the space.

To get anywhere near real holodeck virtual-reality would require “breaking physics,”Knapp said.

While science fiction writers can do that, VR researchers have to stay within the bounds of what actual technology can accomplish. Still, that’s actually pretty impressive.

For instance, biosensors combined with computers processing their data can approach mind-reading by assessing a person’s emotional state via face flushing, heart rate, and sweating. Multiple speakers strategically placed in create location-based sound. Models of reality based on laser scanned environments such as buildings can reproduce those environments faithfully.

Are there practical uses?

So just what are the practical uses for these advanced virtual-reality environments Knapp believes will “spring up everywhere,” as the technologies progress?

They can be a very effective way to explore data, he explains. Using data from the National Oceanic and Atmospheric Administration VR can allow researchers to experience what it is like to be inside a tornado and even look up the funnel. “You can see all the different aspects of her tornado,”Knapp said.

Stadium builders can experience what crowd noise actually sounds like during a football game – important because crowd noise can affect game outcomes (they wanted loud).

Construction companies, builders, architects, and real estate agents, could walk through VR renderings of homes office buildings and other structures. In one case,Knapp said, builders of a senior living facility in Ireland actually discovered a flaw in a proposed building revealed by a VR walk-through.

Designers can create environments to experience the coastal estuaries of North Carolina without getting sand in their toes.

“You can experience those things in the same way by moving a mouse or pointing at the screen to navigate an environment,”Knapp said.

The next step

Knapp’s background in the field is very impressive. He holds twelve patents and is the co-inventor of the BioMuse system, which enables artists to use gesture, cognition, and emotional state to interact with audio and video media.

As the director of technology at MOTO Development Group in San Francisco, he managed teams of engineers and designers developing human-computer interaction systems for companies such as Sony, Microsoft, and Logitech. He co-founded BioControl Systems, a company that develops mobile bioelectric measurement devices for artistic interaction.

He said the next step in virtual-reality development will be combining VR with the real world. By that he means you might be anywhere in the world and still be able to sit down with another researcher who was in a real facility and interact with them as if you were there.

Knapp said during the question-and-answer period at Virginia Tech is involved in some collaborations with other universities and recognizes that the University of North Carolina at Chapel Hill, for instance, has an impressive VR research program.