Redefining Reality

The line between our physical world and the digital world is continually blurring. While virtual reality completely immerses you in a computer-generated environment, augmented reality (AR)—overlaying virtual information on the real world—has shown more potential. As the technology behind AR continues to rapidly improve, companies and consumers are taking notice.

Apple is rumored to be working on augmented/mixed reality glasses, while Facebook announced this month that it's building AR hardware. And of course there’s Pokémon GO. The hugely popular gaming phenomenon that had millions of people glued to their phones beginning last summer made augmented reality mainstream, and gave consumers a glimpse of what’s to come in the future. Augmented reality isn’t all fun and games, though. Researchers at the Georgia Institute of Technology are studying the technology in diverse domains.
 

Beyond Gaming

Medical error is the eighth most common cause of death the U.S., with up to 98-thousand patients dying each year from preventable mistakes. While teamwork and communication skills are crucial in reducing these errors, interdisciplinary teams of healthcare staff that work together in hospitals are often trained separately. Another problem: training focuses on power differences between healthcare workers.

While augmented reality has been used in medical technical skills training, it hasn’t seen widespread use in nontechnical and team training. Georgia Tech researchers are working with faculty, residents, and students from Emory School of Medicine’s Department of Anesthesiology to develop an AR scenario that puts operating room trainees in each other’s shoes.

“A lot of the bad outcomes in a surgery are because of communication breakdown,” said Maribeth Gandy, director of Georgia Tech’s Interactive Media Technology Center and Wearable Computing Center. “What we want is an environment where you can become the surgeon and see the surgery from their point of view.”

Participants in the simulation acted out emergency situations in a real-life operating room where they were able to see virtual co-workers and hear their inner thoughts. It’s all about empathy according to Gandy. “That guy looked like a jerk, but when I’m in his mind I’m realizing that he’s freaked out because the patient has been under anesthesia for too long.”

After completing the scenario, participants said they saw the potential of augmented reality training for practicing new medical skills, crisis and code simulation training, and improving reaction time in critical situations. They also believed that compared to traditional training, AR could better simulate the emotional nature of clinical scenarios. Researchers are now working toward full mission simulation—realistically simulating an entire medical intervention with a full healthcare team in the operating room.

                                                                 *  *  *

Gandy and IMTC are also working with Georgia State to use augmented reality in social psychology research. They’re trying to figure out what leads people to engage in either peaceful protests or violent attacks, specifically, terrorist acts.

For years, Georgia State has used online text-based scenarios to collect data from study participants, but they wanted to administer more realistic experiments. So researchers collaborated with a screenwriter and filmed scenes with professional actors to create an AR experience. Through video montages and first-person narration, participants took on the persona of a member of a fictional oppressed minority group. They also listened to dialogue from two virtual actors who try to persuade them to join a peaceful, student-led protest, or join a violent resistance movement.

“You can imagine this AR experience being applicable to all kinds of scenarios,” said Gandy. “It’s really relevant in our current world where we’re having a lot of cultural conflict.”

GSU and Georgia Tech will soon start a study using the augmented reality system, and will compare the results to the online version.

Creating Augmented Reality Experiences

Using AR to present educational content has been around for years. It’s shown to be effective and compelling, and at least one school in the Atlanta area has adopted the technology in its kindergarten class. However, designing successful educational content for AR is another story. It requires input from a diverse and essential group of people—teachers, students, technologists and artists.

Three years ago, Georgia Tech and the Verizon Foundation began a partnership to explore the elements of good AR design for kids in school. Using Georgia curriculum guidelines, researchers developed a STEM-related augmented reality application, and built and tested six different tablet-based AR prototypes based on physics concepts. Some of the prototypes are very hands-on—launching a catapult or manipulating a cube—while others have more virtual content. Other design choices include group vs. single user, and game elements vs. no game mechanics.

After testing the prototypes with students and teachers, researchers realized there were some factors that they missed—how AR fits into the classroom environment, the possible overstimulation of kids, and their physical limitations. During the testing sessions it became clear that young students, with their small hands, couldn’t hold the tablet for long periods of time.

Researchers will use the knowledge they gained to create better educational AR experiences. It’s also knowledge that can be shared. “The impact of this kind of project is this body of knowledge that can inform game designers who want to build AR for kids, or teachers wanting to understand how these things might be integrated into the classroom,” said Gandy.

                                                                 *  *  *

Georgia Tech is also making it possible for people to create augmented reality experiences using web technologies. Argon, an augmented reality web browser, is on its recently-released fourth iteration. The goal of the project is to make AR accessible to the widest possible group of developers and users through a web-centric platform.

Blair MacIntyre, professor in the School of Interactive Computing and director of the Augmented Environments Lab for over a decade, started working on Argon in 2009.

“We started thinking about—from a research perspective—what are the things that both makes the web a unique and powerful platform for doing AR?” he said. “With the web ecosystem, the fact that you don’t have to build and deploy apps is a unique capability that makes it easier to produce augmented reality experiences.”

Using the web can also be more secure and cost-effective than building an app, and easier for people who don’t have a lot of development experience.

MacIntyre and Gheric Speiginer, Georgia Tech PhD student in human centered computing, are also exploring how Argon can support users running multiple independently-authored AR applications at the same time, and across different platforms such as mobile phones or head-worn displays. “In the future, augmented reality is going to only really be useful—especially for consumers—if you can see virtual content from a lot of applications or services at once,” said MacIntyre.

The team is working toward making AR an add-on feature for webpages, similar to adding video or a social media button. This opens up more possibilities for developers and consumers, like taking an augmented reality tour of Georgia Tech even if you’re not physically on campus.

The Future of Augmented Reality

Scientists will soon be able to use augmented reality in an unexpected way - to support their research. MacIntyre, Speiginer and Georgia Tech College of Engineering Professor Magnus Egerstedt are developing a tool that would allow scientists to debug their robots.

“We didn’t set out to use augmented reality as a debugging tool,” said Egerstedt. “But now, in a very concrete way, we can use it to see what the robots are thinking and why they’re not acting right. This has been extremely powerful and a little surprising.”

Using a smartphone with the AR debugging interface, scientists can see things like a robot’s sensor measurements, power levels and communication functions. The interface is currently only available to Georgia Tech researchers, but Egerstedt expects to make it more widely available this year.

The interface is a tool in the Robotarium, a one-of-a-kind lab that allows roboticists from around the world to remotely conduct experiments on over 100 ground and aerial robots located at Georgia Tech. They upload their own programming code, watch the robots in real-time via livestream and receive data of the results. To date, over 200 researchers, including those from universities in the United States, Japan, Canada and England have used the lab.

                                                                 *  *  *

Phones and tablets have been the entry point for consumers using augmented reality, and there are already a multitude of available AR apps, from games and tours to virtual car maintenance. Head mounts like Microsoft’s HoloLens could be the next wave for consumers, but MacIntyre says developers first have to ask, what are the things that we would want access to all the time? He believes vertical market applications will come first, perhaps with hospital workers or delivery drivers using head mounts.

“It’s easy to imagine wearing a HoloLens for a few hours here and there as part of your job if it’s giving you some very specific benefits. And then once people get used to that, we’ll start seeing the add-ons.”

Vu Ha agrees. Ha, a software engineer and Georgia Tech graduate in Computational Science and Computational Media, has created augmented reality experiences for independent game studio Funomena. He thinks that the next AR milestone will be replacing mobile phones and monitor-based laptops and desktops with virtual displays.

“Eventually we won’t have a giant machine that we do most of our work on. Instead, we’ll have virtual displays that are all around us any time we want. And this affects every part of software, from games to enterprise to education – anything that’s currently digital,” said Ha.

Removing the hardware could also create more socialization in computing, says Ha, an advantage that augmented reality has over virtual reality.

“Virtual reality technology is very singular; the person in the headset is really the only person enjoying the experience. While ideally, augmented reality allows you to interact with other people, see their faces and communicate directly.”

Privacy and security may soon be a big concern with augmented reality. Not only do AR devices have cameras, they also build 3D models of your space. Also, what are app makers doing with the data collected from your device’s camera? “Over time, this is going to be a much bigger issue,” said MacIntyre. “Not a huge nightmare issue, just something we have to be aware of and help consumers understand, just like other security issues on the web.”


Photos by: Joshua Preston, Rob Felt
Videos by: Ashton Pellom, IMTC
Graphics by: Raul Perez