Home » Uncategorized » Augmented and Virtual Reality in Medicine: 6 Applications We’re Keeping Our Eye On

Augmented and Virtual Reality in Medicine: 6 Applications We’re Keeping Our Eye On

If talk of augmented and virtual reality brings to mind comically unfashionable goggles, a dearth of useful applications, and headache-inducing graphics, we’d understand. Until two or three years ago, we thought the same—they were technologies of the future and always would be.

But we’ve come full circle in our thinking and so should you.

Consider this: Consulting firm Deloitte Global, as part of its 2016 outlook, forecasts that this year will be VR’s first billion-dollar year. Looking ahead, Goldman Sachs analyst Heather Bellini predicts that AR and VR could easily generate up to $80 billion in revenue, including hardware and software, by 2025. Facebook’s Mark Zuckerberg has a $2 billion bet on virtual reality by way of his 2014 Oculus acquisition. And Magic Leap, an augmented reality startup affiliated with Google, recently raised money at a lofty $4.5 billion valuation.

And helping to lead the way? Game developers and the military aside, we’re increasingly seeing hospital clinicians and medical researchers at the forefront of this innovation wave.  They’ve been particularly quick to imagine how augmented and virtual reality technologies could potentially reshape the patient experience, improve clinical outcomes, deliver innovative new therapies, and better train healthcare professionals.

In this article, we overview 6 of the most interesting applications we’ve seen piloted or adopted in healthcare settings. We owe thanks to Dr. Justin Barad, Pediatric Orthopaedic Surgery Fellow at Boston Children’s Hospital, who kindly offered us commentary and insight. Dr. Barad is also an editor at Medgadget and an authority on emerging healthcare technologies.

1. Education and training. Clinical procedures are often difficult for physicians to master, yet obviously nobody wants a novice surgeon. In Canada, clinicians affiliated with the University of Montreal have developed a useful but somewhat simple virtual reality simulator, SIM-K, that teaches young doctors how to perform complex knee replacements. The system incorporates a screen as well as haptic sensors that mimic the buzzing of saws and drills. André Blain, president of the company behind the research, says he’s hoping tools like this can change the way many surgeries are taught. He also reports that the team’s now working on similar technologies for neurosurgery and spine surgery.

The study of human anatomy has also emerged as a popular application of augmented and virtual reality technologies. New tools promise to enable medical students to “peel” away skin and muscle and observe the placement and functioning of bones and internal organs. We’re intrigued by the augmented reality collaboration between Microsoft’s HoloLens team and researchers at Case Western Reserve University and Cleveland Clinic.

2. Surgical planning. It shouldn’t be a surprise that complex brain surgeries are planned in excruciating detail.

Now, neurosurgeons at the Mayo Clinic, Ronald Reagan UCLA Medical Center, and a handful of other hospitals are planning their procedures with virtual reality technology. Wearing Facebook’s Oculus Rift goggles, they can virtually navigate through 3D models built from MRI and CT scans—looking, for instance, for the safest path to a hard-to-reach tumor.

Dr. Robert Louis, a neurosurgeon at Hoag Hospital in California who’s trained on Surgical Theater’s Surgical Navigation Advanced Platform (SNAP), is a fan. “It’s a game changer,” he observes. “Kids playing video games have had this technology for 10 years. It’s amazing we only now get it in the [operating room].” He further suggests that patients and their families find it easier to understand his surgical plans when they can virtually experience them.


Photo courtesy of UAB Division of Orthopedic Surgery

3. Telementoring. Imagine you’re a surgeon in the midst of a complex procedure that’s pushing the limits of your training and experience. It’s a scary thought, no? And yet, military surgeons, surgeons in developing countries, and even surgeons in remote parts of the U.S. face this reality all the time. They’ll ideally have consulted ahead of time with a mentor by phone or videoconference, but that kind of advice, if it’s even available, only goes so far—especially if anything unexpected happens during the procedure.

Now, using augmented reality technology developed at the University of Alabama at Birmingham, surgeons can have their mentor virtually join the operation while it’s underway. VIPAAR, short for Virtual Interactive Presence and Augmented Reality, uses Google Glass to superimpose a real-time projection of the mentor’s hands into the surgeon’s field of sight—from across the hall or around the world.

Dr. Brent Ponce, one of the lead VIPAAR researchers, conceived of the use of AR for mentoring from a distance while serving with the military in Iraq. “This could be a lifeline—the ability to dial a friend for assistance,” he notes.

4. Patient experience. Clinical outcomes are paramount, but most hospital quality metrics attempt to consider the larger patient experience. How was I treated and how did you make me feel while I was a patient? Augmented and virtual reality technologies almost certainly have a role to play here.

Tufts Medical Center Virtual Reality

Tufts Medical Center Pre-Surgery Virtual Reality | Photo courtesy of Tufts Medical Center

Boston’s Tufts Medical Center, for instance, will soon be rolling out a virtual reality education experience for pre-surgery patients. “The idea is to make sure patients are familiar with what they’re about to undergo,” says Carey Kimmelstiel, director of the hospital’s Interventional Cardiology Center. “It doesn’t have to be a scary process.”

At Tufts, doctors will use virtual reality technology to show pre-procedural consent, introduce the surgical team, and explain the operating room’s equipment. Kimmelstiel believes this familiarity will be particularly valuable to those undergoing treatment without anesthesia, and who will thus be completely aware of their surroundings during the procedure.

We’re also following an interesting effort underway at C.S. Mott Children’s Hospital in Michigan to explore the benefits of virtual reality amongst pediatric patients confined to their beds and alone for hours on end. It appears that “escaping” to imaginary worlds for recreation could significantly reduce patient boredom and depression.

5. Treatments and therapies. Depression, anxiety, addiction and other mental health conditions affect one in five Americans and represent one of our country’s most troubling health challenges. It’s exciting that researchers increasingly believe that virtual reality technology could provide relief and potential solutions to some of these disorders. Heroin addiction, depression, post-traumatic stress disorder (PTSD), phobias (e.g. fear of heights, fear of flying), phantom pain, stress, and anxiety are amongst the conditions where active research and trials are underway. Early results appear promising.


Photo courtesy of Ari Hollander and Howard Rose, copyright Hunter Hoffman, www.vrpain.com

To treat a fear of heights, for example, a virtual environment of a shopping mall with seven floor levels and partially glass floors can be simulated. The sixth floor includes a section without a safety railing to induce a safe but high level of anxiety. The simulator for those fearful of flying likewise recreates an entire airport and flight experience.

Even victims of severe skin burns are benefiting from virtual reality technology. Hunter Hoffman of the Virtual Reality Analgesia Research Center at the University of Washington has helped pioneer the development of SnowWorld. It’s the first immersive virtual game designed for reducing pain.

“When they put you in SnowWorld, the snowmen throw snowballs at you and you have to throw snowballs back at them,” says Caleb Springer, a burn patient. He notes, “When I was in SnowWorld, I didn’t think about the pain at all.”

6. Augmented reality-enhanced surgery and patient care. Healthcare, by its nature, is extremely information intensive. The challenge, however, is helping clinicians get the right information at the right time and place so they can deliver effective care.

Dr. Steven Horng Using Google Glass at BIDMC

Dr. Steven Horng Using Google Glass |Photo courtesy of Danielle Duffey/BIDMC

In Boston, physicians at Beth Israel Deaconess Medical Center have experimented with secured Google Glass goggles to overlay helpful clinical data in their field of view as they speak with and examine patients. The goggles are triggered by a QR code outside the patient’s room and can pull in electronic health record information ranging from nurses’ notes to lab results. “I believe wearable computing will replace tablet-based computing for many clinicians who need their hands free and instant access to information,” says CIO Dr. John Halamka.

European app maker Droiders wants to bring augmented reality into operating rooms with a Google Glass platform called “MedicAR”. Surgeons direct their goggles at a target on their patient—typically a small temporary tattoo— and can then view a real-time augmented reality display on their screens. To their eyes, they see the procedure illustrated step-by-step with CT/MRI images, notes, and diagnostic information perfectly superimposed over their patient’s skin. “You don’t need to go in blind anymore,” remarks Stanford University’s Dr. Homero Rivas. “Now, we have an educated impression of where a mass is. We can better understand exactly where to make an incision so we can create less trauma.”

The European Space Agency (ESA) is thinking even bigger. They’re testing an augmented reality system to help astronauts who may be called on in emergencies to diagnosis medical conditions and perform basic surgeries. “Although medical expertise will be available among the crew to some extent, astronauts cannot be trained and expected to maintain skills on all the medical procedures that might be needed,” says Arnaud Runge, a biomedical engineer overseeing the project for ESA. The Computer Assisted Medical Diagnosis and Surgery System (CAMDASS) will use ultrasound technology to overlay information and images into a head-mounted display. It’s designed to function even when out of communication with earth-based flight controllers, an advantage for deep space voyages.


James A. Gardner

James A. Gardner

James A. Gardner, @jamesagardner, is a Boston-area sales and marketing professional with a passion for consumer technologies and all things health, web & social. He started his career with Procter & Gamble before earning his MBA at Northwestern University's Kellogg School of Management. Since then, James has served senior clients as a consultant with McKinsey & Company, led complex digital projects with Boston-area agencies, and built several high-performing marketing teams. He’s also been published and quoted in multiple professional publications, most recently CMSWire, CIO.com, eHealthcare Strategy & Trends, and MedTech Boston.

Similar posts


  1. Rachel says:

    Incredibly informative article — thank you! How do you think large commercial availability of AR/VR devices will affect clinical research and testing of these applications?

Leave a Reply

Your email address will not be published. Required fields are marked *

Follow us!

Send this to a friend