Augmented Reality Surgery: How a Mobile Device Can Save Your Life

Scott Jung Senior Editor, MedGadget
Make It Wearable logo

Over the past few weeks, we’ve explored how the mobile device you carry in your pocket every day can be a powerful medical gadget. It’s full of technology that makes it easy to collect information about your health, send updates wirelessly and immediately to your doctor, and receive feedback and reminders on how to improve your health.

For many, however, there will come a time in life when an illness or disease reaches a stage in which medicine and lifestyle changes just can’t do the trick. The only solution is surgery, entrusting your life to a skillful surgeon. Thankfully, mobile technology is also leading to surgical innovations, and tablet PC’s are becoming more and more commonplace in the operating room.

One notable example is a liver cancer surgery that took place last year in Germany. What was unique about the surgery was that augmented reality technology guided the hands of the surgeon.

Surgeons prepping with tablet

By displaying images from a CT scan on a tablet PC, a surgeon can view an overlay of the exact locations of important blood vessels and anatomical features during a procedure as the tablet is held over the patient’s actual liver. This helps ensure that the surgeon doesn’t make any unnecessary cuts and completely removes the cancer.

We reached out to Alexander Köhn, the developer of the augmented reality app and Dr. Andrea Schenk, head of liver research at Fraunhofer MEVIS, the research center in Germany that developed the app. They shed light on how this groundbreaking technology could change the way surgery — and medicine — are done.

What makes a liver cancer operation so delicate that doctors would benefit from using this tablet PC app?

Liver tumors are perfused with a large amount of blood, and unplanned cuts can lead to heavy bleeding. In addition, the remaining liver tissue must be supplied with a sufficient amount of blood and be large enough to ensure patient survival. Computer-aided liver surgery planning can help calculate optimal surgical incisions in advance and have proven their value for this task.

Currently, computer-aided plans are viewed by surgeons before an intervention on a stationary workstation. Some hospitals use two-dimensional images that are available on mobile devices during an operation, and even fewer hospitals use new and highly precise surgical navigation systems to assist doctors during procedures. However, most surgeons still have to rely primarily on their memory and experience during the actual operation to locate the tumor and blood vessels and then make the actual incisions.

What makes a tablet PC the best platform for this technology?

The software that Fraunhofer MEVIS developed helps surgeons in planning a procedure. Once in the operating room, they can adjust these plans quickly and flexibly.

Patients’ radiological images and planning data can be easily viewed on a tablet PC, and all common touch gestures can still be used with a sterile mobile device case. The liver can also be filmed with the tablet computer and, using augmented reality, virtual planning data can be semi-transparently superimposed onto the organ in real-time. In this fashion, a surgeon can immediately compare the liver anatomy shown on the computer model with the actual organ during an intervention.

With relatively little effort, quality checks can be quickly performed. In contrast to large surgical navigation systems, the tablet PC helps to easily access all planning data and to speed up the process of locating risk structures.

What are some other areas of medicine where this type of real-time imaging might be beneficial?

This technology can generally be applied to any other surgical or interventional procedures where extensive pre-operative planning is needed, for example, procedures on the brain, pancreas, lung, and kidneys. In all these cases, the visualization of the planning data in the operating room is helpful.

The application of augmented reality depends largely on the part of the body. Neurosurgery is possible, and is actually less difficult than an organ like the lung, since the lung moves and deforms and appears significantly different between preoperative imaging and the actual surgical procedure.


Ever since he had an atresia surgically repaired at age five, Scott has been fascinated with the field of medical science. Combining his love of consumer electronics and technology with medicine, he studied biomedical engineering at the University of Southern California and graduated in 2009. By day, Scott is a Technical Services Engineer at St. Jude Medical, but moonlights as a senior editor at Medgadget, a leading medical technology and innovation blog. Scott is always searching for the next big thing in medical technology and digital health and looks forward to sharing these life-transforming innovations with iQ by Intel’s audience.

Image courtesy of Fraunhofer MEVIS


Share This Article

Related Topics


Read This Next