Augmenting the senses to improve healthcare

Prof. Nassir Navab leads research into computer-aided medical procedures and augmented reality at the Technical University of Munich in Germany. His groundbreaking work is aimed at enabling medical professionals to use their full range of senses when treating patients.¹

For over 175 years, the people at ZEISS have asked the question: How can we challenge the limits of imagination? In celebration of that vision, ZEISS has partnered with thought leaders and great minds from around the globe for the ZEISS Beyond Talks, giving them center stage to speak about their own work, visions, passion and issues that are affecting our world moving forward.

Medicine excites me because it's at the intersection of the three fields of science: fundamental, translational and radical.

We need the fundamental to make sure we are using the best science and technology. We need translational in order not to be in our ivory tower; to know our impact on society. Then the radical allows us to make inventions that are disruptive and can define a new set of solutions. By working in medicine, I have the privilege of exploring a combination of all three fields.

The first time I went into an operating room – in 1994 in Ontario, Canada – I was completely fascinated that doctors are using technology to make crucial brain surgery decisions. I knew that this is where I want to contribute. It’s a field that will never stop challenging me.

Augmented reality essentially manifests as an extension of the senses.

When we reverse a modern car, for example, we hear a beeping sound that changes as we decrease the distance to an object behind us. This is one of the simplest examples of augmenting your senses. By adding an acoustic dimension to the distance behind the reversing car, you’re able to sense something that you otherwise could not.

When we extend this to the medical world, we want to sense the flow of blood in a vasculature, for example. If you see that a tumor is taking a lot of oxygen, this means it’s very malignant. You want to sense that a given anatomy is not behaving physiologically as it should.

It's a very complex field, but when you see yourself impacting the lives of many people – including your own parents and children – it gives you a good feeling.

In medicine, we already use a multitude of electronic and physical sensors that provide data about the body’s physiology.

If we provide doctors with all this information – in the form of X-rays, CT scans or PET scans – they would have to look at some 12 displays at the same time. But you can never look at so many displays and make a quick decision. Healthcare personnel often only have three to five minutes for a decision, or even less if they are in the middle of an intervention.

Medical augmented reality essentially transforms this data in such a way that it can be experienced by our sensory modalities. By bringing this ocean of information together in a virtual representation of a patient's body, the doctor can see, hear, touch and feel their way to the right decision for the patient.

It’s widely recognized in healthcare today that we are not treating patients – we are treating diseases. This means that we categorize the problem a person has into what we call a disease, for which we have a solution. We do not have a solution for a particular patient.

But every patient differs in their anatomy, diet and daily life, for example, so our disease-centric categorization means that we are not giving the best treatment to each person. Computerizing and allowing access to personal information – including the physiology of your body, your background, and your pathway history – would enable our systems to provide solutions and treatment specific to the individual.

This would have a major impact on healthcare, as we would be treating people rather than treating diseases.

Absolutely. We should even have augmented reality systems in our kindergartens and schools! But when it comes to medical education and training, we need these systems even more.

The question is, can our technology allow a young doctor trainee to do 100 virtual surgeries – using all his or her senses – and get enough feedback and training to do that first real one? This is what we need to explore.

Many things in our society have been democratized over the past 30 years, including access to scientific information. Now we not only have Wikipedia, we also have some of the best lectures from Stanford, MIT and Oxford available to the general public. Many technologies have been democratized too, such as navigation and video communication.

But what’s still to be democratized at the global level is healthcare. I would say that teaching and training for these new technologies is part of this democratization and will allow us to have fewer diseases in the future.

The second part of democratization is making the technology available. You should be able to take a mobile phone picture of your eye, for example, and the system will tell you whether you need to see a doctor because of cataracts.

What makes me excited for the next 100 years are technologies that can be fully integrated into our whole life cycle, accompanying us from the moment we are born to the moment we pass away.

My dream is that every morning when you go and look at yourself in the mirror, it would have built in sensing capabilities to check your skin, your blood pressure and a lot of other measurements. The sensors would record your values and notice when you are stressed or when your diet is not good. This technology would become an integrated part of daily life.

That's what I'm hoping for in 100 years: preventive care driven by technology that’s with us from the very beginning of our lives.