Apple wants Vision Pro to be a medical hub. Here’s how its being used

At its $3,499 starting price point, the Apple Vision Pro isn’t exactly the cheapest consumer tech product. But some healthcare leaders across the U.S. medical system say they’re already embracing the Vision Pro for its spatial computing functions and augmented reality capabilities, which they say hold serious promise.

Apple is pushing the Vision Pro’s potential to be a medical hub with multiple use cases that leverages its infinite field of vision and ultra-high resolution. Earlier this month, the company touted how multiple healthcare organizations were already launching their first programs to use visionOS, putting them to use in clinical and hospital environments for uses as boilerplate as managing medical records to helping educate patients on what happens during and after a heart attack by letting them take a vivid virtual tour inside the human heart.

Senior leaders and executives at academic publishing giant Elsevier, medical technology and imaging firm Philips, the California-based Cedars-Sinai health system, and the Boston Children’s Hospital all detailed their own experiences to date using or experimenting with the Vision Pro platform in interviews with Fast Company.

From potential in the surgical suite to mental health care patient services to real-world nurse training, medical Vision Pro apps have a range of uses, and were all built and being tested by the respective organizations to fit the visionOS ecosystem, setting these early programs up as AR proofs-of-concept that could be applied to medical systems and healthcare industry players across the country, the companies’ leaders said. Here are four detailed looks into some of those applications.

Philips, for instance, is testing ways to port its existing AR platform using the Microsoft HoloLens used for image-guided, minimally invasive surgical procedures to the Vision Pro, and the other organizations have visionOS-specific programs, according to Atul Gupta, chief medical officer and interventional radiologist for Diagnosis & Treatment at Philips. 

Gupta has spearheaded efforts at Philips to use HoloLens AR for procedures such as removing clots with camera-equipped catheters. With the headset, the surgeon can look at a virtual image of the catheter right above its entry point while guiding it through the body, using eye tracking and swiping features to maintain precise movements while also virtually superimposing whatever monitoring charts are necessary into a convenient spot.

“When we started our relationship with Apple five years ago, I thought, let’s see if we can port over some of this stuff onto Vision Pro, because they’ve done a tremendous job increasing the user usability, increasing the field of view, increasing the resolution, making better eye tracking,” Gupta tells Fast Company.

Medical innovation evolves on multiple fronts. It’s not all new apps and AR tech–even base materials, like the medical infusion pumps used to send medications and key nutrients into patients’ bodies, have newer iterations that must integrate into existing systems. And the people who use those base materials–like nurses, who oversee so much of patients’ day-to-day care–need to be comfortable with the latest systems in their own hospitals. At Boston Children’s Hospital (BCH), the Vision Pro is providing the means to give those nurses the real-world training they need through a spatial AR training suite called CyranoHealth.

“Some of the most complex machinery for nurses, IV pumps—really the bread and butter of what they do—those pieces of equipment have become very sophisticated, very complex,” said Peter Weinstock, executive program director for Immersive Design Systems at BCH. “And so what ends up happening is they end up graduating from nursing school, they head to the bedside, they’ve gotten trained in these pumps, and then, you know, the rubber hits the road. They’re now at the bedside of a very sick patient, and they’ve got to make this very complex pump work, and they have to make it work in a timely way.”

With CyranoHealth for the Vision Pro, a nurse can start a training module with standard equipment in a common clinical environment and then switch to the virtual setting for training on how to use the equipment in different environments and with different kinds of patients. 

The virtual elements can also be swapped or adjusted to suit the actual hospital environment. Because of this, Weinstock believes using the Vision Pro will save time and money. 

“We have to use lots of technology in this training and onboarding,” he says. “We have to use computers, we have to use mannequins, sometimes we have to use robots, we have to use, sometimes virtual reality, and we keep swapping these devices, which are costly.”

BCH is now rolling out the CyranoHealth program to a limited number of staff members. “This is in prototype phase,” says John Brownstein, SVP and chief innovation officer at BCH. “So we’re now getting responses from nurses, are collecting feedback, because we’ve developed one training simulation. And the idea is that we expand into other domains. But it’s clearly important to understand how it’s being used, the value that it brings, as we put in more resources… Obviously, this is a totally new environment that we’ve had to learn and develop against. It’s incredibly exciting. Our groups are very excited to continue.”

Medical provider education is one thing. But what about patients themselves?

That’s, appropriately, at the heart of academic publishing giant Elsevier’s own Vision Pro platform app for teaching patients about myocardial infarctions (heart attacks) called Complete HeartX. This is an immersive patient education tool meant to empower people and their loved ones trying to learn more about the human heart, and what happens to it during a traumatic and common medical emergency like a heart attack, says Irene Walsh, senior director of Product & Education Design at Elsevier.

“You launch the app on the Vision Pro, we have one piece of content in there at the moment–one journey around myocardial infarction, which is more commonly known as the heart attack–which explains the healthy heart,” she says. “So you get to get immersed in the heart, you can scale it right up, you can walk into it, and learn about the key concepts and the key structures.”

“So as a member of the general public, you can see it beating, learn about how the valves work together, understand the flow of blood,” says Walsh, adding that 3D artists have been designing more and more intricate models of the heart for more than a decade–designs which have been integrated into the Complete HeartX app “to really see the real resolution of that in such breathtaking fidelity.”

That’s just the first step in the process. After the virtual trek through a healthy heart, a user can see what exactly happens over the course of a heart attack and how different parts of the heart react to its onset. And since the ultimate goal of this platform is patient empowerment within the medical system, it ends with a real-world simulation.

“One of our core beliefs is that it’s so important to train, from the outset, people to think about how they apply theory,” said Walsh. “And that goes beyond an education context, to the act of practicing that in a clinical environment. So it has equal benefit, as a patient, to understand what each of those steps in the clinic are, including things like simulations around taking blood from a heart attack patient. So you can understand what the key steps are, why they’re happening, and basically demystify that clinical experience.”

Mental health services–and access to the professionals who provide them–are a glaring weak point in American medicine. That’s true even at some of the nation’s most prominent health systems, such as the Cedars-Sinai Medical Center in Los Angeles.

“Here in our hospital, we have 1,000 beds. But we have generally only one psychologist who’s available to manage people—and that’s one more than some hospitals have,” says Brennan Spiegel, a director of Health Services Research for Cedars-Sinai. 

To help make up for that disparity while helping patients in mental health need, Cedars-Sinai developed a virtual, self-administered tool called eXtended-Reality Artificially Intelligent Ally, or Xaia, that was already available through other headsets but is now in the Apple Vision Pro ecosystem.

For mental health patients, the platform emulates serene environments tailored to their needs. It’s a dynamic platform that Spiegel says goes one step beyond AR and into a form of generated reality. “What the Vision Pro is achieving is the sense of presence that you have with VR, which in general is always emotionally compelling. It can help get people out of the four walls of their hospital room, into or out of the four walls of their illness experience, and into fantastical new worlds—but with the incredible visual display, we’ve been able to achieve some pretty remarkable effects,” he says.

What makes the tool so compelling, beyond the fact that mental health patients can potentially use it outside of medical settings as a self-administered tool, is that it’s dynamic in responding to each individual patient’s needs. Various customized environments, themes, music, and meditation and breathing exercises were built into the app following years of research from Cedars-Sinai’s medical and tech innovation teams.

“What it’s doing is actually listening to the user and creating contextually relevant worlds around you. So if you’re talking about feeling very anxious and upset, then you find yourself in a rock garden. And there’s certain specific music that has been originally composed just for anxiety—we have all of these hours of music for different emotional states,” says Spiegel. 

“So instead, maybe you’re talking about gender identity, and you’re surrounded by rainbow colored butterflies that sort of flit in and out around you. Depending upon one of more 60 different topics, there’s hundreds of different audio visual effects that we can deploy to surround our users in these generated worlds that are ideally, psychologically, and emotionally appropriate for the moment.”

At its $3,499 starting price point, the Apple Vision Pro isn’t exactly the cheapest consumer tech product. But some healthcare leaders across the U.S. medical system say they’re already embracing the Vision Pro for its spatial computing functions and augmented reality capabilities, which they say hold serious promise.

Apple is pushing the Vision Pro’s potential to be a medical hub with multiple use cases that leverages its infinite field of vision and ultra-high resolution. Earlier this month, the company touted how multiple healthcare organizations were already launching their first programs to use visionOS, putting them to use in clinical and hospital environments for uses as boilerplate as managing medical records to helping educate patients on what happens during and after a heart attack by letting them take a vivid virtual tour inside the human heart.

Senior leaders and executives at academic publishing giant Elsevier, medical technology and imaging firm Philips, the California-based Cedars-Sinai health system, and the Boston Children’s Hospital all detailed their own experiences to date using or experimenting with the Vision Pro platform in interviews with Fast Company.

From potential in the surgical suite to mental health care patient services to real-world nurse training, medical Vision Pro apps have a range of uses, and were all built and being tested by the respective organizations to fit the visionOS ecosystem, setting these early programs up as AR proofs-of-concept that could be applied to medical systems and healthcare industry players across the country, the companies’ leaders said. Here are four detailed looks into some of those applications.

Philips, for instance, is testing ways to port its existing AR platform using the Microsoft HoloLens used for image-guided, minimally invasive surgical procedures to the Vision Pro, and the other organizations have visionOS-specific programs, according to Atul Gupta, chief medical officer and interventional radiologist for Diagnosis & Treatment at Philips. 

Gupta has spearheaded efforts at Philips to use HoloLens AR for procedures such as removing clots with camera-equipped catheters. With the headset, the surgeon can look at a virtual image of the catheter right above its entry point while guiding it through the body, using eye tracking and swiping features to maintain precise movements while also virtually superimposing whatever monitoring charts are necessary into a convenient spot.

“When we started our relationship with Apple five years ago, I thought, let’s see if we can port over some of this stuff onto Vision Pro, because they’ve done a tremendous job increasing the user usability, increasing the field of view, increasing the resolution, making better eye tracking,” Gupta tells Fast Company.

Medical innovation evolves on multiple fronts. It’s not all new apps and AR tech–even base materials, like the medical infusion pumps used to send medications and key nutrients into patients’ bodies, have newer iterations that must integrate into existing systems. And the people who use those base materials–like nurses, who oversee so much of patients’ day-to-day care–need to be comfortable with the latest systems in their own hospitals. At Boston Children’s Hospital (BCH), the Vision Pro is providing the means to give those nurses the real-world training they need through a spatial AR training suite called CyranoHealth.

“Some of the most complex machinery for nurses, IV pumps—really the bread and butter of what they do—those pieces of equipment have become very sophisticated, very complex,” said Peter Weinstock, executive program director for Immersive Design Systems at BCH. “And so what ends up happening is they end up graduating from nursing school, they head to the bedside, they’ve gotten trained in these pumps, and then, you know, the rubber hits the road. They’re now at the bedside of a very sick patient, and they’ve got to make this very complex pump work, and they have to make it work in a timely way.”

With CyranoHealth for the Vision Pro, a nurse can start a training module with standard equipment in a common clinical environment and then switch to the virtual setting for training on how to use the equipment in different environments and with different kinds of patients. 

The virtual elements can also be swapped or adjusted to suit the actual hospital environment. Because of this, Weinstock believes using the Vision Pro will save time and money. 

“We have to use lots of technology in this training and onboarding,” he says. “We have to use computers, we have to use mannequins, sometimes we have to use robots, we have to use, sometimes virtual reality, and we keep swapping these devices, which are costly.”

BCH is now rolling out the CyranoHealth program to a limited number of staff members. “This is in prototype phase,” says John Brownstein, SVP and chief innovation officer at BCH. “So we’re now getting responses from nurses, are collecting feedback, because we’ve developed one training simulation. And the idea is that we expand into other domains. But it’s clearly important to understand how it’s being used, the value that it brings, as we put in more resources… Obviously, this is a totally new environment that we’ve had to learn and develop against. It’s incredibly exciting. Our groups are very excited to continue.”

Medical provider education is one thing. But what about patients themselves?

That’s, appropriately, at the heart of academic publishing giant Elsevier’s own Vision Pro platform app for teaching patients about myocardial infarctions (heart attacks) called Complete HeartX. This is an immersive patient education tool meant to empower people and their loved ones trying to learn more about the human heart, and what happens to it during a traumatic and common medical emergency like a heart attack, says Irene Walsh, senior director of Product & Education Design at Elsevier.

“You launch the app on the Vision Pro, we have one piece of content in there at the moment–one journey around myocardial infarction, which is more commonly known as the heart attack–which explains the healthy heart,” she says. “So you get to get immersed in the heart, you can scale it right up, you can walk into it, and learn about the key concepts and the key structures.”

“So as a member of the general public, you can see it beating, learn about how the valves work together, understand the flow of blood,” says Walsh, adding that 3D artists have been designing more and more intricate models of the heart for more than a decade–designs which have been integrated into the Complete HeartX app “to really see the real resolution of that in such breathtaking fidelity.”

That’s just the first step in the process. After the virtual trek through a healthy heart, a user can see what exactly happens over the course of a heart attack and how different parts of the heart react to its onset. And since the ultimate goal of this platform is patient empowerment within the medical system, it ends with a real-world simulation.

“One of our core beliefs is that it’s so important to train, from the outset, people to think about how they apply theory,” said Walsh. “And that goes beyond an education context, to the act of practicing that in a clinical environment. So it has equal benefit, as a patient, to understand what each of those steps in the clinic are, including things like simulations around taking blood from a heart attack patient. So you can understand what the key steps are, why they’re happening, and basically demystify that clinical experience.”

Mental health services–and access to the professionals who provide them–are a glaring weak point in American medicine. That’s true even at some of the nation’s most prominent health systems, such as the Cedars-Sinai Medical Center in Los Angeles.

“Here in our hospital, we have 1,000 beds. But we have generally only one psychologist who’s available to manage people—and that’s one more than some hospitals have,” says Brennan Spiegel, a director of Health Services Research for Cedars-Sinai. 

To help make up for that disparity while helping patients in mental health need, Cedars-Sinai developed a virtual, self-administered tool called eXtended-Reality Artificially Intelligent Ally, or Xaia, that was already available through other headsets but is now in the Apple Vision Pro ecosystem.

For mental health patients, the platform emulates serene environments tailored to their needs. It’s a dynamic platform that Spiegel says goes one step beyond AR and into a form of generated reality. “What the Vision Pro is achieving is the sense of presence that you have with VR, which in general is always emotionally compelling. It can help get people out of the four walls of their hospital room, into or out of the four walls of their illness experience, and into fantastical new worlds—but with the incredible visual display, we’ve been able to achieve some pretty remarkable effects,” he says.

What makes the tool so compelling, beyond the fact that mental health patients can potentially use it outside of medical settings as a self-administered tool, is that it’s dynamic in responding to each individual patient’s needs. Various customized environments, themes, music, and meditation and breathing exercises were built into the app following years of research from Cedars-Sinai’s medical and tech innovation teams.

“What it’s doing is actually listening to the user and creating contextually relevant worlds around you. So if you’re talking about feeling very anxious and upset, then you find yourself in a rock garden. And there’s certain specific music that has been originally composed just for anxiety—we have all of these hours of music for different emotional states,” says Spiegel. 

“So instead, maybe you’re talking about gender identity, and you’re surrounded by rainbow colored butterflies that sort of flit in and out around you. Depending upon one of more 60 different topics, there’s hundreds of different audio visual effects that we can deploy to surround our users in these generated worlds that are ideally, psychologically, and emotionally appropriate for the moment.”