News & Announcements

Dr. Joaquin Sanchez-Sotelo answers FORE’s questions about applications for this technology in shoulder surgery, as well as how it differs from virtual reality and augmented reality.

FORE: What is augmented (or mixed) reality? How does this differ from virtual reality?

Joaquin Sanchez-Sotelo, MD, PhD: The 3 technologies – virtual reality, augmented reality, and mixed reality – are similar in some ways and different in others. In all 3, objects are created with a computer and projected so that humans can see them. Sound can be paired with these virtual projections, and there may be haptic interactions through hand-held or other devices.

Virtual reality – also known as immersive reality – uses devices that block vision of the real world so that users feel completely immersed in the virtual environment; they can only visualize the virtual world. Virtual reality systems are commonly used for video gaming or education.

Users simultaneously visualize the real world and virtual objects with augmented reality and mixed reality. The difference is that in mixed reality, users can modify virtual objects, while in augmented reality, they cannot.

An example of augmented reality is a smartphone application that displays a graphic representation of a real object when the smartphone is pointed at the object, such as applications that show a graphic of a given building (for example, the Chrysler Building in New York City) when pointed to that building. Users cannot manipulate the graphic representation overlaying the actual object: The graphic will appear and disappear depending on the coordinate relationship between the smartphone and the real object.

In mixed reality, users can modify the size and position of virtual objects and anchor them in specific positions in relation to the real world. This is typically achieved through projection of holograms (3-dimensional images formed by interference of light beams) that users can modify with voice commands or hand gestures. The most common mixed reality device is the HoloLens (Microsoft; Redmond, Washington). Mixed reality holds the greatest promise to improve the field of shoulder arthroplasty (Figure 1).

Figure 1. In this video, Dr. Joaquin Sanchez-Sotelo describes the use of mixed reality in shoulder surgery.

RELATED: Register now for Critical Concepts in Shoulder & Elbow Surgery, October 13-15; click here

FORE: How is mixed reality applied in shoulder arthroplasty? What’s the goal?

Dr. Sanchez-Sotelo: Mixed reality has 2 functions in shoulder arthroplasty, visualization and navigation. Many shoulder surgeons currently utilize CT-based software to assess the operative shoulder and then create a preoperative plan that can be visualized on a computer screen.

Mixed reality visualization consists of a hologram of the patient’s anatomy with renderings of the implants in place. The surgeon can move this hologram in front of the operative field, providing 3-dimensional visualization of the surgical site, and then manipulate the hologram to view the virtual anatomy from multiple angles, with and without the implants. There is no need for the surgeon to raise their eyes from the operative field to look at a monitor for radiographs or other images, which are typically 2-dimensional.

The technology of mixed reality navigation allows the surgeon to accurately execute the preoperative plan. A challenge of shoulder arthroplasty is that even though surgeons have great tools for planning, we are often left hoping that our skills will allow us to accurately execute the plan. That is not always the case.

Navigation is one way to improve accuracy and precision. In essence, navigation is computer-assisted surgery, in which instruments and anatomy are tracked simultaneously to allow the surgeon to place instruments and implants accurately and precisely. With mixed reality, the navigation system tracks the position of markers that are easily identified by the mixed reality device. The hope is that in the future, mixed reality devices will be able to scan and register anatomy without the use of anatomy markers.

FORE: What has been your experience with mixed reality? How does it affect the workflow for shoulder arthroplasty?

Dr. Sanchez-Sotelo: So far, I have only had the opportunity to use mixed reality visualization, but I have practiced mixed reality guidance in cadaver specimens.

The workflow for mixed reality visualization is easy:

• The surgeon’s preoperative plan for the patient is loaded into the HoloLens device.
• The surgeon places the HoloLens device on their head, with the lenses over their eyes.
• The surgeon uses a combination of voice commands and gestures to visualize the elements of the patient’s anatomy and implant placement based on the preoperative plan.

The holograms can be conveniently positioned in the surgical field for easy visualization. I have found this to be particularly helpful in revision shoulder arthroplasty procedures: I can see the bones behind the failed implant in 3 dimensions, as well as virtually remove the implant for even better visualization.

FORE: What does the surgeon gain from using mixed reality in preoperative planning and in the OR? What do you see as the advantages of mixed reality in shoulder arthroplasty?

Dr. Sanchez-Sotelo: In my opinion, my ability to use dedicated software packages for preoperative planning prior to shoulder arthroplasty makes me a better surgeon. I can understand the nuances of the anatomy of each shoulder to be replaced. Being able to visualize a 3-dimensional holographic rendering of the plan with mixed reality right in the surgical field while performing the surgery is very effective.

Mixed reality will provide a number of benefits over other forms of navigation. It not only provides the opportunity for 3-dimensional visualization of the plan, but it is also very time, cost, and space efficient:

• The plan can be created and loaded literally minutes before the procedure.
• HoloLens devices are relatively inexpensive compared with traditional navigation equipment or robots
• The device can be stored in a very small case, saving the substantial space occupied by the footprint of traditional navigation equipment or robots.

FORE: Are there any drawbacks or limitations that you’ve noticed?

Dr. Sanchez-Sotelo: The main drawback is that it can be more difficult to view the surgical field for dissection while wearing the HoloLens device. For that reason, some surgeons use the device only for the critical portions of the operation, when mixed reality visualization or navigation are essential. Although the HoloLens is relatively lightweight, it can increase neck fatigue during surgery.

FORE: What type of training will surgeons need to implement mixed reality?

 Dr. Sanchez-Sotelo: Some of the training will be fairly simple and straightforward, including learning how to use the preoperative planning software package and how to use the HoloLens device. That will take some practice, but it is not difficult to master.

Greater complexity occurs with the addition of navigation. Surgeons will need to learn a specific workflow associated with navigation using the HoloLens device, which includes registration, among other steps, to achieve accurate and precise execution of the preoperative plan.

 FORE: What will it take for mixed reality to be more widely utilized in shoulder procedures?

Dr. Sanchez-Sotelo: Any new technology comes with certain costs. As mentioned previously, mixed reality can be relatively inexpensive compared with alternative navigation methods or robotic equipment.

Robotic systems represent an alternative execution methodology that will play a very important role in the field of shoulder arthroplasty going forward. However, the expense and footprint of most robotic systems may not be as appealing for surgeons who need a less-costly execution solution, such as surgeons in the community setting, where it may not be as easy to absorb the cost of a robot as in an academic center, and surgeons working in ambulatory surgery centers, where space is at a premium and costs can be an issue.

Ultimately, I believe that surgeons already interested in navigation will quickly adopt mixed reality because it provides tools that are both powerful and convenient.

Author Bio

Joaquin Sanchez-Sotelo, MD, PhD, is Chair of the Division of Shoulder and Elbow Surgery, Department of Orthopedic Surgery, at Mayo Clinic, Rochester, Minnesota. As a physician-scientist, he is interested in research related to the application of artificial intelligence in shoulder and elbow surgery, as well as surgical conditions affecting the shoulder and elbow joints and strategies for individualized medicine.

Dr. Sanchez-Sotelo is one of the Course Chairs of FORE’s inaugural Critical Concepts in Shoulder & Elbow Surgery course, taking place October 13-15 in Las Vegas. Visit fore.edusymp.com for more information and to register for the course.