Patient-specific surgical plans

Description

This application claims the benefit of U.S. Provisional Application No. 63/148,806, filed Feb. 12, 2021, the entire contents of which are incorporated by reference herein.

BACKGROUND

Surgical joint repair procedures involve repair and/or replacement of a damaged or diseased joint. Many times, a surgical joint repair procedure, such as joint arthroplasty as an example, involves replacing the damaged joint with a prosthetic that is implanted into the patient's bone. Proper selection of a prosthetic that is appropriately sized and shaped and proper positioning of that prosthetic to ensure an optimal surgical outcome can be challenging. To assist with positioning, the surgical procedure often involves the use of surgical instruments to control the shaping of the surface of the damaged bone and cutting or drilling of bone to accept the prosthetic.

SUMMARY

This disclosure describes a variety of techniques for providing preoperative planning. and/or intraoperative guidance for surgical joint repair procedures (e.g., arthroplasty procedures). The techniques may be used independently or in various combinations to support particular phases or settings for surgical joint repair procedures or provide a multi-faceted ecosystem to support surgical joint repair procedures.

A surgical joint repair procedure may involve a surgeon performing several steps. Different joint repair procedures may involve performance of different standard sets of steps. As one example, to perform a shoulder arthroplasty, a surgeon may perform a first standard set of steps. As another example, to perform an ankle arthroplasty, a surgeon may perform a second standard set of steps. A standard set of steps for a particular joint repair procedure may be the steps required to complete the procedure. In some cases, it may be desirable for a surgeon to perform additional steps, referred to herein as ancillary steps, that are not required to complete a procedure. Performance of ancillary steps may improve patient outcomes (e.g., increased range of motion, longer prosthetic life, etc.).

When planning a joint repair procedure for a particular patient, a surgeon may include a standard set of steps in a surgical plan for the particular patient. The surgeon may also determine whether or not to perform any ancillary steps. However, there may be a large quantity of potential ancillary steps available for some joint repair procedures. Additionally, even when presented with the option to perform a particular ancillary step, the surgeon may not be aware of when the particular ancillary step should or should not be performed.

In accordance with one or more techniques of this disclosure, a computing system may automatically recommend performance of one or more ancillary steps. For instance, the computing system may receive values of a plurality of parameters of a joint of a particular patient on which an arthroplasty procedure is to be performed. The computing system may select, based on the values of the plurality of parameters, one or more ancillary steps of a plurality of ancillary steps. In some examples, the computing system may select the one or more ancillary steps using a machine learning model. The computing system may output a surgical plan for the particular patient that includes the selected one or more ancillary steps. The quantity of ancillary steps evaluated by the computing system for inclusion in the surgical plan may be greater than a quantity that may be evaluated by an actual surgeon. In this way, a large quantity of ancillary steps may be considered for inclusion in a surgical plan for a particular patient, which may improve patient outcomes.

The computing system may recommend performance of the one or more ancillary steps in a specified order (e.g., a specified sequence of steps). In other examples, the computing system may recommend performance of the one or more ancillary steps in an unspecified order (e.g., a list of steps that the surgeon may determine when to perform during the surgical procedure).

The details of various examples of the disclosure are set forth in the accompanying drawings and the description below. Various features, objects, and advantages will be apparent from the description, drawings, and claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an orthopedic surgical system according to an example of this disclosure.

FIG. 2 is a block diagram of an orthopedic surgical system that includes a mixed reality (MR) system, according to an example of this disclosure.

FIG. 3 is a flowchart illustrating example phases of a surgical lifecycle.

FIG. 4 is a flowchart illustrating preoperative, intraoperative and postoperative workflows in support of an orthopedic surgical procedure.

FIG. 5 is a schematic representation of a visualization device for use in a mixed reality (MR) system, according to an example of this disclosure.

FIG. 6 is a block diagram illustrating example components of a visualization device for use in a mixed reality (MR) system, according to an example of this disclosure.

FIG. 7 is a conceptual diagram illustrating an example setting in which a set of users use mixed reality (MR) systems of an orthopedic surgical system during a preoperative phase.

FIG. 8 is a flowchart illustrating example steps in the preoperative phase of the surgical lifecycle.

FIG. 9 illustrates an example welcome page for selecting a surgical case, according to an example of this disclosure.

FIG. 10 illustrates an example of a page of a user interface of a mixed reality (MR) system, according to an example of this disclosure.

FIG. 11 is a flowchart illustrating example stages of a shoulder joint repair surgery.

FIG. 12 is a block diagram illustrating an example system that may assist a user, such as a surgeon, nurse, or other medical technicians, through steps in the workflow steps of orthopedic surgeries, in accordance with a technique of this disclosure.

FIG. 13 is a flowchart illustrating an example operation to assist users, such as surgeons, nurses, or other medical technicians, through steps in the workflows of orthopedic surgeries, in accordance with a technique of this disclosure.

FIG. 14 is a conceptual diagram illustrating an example extended reality (XR) visualization that includes a set of one or more virtual checklist items, as viewed by a user, such as an orthopedic surgeon, nurse, or other medical technician, while performing an orthopedic surgery on a shoulder of a patient.

FIG. 15 is a conceptual diagram illustrating an example system in which a first surgical plan is modified during an intraoperative phase to generate a second surgical plan, in accordance with a technique of this disclosure.

FIG. 16 is a flowchart of an example operation in which a first surgical plan is modified during an intraoperative phase to generate a second surgical plan, in accordance with a technique of this disclosure.

FIG. 17 is a block diagram illustrating an example computing system that implements a neural network usable for selecting and/or recommending ancillary steps for a patient, in accordance with a technique of this disclosure.

FIG. 18 illustrates an example neural network that may be implemented by a computing system with the system of FIG. 17.

FIG. 19 is a flowchart illustrating an example operation of a computing system in using an artificial neural network to determine a classification of a shoulder condition, in accordance with a technique of this disclosure.

FIG. 20 is a flowchart illustrating example stages of an ankle joint repair surgery.

FIGS. 21A and 21B are conceptual diagrams illustrating example attachment of guide pins to a tibia.

FIG. 22 is a conceptual diagram illustrating example drilling of holes in a tibia.

FIG. 23 is a conceptual diagram illustrating example resection of a tibia.

FIGS. 24A and 24B are conceptual diagrams illustrating example guide pins installed in a talus during a talus preparation process.

FIG. 25 is a conceptual diagram illustrating example resection of a talus.

FIG. 26 is a conceptual diagram of an example ankle after performance of a tibial resection and a talar resection.

FIGS. 27A-27C are conceptual diagrams illustrating an example of tibial tray trialing.

FIG. 28 is a conceptual diagram illustrating an example creation of tibial implant anchorage.

FIGS. 29A and 29B are conceptual diagrams illustrating an example attachment of guide pins to a talus.

FIG. 30 is a conceptual diagram of an example chamfer guide on a talus.

FIG. 31 is a conceptual diagram of an example posterior talar chamfer resection.

FIGS. 32 and 33 are conceptual diagrams of example anterior talar chamfer resections.

FIGS. 34 and 35 are conceptual diagrams illustrating an example creation of talar implant anchorage.

FIG. 36 is a conceptual diagram illustrating an example tibial implant.

FIG. 37 is a conceptual diagram illustrating an example of a prepared tibia.

FIG. 38 is a conceptual diagram illustrating example impaction of a tibial implant into a tibia.

FIG. 39 is a conceptual diagram illustrating an example talar implant.

FIG. 40 is a conceptual diagram illustrating example impaction of a talar implant into a talus.

FIG. 41 is a conceptual diagram illustrating an example bearing implanted between a tibial implant and a talar implant.

FIG. 42 is a flowchart illustrating example technique for recommending ancillary steps for inclusion in a surgical plan, in accordance with one or more techniques of this disclosure.

DETAILED DESCRIPTION

Certain examples of this disclosure are described with reference to the accompanying drawings, wherein like reference numerals denote like elements. It should be understood, however, that the accompanying drawings illustrate only the various implementations described herein and are not meant to limit the scope of various technologies described herein. The drawings show and describe various examples of this disclosure.

In the following description, numerous details are set forth to provide an understanding of the present invention. However, it will be understood by those skilled in the art that the present invention may be practiced without these details and that numerous variations or modifications from the described examples may be possible.

Orthopedic surgery, such as a surgical joint repair procedure, can involve performing various steps to prepare bone for implantation of one or more prosthetic devices to repair or replace a patient's damaged or diseased joint. Virtual surgical planning tools may be available that use image data of the diseased or damaged joint to generate an accurate three-dimensional bone model that can be viewed and manipulated preoperatively by the surgeon. These tools can enhance surgical outcomes by allowing the surgeon to simulate the surgery, select or design an implant that more closely matches the contours of the patient's actual bone, and select or design surgical instruments and guide tools that are adapted specifically for repairing the bone of a particular patient. Use of these planning tools typically results in generation of a preoperative surgical plan, complete with an implant and surgical instruments that are selected or manufactured for the individual patient.

As noted above, a surgical joint repair procedure may involve a surgeon performing several steps. Different joint repair procedures may involve performance of different standard sets of steps. The standard set of steps for a particular joint repair procedure may be the steps required to complete the procedure (e.g., steps specifically prescribed by a manufacturer of the prosthetic device to be implanted, such as in a manufacturer surgical technique). In some cases, it may be desirable for a surgeon to perform additional steps, referred to herein as ancillary steps, that are not required to complete the procedure. Performance of ancillary steps may improve patient outcomes (e.g., increased range of motion, longer prosthetic life, etc.).

When planning a joint repair procedure for a particular patient (e.g., using virtual surgical planning tools), a surgeon may include, in a surgical plan for the particular patient, a standard set of steps. The surgeon may also determine whether or not to perform any ancillary steps. However, there may be a large quantity of potential ancillary steps available for some joint repair procedures. Additionally, even when presented with the option to perform a particular ancillary step, the surgeon may not be aware of when the particular ancillary step should or should not be performed. Furthermore, it may not be reasonably possible for a surgeon to evaluate a large quantity of ancillary steps.

In accordance with one or more techniques of this disclosure, a computing system may automatically recommend performance of one or more ancillary steps. For instance, the computing system may receive values of a plurality of parameters of a joint of a particular patient on which an arthroplasty procedure is to be performed. The computing system may select, based on the values of the plurality of parameters, one or more ancillary steps of a plurality of ancillary steps. In some examples, the computing system may select the one or more ancillary steps using a machine learning model. The computing system may output a surgical plan for the particular patient that includes the selected one or more ancillary steps. The quantity of ancillary steps evaluated by the computing system for inclusion in the surgical plan may be greater than a quantity that may be evaluated by an actual surgeon. In this way, a large quantity of ancillary steps may be considered for inclusion in a surgical plan for a particular patient, which may improve patient outcomes.

In some situations, once in the actual operating environment, the surgeon may desire to verify the preoperative surgical plan intraoperatively relative to the patient's actual bone. This verification may result in a determination that an adjustment to the preoperative surgical plan is needed, such as a different implant, a different positioning or orientation of the implant, a different set of ancillary steps, and/or a different surgical guide for carrying out the surgical plan. In addition, a surgeon may want to view details of the preoperative surgical plan relative to the patient's real bone during the actual procedure in order to more efficiently perform standard steps, perform ancillary steps, and accurately position and orient the implant components. For example, the surgeon may want to obtain intra-operative visualization that provides guidance for performing standard or ancillary steps, positioning and orientation of implant components, guidance for preparation of bone or tissue to receive the implant components, guidance for reviewing the details of a procedure or procedural step, and/or guidance for selection of tools or implants and tracking of surgical procedure workflow.

Accordingly, this disclosure describes systems and methods for using a mixed reality (MR) visualization system to assist with creation, implementation, verification, and/or modification of a surgical plan before and during a surgical procedure. Because MR, or in some instances virtual reality (VR), may be used to interact with the surgical plan, this disclosure may also refer to the surgical plan as a “virtual” surgical plan. Visualization tools other than or in addition to mixed reality visualization systems may be used in accordance with techniques of this disclosure. A surgical plan, e.g., as generated by the BLUEPRINT™ system or another surgical planning platform, may include information defining a variety of features of a surgical procedure, such as features of particular surgical procedure steps to be performed on a patient by a surgeon according to the surgical plan including, for example, bone or tissue preparation steps and/or steps for selection, modification and/or placement of implant components. Such information may include, in various examples, dimensions, shapes, angles, surface contours, and/or orientations of implant components to be selected or modified by surgeons, dimensions, shapes, angles, surface contours and/or orientations to be defined in bone or tissue by the surgeon in bone or tissue preparation steps, and/or positions, axes, planes, angle and/or entry points defining placement of implant components by the surgeon relative to patient bone or tissue. Information such as dimensions, shapes, angles, surface contours, and/or orientations of anatomical features of the patient may be derived from imaging (e.g., x-ray, CT, MRI, ultrasound or other images), direct observation, or other techniques.

In this disclosure, the term “mixed reality” (MR) refers to the presentation of virtual objects such that a user sees images that include both real, physical objects and virtual objects. Virtual objects may include text, 2-dimensional surfaces, 3-dimensional models, or other user-perceptible elements that are not actually present in the physical, real-world environment in which they are presented as coexisting. In addition, virtual objects described in various examples of this disclosure may include graphics, images, animations or videos, e.g., presented as 3D virtual objects or 2D virtual objects. Virtual objects may also be referred to as virtual elements. Such elements may or may not be analogs of real-world objects. In some examples, in mixed reality, a camera may capture images of the real world and modify the images to present virtual objects in the context of the real world. In such examples, the modified images may be displayed on a screen, which may be head-mounted, handheld, or otherwise viewable by a user. This type of mixed reality is increasingly common on smartphones, such as where a user can point a smartphone's camera at a sign written in a foreign language and see in the smartphone's screen a translation in the user's own language of the sign superimposed on the sign along with the rest of the scene captured by the camera. In some examples, in mixed reality, see-through (e.g., transparent) holographic lenses, which may be referred to as waveguides, may permit the user to view real-world objects, i.e., actual objects in a real-world environment, such as real anatomy, through the holographic lenses and also concurrently view virtual objects.

The Microsoft HOLOLENS™ headset, available from Microsoft Corporation of Redmond, Washington, is an example of a MR device that includes see-through holographic lenses, sometimes referred to as waveguides, that permit a user to view real-world objects through the lens and concurrently view projected 3D holographic objects. The Microsoft HOLOLENS™ headset, or similar waveguide-based visualization devices, are examples of an MR visualization device that may be used in accordance with some examples of this disclosure. Some holographic lenses may present holographic objects with some degree of transparency through see-through holographic lenses so that the user views real-world objects and virtual, holographic objects. In some examples, some holographic lenses may, at times, completely prevent the user from viewing real-world objects and instead may allow the user to view entirely virtual environments. The term mixed reality may also encompass scenarios where one or more users are able to perceive one or more virtual objects generated by holographic projection. In other words, “mixed reality” may encompass the case where a holographic projector generates holograms of elements that appear to a user to be present in the user's actual physical environment.

In some examples, in mixed reality, the positions of some or all presented virtual objects are related to positions of physical objects in the real world. For example, a virtual object may be tethered to a table in the real world, such that the user can see the virtual object when the user looks in the direction of the table but does not see the virtual object when the table is not in the user's field of view. In some examples, in mixed reality, the positions of some or all presented virtual objects are unrelated to positions of physical objects in the real world. For instance, a virtual item may always appear in the top right of the user's field of vision, regardless of where the user is looking.

Augmented reality (AR) is similar to MR in the presentation of both real-world and virtual elements, but AR generally refers to presentations that are mostly real, with a few virtual additions to “augment” the real-world presentation. For purposes of this disclosure, MR is considered to include AR. For example, in AR, parts of the user's physical environment that are in shadow can be selectively brightened without brightening other areas of the user's physical environment. This example is also an instance of MR in that the selectively-brightened areas may be considered virtual objects superimposed on the parts of the user's physical environment that are in shadow.

Furthermore, in this disclosure, the term “virtual reality” (VR) refers to an immersive artificial environment that a user experiences through sensory stimuli (such as sights and sounds) provided by a computer. Thus, in virtual reality, the user may not see any physical objects as they exist in the real world. Video games set in imaginary worlds are a common example of VR. The term “VR” also encompasses scenarios where the user is presented with a fully artificial environment in which some virtual object's locations are based on the locations of corresponding physical objects as they relate to the user. Walk-through VR attractions are examples of this type of VR.

The term “extended reality” (XR) is a term that encompasses a spectrum of user experiences that includes virtual reality, mixed reality, augmented reality, and other user experiences that involve the presentation of at least some perceptible elements as existing in the user's environment that are not present in the user's real-world environment. Thus, the term “extended reality” may be considered a genus for MR and VR. XR visualizations may be presented in any of the techniques for presenting mixed reality discussed elsewhere in this disclosure or presented using techniques for presenting VR, such as VR goggles.

These mixed reality systems and methods can be part of an intelligent surgical planning system that includes multiple subsystems that can be used to enhance surgical outcomes. In addition to the preoperative and intraoperative applications discussed above, an intelligent surgical planning system can include postoperative tools to assist with patient recovery and which can provide information that can be used to assist with and plan future surgical revisions or surgical cases for other patients.

Accordingly, systems and methods are also described herein that can be incorporated into an intelligent surgical planning system, such as artificial intelligence systems to assist with planning, implants with embedded sensors (e.g., smart implants) to provide postoperative feedback for use by the healthcare provider and the artificial intelligence system, and mobile applications to monitor and provide information to the patient and the healthcare provider in real-time or near real-time.

Visualization tools are available that utilize patient image data to generate three-dimensional models of bone contours to facilitate preoperative planning for joint repairs and replacements. These tools allow surgeons to design and/or select surgical guides and implant components that closely match the patient's anatomy. These tools can improve surgical outcomes by customizing a surgical plan for each patient. An example of such a visualization tool for shoulder repairs is the BLUEPRINT™ system available from Wright Medical Technology, Inc. The BLUEPRINT™ system provides the surgeon with two-dimensional planar views of the bone repair region as well as a three-dimensional virtual model of the repair region. The surgeon can use the BLUEPRINT™ system to select, design or modify appropriate implant components, determine how best to position and orient the implant components and how to shape the surface of the bone to receive the components, and design, select or modify surgical guide tool(s) or instruments to carry out the surgical plan. The information generated by the BLUEPRINT™ system is compiled in a preoperative surgical plan for the patient that is stored in a database at an appropriate location (e.g., on a server in a wide area network, a local area network, or a global network) where it can be accessed by the surgeon or other care provider, including before and during the actual surgery.

FIG. 1 is a block diagram of an orthopedic surgical system 100 according to an example of this disclosure. Orthopedic surgical system 100 includes a set of subsystems. In the example of FIG. 1, the subsystems include a virtual planning system 102, a planning support system 104, a manufacturing and delivery system 106, an intraoperative guidance system 108, a medical education system 110, a monitoring system 112, a predictive analytics system 114, and a communications network 116. In other examples, orthopedic surgical system 100 may include more, fewer, or different subsystems. For example, orthopedic surgical system 100 may omit medical education system 110, monitoring system 112, predictive analytics system 114, and/or other subsystems. In some examples, orthopedic surgical system 100 may be used for surgical tracking, in which case orthopedic surgical system 100 may be referred to as a surgical tracking system. In other cases, orthopedic surgical system 100 may be generally referred to as a medical device system.

Users of orthopedic surgical system 100 may use virtual planning system 102 to plan orthopedic surgeries. Users of orthopedic surgical system 100 may use planning support system 104 to review surgical plans generated using orthopedic surgical system 100. Manufacturing and delivery system 106 may assist with the manufacture and delivery of items needed to perform orthopedic surgeries. Intraoperative guidance system 108 provides guidance to assist users of orthopedic surgical system 100 in performing orthopedic surgeries. Medical education system 110 may assist with the education of users, such as healthcare professionals, patients, and other types of individuals. Pre- and postoperative monitoring system 112 may assist with monitoring patients before and after the patients undergo surgery. Predictive analytics system 114 may assist healthcare professionals with various types of predictions. For example, predictive analytics system 114 may apply artificial intelligence techniques to determine a classification of a condition of an orthopedic joint, e.g., a diagnosis, determine which type of surgery to perform on a patient and/or which type of implant to be used in the procedure, determine types of items that may be needed during the surgery, and so on.

The subsystems of orthopedic surgical system 100 (i.e., virtual planning system 102, planning support system 104, manufacturing and delivery system 106, intraoperative guidance system 108, medical education system 110, pre- and postoperative monitoring system 112, and predictive analytics system 114) may include various systems. The systems in the subsystems of orthopedic surgical system 100 may include various types of computing systems, computing devices, including server computers, personal computers, tablet computers, smartphones, display devices, Internet of Things (IoT) devices, visualization devices (e.g., MR visualization devices, VR visualization devices, holographic projectors, or other devices for presenting XR visualizations), surgical tools, and so on. A holographic projector, in some examples, may project a hologram for general viewing by multiple users or a single user without a headset, rather than viewing only by a user wearing a headset. For example, virtual planning system 102 may include a MR visualization device and one or more server devices, planning support system 104 may include one or more personal computers and one or more server devices, and so on. A computing system is a set of one or more computing systems configured to operate as a system. In some examples, one or more devices may be shared between two or more of the subsystems of orthopedic surgical system 100. For instance, in the previous examples, virtual planning system 102 and planning support system 104 may include the same server devices.

In the example of FIG. 1, the devices included in the subsystems of orthopedic surgical system 100 may communicate using communications network 116. Communications network 116 may include various types of communication networks including one or more wide-area networks, such as the Internet, local area networks, and so on. In some examples, communications network 116 may include wired and/or wireless communication links.

Many variations of orthopedic surgical system 100 are possible in accordance with techniques of this disclosure. Such variations may include more or fewer subsystems than the version of orthopedic surgical system 100 shown in FIG. 1. For example, FIG. 2 is a block diagram of an orthopedic surgical system 200 that includes one or more mixed reality (MR) systems, according to an example of this disclosure. Orthopedic surgical system 200 may be used for creating, verifying, updating, modifying and/or implementing a surgical plan. In some examples, the surgical plan can be created preoperatively, such as by using a virtual surgical planning system (e.g., the BLUEPRINT™ system), and then verified, modified, updated, and viewed intraoperatively, e.g., using MR visualization of the surgical plan. In other examples, orthopedic surgical system 200 can be used to create the surgical plan immediately prior to surgery or intraoperatively, as needed. In some examples, orthopedic surgical system 200 may be used for surgical tracking, in which case orthopedic surgical system 200 may be referred to as a surgical tracking system. In other cases, orthopedic surgical system 200 may be generally referred to as a medical device system.

In the example of FIG. 2, orthopedic surgical system 200 includes a preoperative surgical planning system 202, a healthcare facility 204 (e.g., a surgical center or hospital), a storage system 206, and a network 208 that allows a user at healthcare facility 204 to access stored patient information, such as medical history, image data corresponding to the damaged joint or bone and various parameters corresponding to a surgical plan that has been created preoperatively (as examples). Preoperative surgical planning system 202 may be equivalent to virtual planning system 102 of FIG. 1 and, in some examples, may generally correspond to a virtual planning system similar or identical to the BLUEPRINT™ system.

In the example of FIG. 2, healthcare facility 204 includes a mixed reality (MR) system 212. In some examples of this disclosure, MR system 212 includes one or more processing device(s) (P) 210 to provide functionalities that will be described in further detail below. Processing device(s) 210 may also be referred to as processor(s). In addition, one or more users of MR system 212 (e.g., a surgeon, nurse, or other care provider) can use processing device(s) (P) 210 to generate a request for a particular surgical plan or other patient information that is transmitted to storage system 206 via network 208. In response, storage system 206 returns the requested patient information to MR system 212. In some examples, the users can use other processing device(s) to request and receive information, such as one or more processing devices that are part of MR system 212, but not part of any visualization device, or one or more processing devices that are part of a visualization device (e.g., visualization device 213) of MR system 212, or a combination of one or more processing devices that are part of MR system 212, but not part of any visualization device, and one or more processing devices that are part of a visualization device (e.g., visualization device 213) that is part of MR system 212.

In some examples, multiple users can simultaneously use MR system 212. For example, MR system 212 can be used in a spectator mode in which multiple users each use their own visualization devices so that the users can view the same information at the same time and from the same point of view. In some examples, MR system 212 may be used in a mode in which multiple users each use their own visualization devices so that the users can view the same information from different points of view.

In some examples, processing device(s) 210 can provide a user interface to display data and receive input from users at healthcare facility 204. Processing device(s) 210 may be configured to control visualization device 213 to present a user interface. Furthermore, processing device(s) 210 may be configured to control visualization device 213 to present virtual images, such as 3D virtual models, 2D images, and so on. Processing device(s) 210 can include a variety of different processing or computing devices, such as servers, desktop computers, laptop computers, tablets, mobile phones and other electronic computing devices, or processors within such devices. In some examples, one or more of processing device(s) 210 can be located remote from healthcare facility 204. In some examples, processing device(s) 210 reside within visualization device 213. In some examples, at least one of processing device(s) 210 is external to visualization device 213. In some examples, one or more processing device(s) 210 reside within visualization device 213 and one or more of processing device(s) 210 are external to visualization device 213.

In the example of FIG. 2. MR system 212 also includes one or more memory or storage device(s) (M) 215 for storing data and instructions of software that can be executed by processing device(s) 210. The instructions of software can correspond to the functionality of MR system 212 described herein. In some examples, the functionalities of a virtual surgical planning application, such as the BLUEPRINT™ system, can also be stored and executed by processing device(s) 210 in conjunction with memory storage device(s) (M) 215. For instance, memory or storage system 215 may be configured to store data corresponding to at least a portion of a virtual surgical plan. In some examples, storage system 206 may be configured to store data corresponding to at least a portion of a virtual surgical plan. In some examples, memory or storage device(s) (M) 215 reside within visualization device 213. In some examples, memory or storage device(s) (M) 215 are external to visualization device 213. In some examples, memory or storage device(s) (M) 215 include a combination of one or more memory or storage devices within visualization device 213 and one or more memory or storage devices external to the visualization device.

Network 208 may be equivalent to network 116. Network 208 can include one or more wide area networks, local area networks, and/or global networks (e.g., the Internet) that connect preoperative surgical planning system 202 and MR system 212 to storage system 206. Storage system 206 can include one or more databases that can contain patient information, medical information, patient image data, and parameters that define the surgical plans. For example, medical images of the patient's diseased or damaged bone typically are generated preoperatively in preparation for an orthopedic surgical procedure. The medical images can include images of the relevant bone(s) taken along the sagittal plane and the coronal plane of the patient's body. The medical images can include X-ray images, magnetic resonance imaging (MRI) images, computerized tomography (CT) images, ultrasound images, and/or any other type of 2D or 3D image that provides information about the relevant surgical area. Storage system 206 also can include data identifying the implant components selected for a particular patient (e.g., type, size, etc.), surgical guides selected for a particular patient, and details of the surgical procedure, such as entry points, cutting planes, drilling axes, reaming depths, etc. Storage system 206 can be a cloud-based storage system (as shown) or can be located at healthcare facility 204 or at the location of preoperative surgical planning system 202 or can be part of MR system 212 or visualization device (VD) 213, as examples.

MR system 212 can be used by a surgeon before (e.g., preoperatively) or during the surgical procedure (e.g., intraoperatively) to create, review, verify, update, modify and/or implement a surgical plan. In some examples, MR system 212 may also be used after the surgical procedure (e.g., postoperatively) to review the results of the surgical procedure, assess whether revisions are required, or perform other postoperative tasks. To that end, MR system 212 may include a visualization device 213 that may be worn by the surgeon and (as will be explained in further detail below) is operable to display a variety of types of information, including a 3D virtual image of the patient's diseased, damaged, or postsurgical joint and details of the surgical plan, such as a 3D virtual image of the prosthetic implant components selected for the surgical plan, 3D virtual images of entry points for positioning the prosthetic components, alignment axes and cutting planes for aligning cutting or reaming tools to shape the bone surfaces, or drilling tools to define one or more holes in the bone surfaces, in the surgical procedure to properly orient and position the prosthetic components, surgical guides and instruments and their placement on the damaged joint, and any other information that may be useful to the surgeon to implement the surgical plan. MR system 212 can generate images of this information that are perceptible to the user of the visualization device 213 before and/or during the surgical procedure.

In some examples, MR system 212 includes multiple visualization devices (e.g., multiple instances of visualization device 213) so that multiple users can simultaneously see the same images and share the same 3D scene. In some such examples, one of the visualization devices can be designated as the master device and the other visualization devices can be designated as observers or spectators. Any observer device can be re-designated as the master device at any time, as may be desired by the users of MR system 212.

In this way, FIG. 2 illustrates a surgical planning system that includes a preoperative surgical planning system 202 to generate a virtual surgical plan customized to repair an anatomy of interest of a particular patient. For example, the virtual surgical plan may include a plan for an orthopedic joint repair surgical procedure (e.g., to attach a prosthetic to anatomy of a patient), such as one of a standard total shoulder arthroplasty or a reverse shoulder arthroplasty. In this example, details of the virtual surgical plan may include details relating to at least one of preparation of anatomy for attachment of a prosthetic or attachment of the prosthetic to the anatomy. For instance, details of the virtual surgical plan may include details relating to at least one of preparation of a glenoid bone, preparation of a humeral bone, attachment of a prosthetic to the glenoid bone, or attachment of a prosthetic to the humeral bone. In some examples, the orthopedic joint repair surgical procedure is one of a stemless standard total shoulder arthroplasty, a stemmed standard total shoulder arthroplasty, a stemless reverse shoulder arthroplasty, a stemmed reverse shoulder arthroplasty, an augmented glenoid standard total shoulder arthroplasty, and an augmented glenoid reverse shoulder arthroplasty. In accordance with one or more aspects of this disclosure, preoperative surgical planning system 202 may evaluate and/or recommend one or more ancillary steps for inclusion in a surgical plan, such as a virtual surgical plan, for a particular patient.

The virtual surgical plan may include a 3D virtual model corresponding to the anatomy of interest of the particular patient and a 3D model of a prosthetic component matched to the particular patient to repair the anatomy of interest or selected to repair the anatomy of interest. Furthermore, in the example of FIG. 2, the surgical planning system includes a storage system 206 to store data corresponding to the virtual surgical plan. The surgical planning system of FIG. 2 also includes MR system 212, which may comprise visualization device 213. In some examples, visualization device 213 is wearable by a user. In some examples, visualization device 213 is held by a user, or rests on a surface in a place accessible to the user. MR system 212 may be configured to present a user interface via visualization device 213. The user interface may present details of the virtual surgical plan for a particular patient. For instance, the details of the virtual surgical plan may include a 3D virtual model of an anatomy of interest of the particular patient. The user interface is visually perceptible to the user when the user is using visualization device 213. For instance, in one example, a screen of visualization device 213 may display real-world images and the user interface on a screen. In some examples, visualization device 213 may project virtual, holographic images onto see-through holographic lenses and also permit a user to see real-world objects of a real-world environment through the lenses. In other words, visualization device 213 may comprise one or more see-through holographic lenses and one or more display devices that present imagery to the user via the holographic lenses to present the user interface to the user.

In some examples, visualization device 213 is configured such that the user can manipulate the user interface (which is visually perceptible to the user when the user is wearing or otherwise using visualization device 213) to request and view details of the virtual surgical plan for the particular patient, including a 3D virtual model of the anatomy of interest (e.g., a 3D virtual bone model of the anatomy of interest, such as a glenoid bone or a humeral bone) and/or a 3D model of the prosthetic component selected to repair an anatomy of interest. In some such examples, visualization device 213 is configured such that the user can manipulate the user interface so that the user can view the virtual surgical plan intraoperatively, including (at least in some examples) the 3D virtual model of the anatomy of interest (e.g., a 3D virtual bone model of the anatomy of interest). In some examples, MR system 212 can be operated in an augmented surgery mode in which the user can manipulate the user interface intraoperatively so that the user can visually perceive details of the virtual surgical plan projected in a real environment, e.g., on a real anatomy of interest of the particular patient. In this disclosure, the terms real and real world may be used in a similar manner. For example, MR system 212 may present one or more virtual objects that provide guidance for preparation of a bone surface and placement of a prosthetic implant on the bone surface. Visualization device 213 may present one or more virtual objects in a manner in which the virtual objects appear to be overlaid on an actual, real anatomical object of the patient, within a real-world environment, e.g., by displaying the virtual object(s) with actual, real-world patient anatomy viewed by the user through holographic lenses. For example, the virtual objects may be 3D virtual objects that appear to reside within the real-world environment with the actual, real anatomical object.

FIG. 3 is a flowchart illustrating example phases of a surgical lifecycle 300. In the example of FIG. 3, surgical lifecycle 300 begins with a preoperative phase (302). During the preoperative phase, a surgical plan is developed. The preoperative phase is followed by a manufacturing and delivery phase (304). During the manufacturing and delivery phase, patient-specific items, such as parts and equipment, needed for executing the surgical plan are manufactured and delivered to a surgical site. In some examples, it is unnecessary to manufacture patient-specific items in order to execute the surgical plan. An intraoperative phase follows the manufacturing and delivery phase (306). The surgical plan is executed during the intraoperative phase. In other words, one or more persons perform the surgery on the patient during the intraoperative phase. The intraoperative phase is followed by the postoperative phase (308). The postoperative phase includes activities occurring after the surgical plan is complete. For example, the patient may be monitored during the postoperative phase for complications.

As described in this disclosure, orthopedic surgical system 100 (FIG. 1) may be used in one or more of preoperative phase 302, the manufacturing and delivery phase 304, the intraoperative phase 306, and the postoperative phase 308. For example, virtual planning system 102 and planning support system 104 may be used in preoperative phase 302. Manufacturing and delivery system 106 may be used in the manufacturing and delivery phase 304. Intraoperative guidance system 108 may be used in intraoperative phase 306. Some of the systems of FIG. 1 may be used in multiple phases of FIG. 3. For example, medical education system 110 may be used in one or more of preoperative phase 302, intraoperative phase 306, and postoperative phase 308; pre- and postoperative monitoring system 112 may be used in preoperative phase 302 and postoperative phase 308. Predictive analytics system 114 may be used in preoperative phase 302 and postoperative phase 308.

Various workflows may exist within the surgical process of FIG. 3. For example, different workflows within the surgical process of FIG. 3 may be appropriate for different types of surgeries. FIG. 4 is a flowchart illustrating preoperative, intraoperative and postoperative workflows in support of an orthopedic surgical procedure. In the example of FIG. 4, the surgical process begins with a medical consultation (400). During the medical consultation (400), a healthcare professional evaluates a medical condition of a patient. For instance, the healthcare professional may consult the patient with respect to the patient's symptoms. During the medical consultation (400), the healthcare professional may also discuss various treatment options with the patient. For instance, the healthcare professional may describe one or more different surgeries to address the patient's symptoms.

Furthermore, the example of FIG. 4 includes a case creation step (402). In other examples, the case creation step occurs before the medical consultation step. During the case creation step, the medical professional or other user establishes an electronic case file for the patient. The electronic case file for the patient may include information related to the patient, such as data regarding the patient's symptoms, patient range of motion observations, data regarding a surgical plan for the patient, medical images of the patients, notes regarding the patient, billing information regarding the patient, and so on.

The example of FIG. 4 includes a preoperative patient monitoring phase (404).

During the preoperative patient monitoring phase, the patient's symptoms may be monitored. For example, the patient may be suffering from pain associated with arthritis in the patient's shoulder. In this example, the patient's symptoms may not yet rise to the level of requiring an arthroplasty to replace the patient's shoulder. However, arthritis typically worsens over time. Accordingly, the patient's symptoms may be monitored to determine whether the time has come to perform a surgery on the patient's shoulder. Observations from the preoperative patient monitoring phase may be stored in the electronic case file for the patient. In some examples, predictive analytics system 114 may be used to predict when the patient may need surgery, to predict a course of treatment to delay or avoid surgery or make other predictions with respect to the patient's health.

Additionally, in the example of FIG. 4, a medical image acquisition step occurs during the preoperative phase (406). During the image acquisition step, medical images of the patient are generated. The medical images may be generated in a variety of ways. For instance, the images may be generated using a Computed Tomography (CT) process, a Magnetic Resonance Imaging (MRI) process, an ultrasound process, or another imaging process. The medical images generated during the image acquisition step include images of an anatomy of interest of the patient. For instance, if the patient's symptoms involve the patient's shoulder, medical images of the patient's shoulder may be generated. The medical images may be added to the patient's electronic case file. Healthcare professionals may be able to use the medical images in one or more of the preoperative, intraoperative, and postoperative phases.

Furthermore, in the example of FIG. 4, an automatic processing step may occur (408). During the automatic processing step, virtual planning system 102 (FIG. 1) may automatically develop a preliminary surgical plan for the patient. In some examples of this disclosure, virtual planning system 102 may use machine learning techniques to develop the preliminary surgical plan based on information in the patient's virtual case file.

In accordance with one or more aspects of this disclosure, when performing the automatic processing step, a computing system (e.g., virtual planning system 102) may select, based on the values of the plurality of parameters, one or more ancillary steps of a plurality of ancillary steps for inclusion in the arthroplasty procedure. The plurality of ancillary steps may be different than a standard set of steps included in the arthroplasty procedure (e.g., the plurality of ancillary steps are not included in the standard set of steps).

The example of FIG. 4 also includes a manual correction step (410). During the manual correction step, one or more human users may check and correct the determinations made during the automatic processing step. In some examples of this disclosure, one or more users may use mixed reality or virtual reality visualization devices during the manual correction step. In some examples, changes made during the manual correction step may be used as training data to refine the machine learning techniques applied by virtual planning system 102 during the automatic processing step.

A virtual planning step (412) may follow the manual correction step in FIG. 4. During the virtual planning step, a healthcare professional may develop a surgical plan for the patient. In some examples of this disclosure, one or more users may use mixed reality or virtual reality visualization devices during development of the surgical plan for the patient.

Furthermore, in the example of FIG. 4, intraoperative guidance may be generated (414). The intraoperative guidance may include guidance to a surgeon on how to execute the surgical plan. In some examples of this disclosure, virtual planning system 102 may generate at least part of the intraoperative guidance. In some examples, the surgeon or other user may contribute to the intraoperative guidance.

Additionally, in the example of FIG. 4, a step of selecting and manufacturing surgical items is performed (416). During the step of selecting and manufacturing surgical items, manufacturing and delivery system 106 (FIG. 1) may manufacture surgical items for use during the surgery described by the surgical plan. For example, the surgical items may include surgical implants, surgical tools, and other items required to perform the surgery described by the surgical plan.

In the example of FIG. 4, a surgical procedure may be performed with guidance from intraoperative system 108 (FIG. 1) (418). For example, a surgeon may perform the surgery while wearing a head-mounted MR visualization device of intraoperative system 108 that presents guidance information to the surgeon. The guidance information may help guide the surgeon through the surgery, providing guidance for various steps in a surgical workflow, including sequence of steps, details of individual steps, and tool or implant selection, implant placement and position, and bone surface preparation for various steps in the surgical procedure workflow.

Postoperative patient monitoring may occur after completion of the surgical procedure (420). During the postoperative patient monitoring step, healthcare outcomes of the patient may be monitored. Healthcare outcomes may include relief from symptoms, ranges of motion, complications, performance of implanted surgical items, and so on. Pre- and postoperative monitoring system 112 (FIG. 1) may assist in the postoperative patient monitoring step.

The medical consultation, case creation, preoperative patient monitoring, image acquisition, automatic processing, manual correction, and virtual planning steps of FIG. 4 are part of preoperative phase 302 of FIG. 3. The surgical procedures with guidance steps of FIG. 4 is part of intraoperative phase 306 of FIG. 3. The postoperative patient monitoring step of FIG. 4 is part of postoperative phase 308 of FIG. 3.

As mentioned above, one or more of the subsystems of orthopedic surgical system 100 may include one or more MR systems, such as MR system 212 (FIG. 2). Each MR system may include a visualization device. For instance, in the example of FIG. 2, MR system 212 includes visualization device 213. In some examples, in addition to including a visualization device, an MR system may include external computing resources that support the operations of the visualization device. For instance, the visualization device of an MR system may be communicatively coupled to a computing device (e.g., a personal computer, backpack computer, smartphone, etc.) that provides the external computing resources. Alternatively, adequate computing resources may be provided on or within visualization device 213 to perform necessary functions of the visualization device.

FIG. 5 is a schematic representation of visualization device 213 for use in an MR system, such as MR system 212 of FIG. 2, according to an example of this disclosure. As shown in the example of FIG. 5, visualization device 213 can include a variety of electronic components found in a computing system, including one or more processor(s) 514 (e.g., microprocessors or other types of processing units) and memory 516 that may be mounted on or within a frame 518. Furthermore, in the example of FIG. 5, visualization device 213 may include a transparent screen 520 that is positioned at eye level when visualization device 213 is worn by a user. In some examples, screen 520 can include one or more liquid crystal displays (LCDs) or other types of display screens on which images are perceptible to a surgeon who is wearing or otherwise using visualization device 213 via screen 520. Other display examples include organic light emitting diode (OLED) displays. In some examples, visualization device 213 can operate to project 3D images onto the user's retinas using techniques known in the art.

In some examples, screen 520 may include see-through holographic lenses. sometimes referred to as waveguides, that permit a user to see real-world objects through (e.g., beyond) the lenses and also see holographic imagery projected into the lenses and onto the user's retinas by displays, such as liquid crystal on silicon (LCoS) display devices, which are sometimes referred to as light engines or projectors, operating as an example of a holographic projection system 538 within visualization device 213. In other words, visualization device 213 may include one or more see-through holographic lenses to present virtual images to a user. Hence, in some examples, visualization device 213 can operate to project 3D images onto the user's retinas via screen 520, e.g., formed by holographic lenses. In this manner, visualization device 213 may be configured to present a 3D virtual image to a user within a real-world view observed through screen 520, e.g., such that the virtual image appears to form part of the real-world environment. In some examples, visualization device 213 may be a Microsoft HOLOLENS™ headset, available from Microsoft Corporation, of Redmond, Washington, USA, or a similar device, such as, for example, a similar MR visualization device that includes waveguides. The HOLOLENS™ device can be used to present 3D virtual objects via holographic lenses, or waveguides, while permitting a user to view actual objects in a real-world scene, i.e., in a real-world environment, through the holographic lenses.

Although the example of FIG. 5 illustrates visualization device 213 as a head-wearable device, visualization device 213 may have other forms and form factors. For instance, in some examples, visualization device 213 may be a handheld smartphone or tablet.

Visualization device 213 can also generate a user interface (UI) 522 that is visible to the user, e.g., as holographic imagery projected into see-through holographic lenses as described above. For example, UI 522 can include a variety of selectable widgets 524 that allow the user to interact with a MR system, such as MR system 212 of FIG. 2. Imagery presented by visualization device 213 may include, for example, one or more 3D virtual objects. Details of an example of UI 522 are described elsewhere in this disclosure. Visualization device 213 also can include a speaker or other sensory devices 526 that may be positioned adjacent the user's ears. Sensory devices 526 can convey audible information or other perceptible information (e.g., vibrations) to assist the user of visualization device 213.

Visualization device 213 can also include a transceiver 528 to connect visualization device 213 to a processing device 510 and/or to network 208 and/or to a computing cloud, such as via a wired communication protocol or a wireless protocol, e.g., Wi-Fi, Bluetooth, etc. Visualization device 213 also includes a variety of sensors to collect sensor data, such as one or more optical camera(s) 530 (or other optical sensors) and one or more depth camera(s) 532 (or other depth sensors), mounted to, on or within frame 518. In some examples, the optical sensor(s) 530 are operable to scan the geometry of the physical environment in which a user of MR system 212 is located (e.g., an operating room) and collect two-dimensional (2D) optical image data (either monochrome or color). Depth sensor(s) 532 are operable to provide 3D image data, such as by employing time of flight, stereo or other known or future-developed techniques for determining depth and thereby generating image data in three dimensions. Other sensors can include motion sensors 533 (e.g., Inertial Mass Unit (IMU) sensors, accelerometers, etc.) to assist with tracking movement.

MR system 212 processes the sensor data so that geometric, environmental, textural, or other types of landmarks (e.g., corners, edges or other lines, walls, floors, objects) in the user's environment or “scene” can be defined and movements within the scene can be detected. As an example, the various types of sensor data can be combined or fused so that the user of visualization device 213 can perceive 3D images that can be positioned, or fixed and/or moved within the scene. When a 3D image is fixed in the scene, the user can walk around the 3D image, view the 3D image from different perspectives, and manipulate the 3D image within the scene using hand gestures, voice commands, gaze line (or direction) and/or other control inputs. As another example, the sensor data can be processed so that the user can position a 3D virtual object (e.g., a bone model) on an observed physical object in the scene (e.g., a surface, the patient's real bone, etc.) and/or orient the 3D virtual object with other virtual images displayed in the scene. In some examples, the sensor data can be processed so that the user can position and fix a virtual representation of the surgical plan (or other widget, image or information) onto a surface, such as a wall of the operating room. Yet further, in some examples, the sensor data can be used to recognize surgical instruments and the position and/or location of those instruments.

Visualization device 213 may include one or more processors 514 and memory 516, e.g., within frame 518 of the visualization device. In some examples, one or more external computing resources 536 process and store information, such as sensor data, instead of or in addition to in-frame processor(s) 514 and memory 516. In this way, data processing and storage may be performed by one or more processors 514 and memory 516 within visualization device 213 and/or some of the processing and storage requirements may be offloaded from visualization device 213. Hence, in some examples, one or more processors that control the operation of visualization device 213 may be within visualization device 213, e.g., as processor(s) 514. Alternatively, in some examples, at least one of the processors that controls the operation of visualization device 213 may be external to visualization device 213, e.g., as processor(s) 210. Likewise, operation of visualization device 213 may, in some examples, be controlled in part by a combination one or more processors 514 within the visualization device and one or more processors 210 external to visualization device 213.

For instance, in some examples, when visualization device 213 is in the context of FIG. 2, processing of the sensor data can be performed by processing device(s) 210 in conjunction with memory or storage device(s) (M) 215. In some examples, processor(s) 514 and memory 516 mounted to frame 518 may provide sufficient computing resources to process the sensor data collected by cameras 530, 532 and motion sensors 533. In some examples, the sensor data can be processed using a Simultaneous Localization and Mapping (SLAM) algorithm, or other known or future-developed algorithms for processing and mapping 2D and 3D image data and tracking the position of visualization device 213 in the 3D scene. In some examples, image tracking may be performed using sensor processing and tracking functionality provided by the Microsoft HOLOLENS™ system, e.g., by one or more sensors and processors 514 within a visualization device 213 substantially conforming to the Microsoft HOLOLENS™ device or a similar mixed reality (MR) visualization device.

In some examples, MR system 212 can also include user-operated control device(s) 534 that allow the user to operate MR system 212, use MR system 212 in spectator mode (either as master or observer), interact with UI 522 and/or otherwise provide commands or requests to processing device(s) 210 or other systems connected to network 208. As examples, control device(s) 534 can include a microphone, a touch pad, a control panel, a motion sensor or other types of control input devices with which the user can interact.

FIG. 6 is a block diagram illustrating example components of visualization device 213 for use in a MR system. In the example of FIG. 6, visualization device 213 includes processors 514, a power supply 600, display device(s) 602, speakers 604, microphone(s) 606, input device(s) 608, output device(s) 610, storage device(s) 612, sensor(s) 614, and communication devices 616. In the example of FIG. 6, sensor(s) 616 may include depth sensor(s) 532, optical sensor(s) 530, motion sensor(s) 533, and orientation sensor(s) 618. Optical sensor(s) 530 may include cameras, such as Red-Green-Blue (RGB) video cameras, infrared cameras, or other types of sensors that form images from light. Display device(s) 602 may display imagery to present a user interface to the user.

Speakers 604, in some examples, may form part of sensory devices 526 shown in FIG. 5. In some examples, display devices 602 may include screen 520 shown in FIG. 5. For example, as discussed with reference to FIG. 5, display device(s) 602 may include see-through holographic lenses, in combination with projectors, that permit a user to see real-world objects, in a real-world environment, through the lenses, and also see virtual 3D holographic imagery projected into the lenses and onto the user's retinas, e.g., by a holographic projection system. In this example, virtual 3D holographic objects may appear to be placed within the real-world environment. In some examples, display devices 602 include one or more display screens, such as LCD display screens, OLED display screens, and so on. The user interface may present virtual images of details of the virtual surgical plan for a particular patient.

In some examples, a user may interact with and control visualization device 213 in a variety of ways. For example, microphones 606, and associated speech recognition processing circuitry or software, may recognize voice commands spoken by the user and, in response, perform any of a variety of operations, such as selection, activation, or deactivation of various functions associated with surgical planning, intra-operative guidance, or the like. As another example, one or more cameras or other optical sensors 530 of sensors 614 may detect and interpret gestures to perform operations as described above. As a further example, sensors 614 may sense gaze direction and perform various operations as described elsewhere in this disclosure. In some examples, input devices 608 may receive manual input from a user, e.g., via a handheld controller including one or more buttons, a keypad, a touchscreen, joystick, trackball, and/or other manual input media, and perform, in response to the manual user input, various operations as described above.

As discussed above, surgical lifecycle 300 may include a preoperative phase 302 (FIG. 3). One or more users may use orthopedic surgical system 100 in preoperative phase 302. For instance, orthopedic surgical system 100 may include virtual planning system 102 to help the one or more users generate a virtual surgical plan that may be customized to an anatomy of interest of a particular patient. As described herein, the virtual surgical plan may include a 3-dimensional virtual model that corresponds to the anatomy of interest of the particular patient and a 3-dimensional model of one or more prosthetic components matched to the particular patient to repair the anatomy of interest or selected to repair the anatomy of interest. The virtual surgical plan also may include a 3-dimensional virtual model of guidance information to guide a surgeon in performing the surgical procedure, e.g., in preparing bone surfaces or tissue and placing implantable prosthetic hardware relative to such bone surfaces or tissue.

FIG. 7 is a conceptual diagram illustrating an example setting in which a set of users use MR systems of orthopedic surgical system 100 during preoperative phase 302. In the example of FIG. 7, a surgeon may use (e.g., wear) a visualization device (e.g., visualization device 213) of a first MR system 700A (e.g., MR system 212). The visualization device of MR system 700A may present MR preoperative planning content 702 to the surgeon during preoperative phase 302. As described in detail elsewhere in this disclosure, MR preoperative planning content 702 may help the surgeon plan for a surgery.

Furthermore, in the example of FIG. 7, one or more other users may use visualization devices of MR systems of orthopedic surgical system 100 to view MR preoperative planning content 702. For example, a patient may use a visualization device of a second MR system 700B during preoperative phase 302. The visualization device of MR system 700B may present MR preoperative planning content 702 to the patient. For instance, as described in detail elsewhere in this disclosure, MR preoperative planning content 702 may include virtual 3D model information to be presented using MR to help the patient understand one or more of the patient's current condition and the surgery to be performed on the patient.

In the example of FIG. 7, a nurse or other healthcare professional may use a visualization device of a third MR system 700C during preoperative phase 302. The visualization device of MR system 700C may present MR preoperative planning content 702 to the nurse or other healthcare professional. For instance, in one example, MR preoperative planning content 702 may help the nurse understand a surgery before the surgery happens.

Furthermore, in the example of FIG. 7, a second surgeon may use a visualization device of a fourth MR system 700D. The visualization device of MR system 700D may present MR preoperative planning content 702 to the second surgeon. This may allow the surgeons to collaborate to develop and review a surgical plan for the patient. For instance, surgeons may view and manipulate the same preoperative planning content 702 at the same or different times. MR systems 700A, 700B, 700C, and 700D may collectively be referred to herein as “MR systems 700.”

Thus, as described in the examples above, two or more of the individuals described above (e.g., the first surgeon, the patient, the nurse, and the second surgeon) can view the same or different MR preoperative planning content 702 at the same time. In examples where two or more of the individuals are viewing the same MR preoperative planning content 702 at the same time, the two or more individuals may concurrently view the same MR preoperative guidance content 702 from the same or different perspectives. Moreover, in some examples, two or more of the individuals described above can view the same or different MR preoperative planning content 702 at different times. Preoperative planning content 702 may include an information model of a surgical plan, virtual 3D model information representing patient anatomy, such as bone and/or tissue, alone, or in combination with virtual 3D model information representing surgical procedure steps and/or implant placement and positioning. Examples of preoperative planning content 702 may include a surgical plan for a shoulder arthroplasty, virtual 3D model information representing scapula and/or glenoid bone, or representing humeral bone, with virtual 3D model information of instruments to be applied to the bone or implants to be positioned on or in the bone. In some examples, multiple users may be able to change and manipulate preoperative planning content 702.

FIG. 8 is a flowchart illustrating example steps in preoperative phase 302 of surgical lifecycle 300. In other examples, preoperative phase 302 may include more, fewer, or different steps. Moreover, in other examples, one or more of the steps of FIG. 8 may be performed in different orders. In some examples, one or more of the steps may be performed automatically within a surgical planning system such as virtual planning system 102 (FIG. 1) or 202 (FIG. 2).

In the example of FIG. 8, a model of the area of interest is generated (800). For example, a scan (e.g., a CT scan, MRI scan, or other type of scan) of the area of interest may be performed. For example, if the area of interest is the patient's shoulder, a scan of the patient's shoulder may be performed. Furthermore, a pathology in the area of interest may be classified (802). In some examples, the pathology of the area of interest may be classified based on the scan of the area of interest. For example, if the area of interest is the user's shoulder, a surgeon may determine what is wrong with the patient's shoulder based on the scan of the patient's shoulder and provide a shoulder classification indicating the classification or diagnosis, e.g., such as primary glenoid humeral osteoarthritis (PGHOA), rotator cuff tear arthropathy (RCTA) instability, massive rotator cuff tear (MRCT), rheumatoid arthritis, post-traumatic arthritis, and osteoarthritis.

Additionally, a surgical plan may be selected based on the pathology (804). The surgical plan is a plan to address the pathology. For instance, in the example where the area of interest is the patient's shoulder, the surgical plan may be selected from an anatomical shoulder arthroplasty, a reverse shoulder arthroplasty, a post-trauma shoulder arthroplasty, or a revision to a previous shoulder arthroplasty. The surgical plan may then be tailored to patient (806). For instance, tailoring the surgical plan may involve selecting and/or sizing surgical items needed to perform the selected surgical plan. Additionally, the surgical plan may be tailored to the patient in order to address issues specific to the patient, such as the presence of osteophytes. As described in detail elsewhere in this disclosure, one or more users may use mixed reality systems of orthopedic surgical system 100 to tailor the surgical plan to the patient. As also described elsewhere in this disclosure, tailoring the surgical plan may involve selecting one or more ancillary steps of a plurality of ancillary steps for inclusion in the arthroplasty procedure and including the selected one or more ancillary steps in the selected surgical plan.

The surgical plan may then be reviewed (808). For instance, a consulting surgeon may review the surgical plan before the surgical plan is executed. As described in detail elsewhere in this disclosure, one or more users may use MR systems of orthopedic surgical system 100 to review the surgical plan. In some examples, a surgeon may modify the surgical plan using an MR system by interacting with a UI and displayed elements, e.g., to select a different procedure, change the sizing, shape or positioning of implants, or change the angle, depth or amount of cutting or reaming of the bone surface to accommodate an implant.

Additionally, in the example of FIG. 8, surgical items needed to execute the surgical plan may be requested (810). As described in the following sections of this disclosure, orthopedic surgical system 100 may assist various users in performing one or more of the preoperative steps of FIG. 8.

FIG. 9 illustrates an example welcome page for selecting a surgical case, according to an example of this disclosure. The Welcome page, which may be presented by MR visualization device 213 to a user, displays a menu 904 that allows the user to scroll through and select a specific patient's surgical plan that is stored on and retrieved from storage system 206 in system 200 (FIG. 2) or in memory or storage device 215 of MR visualization device 213 (FIG. 2).

FIG. 10 illustrates an example of a page of a user interface of a mixed reality system, according to an example of this disclosure, e.g. as produced for a particular patient's surgical plan selected from the welcome page of FIG. 9. Using visualization device 213, a user can perceive and interact with UI 522. In the example shown in FIG. 10, UI 522 includes a workflow bar 1000 with selectable buttons 1002 that represent a surgical workflow, spanning various surgical procedure steps for operations on the humerus and glenoid in a shoulder arthroplasty procedure. Selection of a button 1002 can lead to display of various selectable widgets with which the user can interact, such as by using hand gestures, voice commands, gaze direction, connected lens and/or other control inputs. Selection of widgets can launch various modes of operation of MR system 212, display information or images generated by MR system 212, allow the user to further control and/or manipulate the information and images, lead to further selectable menus or widgets, etc.

The user can also organize or customize UI 522 by manipulating, moving and orienting any of the displayed widgets according to the user's preferences, such as by visualization device 213 or other device detecting gaze direction, hand gestures and/or voice commands. Further, the location of widgets that are displayed to the user can be fixed relative to the scene. Thus, as the user's gaze (i.e., eye direction) moves to view other features of the user interface 522, other virtual images, and/or real objects physically present in the scene (e.g., the patient, an instrument set, etc.), the widgets may remain stationary and do not interfere with the user's view of the other features and objects. As yet another example, the user can control the opacity or transparency of the widgets or any other displayed images or information. The user also can navigate in any direction between the buttons 1002 on the workflow bar 1000 and can select any button 1002 at any time during use of MR system 212. Selection and manipulation of widgets, information, images or other displayed features can be implemented based on visualization device 213 or other device detecting user gaze direction, hand motions, voice commands or any combinations thereof.

In the example of FIG. 10, UI 522 is configured for use in shoulder repair procedures and includes, as examples, buttons 1002 on workflow bar 1000 that correspond to a “Welcome” page, a “Planning” page, a “Graft” page, a “Humerus Cut” page, an “Install Guide” page, a “Glenoid Reaming” page, and a “Glenoid Implant” page. The presentation of the “Install Guide” page may be optional as, in some examples, glenoid reaming may be accomplished using virtual guidance and without the application of a glenoid guide.

As shown FIG. 10, the “Planning” page in this example of UI 522 displays various information and images corresponding to the selected surgical plan, including an image 1006 of a surgical plan file (e.g., a pdf file or other appropriate media format) that corresponds to the selected plan (including preoperative and postoperative information); a 3D virtual bone model 1008 and a 3D virtual implant model 1010 along with a 3D image navigation bar 1012 for manipulating the 3D virtual models 1008, 1010 (which may be referred to as 3D images); a viewer 1014 and a viewer navigation bar 1016 for viewing a multi-planar view associated with the selected surgical plan. MR system 212 may present the “Planning” page as a virtual MR object to the user during preoperative phase 302 (FIG. 3). For instance, MR system 212 may present the “Planning” page to the user to help the user classify a pathology, select a surgical plan, tailor the surgical plan to the patient, revise the surgical plan, and review the surgical plan, as described in steps 802, 804, 806, and 808 of FIG. 8.

The surgical plan image 1006 may be a compilation of preoperative (and, optionally, postoperative) patient information and the surgical plan for the patient that are stored in a database in storage system 206. In some examples, surgical plan image 1006 can correspond to a multi-page document through which the user can browse. For example, further images of pages can display patient information, information regarding the anatomy of interest, postoperative measurements, and various 2D images of the anatomy of interest. Yet further page images can include, as examples, planning information associated with an implant selected for the patient, such as anatomy measurements and implant size, type and dimensions; planar images of the anatomy of interest; images of a 3D model showing the positioning and orientation of a surgical guide selected for the patient to assist with execution of the surgical plan; etc.

It should be understood that the surgical plan image 1006 can be displayed in any suitable format and arrangement and that other implementations of the systems and techniques described herein can include different information depending upon the needs of the application in which the plan image 1006 is used.

Referring again FIG. 10, the Planning page of UI 522 also may provide images of the 3D virtual bone model 1008 and the 3D model of the implant components 1010 along with navigation bar 1012 for manipulating 3D virtual models 1008, 1010. For example, selection or de-selection of the icons on navigation bar 1012 allow the user to selectively view different portions of 3D virtual bone model 1008 with or without the various implant components 1010. For example, the scapula of virtual bone model 1008 and the glenoid implant of implant model 1010 have been de-selected, leaving only the humerus bone and the humeral implant components visible. Other icons can allow the user to zoom in or out, and the user also can rotate and re-orient 3D virtual models 1008, 1010, e.g., using gaze detection, hand gestures and/or voice commands.

Returning to the example of FIG. 10, the Planning page presented by visualization device 213 also includes multi-planar image viewer 1014 (e.g., a DICOM viewer) and navigation bar 1016 that allow the user to view patient image data and to switch between displayed slices and orientations. For example, the user can select 2D Planes icons 1026 on navigation bar 1016 so that the user can view the 2D sagittal and coronal planes of the patient's body in multi-planar image viewer 1014.

Workflow bar 1000 in FIG. 10 includes further pages that correspond to steps in the surgical workflow for a particular orthopedic procedure (here, a shoulder repair procedure). In the example of FIG. 10, workflow bar 1000 includes elements labeled “Graft,” “Humerus Cut,” “Install Guide,” “Glenoid Reaming,” and “Glenoid Implant” that correspond to workflow pages for steps in the surgical workflow for a shoulder repair procedure. In general, these workflow pages include information that can be useful for a health care professional during planning of or during performance of the surgical procedure, and the information presented upon selection of these pages is selected and organized in a manner that is intended to minimize disturbances or distractions to the surgeon during a procedure. Thus, the amount of displayed information is optimized and the utility of the displayed information is maximized. These workflow pages may be used as part of intraoperative phase 306 (FIG. 3) to guide a surgeon, nurse or other medical technician through the steps in a surgical procedure. In some examples, these workflow pages may be used as part of preoperative phase 302 (FIG. 3) to enable a user to visualize 3-dimensional models of objects involved in various steps of a surgical workflow.

In the example shown, each workflow page that can be selected by the user (e.g., a surgeon) can include an Augment Surgery widget that, when selected, launches an operational mode of MR system 212 in which a user using (e.g., wearing) visualization device 213 (FIG. 2) can see the details (e.g., virtual images of details) of the surgical plan projected and matched onto the patient bone and use the plan intraoperatively to assist with the surgical procedure. In general, the Augment Surgery mode allows the surgeon to register the virtual 3D model of the patient's anatomy of interest (e.g., glenoid) with the observed real anatomy so that the surgeon can use the virtual surgical planning to assist with implementation of the real surgical procedure, as will be explained in further detail below. There may be different Augment Surgery widgets for each of the steps of the surgery that the surgeon uses during actual surgery. The Augment Surgery widgets for different steps may include different text, control, icons, graphics, etc.

In this example of a shoulder repair procedure, and with reference FIG. 10, the workflow pages of UI 522 that can be used by the surgeon include “Graft”, “Humerus Cut”, “Install Guide”, “Glenoid Reaming”, and “Glenoid Implant”. The “Graft” step and “Install Guide” steps may be optional. For example, it may not be necessary to take a graft in every procedure and the use of a glenoid reaming guide may not be necessary if MR reaming axis guidance is presented to the user by visualization device 213. A user may view the workflow pages during the preoperative phase 302, during the intraoperative phase 306, or at other times. It may be helpful to a surgeon to view the workflow pages during the preoperative phase 302 in order to tailor a surgical plan for the patient, to review the steps of a surgical plan, or perform other tasks. It may be helpful to a surgeon to view the workflow pages in the intraoperative phase 306 to refresh the surgeon on the anatomy of the patient involved in the corresponding surgical steps, to obtain information on how to perform certain actions during the corresponding surgical steps, to take inventory of surgical instruments, implants or other surgical items needed in the surgical steps, and so on. As mentioned, each of the workflow pages generally corresponds to a step in the workflow for the particular surgical procedure.

The images displayed on UI 522 of MR system 212 can be viewed outside or within the surgical operating environment and, in spectator mode, can be viewed by multiple users outside and within the operating environment at the same time. In some circumstances, such as in the operating environment, the surgeon may find it useful to use a control device 534 to direct visualization device 213 such that certain information should be locked into position on a wall or other surface of the operating room, as an example, so that the information does not impede the surgeon's view during the procedure. For example, relevant surgical steps of the surgical plan can be selectively displayed and used by the surgeon or other care providers to guide the surgical procedure.

In various some examples, the display of surgical steps can be automatically controlled so that only the relevant steps are displayed at the appropriate times during the surgical procedure.

As discussed above, surgical lifecycle 300 may include an intraoperative phase 306 during which a surgical operation is performed. One or more users may use orthopedic surgical system 100 in intraoperative phase 306. In some examples, one or more users, including at least one surgeon, may use orthopedic surgical system 100 in an intraoperative setting to perform shoulder surgery. FIG. 11 is a flowchart illustrating example stages of a shoulder joint repair surgery. As discussed above, FIG. 11 describes an example surgical process for a shoulder surgery. The surgeon may wear or otherwise use visualization device 213 during each step of the surgical process of FIG. 10. In other examples, a shoulder surgery may include more, fewer, or different steps. For example, a shoulder surgery may include a step for adding a bone graft, adding cement, and/or other steps. In some examples, visualization device 213 may present virtual guidance to guide the surgeon, nurse, or other users, through the steps in the surgical workflow.

The stages (e.g., steps) of FIG. 11 may represent one example of a standard set of steps of a should arthroplasty. As discussed herein, one or more processors of a computing system may select and/or recommend one or more ancillary steps for performance in addition to a standard set of steps. For instance, one or more processors of preoperative surgical planning system 202 may select and/or recommend one or more ancillary steps for performance during a shoulder arthroplasty in addition to the standard set of steps shown in FIG. 11.

In the example of FIG. 11, a surgeon performs an incision process (1900). During the incision process, the surgeon makes a series of incisions to expose a patient's shoulder joint. In some examples, an MR system (e.g., MR system 212, MR system 1800A, etc.) may help the surgeon perform the incision process, e.g., by displaying virtual guidance imagery illustrating how to where to make the incision.

Furthermore, in the example of FIG. 11, the surgeon may perform a humerus cut process (1902). During the humerus cut process, the surgeon may remove a portion of the humeral head of the patient's humerus. Removing the portion of the humeral head may allow the surgeon to access the patient's glenoid. Additionally, removing the portion of the humeral head may allow the surgeon to subsequently replace the portion of the humeral head with a humeral implant compatible with a glenoid implant that the surgeon plans to implant in the patient's glenoid.

As discussed above, the humerus preparation process may enable the surgeon to access the patient's glenoid. In the example of FIG. 11, after performing the humerus preparation process, the surgeon may perform a registration process that registers a virtual glenoid object with the patient's actual glenoid bone (1904) in the field of view presented to the surgeon by visualization device 213.

The surgeon may perform a reaming axis drilling process (1906). During the reaming axis drilling process, the surgeon may drill a reaming axis guide pin hole in the patient's glenoid to receive a reaming guide pin. In some examples, at a later stage of the shoulder surgery, the surgeon may insert a reaming axis pin into the reaming axis guide pin hole. In some examples, the reaming axis pin may itself be the drill bit that is used to drill the reaming axis guide pin hole (e.g., the reaming axis pin may be self-tapping). Thus, in such examples, it may be unnecessary to perform a separate step of inserting the reaming axis pin. In some examples, an MR system (e.g., MR system 212, MR system 1800A, etc.) may present a virtual reaming axis to help the surgeon perform the drilling in alignment with the reaming axis and thereby place the reaming guide pin in the correct location and with the correct orientation.

The surgeon may perform the reaming axis drilling process in one of various ways. For example, the surgeon may perform a guide-based process to drill the reaming axis pin hole. In the case, a physical guide is placed on the glenoid to guide drilling of the reaming axis pin hole. In other examples, the surgeon may perform a guide-free process, e.g., with presentation of a virtual reaming axis that guides the surgeon to drill the reaming axis pin hole with proper alignment. An MR system (e.g., MR system 212, MR system 1800A, etc.) may help the surgeon perform either of these processes to drill the reaming axis pin hole.

Furthermore, in the surgical process of FIG. 11, the surgeon may perform a reaming axis pin insertion process (1908). During the reaming axis pin insertion process, the surgeon inserts a reaming axis pin into the reaming axis pin hole drilled into the patient's scapula. In some examples, an MR system (e.g., MR system 212, MR system 1800A, etc.) may present virtual guidance information to help the surgeon perform the reaming axis pin insertion process.

After performing the reaming axis insertion process, the surgeon may perform a glenoid reaming process (1910). During the glenoid reaming process, the surgeon reams the patient's glenoid. Reaming the patient's glenoid may result in an appropriate surface for installation of a glenoid implant. In some examples, to ream the patient's glenoid, the surgeon may affix a reaming bit to a surgical drill. The reaming bit defines an axial cavity along an axis of rotation of the reaming bit. The axial cavity has an inner diameter corresponding to an outer diameter of the reaming axis pin. After affixing the reaming bit to the surgical drill, the surgeon may position the reaming bit so that the reaming axis pin is in the axial cavity of the reaming bit. Thus, during the glenoid reaming process, the reaming bit may spin around the reaming axis pin. In this way, the reaming axis pin may prevent the reaming bit from wandering during the glenoid reaming process. In some examples, multiple tools may be used to ream the patient's glenoid. An MR system (e.g., MR system 212, MR system 1800A, etc.) may present virtual guidance to help the surgeon or other users to perform the glenoid reaming process. For example, the MR system may help a user, such as the surgeon, select a reaming bit to use in the glenoid reaming process. In some examples, the MR system presents virtual guidance to help the surgeon control the depth to which the surgeon reams the user's glenoid. In some examples, the glenoid reaming process includes a paleo reaming step and a neo reaming step to ream different parts of the patient's glenoid.

Additionally, in the surgical process of FIG. 11, the surgeon may perform a glenoid implant installation process (1912). During the glenoid implant installation process, the surgeon installs a glenoid implant in the patient's glenoid. In some instances, when the surgeon is performing an anatomical shoulder arthroplasty, the glenoid implant has a concave surface that acts as a replacement for the user's natural glenoid. In other instances, when the surgeon is performing a reverse shoulder arthroplasty, the glenoid implant has a convex surface that acts as a replacement for the user's natural humeral head. In this reverse shoulder arthroplasty, the surgeon may install a humeral implant that has a concave surface that slides over the convex surface of the glenoid implant. As in the other steps of the shoulder surgery of FIG. 11, an MR system (e.g., MR system 212, MR system 1800A, etc.) may present virtual guidance to help the surgeon perform the glenoid installation process.

In some examples, the glenoid implantation process includes a process to fix the glenoid implant to the patient's scapula (1914). In some examples, the process to fix the glenoid implant to the patient's scapula includes drilling one or more anchor holes or one or more screw holes into the patient's scapula and positioning an anchor such as one or more pegs or a keel of the implant in the anchor hole(s) and/or inserting screws through the glenoid implant and the screw holes, possibly with the use of cement or other adhesive. An MR system (e.g., MR system 212, MR system 1800A, etc.) may present virtual guidance to help the surgeon with the process of fixing the glenoid implant the glenoid bone, e.g., including virtual guidance indicating anchor or screw holes to be drilled or otherwise formed in the glenoid, and the placement of anchors or screws in the holes.

Furthermore, in the example of FIG. 11, the surgeon may perform a humerus preparation process (1916). During the humerus preparation process, the surgeon prepares the humerus for the installation of a humerus implant. In instances where the surgeon is performing an anatomical shoulder arthroplasty, the humerus implant may have a convex surface that acts as a replacement for the patient's natural humeral head. The convex surface of the humerus implant slides within the concave surface of the glenoid implant. In instances where the surgeon is performing a reverse shoulder arthroplasty, the humerus implant may have a concave surface and the glenoid implant has a corresponding convex surface. As described elsewhere in this disclosure, an MR system (e.g., MR system 212, MR system 1800A, etc.) may present virtual guidance information to help the surgeon perform the humerus preparation process.

Furthermore, in the example surgical process of FIG. 11, the surgeon may perform a humerus implant installation process (1918). During the humerus implant installation process, the surgeon installs a humerus implant on the patient's humerus. As described elsewhere in this disclosure, an MR system (e.g., MR system 212, MR system 1800A, etc.) may present virtual guidance to help the surgeon perform the humerus preparation process.

After performing the humerus implant installation process, the surgeon may perform an implant alignment process that aligns the installed glenoid implant and the installed humerus implant (1920). For example, in instances where the surgeon is performing an anatomical shoulder arthroplasty, the surgeon may nest the convex surface of the humerus implant into the concave surface of the glenoid implant. In instances where the surgeon is performing a reverse shoulder arthroplasty, the surgeon may nest the convex surface of the glenoid implant into the concave surface of the humerus implant. Subsequently, the surgeon may perform a wound closure process (1922). During the wound closure process, the surgeon may reconnect tissues severed during the incision process in order to close the wound in the patient's shoulder.

As mentioned elsewhere in this disclosure, a user interface of MR system 212 may include workflow bar 1000. Workflow bar 1000 include icons corresponding to workflow pages. In some examples, each workflow page that can be selected by the user (e.g., a surgeon) can include an Augment Surgery widget, that, when selected, launches an operational mode of MR system 212 in which a user wearing or otherwise using visualization device 213 can see the details (e.g., virtual images of details) of the surgical plan projected and matched onto the patient bone and use the plan intraoperatively to assist with the surgical procedure. In general, the Augment Surgery mode allows the surgeon to register the virtual 3D model of the patient's anatomy of interest (e.g., glenoid) with the observed real anatomy so that the surgeon can use the virtual surgical planning to assist with implementation of the real surgical procedure, as will be explained in further detail below.

For a shoulder arthroplasty application, the registration process may start by virtualization device 213 presenting the user with 3D virtual bone model 1008 of the patient's scapula and glenoid that was generated from preoperative images of the patient's anatomy, e.g., by surgical planning system 102. The user can then manipulate 3D virtual bone model 1008 in a manner that aligns and orients 3D virtual bone model 1008 with the patient's real scapula and glenoid that the user is observing in the operating environment. As such, in some examples, the MR system may receive user input to aid in the initialization and/or registration. However, as discussed above, in some examples, the MR system may perform the initialization and/or registration process automatically (e.g., without receiving user input to position the 3D bone model). For other types of arthroplasty procedures, such as for the knee, hip, foot, ankle or elbow, different relevant bone structures can be displayed as virtual 3D images and aligned and oriented in a similar manner with the patient's actual, real anatomy.

Regardless of the particular type of joint or anatomical structure involved, selection of the augment surgery mode initiates a procedure where 3D virtual bone model 1008 is registered with an observed bone structure. In general, the registration procedure can be considered as a classical optimization problem (e.g., either minimization or maximization). For a shoulder arthroplasty procedure, known inputs to the optimization (e.g., minimization) analysis are the 3D geometry of the observed patient's bone (derived from sensor data from the visualization device 213, including depth data from the depth camera(s) 532) and the geometry of the 3D virtual bone derived during the virtual surgical planning state (such as by using the BLUEPRINT™ system). Other inputs include details of the surgical plan (also derived during the virtual surgical planning stage, such as by using the BLUEPRINT™ system), such as the position and orientation of entry points, cutting planes, reaming axes and/or drilling axes, as well as reaming or drilling depths for shaping the bone structure, the type, size and shape of the prosthetic components, and the position and orientation at which the prosthetic components will be placed or, in the case of a fracture, the manner in which the bone structure will be rebuilt.

Upon selection of a particular patient from the welcome page of UI 522 of MR system 212 (FIG. 5), the surgical planning parameters associated with that patient are connected with the patient's 3D virtual bone model 1008, e.g., by one or more processors of visualization device 213. In the Augment Surgery mode, registration of 3D virtual bone model 1008 (with the connected preplanning parameters) with the observed bone by visualization device 213 allows the surgeon to visualize virtual representations of the surgical planning parameters on the patient.

On occasion, during a surgery, the surgeon may determine that there is a need to modify the preoperative surgical plan. MR system 212 allows for intraoperative modifications to the surgical plan that then can be executed in the Augmented Surgery Mode. For instance, in some examples, the user can manipulate the user interface so that the user can view the virtual surgical plan intraoperatively, including at least the 3D virtual bone anatomy of interest. In such examples, the user can manipulate the user interface so that the user can modify the virtual surgical plan intraoperatively. As an example, selection of the Planning page on the workflow bar 1000 of the UI 522 shown in FIG. 10, which allows the surgeon to view and manipulate 3D virtual bone model 1008 of the patient's anatomy and the prosthetic implant components 1010. Using UI 522, the surgeon can rotate and translate the implant components 1010 and change their type and size if desired. If changes are made, the virtual surgical plan is automatically updated with the new parameters, which can then be connected with 3D virtual bone model 1008 when in the Augment Surgery mode. If registration has previously been completed with the prior version of the virtual surgical plan, the planning parameters can be updated. If the modifications to the virtual surgical plan require the surgeon to repeat the registration process, MR system 212 can prompt the surgeon to do so.

As discussed elsewhere in this disclosure, orthopedic surgical procedures may involve performing various work on a patient's anatomy. Some examples of work that may be performed include, but are not necessarily limited to, cutting, drilling, reaming, screwing, adhering, and impacting. In general, it may be desirable for a practitioner (e.g., surgeon, physician's assistant, nurse, etc.) to perform the work as accurately as possible. For instance, if a surgical plan for implanting a prosthetic in a particular patient specifies that a portion of the patient's anatomy is to be reamed at a particular diameter to a particular depth, it may be desirable for the surgeon to ream the portion of the patient's anatomy to as close as possible to the particular diameter and to the particular depth (e.g., to increase the likelihood that the prosthetic will fit and function as planned and thereby promote a good health outcome for the patient).

In some examples, a surgeon may perform one of more work operations “free hand” (i.e., by applying or otherwise using a tool without mechanical or visual guides/aids for the tool). In some examples, in the course of an orthopedic surgical procedure, a surgeon may perform one of more work operations, which also may be referred to as surgical steps, with the assistance of a mechanical guide. In some examples, a visualization system, such as MR visualization system 212, may be configured to display virtual guidance including one or more virtual guides for performing work on a portion of a patient's anatomy.

For instance, the visualization system may display a virtual cutting plane overlaid on an anatomic neck of the patient's humerus. In some examples, a user such as a surgeon may view real-world objects in a real-world scene. The real-world scene may be in a real-world environment such as a surgical operating room. In this disclosure, the terms real and real-world may be used in a similar manner. The real-world objects viewed by the user in the real-world scene may include the patient's actual, real anatomy, such as an actual glenoid or humerus, exposed during surgery. The user may view the real-world objects via a see-through (e.g., transparent) screen, such as see-through holographic lenses, of a head-mounted MR visualization device, such as visualization device 213, and also see virtual guidance such as virtual MR objects that appear to be projected on the screen or within the real-world scene, such that the MR guidance object(s) appear to be part of the real-world scene, e.g., with the virtual objects appearing to the user to be integrated with the actual, real-world scene. For example, the virtual cutting plane/line may be projected on the screen of a MR visualization device, such as visualization device 213, such that the cutting plane is overlaid on, and appears to be placed within, an actual, observed view of the patient's actual humerus viewed by the surgeon through the transparent screen, e.g., through see-through holographic lenses. Hence, in this example, the virtual cutting plane/line may be a virtual 3D object that appears to be part of the real-world environment, along with actual, real-world objects.

A screen through which the surgeon views the actual, real anatomy and also observes the virtual objects, such as virtual anatomy and/or virtual surgical guidance, may include one or more see-through holographic lenses. The holographic lenses, sometimes referred to as “waveguides,” may permit the user to view real-world objects through the lenses and display projected holographic objects for viewing by the user. As discussed above, an example of a suitable head-mounted MR device for visualization device 213 is the Microsoft HOLOLENS™ headset, available from Microsoft Corporation, of Redmond, Washington, USA. The HOLOLENS™ headset includes see-through, holographic lenses, also referred to as waveguides, in which projected images are presented to a user. The HOLOLENS™ headset also includes an internal computer, cameras and sensors, and a projection system to project the holographic content via the holographic lenses for viewing by the user. In general, the Microsoft HOLOLENS™ headset or a similar MR visualization device may include, as mentioned above, LCoS display devices that project images into holographic lenses, also referred to as waveguides, e.g., via optical components that couple light from the display devices to optical waveguides. The waveguides may permit a user to view a real-world scene through the waveguides while also viewing a 3D virtual image presented to the user via the waveguides. In some examples, the waveguides may be diffraction waveguides.

The visualization system (e.g., MR system 212/visualization device 213) may be configured to display different types of virtual guides. Examples of virtual guides include, but are not limited to, a virtual point, a virtual axis, a virtual angle, a virtual path, a virtual plane, and a virtual surface or contour. As discussed above, the visualization system (e.g., MR system 212/visualization device 213) may enable a user to directly view the patient's anatomy via a lens by which the virtual guides are displayed, e.g., projected. The virtual guides may guide or assist various aspects of the surgery. For instance, a virtual guide may guide at least one of preparation of anatomy for attachment of the prosthetic or attachment of the prosthetic to the anatomy.

The visualization system may obtain parameters for the virtual guides from a virtual surgical plan, such as the virtual surgical plan described herein. Example parameters for the virtual guides include, but are not necessarily limited to a guide location, a guide orientation, a guide type, a guide color, etc.

The visualization system may display a virtual guide in a manner in which the virtual guide appears to be overlaid on an actual, real anatomical object of the patient, within a real-world environment, e.g., by displaying the virtual guide(s) with actual, real-world patient anatomy (e.g., at least a portion of the patient's anatomy) viewed by the user through holographic lenses. For example, the virtual guides may be 3D virtual objects that appear to reside within the real-world environment with the actual, real anatomical object. The visualization system may display virtual guidance for any combination of standard steps and ancillary steps.

The techniques of this disclosure are described below with respect to a shoulder arthroplasty surgical procedure. Examples of shoulder arthroplasties include, but are not limited to, reversed arthroplasty, augmented reverse arthroplasty, standard total shoulder arthroplasty, augmented total shoulder arthroplasty, and hemiarthroplasty. However, the techniques are not so limited, and the visualization system may be used to provide virtual guidance information, including virtual guides in any type of surgical procedure. Other example procedures in which a visualization system, such as MR system 212, may be used to provide virtual guides include, but are not limited to, other types of orthopedic surgeries; any type of procedure with the suffix “plasty,” “stomy,” “ectomy,” “clasia,” or “centesis,”; orthopedic surgeries for other joints, such as elbow, wrist, finger, hip, knee, ankle or toe, or any other orthopedic surgical procedure in which precision guidance is desirable.

A typical shoulder arthroplasty includes various work on a patient's scapula and performing various work on the patient's humerus. The work on the scapula may generally be described as preparing the scapula (e.g., the glenoid cavity of the scapula) for attachment of a prosthesis and attaching the prosthesis to the prepared scapula. Similarly, the work on the humerus may generally be described as preparing the humerus for attachment of a prosthesis and attaching the prosthesis to the prepared humerus. As described herein, the visualization system may provide guidance for any or all work performed in such an arthroplasty procedure.

During intraoperative phase 306 (FIG. 3), a surgeon may perform an orthopedic surgery that follows a complex series of workflow steps, some or all of which may proceed according to prescribed sequence. Although surgeons are typically well trained to perform the steps of an orthopedic surgery, incidents may occur where one or more steps of the orthopedic surgery are accidentally omitted or performed in an incorrect manner. Moreover, incidents may occur where the surgeon remembers to perform a step of an orthopedic surgery, but misremembers how to perform one or more aspects the step, or would otherwise benefit from a reminder of how to perform the step or guidance about one or more aspects of the step, some of which may vary according to a specific surgical plan generated for a particular patient. For instance, while a surgeon may remember how to perform standard steps of a procedure, the surgeon may misremember how to perform ancillary steps (e.g., because ancillary steps may be less frequently performed than standard steps).

Such incidents could impact surgical procedure efficiency, surgical procedure efficacy or patient outcomes (e.g., range of motion, infection, etc.). Paper checklists may be used to remind surgeons and operating room (OR) staff to perform the steps of the surgery. However, there are numerous drawbacks to the use of paper checklists. For example, it might not be easy for a surgeon to see a paper checklist, especially when the surgeon is holding surgical tools. Moreover, because the surgeon must preserve sterility, it may be impossible for the surgeon to use her or his hands to mark off items in the paper checklist. Additionally, a paper checklist includes visual clutter regarding completed and upcoming steps and it may take the surgeon valuable time to process the paper checklist. Moreover, the visual clutter in the paper checklist may allow the surgeon to visually skip over a step of the surgery. Having a nurse maintain and read back the checklist to the surgeon may increase expenses because this approach may require a dedicated nurse to be present for this purpose. Furthermore, a paper checklist does not allow the surgeon to access any more information than the information that is provided on the paper itself.

Presenting a computerized version of a paper checklist on a computer screen in the OR may have many of the same problems as a conventional paper checklist. For example, presenting all steps of the surgery on the computer screen may result in visual clutter that may delay or confuse the surgeon, and may require the surgeon to look away from a surgical site. Moreover, a dedicated nurse may be required to control the checklist presented on the computer screen. Furthermore, although a computerized version of a paper checklist may, in principle, allow the surgeon to access more information during the surgery, the surgeon would have difficulty requesting the display of such information. For instance, because of the need to preserve sterility, the surgeon may be unable to use a mouse or keyboard to control the computer. The surgeon may be able to provide voice instructions to the computer or a nurse, but voice commands may be misinterpreted by computers and nurses, which may result in lost time, or may again require a dedicated nurse be available to control the computer just in case the surgeon wants the computer to display additional information.

An additional problem with checklists presented on computer screens is that the computer screens are frequently somewhat distant from the orthopedic surgeon during a surgery in order to ensure that the orthopedic surgeon and other healthcare professionals have adequate room to work around the patient. For instance, the computer monitors may be mounted to the walls of the operating room. However, this alone may cause certain problems for orthopedic surgeons. For instance, in one example, some orthopedic surgeons may wear magnifying loupe eyewear or standard corrective eyewear to help the surgeon better see the surgical site on the patient. However, since the surgical site on the patient is typically much closer to the orthopedic surgeon than the location where the computer monitor is located, the surgeon's eyewear may make it difficult to see the computer monitor that is at a further distance. In other words, the difference in focal distance between the surgical site and the computer monitor may be so great that the orthopedic surgeon cannot clearly see both the surgical site and the computer monitor when wearing (or without wearing) particular eyewear. Thus, the surgeon may need to remove or change the eyewear in order to see the computer monitor. Removing or changing eyewear may consume time and may introduce another potential vector for the spread of germs and other types of contamination.

This disclosure describes techniques that use XR to assist users (e.g., surgeons or other types of persons) through the workflow steps of orthopedic surgeries in a way that may address challenges such as those mentioned above. As described elsewhere in this disclosure, XR may include VR, MR, and AR. In examples where XR is used to assist a user through the workflow steps of an orthopedic surgery and XR takes the form of VR, the user may be performing a simulation of the orthopedic surgery or may be performing the orthopedic surgery remotely. In examples where XR is used to assist a user through workflow steps of an orthopedic surgery and XR takes the form of MR or AR, the surgeon may concurrently perceive real-world objects and virtual objects during the orthopedic surgery.

For instance, FIG. 12 is a block diagram illustrating an example system 11700 that may assist a user, such as a surgeon, nurse or other medical technician, through steps in the workflow steps of orthopedic surgeries, in accordance with a technique of this disclosure. In the example of FIG. 12, system 11700 includes an XR visualization device 11702, a communication network 11704, and one or more computing systems 11706A through 11706N (collectively, “computing systems 11706”). In other examples, system 11700 may include more, fewer, or different devices and systems.

Computing systems 11706 may include various types of computing devices, such as server computers, personal computers, smartphones, laptop computers, and other types of computing devices. For case of explanation, this disclosure may describe actions performed by processing circuits, data storage systems, and communication interfaces of XR visualization device 11702 and computing systems 11706 as being performed by XR visualization device 11702 and computing systems 11706 as a whole. Communication network 11704 may include various types of networks, such as a local area network, the Internet, and/or other types of communication networks. Various computing systems of orthopedic surgical system 100 (FIG. 1) may include system 11700. For example, intraoperative guidance system 108 may include system 11700. In some examples, XR visualization device 11702 may be implemented as shown in the example of FIG. 5.

In the example of FIG. 12, XR visualization device 11702 may be worn by an orthopedic surgeon and may output an XR visualization for display. In some examples, XR visualization device 11702 is worn by another type of user (e.g., a nurse, medical technician, or other type of user) and may output the XR visualization for display. XR visualization device 11702 may access information about workflow steps of an orthopedic surgery from one or more of computing systems 11706. XR visualization devices substantially similar or identical to XR visualization device 11702 may be worn by other persons in a surgical operating room, such as one or more nurses, one or more medical technicians, or one or more additional surgeons, such that one or multiple persons may observe AR, MR and/or VR imagery providing guidance regarding workflow steps of an orthopedic surgery.

In an example, the XR visualization includes a set of one or more virtual checklist items. Each of the one or more virtual checklist items corresponds to an item in a checklist of steps of an orthopedic surgery (e.g., a shoulder arthroplasty, an ankle arthroplasty, or any other type of orthopedic surgery). The steps may be arranged in sequence according to an order in which the steps are to be performed or may be arranged topically into step topics according to physical or functional characteristics of the procedural steps. In accordance with techniques of this disclosure, the steps may include standard steps alongside ancillary steps. Furthermore, in this example, a camera or other sensor of XR visualization device 11702 may detect a command of the orthopedic surgeon or other user to select a virtual checklist item in the set of virtual checklist items. In some examples, one or more devices separate from XR visualization device 11702 may detect the command. For instance, one or more cameras, microphones, or other types of devices positioned in an operating room may detect the command. Nevertheless, for case of explanation, this disclosure describes XR visualization device 11702 as detecting the command. However, discussion of XR visualization device 11702 detecting the command may apply to one or more other types of devices detecting the command. In response to detecting the command, XR visualization device 11702 may update the XR visualization to include additional information regarding a step of the orthopedic surgery corresponding to the selected virtual checklist item.

In some examples, the command is a contactless command. The contactless command is performed without the orthopedic surgeon or other user touching any solid object. For example, the contactless command may be a hand gesture performed in the air. For instance, the contactless command may be a pinching gesture directed to a virtual element of the XR visualization. In some examples, the contactless command is a voice command.

In other examples, XR visualization device 11702 may detect a command that involves contact. For example, XR visualization device 11702 may detect a first command in the form of a hand gesture in which a user (e.g., the orthopedic surgeon or other type of user) taps one or more fingers of the user's left hand onto the back of the user's right hand. In this example, XR visualization device 11702 may detect a second commend in the form of a hand gesture in which the user taps one or more fingers of the user's right hand onto the back of the user's left hand. In another example, the user may tap on a surface (e.g., a sterilized surface), such as a corner of the operating table or a surgical item.

In examples where the command is a contactless command, because the orthopedic surgeon or other user is able to view the additional information without touching any solid object, there is no additional risk of contamination from the orthopedic surgeon or other user touching a physical computer user interface device, such as a mouse, keyboard or touchscreen. Similar considerations regarding reduction of risk of contamination may apply with contact-based commands on surfaces that are already sterilized, such as the surgeon's own hands or a sterilized surgical item. Additionally, in examples where the command (e.g., a contactless command) is a hand gesture, the use of the hand gesture to access the information may avoid the problems with voice commands discussed above, although voice commands also may be used in some examples. In this way, a computing system (e.g., XR visualization device 11702, one of computing system 11706, etc.) that handles the workflow checklist for the orthopedic surgery may be improved by decreasing the computing system's risk of acting as a vector for spreading contamination. Furthermore, because the virtual checklist items may be presented in an XR visualization in a focal plane that is relatively close to a focal plane of the surgical site, the orthopedic surgeon may not need to change or remove eyewear in order to see the virtual checklist items. Thus, in various examples, the techniques of this disclosure may improve the accessibility of the computing system, especially for orthopedic surgeons or other users with vision impairments and may enable the orthopedic surgeon or other users to skip steps otherwise needed to access the virtual checklist items, thereby improving efficiency of use of the computing system.

XR visualization device 11702 may present various virtual checklist items in the XR visualization. For example, the XR visualization may include virtual checklist items corresponding to a current step of the orthopedic surgery that the orthopedic surgeon is performing. In this example, the XR visualization may exclude virtual checklist items corresponding to completed steps of the orthopedic surgery. Furthermore, in some examples, the XR visualization may exclude virtual checklist items corresponding to one or more steps of the orthopedic surgery occurring after the current step of the orthopedic surgery. For instance, in one example, XR visualization device 11702 may exclude virtual checklist items corresponding to all steps of the orthopedic surgery occurring after the current step. In another example, the XR visualization may exclude checklist items corresponding to all steps of the orthopedic surgery except a next step of the orthopedic surgery. In some examples, the displayed set of virtual checklist items in the XR visualization is user configurable.

There are different standard steps for different orthopedic surgeries. For instance, a standard set of steps to perform a knee arthroplasty is different from a standard set of steps required to perform a reverse shoulder arthroplasty. In one example, a standard set of steps for a shoulder arthroplasty may include: an incision step, a step of severing the humeral head, a step of drilling a hole for a reaming guide pin, a step for performing a glenoid reaming process, a step for installing a glenoid implant, a step for preparing the humerus, a step for installing a humeral implant, and a step for closing the incision. In some examples, the steps may include nested sub-steps. In some examples, the series of steps may include the graft, humerus cut, install guide, glenoid reaming, and glenoid implant steps of FIG. 10. In some such examples, the next step of the orthopedic surgery may be a sub-step. Steps such as those described above may be presented as checklist items, e.g., using AR, MR, or VR visualization. Of course, the steps for ankle arthroplasty would also differ from those of other types of orthopedic surgeries, and virtual workflow guidance and virtual checklists may be defined to guide and track the specific steps defined for a given type of ankle arthroplasty or other orthopedic surgical procedure.

As discussed above, a particular type of orthopedic surgical procedure may include a standard set of steps (e.g., a shoulder arthroplasty surgery may include the standard set of steps shown in FIG. 11). As also discussed above, an orthopedic surgery may be customized to a particular patient. For instance, one or more ancillary steps may be included in a surgical plan for the particular patient, in addition to the standard set of steps. As one example, an ancillary step of removing osteophytes may be required in a shoulder arthroplasty surgery for one patient but might not be required in the shoulder arthroplasty surgery of another patient. That fact that there may be variations within the same surgery for different patients may increase the mental load on surgeons and operating room personnel. Accordingly, the XR visualization may include virtual checklist items corresponding to patient-specific steps (e.g., patient-specific ancillary steps) or sub-steps of the orthopedic surgery customized for the particular patient. Including virtual checklist items corresponding to patient-specific steps or sub-steps may be helpful in reminding surgical personnel of how the orthopedic surgery is to be performed for the individual patient.

The XR visualization may present various types of information in the virtual checklist items. For instance, in one example, a virtual checklist item in the XR visualization may include text identifying a corresponding step of the orthopedic surgery. In some examples, a virtual checklist item in the XR visualization may include an icon or other non-text graphic representing the corresponding step of the orthopedic surgery. In some examples, a virtual checklist item may specify which surgical items (e.g., tools, implants, etc.) to use in the corresponding step of the orthopedic surgery. The XR visualization may present the virtual checklist items according to various formats. For example, the XR visualization may present the virtual checklist items as a row or a column of items, each of which may contain text, icons, or other information. In some examples, the XR visualization may present the virtual checklist items in a 3-dimensional stack-of-cards format, where virtual checklist items are in different virtual cards that may be removed or added to the stack.

In some examples, the virtual checklist items may correspond to steps of the orthopedic surgery that are specific to an individual patient. In other words, a virtual checklist item may correspond to a step included in a version of an orthopedic surgery specific to a patient that is not included in a version of the orthopedic surgery of other patients. For example, in the example of the shoulder arthroplasty surgery provided above, the XR visualization may include a virtual checklist item that corresponds to a step of removing osteophytes during the surgery for a first patient but does not include this virtual checklist item in the surgery for a second patient who does not have osteophytes that need to be removed.

XR visualization device 11702 may present various types of additional information in the XR visualization. In one example, the additional information may include additional text describing a step of the orthopedic surgery. For instance, the additional information may include previously prepared notes by the orthopedic surgeon or other person. In some examples, the previously prepared notes may be specific to the patient on which the orthopedic surgeon is operating. For instance, the previously prepared notes specific to the patient may tell the orthopedic surgeon to install screws in a particular way so as to avoid cysts present at particular positions in a bone of a first patient. In this example, other patients may not have cysts at the same positions as the first patient. In some examples, the previously prepared notes may be common across patients but specific to an individual surgeon, hospital, care network, or other grouping. For instance, a surgeon may always want to have access to his or her personal notes regarding a particular step of the surgery.

In some examples, the additional information may include one or more virtual 3-dimensional objects associated with the step of the orthopedic surgery. In other words, XR visualization device 11702 may update the XR visualization to include one or more virtual 3-dimensional objects associated with the step of the orthopedic surgery. The 3-dimensional objects may include one or more virtual 3-dimensional models of bones involved with the step of the orthopedic surgery. For instance, in the context of a shoulder arthroplasty surgery described elsewhere in this disclosure, the 3-dimensional objects may include a 3-dimensional model of a patient's scapula. In this example, the XR visualization may include one or more virtual 3-dimensional objects that do not correspond to real-world objects. For instance, the virtual 3-dimensional objects may include a virtual reaming axis. The virtual 3-dimensional models shown in the XR visualization during the surgery may be the same as virtual 3-dimensional models shown during preoperative phase 302. Thus, an orthopedic surgeon or other user may already be familiar with the virtual 3-dimensional models because the orthopedic surgeon may have used the virtual 3-dimensional models during the preoperative planning phase. In some examples, one or more of the 3-dimensional models is specific to an individual patient undergoing surgery. In some examples, one or more of the 3-dimensional models is generic across patients.

In some examples, the XR visualization may also include virtual controls for controlling the virtual 3-dimensional objects. The orthopedic surgeon or other user may use contactless commands (e.g., hand gestures or voice commands) or contact-based commands to select the virtual controls. The virtual controls may allow the orthopedic surgeon or other user to hide or display individual virtual 3-dimensional objects, rotate the virtual 3-dimensional objects, scale the virtual 3-dimensional objects or otherwise control the virtual 3-dimensional objects. In some examples, XR visualization device 11702 may update the XR visualization to rotate or scale the one or more virtual 3-dimensional objects in response to a command from the orthopedic surgeon or other user. In some such examples, the command may be a contactless command, such as a hand gesture or a voice command.

Furthermore, in some examples, XR visualization device 11702 may update the XR visualization to show to the orthopedic surgeon or other user an animation of the step of the orthopedic surgery as applied to the one or more virtual 3-dimensional objects. The animation may show progressive movement of one or more virtual 3D objects, such as tools or implants, from initial positions, to intermediate positions, and to final positions to represent the operations needed to complete the step in the procedure. For example, in the context of a shoulder arthroplasty, the XR visualization may show an animation of how to perform a step of cutting a humeral head. This may help the orthopedic surgeon or other user remember how to perform a step of the orthopedic surgery, or how to perform the step in a particular way in accordance with a patient-specific surgical plan. In some examples, the animation is specific to an individual patient. In some examples, the animation is generic across a set of patients.

In some examples, XR visualization device 11702 may detect a command (e.g., a contactless command or contact-based command) from the orthopedic surgeon or other user to mark the step of the orthopedic surgery complete. In examples where the command is a contactless command, the contactless command may be a hand gesture or a voice command. Marking a step as complete may comprise storing data (e.g., by one of computing systems 11706) indicating that the step is complete. Furthermore, based on the command, XR visualization device 11702 may update the XR visualization to include a virtual checklist item corresponding to a next step in the checklist of steps of the orthopedic surgery. For instance, XR visualization device 11702 may update the XR visualization to mark the current step of the orthopedic surgery complete in response to detecting the command from the orthopedic surgeon.

In some examples, one or more of computing systems 11706 may display virtual checklist items to one or more persons other than the orthopedic surgeon. For instance, an XR visualization device or computer may display the same virtual checklist items that XR visualization device 11702 displays to the orthopedic surgeon. This may allow the other person to track the progress of the surgery. Such other persons may include nurses, product representatives, students, other surgeons, and so on. Techniques for intraoperative collaboration and education are described elsewhere in this disclosure.

In some examples, one of computing systems 11706 may control various surgical tools based on which step of the surgical procedure is the current step of the surgical procedure. For example, the computing system may disable a particular tool if the tool is not supposed to be used during the current step. For instance, in the context of a shoulder arthroplasty surgery, the computing system may disable a reaming drill during a step of cutting the patient's humeral head if the reaming drill should not be used during the step of cutting the patient's humeral head. This may help prevent the surgeon from skipping a step of the orthopedic surgery. To control which surgical tools may be used with a particular step, computing system 11706 may access a predefined list of surgical tools that may be used in particular steps of the surgical procedure.

In some examples, the workflow management process described herein may guide a process for helping a nurse provide the correct tools to the surgeon, as described elsewhere in this disclosure. For instance, a virtual object in an XR visualization presented to the nurse may indicate to the nurse which tools to provide to the surgeon for the next step of the orthopedic surgery. In another example, lights built into the surgical tools or trays may indicate to the nurse which tools are to be used in the next step of the orthopedic surgery. Such examples are described in greater detail elsewhere in this disclosure.

In some examples, a computing system (e.g., one of computing systems 11706) outputs checklist items for display to a nurse or other individual. In some such examples, rather than XR visualization device 11702 detecting a command to mark a step of the orthopedic surgery complete, the computing system may mark the step of the orthopedic surgery complete when the nurse uses a device of the computing system to optically scan a code (e.g., a bar code) of a surgical item (e.g., surgical tool, implant, box of surgical items, etc.) used in the next step of the orthopedic surgery. This may have the combined effect of advancing the checklist and updating an inventory of available surgical items. In this example, the computing system may warn the nurse if the nurse did not scan the expected surgical item for a next step of the orthopedic surgery.

FIG. 13 is a flowchart illustrating an example operation to assist users, such as surgeons, nurses, or other medical technicians, through steps in the workflows of orthopedic surgeries, in accordance with a technique of this disclosure. In the example of FIG. 13, a XR visualization device 11702 worn by an orthopedic surgeon may output a XR visualization for display (11800). The XR visualization includes a set of one or more virtual checklist items. Each of the one or more virtual checklist items corresponds to an item in a checklist of steps of an orthopedic surgery, such as ankle arthroplasty or shoulder arthroplasty.

Furthermore, XR visualization device 11702 may detect a contactless command (e.g., a hand gesture or voice command) of the orthopedic surgeon to select a virtual checklist item in the set of virtual checklist items (11802). The contactless command is performed without the orthopedic surgeon touching any solid object and may be detected by a camera, microphone, or other sensor associated with XR visualization device 11702 worn by the surgeon, an XR visualization device worn by other person in the operating room, or another sensor not carried by an XR visualization device. Furthermore, in response to detecting the contactless command, XR visualization device 11702 may update the XR visualization to include additional information regarding a step of the orthopedic surgery corresponding to the selected virtual checklist item (11804).

In some examples, XR visualization device 11702 is a MR visualization device or AR visualization device that the orthopedic surgeon wears or otherwise uses during surgery. In some examples, XR visualization device 11702 is a VR visualization device. In such examples, the orthopedic surgeon may use the VR visualization device to perform a simulation of the orthopedic surgery. During such a simulation, it may be helpful to the orthopedic surgeon to see the virtual checklist items in the same way that the orthopedic surgeon would see the virtual checklist items in a MR visualization device or an AR visualization device.

FIG. 14 is a conceptual diagram illustrating an example XR visualization 11900 that includes a set of one or more virtual checklist items, as viewed by a user, such as an orthopedic surgeon, nurse, or other medical technician, while performing an orthopedic surgery on a shoulder of a patient. In the example of FIG. 14, the patient is lying on an operating table and the orthopedic surgeon is viewing the patient from above. The orthopedic surgeon has already made an incision 11902 in the patient's shoulder and exposed a glenoid surface 11904.

In accordance with a technique of this disclosure, XR visualization 11900 includes a set of virtual checklist items 11906. Virtual checklist items 11906 correspond to items in a checklist of steps of the orthopedic surgery that the orthopedic surgeon is performing on the patient. In the example of FIG. 14, virtual checklist items 11906 are presented in the form of a stack of virtual cards, each corresponding to a step (or sub-step) of the orthopedic surgery. In other examples, one or more virtual checklist items 11906 may be presented in an XR visualization in a column form. In some examples, one or more virtual checklist items 11906 are in a row form, similar to the arrangement of selectable buttons 1002 in FIG. 10.

A virtual checklist item 11908 includes text describing a step of the orthopedic surgery and a graphic corresponding to the step. In particular, virtual checklist item 11908 includes text describing a step for installing a reaming guide pin. Furthermore, as shown in the example of FIG. 14, virtual checklist item 11908 includes patient-specific content, namely that the patient has a bone cyst on a coracoid process and to avoid accidental contact. Such patient-specific content would not be present for all patients. In some examples, virtual checklist item 11908 may include one or more warnings associated with the current step. For instance, a virtual checklist item associated with a step of installing fixation screws 16102A and 16102B of FIG. 31 may include a textual or graphical warning against over torqueing fixation screws 16102A and 16102B.

As described elsewhere in this disclosure, a surgical plan may be developed for an orthopedic surgery during preoperative phase 302 (FIG. 3). For instance, the surgical plan may be developed using the BLUEPRINT™ system. However, a surgeon may deviate from the surgical plan when performing the orthopedic surgery. In other words, the surgeon's actions during intraoperative phase 306 (FIG. 3) may be different from the actions set forth in the surgical plan defined during preoperative phase 302 (FIG. 3).

There are many reasons why a surgeon may deviate from the surgical plan. For example, the surgical plan developed during preoperative phase 302 may be based on incorrect estimations or assumptions regarding the anatomy of the patient. For instance, in one example, the surgical plan developed during preoperative phase 302 may be based on incorrect estimates of the positions or sizes of osteophytes.

In another example of why a surgeon may deviate from the surgical plan developed during preoperative phase 302, the surgical plan may be based on incorrect estimates of the patient's pathology. For instance, the surgeon may discover upon opening the patient's shoulder joint that a level of glenoid erosion in the patient's shoulder joint is consistent with a different category in the Walch glenohumeral osteoarthritis classification system than was understood from previous imaging of the patient's shoulder joint. In the case of a shoulder repair surgery, an incorrect estimate of the patient's pathology may require the surgeon to perform a reverse shoulder arthroplasty instead of a standard anatomic shoulder arthroplasty, or vice versa.

In another example of why a surgeon may deviate from the surgical plan developed during preoperative phase 302, the surgical plan may be based on incorrect estimates of bone density. For instance, in this example, if bone density is too low in a particular area of a bone, such as the humerus or scapula, the bone may not be able to adequately support an implant, such as the implant type planned during preoperative phase 302. The bone may fracture if the bone is not able to adequately support the implant or it may be impossible to securely mount the implant onto the bone. As a result, the surgeon may be compelled to cut, drill, or ream the bone in a manner that differs from the surgical plan to improve the seating of the implant, or use a different implant type. For instance, in one example, the initial surgical plan may call for the use of a stemless humeral implant, but if the bone density of the humerus is too low in certain areas, the surgeon may want to switch to a stemmed humeral implant.

In other examples of why a surgeon may deviate from the surgical plan developed during the preoperative phase 302, the surgeon may simply realize that there were errors in the surgical plan. The surgeon may also deviate from the surgical plan inadvertently. For example, the surgeon may cut a bone at an angle that differs somewhat from the angle indicated by the surgical plan. For instance, the surgeon may cut away too much of the humeral head. In some instances, different angles of a cut may result in different potential ranges of motion if not compensated for by using different implants.

In accordance with a technique of this disclosure, a computing system may obtain an information model specifying a first surgical plan, which may be a preoperatively defined surgical plan. The computing system may modify the first surgical plan during intraoperative phase 306 of an orthopedic surgery to generate a second surgical plan (FIG. 3). For instance, the computing system may insert one or more ancillary steps into, and/or remove one or more ancillary steps from the first surgical plan to generate the second surgical plan. Furthermore, in accordance with a technique of this disclosure, an XR system may present an XR visualization for display that is based on the second surgical plan. This may allow the surgeon to start using XR visualizations to help the surgeon execute the second surgical plan.

As noted above, the computing system may insert one or more ancillary steps into, and/or remove one or more ancillary steps from the first surgical plan to generate the second surgical plan. As one example, based on an intra operative evaluation, the computing system may intra operatively insert an ancillary step of fusing of three midfoot joints (e.g., as opposed to fusing a single midfoot joint or as opposed to fusing zero midfoot joints). As another example, based on an intra operative bone quality evaluation, the computing system may intra operatively insert an ancillary step of using a biologic bone void filler. As another example, based on an intra operative evaluation indicating that laxity is otherwise appropriate, the computing system may remove an ancillary step of soft tissue release.

FIG. 15 is a conceptual diagram illustrating an example system 12000 in which a first surgical plan is modified during intraoperative phase 306 to generate a second surgical plan, in accordance with a technique of this disclosure. System 12000 includes a computing system 12002, an XR system 12004, a storage system 12006, and a network 12008. In the example of FIG. 15, XR system 12004 includes a visualization device 12010. Similar to MR system 212 of FIG. 2. XR system 12004 may also include other components, such as one or more processors and storage devices separate from visualization device 12010. As shown in the example of FIG. 15, visualization device 12010 may be a head-mounted visualization device. In other examples, visualization device 12010 may be a holographic projector or other type of device that enables a user to visualize an MR visualization. In some examples, the functions of computing system 12002 are performed by one or more processors included in visualization device 12010 or XR system 12004. In other examples, some or all functions of computing system 12002 are performed by processors external to visualization device 12010.

As discussed elsewhere in this disclosure, computing system 12002 may comprise one or more computing devices operating as a system. In some examples, computing system 12002 is part of virtual planning system 102 (FIG. 1) or intraoperative guidance system 108 (FIG. 1). Computing system 12002 may obtain an information model that describes a surgical plan. For example, computing system 12002 may generate the information model e.g., using surgical planning software, such as BLUEPRINT™. In some examples, computing system 12002 may obtain the information model from a computer readable medium, such as a communication medium or non-transitory computer-readable medium. Storage system 12006 may store the information model.

FIG. 16 is a flowchart of an example operation in which a first surgical plan is modified during intraoperative phase 306 to generate a second surgical plan, in accordance with a technique of this disclosure. In the example of FIG. 16, computing system 12002 may obtain an information model specifying a first surgical plan for an orthopedic surgery to be performed on a patient (12100). For instance, computing system 12002 may generate or receive the first surgical plan as described elsewhere in this disclosure. Furthermore, in the example of FIG. 16, computing system 12002 may modify the first surgical plan during intraoperative phase 306 (FIG. 3) of the orthopedic surgery to generate a second surgical plan (12102). Visualization device 12010 may present, during intraoperative phase 306 of the orthopedic surgery, a visualization for display that is based on the second surgical plan (12104). The visualization may be an XR visualization, such as an MR visualization or a VR visualization.

In some examples, the first surgical plan is a preoperatively-defined surgical plan for the orthopedic surgery to be performed on the patient. In other words, the first surgical plan may be defined during preoperative phase 302 (FIG. 3). In some examples, the first surgical plan is an intraoperatively-defined surgical plan for the orthopedic surgery to be performed on the patient. For instance, in one example, a preoperatively-defined surgical plan may have been modified during the course of the orthopedic surgery to generate the first surgical plan. which may be further modified to generate the second surgical plan.

The first surgical plan may differ from the second surgical plan in various ways. For example, the first surgical plan may specify use of a first surgical item, such as a surgical tool or implant. In this example, computing system 12002 may modify the first surgical plan such that the second surgical plan specifies use of a second surgical item instead of the first surgical item. In this example, the second surgical item is different from the first surgical item. For instance, the first surgical item and the second surgical item may be different sizes of the same type of item or may be different types of items altogether. For instance, the first surgical item may be a stemless humeral implant and the second surgical item may be a stemmed humeral implant, or vice versa. In some examples, the first surgical plan may differ from the second surgical plan with respect to a cutting angle or position (e.g., a cutting angle or position of a resection of the humeral head).

In some examples, the first surgical plan may specify performance of a first type of surgery. In this example, computing system 12002 may modify the first surgical plan such that the second surgical plan specifies a second type of surgery that is different from the first type of surgery. For instance, the first type of surgery and the second type of surgery may be different ones of: a stemless standard total shoulder arthroplasty, a stemmed standard total shoulder arthroplasty, a stemless reverse shoulder arthroplasty, a stemmed reverse shoulder arthroplasty, an augmented glenoid standard total shoulder arthroplasty, or an augmented glenoid reverse shoulder arthroplasty. In other examples, the first type of surgery and the second type of surgery may be different types of ankle surgery or other types of joint surgery. Although techniques are described with reference to shoulder surgical procedures, similar interoperative changes to the surgical plan may also occur with regard to other types of orthopedic surgical procedures, such as for an ankle arthroplasty procedure.

In some examples where the orthopedic surgery is a shoulder repair surgery, the first surgical plan may specify a first reaming axis on a glenoid of the patient. In this example, the second surgical plan may specify a second reaming axis on the glenoid of the patient different from the first reaming axis. For instance, the first reaming axis and the second reaming axis may have different parameters, such as angles, entry points, and so on. In some examples where the orthopedic surgery is a shoulder repair surgery, the first surgical plan may specify a first reaming depth on a glenoid of the patient. In this example, the second surgical plan may specify a second reaming depth on the glenoid of the patient different from the first reaming depth.

Furthermore, in some examples where the orthopedic surgery is a shoulder repair surgery, the first surgical plan may specify a first humerus cut depth and the second surgical plan may specify a second humerus cut depth different from the first humerus cut depth. In some examples where the orthopedic surgery is a shoulder repair surgery, the first surgical plan specifies a first humerus cut angle and the second surgical plan specifies a second humerus cut angle different from the first humerus cut angle.

In some examples, the first surgical plan may specify a first implant position and the second surgical plan specifies a second implant position different from the first implant position. For instance, in one example, the first implant position and the second implant position may be different positions at which to mount an implant to a bone. In another example, the first implant position and the second implant position may be different depths of a stem of the implant, such as a humeral implant.

Computing system 12002 may perform various actions to determine whether and how to modify the first surgical plan to generate the second surgical plan. For example, computing system 12002 may generate, based on the first surgical plan, a planned depth map that represents a 3-dimensional shape that a surface of a bone of the patient will have after performance of a step of the orthopedic surgery if the step of the orthopedic surgery is performed according to the first surgical plan. Furthermore, computing system 12002 may generate a realized depth map that represents a 3-dimensional shape of the surface of the bone of the patient after a surgeon has actually performed the step of the orthopedic surgery. In this example, computing system 12002 may use information provided by one or more depth cameras (e.g., depth camera(s) 532 (FIG. 5)) of visualization device 12010 to generate the depth maps (e.g., the realized depth maps). Computing system 12002 may determine differences between the realized depth map and the planned depth map. For instance, computing system 12002 may determine differences in angles or depths between the realized depth map and the planned depth map. Computing system 12002 may generate the second surgical plan based on the determined differences between the realized depth map and the planned depth map. For instance, computing system 12002 may determine that different type of surgery or different surgical item may be preferable given the differences between the realized depth map and the planned depth map. In some examples, computing system 12002 uses markers (e.g., marker 3010 of FIG. 30) to determine depths and orientations of objects, such as bones and surgical tools.

In some examples, the orthopedic surgery is a shoulder repair surgery that includes a step of severing a humeral head (e.g., resecting a humeral head) with an oscillating saw tool. In such examples, a planned cut plane is a plane along which the oscillating saw tool will cut the humeral head if the step of the step of severing the humeral head is performed according to the first surgical plan. Furthermore, in this example, a realized cut plane is a plane along which the oscillating saw tool actually cut the humeral head during performance of the step of severing the humeral head. In this example, computing system 12002 may determine differences between the planned cut plane and the realized cut plane. In addition, computing system 12002 may generate the second surgical plan based on the determined differences between the realized cut plane and the planned cut plane.

In some examples where the orthopedic surgery is a shoulder repair surgery that includes a step of severing a humeral head with an oscillating saw tool, computing system 12002 may generate a depth map that represents 3-dimensional shapes within a scene that includes the humeral head and the oscillating saw tool. In such examples, computing system 12002 may determine the realized cut plane based on the depth map. In some examples, a marker is fixed to the oscillating saw tool and computing system 12002 determines the realized cut plane based on video images (e.g., RGB images) of the marker captured during performance of the step of severing the humeral head.

Furthermore, in one example where the orthopedic surgery is a shoulder repair surgery that includes a step of severing a humeral head with an oscillating saw tool, the first surgical plan specifies use of a first humeral implant. In this example, computing system 12002 may determine, based on the realized cut plane, a first range of motion that would result from using the first humeral implant. Additionally, computing system 12002 may determine that use of a second humeral implant would result in a second range of motion instead of the first range of motion given the realized cut plane, where the second humeral implant is different from the first humeral implant. In this example, computing system 12002 may determine the first range of motion using a database of case studies that maps realized cut planes to ranges of motion, using a mathematical model, or based on other algorithms. Furthermore, in this example, computing system 12002 may generate the second surgical plan such that the second surgical plan specifies use of the second humeral implant instead of the first humeral implant.

In some examples where the orthopedic surgery is a shoulder repair surgery that includes a step of reaming a glenoid of the patient with a reaming tool, a planned reaming depth is a depth to which the reaming tool will ream the glenoid if the step of reaming the glenoid is performed according to the first surgical plan. In such examples, a realized reaming depth is a depth to which the reaming tool actually reamed the glenoid during performance of the step of reaming the glenoid. In one such example, computing system 12002 may determine differences between the planned reaming depth and the realized reaming depth. Furthermore, in this example, computing system 12002 may generate the second surgical plan based on the determined differences between the realized reaming depth and the planned reaming depth.

Furthermore, in one example where the orthopedic surgery is a shoulder repair surgery that includes a step of reaming the glenoid of the patient, computing system 12002 may generate a depth map that represents 3-dimensional shapes within a scene that includes the glenoid and the reaming tool. In this example, computing system 12002 may determine the realized reaming depth based on the depth map. For instance, computing system 12002 may use the depth values in the depth map that correspond to the glenoid after reaming to determine how deeply the glenoid was reamed. In some examples, a first marker is fixed to the reaming tool and a second marker is fixed to a scapula that has the glenoid. In such examples, computing system 12002 may determine the realized reaming depth based on video images of the first marker and the second marker captured during performance of the step of reaming the glenoid.

Furthermore, in one example where the orthopedic surgery is a shoulder repair surgery that includes a step of reaming the glenoid, the first surgical plan may specify use of a first glenoid implant. In this example, computing system 12002 may determine, based on the realized reaming depth, a first range of motion that would result from using the first glenoid implant. Additionally, in this example, computing system 12002 may determine that use of a second glenoid implant would result in a second range of motion instead of the first range of motion given the realized reaming depth, where the second glenoid implant is different from the first glenoid implant. Computing system 12002 may determine the ranges of motion in accordance with any of the examples provided elsewhere in this disclosure for determining ranges of motion. Furthermore, computing system 12002 may generate the second surgical plan such that the second surgical plan specifies use of the second glenoid implant instead of the first glenoid implant. In some examples, the second range of motion is greater than the first range of motion.

As mentioned above, visualization device 12010 may present an XR visualization based on the second surgical plan. The XR visualization based on the second surgical plan may include various types of content, such as any of the intraoperative MR content described elsewhere in this disclosure. For instance, the XR visualization may comprise one or more virtual checklist items corresponding to items in a checklist of steps of the orthopedic surgery that conforms to the second surgical plan. In examples where the visualization is an MR visualization, the MR visualization may include virtual checklist items and also include images of real-world objects. In such examples, the images of the real-worlds objects may be the images of the patient and operating room as seen by the surgeon through a see-through holographic lens of visualization device 12010 or as seen by the surgeon on a screen of visualization device 12010.

In some examples, system 12000 may enable a surgeon to assess the consequences of deviating from a surgical plan before the surgeon deviates from the surgical plan. For example, the surgeon may want to assess what the consequences would be of using a different cutting angle or depth, different reaming depth, or different surgical item than specified in the surgical plan e.g., because the surgeon has discovered certain osteophytes, different bone density than expected, or other factors. In some examples where system 12000 enables the surgeon to assess the consequences of deviating from the surgical plan, computing system 12002 may generate, during intraoperative phase 306 (FIG. 3), a modified surgical plan based on a proposed deviation from the surgical plan. The surgeon may then have the opportunity to review the modified surgical plan intraoperatively. For instance, visualization device 12010 may present an XR visualization based on the modified surgical plan. In some examples, system 12000 may determine one or more ranges of motion that may result from deviating from the surgical plan. The ranges of motion may be determined based on data in a database of historical cases, based on a mathematical model, based on a neural network model, or using another type of algorithm. An XR visualization may present the ranges of motion.

The surgeon may specify a proposed deviation from the surgical plan in various ways. For instance, in an example where the surgeon is using an oscillating saw tool, the surgeon may hold the oscillating saw tool at a particular angle with respect to a cutting plane specified by a surgical plan for a bone. In this example, the surgeon may then provide input that asks system 12000 to determine the consequences of cutting the bone at the particular angle. For instance, the surgeon may provide a voice command saying, “what happens if I use this cutting plane instead?” In an example where the surgeon is using drilling tool, the surgeon may issue a voice command indicating how the surgeon proposes changing a position or depth of a drilling hole. Similarly, where the surgeon is using a reaming tool, the surgeon may issue a voice command indicating a proposed alternative reaming depth.

When computing system 12002 modifies a surgical plan, computing system 12002 may attempt to optimize one or more surgical parameters of the orthopedic surgery given the constraints imposed by the deviation from the surgical plan. For example, computing system 12002 may attempt to optimize the anticipated range of motion, optimize contact with cortical bone, optimize contact points of implants with high quality (e.g., dense) portions of the bone, or other factors. These factors may drive positions of implants, stem sizes of implants, surface sizes of implants, selection of ancillary steps, and other surgical parameters.

This disclosure describes surgical planning, such as selection of ancillary steps, using artificial intelligence (AI) techniques such as neural networks. In some examples, such AI techniques may be employed during preoperative phase 302 (FIG. 3) or another phase of a surgical lifecycle. Artificial neural networks (ANNs), including deep neural networks (DNNs), have shown great promise as classification tools. A DNN includes an input layer, an output layer, and one or more hidden layers between the input layer and the output layer. ANNs and DNNs may also include one or more other types of layers, such as pooling layers.

Each layer may include a set of artificial neurons, which are frequently referred to simply as “neurons.” Each neuron in the input layer receives an input value from an input vector. Outputs of the neurons in the input layer are provided as inputs to a next layer in the ANN. Each neuron of a layer after the input layer may apply a propagation function to the output of one or more neurons of the previous layer to generate an input value to the neuron. The neuron may then apply an activation function to the input to compute an activation value. The neuron may then apply an output function to the activation value to generate an output value for the neuron. An output vector of the ANN includes the output values of the output layer of the ANN.

There have been several challenges associated with application of ANNs to planning orthopedic surgery. For example, some challenges relate to how to structure and train an ANN so that the ANN is able to provide meaningful output regarding shoulder pathology. In another example of a challenge associated with application of ANNs to planning orthopedic surgery, patients and healthcare professionals are understandably reluctant to trust decisions made by a computer, especially when it is unclear how the computer made those decisions. There are therefore problems about how to generate output in a way that helps ensure that patients and healthcare professionals are comfortable in trusting the output of an ANN.

This disclosure describes techniques that may resolve these challenges and provide an ANN structure that provides meaningful output regarding shoulder pathology. For example, an ANN, such as a DNN, has an input layer, an output layer, and one or more hidden layers between the input layer and the output layer. The input layer includes a plurality of input layer neurons. Each input layer neuron in the plurality of input layer neurons corresponds to a different input element in a plurality of input elements (e.g., parameters of a particular patient). The output layer includes a plurality of output layer neurons.

Each output layer neuron in the plurality of output layer neurons corresponds to a different output element in a plurality of output elements. Each output element in the plurality of output elements may correspond to an ancillary step that may be selected for inclusion in a surgical plan. In this example, a computing system may generate a plurality of training datasets from past shoulder surgery cases. Each respective training dataset corresponds to a different training data patient in a plurality of training data patients and comprises a respective training input vector and a respective target output vector.

For each respective training dataset, the training input vector of the respective training dataset comprises a value for each element of the plurality of input elements. For each respective training dataset, the target output vector of the respective training dataset comprises a value for each element of the plurality of output elements. In this example, the computing system may use the plurality of training datasets to train the neural network. Additionally, in this example, the computing system may obtain a current input vector that corresponds to a current patient. The computing system may apply the ANN to the current input vector to generate a current output vector. The computing system may then determine, based on the current output vector, one or more ancillary steps that are recommended for inclusion in a surgical plan.

In this example, by having different output elements in the plurality of output elements correspond to different ancillary steps, the ANN may be able to provide meaningful output information that can be used in the classification of shoulder conditions of patients. Furthermore, in some examples, the output values of neurons in the output layer indicate measures of confidence that the corresponding ancillary step should be performed. Such confidence values may help users consider whether or not to perform the ancillary step determined by the computing system using the ANN.

FIG. 17 is a block diagram illustrating an example computing system 12202 that implements an ANN usable for selecting and/or recommending ancillary steps for a patient, in accordance with a technique of this disclosure. Computing system 12202 may be part of orthopedic surgical system 100 (FIG. 1). Computing system 12202 may use the ANN to select or recommend ancillary steps as party of a tailoring step 806 of FIG. 8. In some examples, computing system 12202 includes a XR visualization device (e.g., an MR visualization device or a VR visualization device) that includes one or more processors that perform operations of computing system 12202.

As shown in the example of FIG. 17, system 10900 includes a computing system 12202, a set of one or more client devices (collectively, “client devices 12204”). In other examples, system 12200 may include more, fewer, or different devices and systems. In some examples, computing system 12202 and client devices 12204 may communicate via one or more communication networks, such as the Internet.

Computing system 12202 may include one or more computing devices. Computing system 12202 and client devices 12204 may include various types of computing devices, such as server computers, personal computers, smartphones, laptop computers, and other types of computing devices. In the example of FIG. 17, computing system 12202 includes one or more processing circuits 12206, a data storage system 12208, and a set of one or more communication interfaces 12210A through 12210N (collectively, “communication interfaces 12210”). Data store 12208 is configured to store data. Communication interfaces 12210 may enable computing system 12202 to communicate (e.g., wirelessly or using wires) to other computing systems and devices, such as client devices 12204. For ease of explanation, this disclosure may describe actions performed by processing circuits 12206, data store 12208, and communication interfaces 12210 as being performed by computing system 10902 as a whole. One or more sub-systems of orthopedic surgical system 100 (FIG. 1) may include computing system 12202 and client devices 12204. For example, virtual planning system 102 may include computing system 12202 and client devices 12204.

Users may use client devices 12204 to access information generated by computing system 12202. For example, computing system 12202 may determine a classification of a shoulder condition of a current patient. The classification may be represented by a shoulder class among a plurality of shoulder classes in a shoulder pathology classification system. In this example, users may use client devices 12204 to access information regarding the classification. Because computing system 12202 may be remote from client devices 12204, users of client devices 12204 may consider computing system 12202 to be in a cloud-based computing system. In other examples, some or all the functionality of computing system 12202 may be performed by one or more of client devices 12204.

Computing system 12202 may implement a neural network (NN), such as a DNN. Storage system 12208 may comprise one or more computer-readable data storage media. Storage system 12208 may store parameters of the NN. For instance, storage system 12208 may store weights of neurons of the NN, bias values of neurons of the NN, and so on. In some examples, storage system 12208 may store multiple NNs. For instance, storage system 12208 may store a different NN for type of surgery (e.g., a different NN for each different standard set of steps).

Computing system 12202 may select or recommend ancillary steps for a patient based on output of the NN. In accordance with a technique of this disclosure, output elements of the NN include output elements corresponding to different ancillary steps. In some examples, computing system 12202 may select the ancillary steps based on a comparison of the values of the output elements generated by the NN. For example, the values of the output elements may correspond to confidence values that indicate levels of confidence that an ancillary step should be performed. For instance, the values of the output elements may be the confidence values or computing system 12202 may calculate the confidence values based on the values of the output elements.

In some examples, the output function of the output layer neurons generates the confidence values. Furthermore, computing system 12202 may rank the confidence values. In this example, computing system 12202 may make a determination to recommend performance of ancillary steps with confidence values greater than a threshold confidence value. In some examples, if none of the confidence values is above the threshold confidence value, computing system 12202 may generate output indicating that no ancillary steps are recommended.

As mentioned above, in some examples, the output elements of the NN include confidence values. In one such example, a confidence value function outputs confidence values. The confidence value function may be the output function of the output layer neurons of the NN. In this example, all possible confidence values output by the confidence value function are within a predefined range. Furthermore, in this example, computing system 12202 may apply the NN to an input vector to generate an output vector. As part of applying the NN, computing system 12202 may, for each respective output layer neuron in the plurality of output layer neurons, calculate an output value of the respective output layer neuron.

Computing system 12202 may use various confidence value functions. For example, computing system 12202 may apply a hyperbolic tangent function, a sigmoid function, or another type of function that output values that are within a predefined range. The hyperbolic tangent function (tanh) has the form γ(c)=tan h(c)=(e^c−e^−c)/(e^c+e^−c). The hyperbolic tangent function takes real-valued arguments, such as output values of output layer neurons, and transforms them to the range (−1, 1). The sigmoid function has the form γ(c)=1/(1+e^−c). The sigmoid function takes real-valued arguments, such as output values of output layer neurons, and transforms them to the range (0, 1).

Computing system 12202 may use a plurality of training datasets to train the NN. Each respective training dataset may correspond to a different training data patient in a plurality of previously classified training data patients. For instance, a first training dataset may correspond to a first training data patient, a second training dataset may correspond to a second training data patient, and so on. A training dataset may correspond to a training data patient in the sense that the training dataset may include information regarding the patient. The training data patients may be real patients on which ancillary steps have been performed. In some examples, the training data patients may include simulated patients.

Each respective training dataset may include a respective training input vector and a respective target output vector. For each respective training dataset, the training input vector of the respective training dataset comprises a value for each element of the plurality of input elements. In other words, the training input vector may include a value for each input layer neuron of the NN. For each respective training dataset, the target output vector of the respective training dataset may comprise a value for each element of the plurality of output elements. In other words, the target output vector may include a value for each output layer neuron of the NN.

In some examples, the values in the target output vector are based on confidence values. Such confidence values may, in turn, be based on levels of confidence expressed by one or more trained healthcare professionals, such as orthopedic surgeons. For instance, a trained healthcare professional may be given the information in the training input vector of a training dataset (or information from which the training input vector of the training dataset is derived) and may be asked to provide a confidence level that a particular ancillary step should be performed on the training data patient.

Furthermore, in some examples, computing system 12202 may provide confidence values for output to one or more users. For instance, computing system 12202 may provide the confidence values to client devices 12204 for display to one or more users. In this way, the one or more users may be better able to understand how computing system 12202 may have arrived at the classification of the shoulder condition of a patient.

In some examples, the training datasets are weighted based on health outcomes of the training data patients. For example, a training dataset may be given higher weight if the training data patient associated with the training dataset had all positive health outcomes. However, a training dataset may be given a lower weight if the associated training data patient had less positive health outcomes. During training, computing system 12202 may use a loss function that weights the training datasets based on the weights given to the training datasets.

In some examples, as part of generating the training datasets, computing system 12202 may select the plurality of training datasets from a database of training datasets based on one or more training dataset selection criteria. In other words, computing system 12202 may exclude certain training datasets from the training process of the NN if the training datasets do not satisfy the training dataset selection criteria. In the example of FIG. 17, data storage system 12208 stores a database 12212 that contains training datasets from past surgery cases.

In some examples, the one or more training dataset selection criteria include postoperative health outcomes of the training data patients. In such examples, the postoperative health outcomes of the training data patients may include one or more of: postoperative range of motion, presence of postoperative infection, or postoperative pain. Thus, in such examples, the training datasets upon which the NN is trained may exclude training datasets where adverse health outcomes occurred.

Additional training datasets may be added to the database over time and computing system 12202 may use the additional training datasets to train the NN. Thus, the NN may continue to improve over time as more training datasets are added to the database.

Computing system 12202 may apply one of various techniques to use the training datasets to train the NN. For example, computing system 12202 may use one of the various standard backpropagation algorithms known in the art. For instance, as part of training the NN, computing system 12202 may apply a cost function to determine cost values based on differences between the output vector generated by the NN and the target output vector. Computing system 12202 may then use the cost values in a backpropagation algorithm to update the weights of neurons in the NN.

FIG. 18 illustrates an example NN 12300 that may be implemented by computing system 12202 with the system of FIG. 17. In the example of FIG. 18, NN 12300 includes an input layer 12302, an output layer 12304, and one or more hidden layers 12306 between input layer 12302 and output layer 12304. In the example of FIG. 18, neurons are represented as circles. Although in the example of FIG. 18, each layer is shown as including six neurons, layers in NN 12300 may include more or fewer neurons. Furthermore, although NN 12300 is shown in FIG. 18 as being a fully connected network, NN 12300 may have a different architecture. For instance, NN 12300 may not be a fully connected network, may have one or more convolutional layers, or may otherwise have a different architecture from that shown in FIG. 18.

In some implementations, NN 12300 can be or include one or more artificial neural networks (also referred to simply as neural networks). A neural network can include a group of connected nodes, which also can be referred to as neurons or perceptrons. A neural network can be organized into one or more layers. Neural networks that include multiple layers can be referred to as “deep” networks. A deep network can include an input layer, an output layer, and one or more hidden layers positioned between the input layer and the output layer. The nodes of the neural network can be connected or non-fully connected.

NN 12300 can be or include one or more feed forward neural networks. In feed forward networks, the connections between nodes do not form a cycle. For example, each connection can connect a node from an earlier layer to a node from a later layer.

In some instances, NN 12300 can be or include one or more recurrent neural networks. In some instances, at least some of the nodes of a recurrent neural network can form a cycle. Recurrent neural networks can be especially useful for processing input data that is sequential in nature. In particular, in some instances, a recurrent neural network can pass or retain information from a previous portion of the input data sequence to a subsequent portion of the input data sequence through the use of recurrent or directed cyclical node connections.

In some examples, sequential input data can include time-series data (e.g., sensor data versus time or imagery captured at different times). For example, a recurrent neural network can analyze sensor data versus time to detect or predict a swipe direction, to perform handwriting recognition, etc. Sequential input data may include words in a sentence (e.g., for natural language processing, speech detection or processing, etc.); notes in a musical composition; sequential actions taken by a user (e.g., to detect or predict sequential application usage); sequential object states; etc. Example recurrent neural networks include long short-term (LSTM) recurrent neural networks; gated recurrent units; bi-direction recurrent neural networks; continuous time recurrent neural networks; neural history compressors; echo state networks; Elman networks; Jordan networks; recursive neural networks; Hopfield networks; fully recurrent networks; sequence-to-sequence configurations; etc.

In some implementations, NN 12300 can be or can include one or more convolutional neural networks. In some instances, a convolutional neural network can include one or more convolutional layers that perform convolutions over input data using learned filters. Filters can also be referred to as kernels. Convolutional neural networks can be especially useful for vision problems such as when the input data includes imagery such as still images or video. However, convolutional neural networks can also be applied for natural language processing.

NN 12300 may be or include one or more other forms of artificial neural networks such as, for example, deep Boltzmann machines; deep belief networks; stacked autoencoders; etc. Any of the neural networks described herein can be combined (e.g., stacked) to form more complex networks.

In the example of FIG. 18, an input vector 12308 includes a plurality of input elements. Each of the input elements may be a numerical value. Input layer 12302 includes a plurality of input layer neurons. Each input layer neuron in the plurality of input layer neurons included in input layer 12302 may correspond to a different input element in a plurality of input elements. In other words, input layer 12302 may include a different neuron for each input element in input vector 12308.

Furthermore, in the example of FIG. 18, an output vector 12310 includes a plurality of output elements. Each of the output elements may be a numerical value. Output layer 12304 includes a plurality of output layer neurons. Each output layer neuron in the plurality of output layer neurons corresponds to a different output element in the plurality of output elements. In other words, each output layer neuron in output layer 12304 corresponds to a different output element in output vector 12310.

Input vector 12308 may include a wide variety of information. For example, input vector 12308 may include morphological measurements of the patient. In some examples where input vector 12308 includes measurements of the patient's morphology, input vector 12308 may determine the measurements based on medical images of the patient, such as CT images, MRI images, or other types of images. For instance, computing system 12202 may obtain the medical images of a current patient. For instance, computing system 12202 may obtain the medical images from an imaging machine (e.g., a CT machine, MRI machine, or other type of imaging machine), an electronic medical record of the patient, or another data source. In this example, computing system 12202 may segment the medical images to identify internal structures of the current patient, such as soft tissue and bone. For instance, in one example, computing system 12202 may apply an artificial neural network trained to identify portions of medical images that correspond to bones or soft tissue. Furthermore, in this example, computing system 12202 may determine the plurality of measurements based on relative positions of the identified internal structures of the current patient. In this example, the plurality of input elements may include an input element for each measurement in the plurality of measurements.

As mentioned elsewhere in this disclosure, computing system 12202 may include one or more computing devices. Hence, various functions of computing system 12202 may be performed by various combinations of the computing devices of computing system 12202. For instance, in some examples, a first computing device of computing system 12202 may segment the images, a second computing device of computing system 12202 may train the DNN, a third computing device of computing system 12202 may apply the DNN, and so on. In other examples, a single computing device of computing system 12202 may segment the images, train the DNN, and apply the DNN.

In some examples, input vector 12308 may include (e.g., in combination with zero or more other example types of input data described herein) patient range of motion information. In some examples, the patient range of motion information may be generated using a motion tracking device.

In some examples, input vector 12308 may include information (e.g., in combination with zero or more other example types of input data described herein) that specifies bone density scores. Other information included in input vector 12308 may include demographic information, such as patient age, patient activities, patient gender, patient body mass index (BMI), and so on. In some examples, input vector 12308 may include information regarding the speed of onset of the symptoms (e.g., gradual or sudden). The plurality of input elements in input vector 12308 also may include patient objectives for participation in activities such as particular exercises/sport types, ranges of motion, etc.

FIG. 19 is a flowchart illustrating an example operation of computing system 12202 in using an ANN to recommend ancillary steps, in accordance with a technique of this disclosure. In the example of FIG. 19, computing system 12202 generates a plurality of training datasets from past surgery cases (e.g., past surgical procedures in which at least a particular standard set of steps and one or more ancillary steps was performed) (12400). In some examples, generating the plurality of training datasets comprises retrieving selected training datasets from a database of historical cases. In the example of FIG. 19, an ANN, such as NN 12300 (FIG. 18) has an input layer, an output layer, and one or more hidden layers between the input layer and the output layer. The input layer includes a plurality of input layer neurons. Each input layer neuron in the plurality of input layer neurons corresponds to a different input element in a plurality of input elements. The output layer includes a plurality of output layer neurons. Each output layer neuron in the plurality of output layer neurons corresponds to a different output element in a plurality of output elements. Each output element in the plurality of output elements corresponds to a different ancillary step of a plurality of ancillary steps.

Each respective training dataset corresponds to a different training data patient in a plurality of training data patients from past surgery cases and comprises a respective training input vector and a respective target output vector. For each respective training dataset, the training input vector of the respective training dataset comprises a value for each element of the plurality of input elements. For each respective training dataset, the target output vector of the respective training dataset comprises a value for each element of the plurality of output elements.

Furthermore, in the example of FIG. 19, computing system 12202 uses the plurality of training datasets to train the NN (12402). Additionally, computing system 12202 may obtain a current input vector that corresponds to a current patient (12404), e.g., including a combination of various patient inputs as described above. Computing system 12202 may apply the NN to the current input vector to generate a current output vector (12406). For instance, computing system 12202 may provide the current input vector to the input layer of the NN and may perform forward propagation through the NN to generate the current output vector.

Furthermore, computing system 12202 may determine, based on the current output vector, one or more ancillary steps recommended for performance on the current patient (12408). The one or more ancillary steps may be selected from a plurality of ancillary steps. In this way, computing system 12202 may evaluate ancillary steps of a plurality of ancillary steps for inclusion in a surgical plan for a patient.

Where the surgery is an ankle arthroplasty, the plurality of ancillary steps may include hindfoot corrective osteotomies, hindfoot corrective arthrodesis, midfoot/forefoot osteotomies, midfoot/forefoot arthrodesis, tibia/fibula osteotomies, soft tissue procedures, and miscellaneous steps (e.g., use of biological products).

Example hindfoot corrective osteotomies include, but are not limited to:

• • 1) Calcaneal osteotomy-“axial fluoroscopic image strongly encouraged for determining type of osteotomy”
    • a) Lateralizing (varus deformity)
      • i) Rotation only
        • (1) Dwyer-lateral closing wedge
        • (2) Malerba Z-type
      • ii) Rotation+translation→consider tarsal tunnel release to avoid tension on tibial nerve
        • (1) Modified Dwyer-closing wedge and lateral translation
        • (2) Oblique (in sagittal) osteotomy and lateral translate
    • b) Medializing (valgus deformity)
      • i) Medial Displacement (MDCO)
      • ii) Malerba Z-type (rotational osteotomy)
    • c) Vertical slide (may also incorporate a closing wedge)
      • i) Superior translation
      • ii) Inferior translation
    • d) Lateral Column Lengthening
      • i) Evans Opening Wedge (anterior neck of the calc)
  • 2) Cuboid—less power because less maximum translation.
    • a) Vertical slide
    • b) Closing wedge vertical slide (adducted foot condition)

Example hindfoot corrective arthrodeses include, but are not limited to:

• • 1. Subtalar joint (e.g., lateral approach)
    • a. Reduced joint space, arthritis→can be fused without correction
    • b. Closing wedge osteotomy (rotation)
    • c. Medial translation osteotomy
  • 2. Talonavicular joint
    • a. Reduced joint space, arthritis→can be fused without correction
    • b. Rotation correction (abduction or adduction of foot, correction of plantarflexed talus)
  • 3. Calcaneocuboid (C-C) arthrodesis—varus deformity/coronal plane z-deformity. Rigid joint, but extremely rare to have an isolated fusion without a slide
    • a. Vertical slide
    • b. Closing wedge vertical slide (adducted foot condition)
    • c. Can fuse for isolated DJD. Also aides in NC fusions
  • 4. Naviculocuneiform (NC) fusion (e.g., to achieve extra deformity correction after C-C; sometimes performed independently)
    • a. Closing wedge osteotomy (rotation)
    • b. Shortening osteotomy (option to balance out C-C fusion-foot length)
    • c. Dorsiflexion osteotomy (residual cavus foot option)

Example midfoot/forefoot osteotomies, include, but are not limited to:

• • 1) Medial cuneiform osteotomy
    • a) Cotton opening wedge
    • b) Closing wedge
  • 2) First metatarsal
    • a) Dorsiflexion osteotomy

Example midfoot/forefoot arthrodesis, include, but are not limited to:

• • 1) First metatarsocuneiform joint (first TMT-tarsometatarsal)
    • a) Dorsal closing wedge osteotomy (cavus, larger deformity (than dorsiflexion osteo) or arthritis)
    • b) Plantar closing wedge osteotomy (flat foot, hypermobile first metatarsocuneiform joint, bunion deformity combined with Lapidus procedure e.g. LAPIFUSE)
    • c) With Hallux Valgus
  • 2) Second and third tarsometatarsal joint
    • a) Transverse plane osteotomy for metatarsus adductus
    • b) Closing or opening (dorsal) wedge osteotomy to balance metatarsal heads

Example tibia/fibula osteotomies, include, but are not limited to:

• • 1) Tibial osteotomy
    • a) Medial malleolus
      • i) Vertical slide
      • ii) Opening wedge
      • iii) Closing wedge
    • b) Intra-articular reshaping for congruent deformity
      • i) Take down lateral tibial plafond combined with medial fibular ostectomy for varus-sloped tibia plafond (varus)
        • (1) Possible partial deltoid release
        • (2) Possible posterior tibial lengthening
      • ii) Take down medial tibial plafond combined with medial malleolar ostectomy for valgus-sloped tibia plafond (valgus)
    • c) Supramalleolar (bottom 4-5 cm of tibia)
      • i) Opening or closing wedge, with or without fibula osteotomy
      • ii) Dome osteotomy
      • iii) Distraction osteogenesis or rotational correction via fixator frame
    • d) Midshaft or Proximal (post-traumatic genu valgum, recurvatum, procurvatum, tibia vara)
      • i) Intramedullary rods
      • ii) Fixator frame
      • iii) Plating
  • 2) Fibular osteotomy
    • a) Lengthening—bone graft
      • i) Transverse osteotomy (rotation)
      • ii) Oblique/Step cut osteotomy (translation)
    • b) Shortening
      • i) Transverse osteotomy (rotation)
      • ii) Oblique/Step cut osteotomy (translation)
    • c) Syndesmosis reconstruction

Example soft tissue procedures, include, but are not limited to:

• • 1) Achilles tendon lengthening (Hoke)
  • 2) Gastrocnemius recession (Strayer)
  • 3) Tendon transfer
    • a) Peroneus longus to brevis—cavus foot
    • b) Peroneus Brevis to longus-flat foot
    • c) Posterior tib to Peroneus brevis-in varus with tight posterior tib and weak peroneals
    • d) Flexor Digitorum Longus transfer to navicular—adult flat foot
    • e) Posterior tibial tendon transfer through intraosseous membrane-foot drop
      • i) Bridle procedure
  • 4) Lateral ligament reconstruction
    • a) Anatomic repair
      • i) Modified Bröstrom-Gould
      • ii) Reverse Modified Bröstrom-Gould—congruent varus and incongruent valgus. Intentionally cut the ligaments then reattach with suture anchors.
    • b) Non-anatomic repair
      • i) Chrisman Snook or version thereof
  • 5) Deltoid ligament repair
    • a) Anatomic repair
      • i) Imbrication
      • ii) Plication
    • b) Non-anatomic repair
      • i) Allograft tendon transfer techniques
  • 6) Deltoid ligament partial release-varus
  • 7) Talonavicular joint capsular release dorsal and medial—varus
  • 8) Posterior capsulotomy for contracture with FHL tenosynovectomy
  • 9) Complete posterior tibial tendon release vs posterior tibial tendon lengthening

Example miscellaneous procedures, include, but are not limited to:

• • 1) Hardware removal
    • a) For visibility of intra-operative steps
  • 2) Gutter debridement or medial/lateral osteophyte removal
    • a) Separate medial/lateral approach
    • b) Direct anterior approach with or without polyethylene exchange (revision).
    • c) Lateral wall ostectomy of the calcaneus (more generous than a gutter debridement, calc-fibula). May require separate lateral approach. Modified Bröstrom can be through that same incision, if the lateral ligaments are weakened.
    • d) In severe varus my need to do before deformity correction
  • 3) Bone void filling (e.g., biologic product)
    • a) Bone grafting (e.g. morselized resection autograft)
    • b) Bone graft substitute (Prodense, etc.)
    • c) Bone cement
      • i) Cement alone
      • ii) Rebar technique
    • d) Biologic-based fusion products
    • e) Soft tissue reinforcement with treated biological tissue
  • 4) Medial malleolar reinforcement screws or K-wires
    • a) Temporary
    • b) Permanent
  • 5) Syndesmosis repair
  • 6) Syndesmosis fusion
  • 7) Subluxation
    • a) Anterior
      • i) Achilles tendon lengthening
      • ii) Gutter debridement
    • b) Posterior
      • i) Achilles tendon lengthening
      • ii) Gutter debridement
  • 8) Fusion takedown
    • a) Subtalar fusion, cement spacer after breaking the fusion
  • 9) Cement spacer

As noted above, the techniques of this disclosure may be applicable to ankle surgery (e.g., total ankle arthroplasty). In the example of a total ankle arthroplasty, a surgeon may perform a distal tibial cut, a proximal calcaneus cut, and two other medial/lateral cuts. To do so, the surgeon may need to place a cutting guide on the ankle joint. The cutting guide is placed so that the cuts will be perpendicular to the mechanical axis of the tibia. The placement of the cutting guide is then refined by adjusting three angles relative to the three anatomical planes (axial, sagittal and coronal). The surgeon can perform these cuts using a cut jig or can perform these cuts directly using an oscillating saw. Next, the surgeon performs the posterior and anterior talar chamfer cut.

Many of the examples provided above with regards to cutting and drilling are applicable to the cutting and drilling operations performed during a total ankle arthroplasty. For example, during preoperative phase 302 (FIG. 3) and intraoperative phase 306 (FIG. 3), orthopedic surgical system 100 (FIG. 1) may provide XR visualizations (e.g., MR visualizations or VR visualizations) that include patient-specific virtual 3D models of a patient's ankle anatomy. This may help surgeons plan and perform total ankle arthroplasties.

Furthermore, during the intraoperative phase 306 (FIG. 3) of a total ankle arthroplasty, visualization device 213 of MR system 212 may present an MR visualization that includes virtual guidance, such as virtual cutting planes, virtual drilling axes, and virtual entry points that help the surgeon perform precise cuts, drill holes, and position or place prosthetic components. For instance, the MR visualization may include cutting planes for the distal tibial cut, the proximal calcaneus cut, and so on. Prosthetic implant components for ankle arthroplasty may include, in one example, a talar dome, a tibial tray, and associated pegs or other anchor components. Moreover, a registration process similar to that described elsewhere in this disclosure with respect to shoulder repair surgery may be used in the context of total ankle arthroplasty. For instance, instead of using a center of a glenoid as a landmark for aligning a virtual 3D model with the patient's real anatomy, another landmark (e.g., the bottom of the tibia) on the patient's ankle may be used.

FIG. 20 is a flowchart illustrating an example standard set of steps of an ankle joint repair surgery. The surgeon may wear or otherwise use a visualization device, such as visualization device 213, during some or all of the steps of the surgical process of FIG. 20. In other examples, an ankle surgery may include more, fewer, or different steps. For example, an ankle surgery may include steps for adding cement, and/or other steps. In some examples, visualization device 213 may present virtual guidance to guide the surgeon, nurse, or other users through the steps in the surgical workflow.

In the example of FIG. 20, a surgeon performs an incision process (15002). During the incision process, the surgeon makes a series of incisions to expose a patient's ankle joint (e.g., to expose at least a portion of the patient's tibia and at least a portion of the patient's talus). In some examples, an MR system (e.g., MR system 212, MR system 1800A, etc.) may help the surgeon perform the incision process, e.g., by displaying virtual guidance imagery illustrating how and/or where to make the incision. As discussed above, MR system 212 may display a virtual checklist, with each item on the checklist items corresponding to an item in a checklist of steps of an orthopedic surgery. For instance, MR system 212 may display a virtual checklist having a checklist item specifying a current step of performing an incision process.

The surgeon may perform a registration process that registers a virtual tibia object with the patient's actual tibia bone (15004) in the field of view presented to the surgeon by visualization device 213. For instance, MR system 212 may obtain the virtual tibia object from storage system 206 of FIG. 2. Similar to the virtual glenoid object discussed above, the virtual tibia object may be generated based on pre-operative imaging (e.g., CT imaging) of the patient's tibia. MR system 212 may perform the registration using any suitable process. For instance, MR system 212 may perform the registration of the virtual tibia object with the patient's actual tibia bone using any of the registration techniques discussed above with reference to FIGS. 20A-31. As discussed above, the registration may produce a transformation matrix between the virtual tibia object with the patient's actual tibia bone. As discussed above, MR system 212 may display an animation, video, or text to describe how a particular step or steps are to be performed. For instance, MR system 212 may cause visualization device 213 to display a diagram or animation showing how the registration process is to be performed. As also discussed above, MR system 212 may display a virtual checklist, with each item on the checklist items corresponding to an item in a checklist of steps of an orthopedic surgery. For instance, MR system 212 may display a virtual checklist having a checklist item specifying a current step of registering a virtual tibia object with the patient's actual tibia bone.

The surgeon may perform various work steps to prepare the tibia bone (15006). Example work steps to prepare the tibia bone include, but are not limited to, installing one or more guide pins into the tibia bone, drilling one or more holes in the tibia bone, and/or attaching one or more guides to the tibia bone. MR system 212 may provide virtual guidance to assist the surgeon with the various work steps to prepare the tibia bone. As discussed above, MR system 212 may display an animation, video, or text to describe how a particular step or steps are to be performed. For instance, MR system 212 may cause visualization device 213 to display a diagram or animation showing how the tibia is to be prepared. As also discussed above, MR system 212 may display a virtual checklist, with each item on the checklist items corresponding to an item in a checklist of steps of an orthopedic surgery. For instance, MR system 212 may display a virtual checklist having a checklist item specifying a current step, or sequence of steps, of preparing the tibia bone.

FIGS. 21A and 21B are conceptual diagrams illustrating example attachment of guide pins to a tibia. The incision process may expose at least a portion of tibia 15102, fibula 15110, and talus 15108 of ankle 15100. After performing the incision process, the surgeon may install guide pins 15104A, 15104B, 15106A, and 15106B into tibia 15102.

In some examples, such as the example of FIG. 21B, the surgeon may install guide pins 15104A, 15104B, 15106A, and 15106B using a physical guide. For instance, the surgeon may place tibial guide 15112 on tibia 15102 and utilize one or more holes in tibial guide 15112 to guide installation of guide pins 15104A, 15104B, 15106A, and 15106B. In some examples, tibial guide 15112 may be a patient-specific guide that is manufactured with a surface designed to conform with the contours of tibia 15102. One example of such a patient specific guide is the Prophecy Tibial Alignment Guide of the Prophecy® Infinity® Total Ankle system produced by Wright Medical Group N.V.

In addition to, or in place of tibial guide 15112, MR system 212 may provide virtual guidance to assist the surgeon with the installation of guide pins 15104A, 15104B, 15106A, and 15106B. For instance, visualization device 213 may display a virtual marker that guides a surgeon in installing a guide pin. Visualization device 213 may display the virtual marker with an appearance that the virtual marker is overlaid on tibia 15102 (e.g., to indicate the position and/or orientation at which the guide pin is to be installed). The virtual marker may be a virtual axis at a point on tibia 15102 that guides a surgeon in installing a guide pin. For instance, as shown in FIG. 21A, visualization device 213 may display virtual axes 15114A, 15114B, 15116A, and 15116B to respectively guide installation of guide pins 15104A. 15104B, 15106A, and 15106B, e.g., along the axes. While virtual axes 15114A, 15114B, 15116A, and 15116B are illustrated in FIG. 21A as being displayed with an appearance similar to guide pins 15104A, 15104B, 15106A, and 15106B of FIG. 21B, the display of virtual markers that guide installation of guide pins (e.g., guide pins 15104A, 15104B, 15106A, and 15106B) is not so limited. Other examples of virtual markers that MR system 212 may display include, but are not limited to axes, arrows, points, circles, rings, polygons, X shapes, crosses, targets, or any other shape or combination of shapes. MR system 212 may display the virtual markers as static features or with various animations or other effects.

MR system 212 may utilize different types of virtual markers depending on whether or not a physical guide is also used. As one example, in the example of FIG. 21B where tibial guide 15112 is used, MR system 212 may utilize an arrow to guide installation of a guide pin is to be installed. As shown in FIG. 21B, visualization device 213 may display an arrow to guide installation of guide pin 15106A via a particular hole of tibial guide 15112. As another example, in the example of FIG. 21A where tibial guide 15112 is not used, MR system 212 may utilize a virtual axis to guide installation of a guide pin. As shown in FIG. 21A, visualization device 213 may display virtual axis 15116A to guide installation of guide pin 15106A.

In examples where multiple guide pins are to be installed, visualization device 213 may display a respective virtual marker for each guide pin. In the example of FIG. 151, visualization device 213 may display multiple virtual markers to guide installation of guide pins 15104A, 15104B, 15106A, and 15106B. In some examples, visualization device 213 may display the virtual markers concurrently. For instance, visualization device 213 may display virtual axes 15114A, 15114B, 15116A, and 15116B, e.g., for alignment of guide pins, at the same time. In other examples, visualization device 213 may display fewer than all of the virtual markers at a particular time. For instance, visualization device 213 may display the virtual markers sequentially. As one example, at a first time, visualization device 213 may display a first virtual marker that guides installation of a first guide pin (e.g., guide pin 15104A). At a second time that is after the first time (e.g., after guide pin 15104A has been installed), visualization device 213 may display a second virtual marker that guides installation of a second guide pin (e.g., guide pin 15104B). In other words, responsive to determining that guide pin 15404A has been installed, visualization device 213 may cease to display the virtual marker that guided installation of guide pin 15404A and display a virtual marker to a next guide pin to be installed. Visualization device 213 may continue to sequentially display virtual markers until all necessary guide pins are installed (e.g., until guide pins 15104A, 15104B, 15106A, and 15106B are installed). In this way, MR system 212 may display a plurality of virtual axes each having parameters obtained from the virtual surgical plan, each of the virtual axes configured to guide installation of a respective guide pin of a plurality of pins in the tibia.

MR system 212 may display the virtual markers with particular colors. For instance, in some examples, MR system 212 may preferably display the virtual markers in a color other than red, such as green, blue, yellow, etc. Displaying the virtual markers in a color or colors other than red may provide one or more benefits. For instance, as blood appears red and blood may be present on or around the anatomy of interest, a red colored virtual marker may not be visible.

In some examples, such as where visualization system 213 displays multiple virtual markers at the same time, visualization system 213 may alter or otherwise modify the display of a virtual marker after the surgeon has completed a corresponding work step. Alterations of the display of virtual markers may include, but are not limited to, changing a color, changing a marker type, animating (e.g., blinking or flashing), displaying an additional element (e.g., an X or a checkmark on or near the virtual marker) or any other visually perceptible alteration. For instance, visualization system 213 may initially display a first virtual marker to guide installation of guide pin 15104A as a virtual axis and a second virtual marker to guide installation of guide pin 15104B as a virtual axis. After the surgeon installs guide pin 15104A, visualization system 213 may modify the first virtual marker displayed to guide installation of guide pin 15104A (e.g., changing from a virtual axis to a reticle) while maintaining the display of the second virtual marker as a virtual axis.

MR system 212 may provide other virtual guidance in addition to, or in place of, the virtual markers. For instance, MR system 212 may display depth guidance to enable the surgeon to install the guide pins to a target depth (e.g., depth guidance similar to the depth guidance discussed above with reference to FIGS. 66-68). As another example, MR system 212 may provide targeting guidance. For instance, MR system 212 may display one or both of a virtual marker that identifies a center point or prescribed axis of the pin installation and/or an indication of whether the guide pin is aligned with the prescribed axis. As discussed above, MR system 212 may determine whether the guide pin is aligned with the prescribed axis by monitoring a position/orientation of the guide pin and/or a drill driving the guide pin, and comparing the monitored position/orientation with the prescribed axis.

The surgeon may install guide pins 15104A, 15104B, 15106A, and 15106B using the virtual guidance. In examples where tibial guide 15112 was used, the surgeon may remove tibial guide 15112 after installation of guide pins 15104A, 15104B, 15106A, and 15106B.

FIG. 22 is a conceptual diagram illustrating example drilling of holes in a tibia. As shown in FIG. 22, the surgeon may install drilling guide 15202 onto tibia 15102 using guide pins 15104A, 15104B, 15106A, and 15106B. Drilling guide 15202 includes one or more channels that guide drilling of holes into tibia 15102. For instance, as shown in FIG. 22, drilling guide 15202 include first channel 15204A and second channel 15204B. The surgeon may utilize a drill (e.g., a surgical motor with tibial corner drill bit) to drill a hole using each of first channel 15204A and second channel 15204B. In this way, the surgeon may bi-cortically drill both proximal corners of tibia 15102.

In addition to, or in place of drilling guide 15202, MR system 212 may provide virtual guidance to assist the surgeon with the drilling of the proximal corners of tibia 15102. For instance, visualization device 213 may display a virtual marker that guides a surgeon in drilling a hole in tibia 15102. Visualization device 213 may display the virtual marker overlaid on tibia 15102 (e.g., to indicate the position and/or orientation at which the hole is to be drilled). The virtual marker may be a virtual drilling axis at a point on tibia 15102 that guides a surgeon in performing the drilling. Similar to the virtual markers discussed above that guide installation of guide pins, visualization device 213 device may display the virtual markers that guide the drilling of the proximal corners of tibia 15102 concurrently or sequentially, and the virtual markers that guide the drilling at each respective proximal corner of the tibia.

MR system 212 may provide other virtual guidance in addition to, or in place of, the virtual markers. For instance, MR system 212 may display depth guidance to enable the surgeon to drill the holes to a target depth (e.g., depth guidance similar to the depth guidance discussed above with reference to FIGS. 66-68). As another example, MR system 212 may provide targeting guidance. For instance, MR system 212 may display one or both of a virtual marker that identifies a center point or prescribed axis of the drilling, e.g., into the tibia or talus, and/or an indication of whether the drill bit is aligned with the prescribed axis. As discussed above, MR system 212 may determine whether the drill bit is aligned with the prescribed axis by monitoring a position/orientation of the drill bit and/or a drill driving the drill bit, and comparing the monitored position/orientation with the prescribed axis.

In some examples, MR system 212 may utilize the surgical item tracking techniques described in this disclosure (e.g., with reference to FIGS. 83-108) to assist with the ankle arthroplasty. For instance, MR system 212 may select a surgical item of a plurality of surgical items. The plurality of surgical items may be included in one or more trays for performing the ankle arthroplasty procedure. Example surgical items include, but are not limited to, tools, instruments, implants, associated hardware. MR system 212 may cause a second visualization device (e.g., worn by a nurse or someone other than the surgeon) to display virtual information that identifies the selected surgical item among the plurality of surgical items, wherein the virtual information is presented on or adjacent to a position of the selected surgical item visible via the second visualization device. MR system 212 may select the surgical item as a surgical item associated with a current step of the ankle arthroplasty procedure. For instance, in the example of FIG. 22 where the surgeon may use drilling guide 15202 to drill the tibial corners, MR system 212 may select drilling guide 15202 as the selected surgical item. The second visualization device may display virtual information that identifies drilling guide 15202 (e.g., highlight or otherwise identify drilling guide 15202 in a manner similar to FIGS. 105-108)

With continued reference to the stages of an ankle joint repair surgery of FIG. 20, the surgeon may perform a tibia resection process (15008). For instance, the surgeon may remove a portion of tibia 15102 to make room for subsequent installation of a tibial implant. In some examples, the surgeon may perform the tibial resection by making three cuts (e.g., a proximal cut, a medial cut, and a lateral cut) in tibia 15102 to remove a portion of tibia 15102 and create a space for subsequent installation of a tibial implant. As discussed above, MR system 212 may display an animation, video, or text to describe how a particular step or steps are to be performed. For instance, MR system 212 may cause visualization device 213 to display a diagram or animation showing how the tibia resection is to be performed. As also discussed above, MR system 212 may display a virtual checklist, with each item on the checklist items corresponding to an item in a checklist of steps of an orthopedic surgery. For instance, MR system 212 may display a virtual checklist having a checklist item specifying a current step, or sequence of steps, of performing the tibial resection. Checklist items may be standard steps or ancillary steps.

FIG. 23 is a conceptual diagram illustrating example resection of a tibia. As shown in FIG. 23, the surgeon may install resection guide 15302 onto tibia 15102 using guide pins 15104A, 15104B, 15106A, and 15106B. Resection guide 15302 includes one or more channels that guide performing cuts into tibia 15102. For instance, as shown in FIG. 23, resection guide 15202 include first channel 15306A that guides performance of a medial cut, second channel 15306B that guides performance of a proximal cut, and third channel 15306C that guides performance of a lateral cut. In some examples, resection guide 15302 may include a fourth channel that guides performance of a resection of talus 15108. For instance, as shown in FIG. 23, resection guide 15302 may include fourth channel 15304. The surgeon may utilize a saw blade (e.g., an oscillating bone saw) to perform the medial, lateral, and proximal cuts using channels 15306A-15306C. In this way, the surgeon may perform a resection of tibia 15102.

In addition to, or in place of resection guide 15302, MR system 212 may provide virtual guidance to assist the surgeon with performing the resection of tibia 15102. For instance, visualization device 213 may display a virtual marker that guides a surgeon in performing a cut in tibia 15102. Visualization device 213 may display the marker overlaid on tibia 15102 (e.g., to indicate the position and/or orientation at which the cut is to be made). The virtual marker may be a virtual cutting line, a virtual cutting surface or a virtual cutting plane (e.g., similar to virtual cutting plane 4200 of FIGS. 42A and 42B) at a point on tibia 15102 that guides a surgeon in performing the cut. Similar to the virtual markers discussed above that guide installation of guide pins, visualization device 213 device may display the virtual markers that guide the performance of the proximal, medial, and lateral cuts concurrently or sequentially. In this way, MR system 212 may display a plurality of virtual cutting surfaces each having parameters obtained from the virtual surgical plan, the plurality of virtual cutting surfaces configured to guide resection of the tibia.

MR system 212 may provide other virtual guidance in addition to, or in place of, the virtual markers. For instance, MR system 212 may display depth guidance to enable the surgeon to perform the cuts to a target depth (e.g., depth guidance similar to the depth guidance discussed above with reference to FIGS. 66-68). As another example, MR system 212 may provide targeting guidance. For instance, MR system 212 may display one or both of a virtual marker that identifies a prescribed plane of the cutting and/or an indication of whether the saw blade is aligned with the prescribed plane. As discussed above, MR system 212 may determine whether the saw blade is aligned with the prescribed plane by monitoring a position/orientation of the saw blade and/or a motor driving the saw blade the guide pin, and comparing the monitored position/orientation with the prescribed plane.

The surgeon may remove the resection (i.e., the portion of tibia 15102 separated via the cuts). Guide pins 15104A and 15104B may be attached to the resection and removed as a consequence of the resection removal.

As discussed above, MR system 212 may cause the second visualization device to display virtual information that identifies surgical item selected for a current step of the ankle arthroplasty procedure. For instance, in the example of FIG. 23 where the surgeon may use resection guide 15302 to perform the tibial resection, MR system 212 may select resection guide 15302 as the selected surgical item.

Furthermore, with reference to the stages of the ankle joint repair surgery of FIG. 20, the surgeon may perform a registration process that registers a virtual talus object with the patient's actual talus bone (15010) in the field of view presented to the surgeon by visualization device 213. For instance, MR system 212 may obtain the virtual talus object from storage system 206 of FIG. 2. Similar to the virtual tibia object discussed above, the virtual talus object may be generated based on pre-operative imaging (e.g., CT imaging) of the patient's talus. MR system 212 may perform the registration using any suitable process. For instance, MR system 212 may perform the registration of the virtual talus object with the patient's actual talus bone using any of the registration techniques discussed above with reference to FIGS. 20A-31. As discussed above, the registration may produce a transformation matrix between the virtual talus object with the patient's actual talus bone. As also discussed above, MR system 212 may display a virtual checklist, with each item on the checklist items corresponding to an item in a checklist of steps of an orthopedic surgery. For instance, MR system 212 may display a virtual checklist having a checklist item specifying a current step, or sequence of steps, of registering a virtual talus object with the patient's actual talus bone.

Additionally, in the example of FIG. 20, the surgeon may perform various work steps to prepare the talus bone (15012). Example work steps to prepare the talus bone include, but are not necessarily limited to, installing one or more guide pins into the talus bone, drilling one or more holes in the talus bone, and/or attaching one or more guides (e.g., cutting guides, drilling guides, reaming guides, etc.) to the talus bone. MR system 212 may provide virtual guidance to assist the surgeon with the various work steps to prepare the talus bone. As discussed above, MR system 212 may display an animation, video, or text to describe how a particular step or steps are to be performed. For instance, MR system 212 may cause visualization device 213 to display a diagram or animation showing how the talus is to be prepared. As also discussed above, MR system 212 may display a virtual checklist, with each item on the checklist items corresponding to an item in a checklist of steps of an orthopedic surgery. For instance, MR system 212 may display a virtual checklist having a checklist item specifying a current step, or sequence of steps, of preparing the talus bone.

FIGS. 24A and 24B are conceptual diagrams illustrating example guide pins installed in a talus during the talus preparation process. As shown in FIGS. 24A and 24B, the surgeon may install guide pins 15402A and 15402B into talus 15108.

In some examples, such as the example of FIG. 24B, the surgeon may install guide pins 15402A and 15402B using a physical guide. For instance, the surgeon may place talar guide 15404 on talus 15108 and utilize one or more holes in talar guide 15404 to guide installation of guide pins 15402A and 15402B. In some examples, talar guide 1540 may be a patient-specific guide that is manufactured with a surface designed to conform with the contours of talus 15108. One example of such a patient-specific guide is the Prophecy Talus Alignment Guide of the Prophecy® Infinity® Total Ankle system produced by Wright Medical Group N.V.

In addition to, or in place of talar guide 15404, MR system 212 may provide virtual guidance to assist the surgeon with the installation of guide pins 15402A and 15402B. For instance, visualization device 213 may display one or more virtual markers that guide a surgeon in installing a guide pin of guide pins 15402A and 15402B. For instance, as shown in FIG. 24A, visualization device 213 may display virtual axes 15406A and 15406B to respectively guide installation of guide pins 15402A and 15402B. Visualization device 213 may display the virtual markers in a manner similar to that described above with reference to FIGS. 21A and 21B. MR system 212 may provide other virtual guidance to assist with the installation of guide pins 15402A and 15402B in addition to, or in place of, the virtual markers. For instance, MR system 212 may provide any of the additional virtual guidance (e.g., depth guidance, targeting guidance, etc.) discussed above. In this way, MR system 212 may display a plurality of a virtual axes each having parameters obtained from the virtual surgical plan, and each of the virtual axes configured to guide installation of a respective guide pin in the talus. A virtual axis may guide installation of a corresponding guide pin by providing a visual reference with which a surgeon may align the physical guide pin during installation of the guide pin. As discussed herein, in some examples, MR system 212 may provide feedback as to whether the physical guide pin is actually aligned with the virtual axis.

The surgeon may install guide pins 15402A and 15402B using the virtual guidance. For example, the surgeon may align guide longitudinal axes of pins 15402A and 15402B with respective virtual axes to place the pins in bone. In examples where talar guide 15404 was used, the surgeon may remove talar guide 15404 after installation of guide pins 15402A and 15402B.

As discussed above, MR system 212 may cause the second visualization device to display virtual information that identifies a surgical item selected for a current step of the ankle arthroplasty procedure. For instance, in the example of FIG. 154B where the surgeon may use talar guide 15404 to install guide pins 15402A and 15402B, MR system 212 may select talar guide 15404 as the selected surgical item.

With continued reference to FIG. 20, after performing the talus preparation process, the surgeon may perform various perform a talus resection process (15014). For instance, the surgeon may remove a portion of talus 15108 to make room for subsequent installation of a talus implant. In some examples, the surgeon may perform the talus resection by making a single cut in talus 15108 to remove a portion of talus 15108 and create a space for subsequent installation of a talus implant. As discussed above, MR system 212 may display an animation, video, or text to describe how a particular step or steps are to be performed. For instance, MR system 212 may cause visualization device 213 to display a diagram or animation showing how the talus resection is to be performed. As also discussed above, MR system 212 may display a virtual checklist, with each item on the checklist items corresponding to an item in a checklist of steps of an orthopedic surgery. For instance, MR system 212 may display a virtual checklist having a checklist item specifying a current step, or sequence of steps, of performing the talar resection.

FIG. 25 is a conceptual diagram illustrating example resection of a talus. As shown in FIG. 25, the surgeon may install resection guide 15302 onto talus 15108 using guide pins 15402A and 15402B. In the example of FIG. 25, the surgeon may utilize the same resection guide (i.e., resection guide 15302) as was used to perform the tibial resection. In other examples, a talus specific resection guide may be used. The surgeon may perform the talus resection using resection guide 15302. For instance, the surgeon may utilize a saw blade (e.g., an oscillating bone saw) to perform a cut using channel 15304. In this way, the surgeon may perform a resection of talus 15108.

In addition to, or in place of resection guide 15302, MR system 212 may provide virtual guidance to assist the surgeon with performing the resection of talus 15308. For instance, visualization device 213 may display a virtual marker that guides a surgeon in performing a cut in talus 15108. Visualization device 213 may display the marker overlaid on talus 15108 (e.g., to indicate the position and/or orientation at which the cut is to be made). The virtual marker may be a virtual cutting line, virtual cutting surface or virtual cutting plane (e.g., similar to virtual cutting plane 4200 of FIGS. 42A and 42B) at a point on talus 15108 that guides a surgeon in performing the cut. In this way, MR system 212 may display a virtual cutting surface having parameters obtained from the virtual surgical plan, the virtual cutting surface configured to guide primary resection of the talus.

MR system 212 may provide other virtual guidance in addition to, or in place of, the virtual markers. For instance, MR system 212 may display depth guidance to enable the surgeon to perform the cut to a target depth (e.g., depth guidance similar to the depth guidance discussed above with reference to FIGS. 66-68). As another example, MR system 212 may provide targeting guidance. For instance, MR system 212 may display one or both of a virtual marker that identifies a prescribed plane of the cutting and/or an indication of whether the saw blade is aligned with the prescribed plane. As discussed above, in some examples, MR system 212 may determine whether the saw blade is aligned with the prescribed plane by registering the saw blade or something connected thereto (e.g., a saw motor body, a saw handle, a physical registration marker, etc.) with a corresponding virtual model, and comparing the position of the corresponding virtual model with the prescribed plane.

The surgeon may remove the resection (i.e., the portion of talus 15108 separated via the cuts). In some examples, the surgeon may use various tools (e.g., a reciprocating saw or bone rasp) to remove any excess bone left after the resection has been removed. FIG. 26 is a conceptual diagram of an example ankle after performance of a tibial resection and a talar resection.

The surgeon may perform one or more additional work steps on one or both of tibia 15102 and/or talus 15108 to prepare tibia 15102 and/or talus 15108 to receive implants. Example additional work steps include, but are not necessarily limited to, tibial tray trialing, tibial peg broaching, talar chamfer resections, and talar peg drilling.

FIGS. 27A-27C are conceptual diagrams illustrating an example of tibial tray trialing. In some examples, it may be desirable to ensure that, when installed, a posterior edge of the tibial implant will at least reach the posterior portion of tibia 15102. Additionally, in some examples, there may be multiple size tibial implants available. As such, it may be desirable for the surgeon to determine which size tibial implant to utilize. To ensure that the posterior edge of the tibial implant will at least reach the posterior portion of tibia 15102 and/or to determine which size tibial implant to utilize, the surgeon may perform tibial tray trialing.

To perform tibial tray trialing, the surgeon may attach tibial tray trial 15702 to tibia 15102. As shown in FIG. 27A, tibial tray trial 15702 may include posterior edge 15704, indicator 15710, guide pin holes 15712A and 15712B, broaching holes 15714A and 15714B (an additional anterior broaching hole 15714C is not shown), and anterior surface 15716. The surgeon may attach tibial tray trial 15702 to tibia 15102 by sliding guide pins 15106A and 15106B into corresponding guide pin holes 15712A and 15712B. In some examples, after attaching tibial tray trial 15702, the surgeon may trim guide pins 15106A and 15106B to be flush with anterior surface 15716 of tibial tray trial 15702 (e.g., as shown in FIG. 28).

In some examples, the surgeon may utilize fluoroscopy to perform the tibial tray trialing. For instance, the surgeon may utilize fluoroscopy to determine the relative positions of tibial tray trial 15702 and tibia 15102.

MR system 212 may provide virtual guidance to assist with tibial tray trialing. As one example, visualization device 213 may display a synthesized view showing the relative positions of tibial tray trial 15702 and tibia 15102. For instance, MR system 212 may register tibial tray trial 15702 to a corresponding virtual model of tibial tray trial and utilize the registered virtual models of tibial tray trial 15702 and tibia 15102 to synthesize a view showing the relative positions of the virtual models of tibial tray trial 15702 and tibia 15102. As the virtual models of tibial tray trial 15702 and tibia 15102 are respectively registered to tibial tray trial 15702 and tibia 15102, the relative positions of the virtual models of tibial tray trial 15702 and tibia 15102 corresponds to the relative positions of tibial tray trial 15702 and tibia 15102. The synthesized views may appear similar to the conceptual diagrams of FIGS. 27B and 27C.

The surgeon may utilize the synthesized view to perform one or more adjustments on tibial tray trial 15702. For instance, if the synthesized view indicates that posterior edge 15704 of tibial tray trial 15702 extends past posterior edge 15706 of tibia 15102, the surgeon may adjust tibial tray trial 15702 to anteriorly advance posterior edge 15704 of tibial tray trial 15702. For instance, the surgeon may utilize tool 15708 to anteriorly translate tibial tray trial 15702.

The surgeon may utilize the synthesized view to determine which size tibial implant is to be utilized. For instance, if the synthesized view indicates that indicator 15710 (illustrated in FIG. 27C as a notch) of tibial tray trial 15702 extends past posterior edge 15706 of tibia 15102, the surgeon may determine that a first size tibial implant (e.g., a standard size) is to be utilized. If the synthesized view indicates that indicator 15710 of tibial tray trial 15702 does not extend past posterior edge 15706 of tibia 15102, the surgeon may determine that a second size tibial implant (e.g., a long size) is to be utilized.

As described above, MR system 212 may enable the surgeon to perform tibial tray trialing using virtual guidance. In some examples, MR system 212 may enable the surgeon to perform tibial tray trialing without using fluoroscopy.

As discussed above, MR system 212 may cause the second visualization device to display virtual information that identifies surgical item selected for a current step of the ankle arthroplasty procedure. For instance, in the example of FIGS. 27A-27C where the surgeon may use tibial tray trial 15702, MR system 212 may select tibial tray trial 15702 as the selected surgical item.

The surgeon may create anchorage points for the tibial implant. For instance, the surgeon may utilize a tibial tray trial to perform tibial peg broaching. FIG. 28 is a conceptual diagram illustrating an example creation of tibial implant anchorage. As shown in FIG. 28, the surgeon may utilize anterior tibial peg broach 15802A to broach a first anterior hole in tibia 15102 using broaching hole 15714A, utilize anterior tibial peg broach 15802A to broach a second anterior hole in tibia 15102 using broaching hole 15714C, and utilize posterior tibial peg broach 15802B to broach a hole in tibia 15102 using broaching hole 15714B. The holes broached in tibia 15102 may constitute anchorage points for the tibial implant.

As discussed above, MR system 212 may cause the second visualization device to display virtual information that identifies surgical item selected for a current step of the ankle arthroplasty procedure. For instance, in the example of FIG. 28 where the surgeon may use anterior tibial peg broach 15802A and posterior tibial peg broach 15802B, MR system 212 may select anterior tibial peg broach 15802A and posterior tibial peg broach 15802B as the selected surgical item (or items). As discussed above, MR system 212 may cause the second visualization device, and/or visualization device 213, to visually distinguish the selected surgical items (i.e., anterior tibial peg broach 15802A and posterior tibial peg broach 15802B).

The surgeon may perform one or more talar chamfer resections to further prepare talus 15108 to receive the talar implant. In some examples, the surgeon may perform an anterior talar chamfer resection and a posterior talar chamfer resection. To perform the one or more talar resections, the surgeon may attach one or more guide pins to talus 15108.

FIGS. 29A and 29B are conceptual diagrams illustrating an example attachment of guide pins to talus 15108. MR system 212 may provide virtual guidance to guide the surgeon in attaching guide pins 15904A and 15904B to talus 15108. For instance, as shown in FIG. 29A, visualization device 213 may display virtual axes 15902A and 15902B overlaid on talus 15108 to guide installation of guide pins 15904A and 15904B to talus 15108. While illustrated in FIG. 29A as virtual axes, visualization device 213 may display any of the virtual markers described herein to guide installation of guide pins 15904A and 15904B to talus 15108.

In some examples, the surgeon may utilize a physical guide to assist with the installation of guide pins 15904A and 15904B to talus 15108. For instance, the surgeon may utilize fluoroscopy to position a talar dome trial component. When the talar dome trial component is positioned, the surgeon may utilize holes in the talar dome trial component to guide the installation of guide pins 15904A and 15904B.

The surgeon may perform the talar chamfer resections using guide pins 15904A and 15904B. For instance, as shown in FIG. 30, the surgeon may position talar resection guide base 16002 on talus 15108 using guide pins 15904A and 15904B. The surgeon may utilize one or more components to secure talar resection guide base 16002 to talus 15108. For instance, as shown in FIG. 31, the surgeon may install fixation screws 16102A and 16102B through resection guide base 16002 into talus 15108.

MR system 212 may provide virtual guidance to assist the surgeon with the installation of fixation screws 16102A and 16102B. As one example, visualization device 213 may display virtual markers that indicate the location and axis at which fixation screws 16102A and 16102B are to be installed. As another example, visualization device 213 may provide depth guidance to enable the surgeon to install fixation screws 16102A and 16102B to a target depth (e.g., depth guidance similar to the depth guidance discussed above with reference to FIGS. 66-68). In some examples, MR system 212 may utilize closed-loop tool control to positively control a drill used to attach fixation screws 16102A and 16102B. For instance, MR system 212 may utilize the closed-loop tool control techniques discussed above, e.g., with reference to FIG. 72, to reduce a speed of and/or stop the drill used to attach fixation screws 16102A and 16102B when a desired depth and/or torque is reached.

The surgeon may utilize talar resection guide base 16002 to perform the posterior talar chamfer resection. For instance, as shown in FIG. 31, the surgeon may insert saw blade 16104 into slot 16004 of talar resection guide base 16002 to perform the posterior talar chamfer resection.

As discussed above, MR system 212 may cause the second visualization device to display virtual information that identifies surgical item selected for a current step of the ankle arthroplasty procedure. For instance, in the example of FIG. 31 where the surgeon may use talar resection guide base 16002, MR system 212 may select talar resection guide base 16002 as the selected surgical item.

In addition to, or in place of talar resection guide base 16002, MR system 212 may provide virtual guidance to assist the surgeon with performing the posterior talar chamfer resection. For instance, visualization device 213 may display a virtual marker that guides a surgeon in performing the posterior talar chamfer resection. Visualization device 213 may display the marker overlaid on talus 15108 (e.g., to indicate the position and/or orientation at which the cut is to be made). The virtual marker may be a virtual surface or virtual cutting plane (e.g., similar to virtual cutting plane 4200 of FIGS. 42A and 42B) at a point on talus 15108 that guides a surgeon in performing the cut.

The surgeon may utilize talar resection guide base 16002 to perform the anterior talar chamfer resection. For instance, as shown in FIG. 32, the surgeon may attach anterior talar guide 16202 to talar resection guide base 16002. The surgeon may utilize a drill with talar reamer 16204 to ream the anterior surface of talus 15108. For instance, the surgeon may slide talar reamer 16204 horizontally through anterior talar guide 16202 to prepare the surface of talus 15108 for an anterior flat of the talar implant. As shown in FIG. 32, talar reamer 16204 may include depth stop 16206 that engages surface 16208 of anterior talar guide 16202 to control the reaming depth. The surgeon may rotate talar guide 16202 180 degrees and again slide talar reamer 16204 horizontally through (the now rotated) anterior talar guide 16202 to prepare the surface of talus 15108 for an anterior chamfer of the talar implant. As discussed above, talar reamer 16204 may include depth stop 16206 that engages surface 16208 of anterior talar guide 16202 to control the reaming depth.

In some examples, for one or both of the anterior flat and anterior chamfer preparation, the surgeon may perform plunge cuts (e.g., using talar reamer 16204) to prepare talus 15108 for reaming. For instance, the surgeon may attach a pilot guide with holes that guide performance of the plunge cuts. Depth stop 16206 of talar reamer 16204 may engage with a surface of the pilot guide the control the plunge depth.

In addition to, or in place of talar resection guide base 16002, MR system 212 may provide virtual guidance to assist the surgeon with performing the anterior talar chamfer resection. For instance, visualization device 213 may display one or more virtual markers that guide a surgeon in performing the plunge cuts and/or horizontal reaming. As one example, visualization device 213 may display a respective virtual axis for each of the plunge cuts. MR system 212 may provide other virtual guidance to assist with performing the plunge cuts and/or horizontal reaming in addition to, or in place of, the virtual markers. For instance, MR system 212 may provide any of the additional virtual guidance (e.g., depth guidance, targeting guidance, etc.) discussed above.

The surgeon may perform talar peg drilling to create anchorage points in talus 15108 for the talar implant. MR system 212 may provide virtual guidance to assist the surgeon with performing the anterior talar chamfer resection. For instance, visualization device 213 may display one or more virtual markers that guide a surgeon in drilling holes in talus 15108. As shown in FIG. 34, visualization device 213 may display virtual axes 16402A and 16402B that guide drilling of peg holes 16502A and 16502B of FIG. 5. MR system 212 may provide other virtual guidance to assist with creating the anchorage in addition to, or in place of, the virtual markers. For instance, MR system 212 may provide any of the additional virtual guidance (e.g., depth guidance, targeting guidance, etc.) discussed above. In this way, MR system 212 may display a plurality of virtual drilling axes each having parameters obtained from the virtual surgical plan, each of the virtual drilling axes configured to guide drilling of an anchorage point in the talus.

With continued reference to FIG. 20, the surgeon may perform a tibia implant installation process (15016). FIG. 36 is a conceptual diagram illustrating an example tibial implant. As shown in FIG. 36, tibial implant 16602 includes posterior peg 16604A, and anterior pegs 16604B and 16604C. FIG. 37 is a conceptual diagram illustrating an example tibia as prepared using the steps described above. As shown in FIG. 37, tibia 15102 includes peg holes 16702A-16702C that were created during the broaching process described above with reference to FIG. 28.

The surgeon may install tibial implant 16602 such that posterior peg 16604A, and anterior pegs 16604B and 16604C of tibial implant 16602 engage with peg holes 16702A-16702C of tibia 15102. For instance, the surgeon may position tibial implant 16602 such that posterior peg 16604A lines up with peg hole 16702A, anterior peg 16604B lines up with peg hole 16702B, and anterior peg 16604C lines up with peg hole 16702C. Once the pegs are lined up with their corresponding peg holes, the surgeon may impact tibial implant 16602 into tibia 15102. As discussed above, MR system 212 may display an animation, video, or text to describe how a particular step or steps are to be performed. For instance, MR system 212 may cause visualization device 213 to display a diagram or animation showing how tibial implant 16602 is to be installed. As also discussed above, MR system 212 may display a virtual checklist, with each item on the checklist items corresponding to an item in a checklist of steps of an orthopedic surgery. For instance, MR system 212 may display a virtual checklist having a checklist item specifying a current step, or sequence of steps, of installing the tibial implant.

FIG. 38 is a conceptual diagram illustrating example impaction of a tibial implant into a tibia. As shown in FIG. 38, the surgeon may utilize tray impactor 16802 to impact tibial implant 16602 into tibia 15102. For instance, the surgeon may place tip 16806 of tray impactor 16802 on tibial implant 16602 and strike one or both of impaction points 16804A and/or 16804B with an impactor (e.g., a hammer).

With continued reference to FIG. 20, the surgeon may perform a talus implant installation process (15018). FIG. 39 is a conceptual diagram illustrating an example talar implant. As shown in FIG. 39, talar implant 16902 includes first peg 16904A and second peg 16904B.

The surgeon may install talar implant 16902 such that first peg 16904A and second peg 16904B of talar implant 16902 engage with peg holes 16502A and 16502B of talus 15108. For instance, the surgeon may position talar implant 16902 such that first peg 16904A lines up with peg hole 16502A, and second peg 16904B of talar implant 16902 lines up with peg hole 16502B. Once the pegs are lined up with their corresponding peg holes, the surgeon may impact talar implant 16902 into talus 15108.

As discussed above, MR system 212 may cause the second visualization device to display virtual information that identifies a surgical item selected for a current step of the ankle arthroplasty procedure. For instance, in the example of FIG. 38 where the surgeon may use tray impactor 16802. MR system 212 may select tray impactor 16802 as the selected surgical item.

FIG. 40 is a conceptual diagram illustrating example impaction of a talar implant into a talus. As shown in FIG. 40, the surgeon may utilize talar impactor 17002 to impact talar implant 16902 into talus 15108. For instance, the surgeon may place tip 17004 of talar impactor 17002 on talar implant 16902 and strike an impaction point of talar impactor 17002 with an impactor (e.g., a hammer).

As discussed above, MR system 212 may cause the second visualization device to display virtual information that identifies surgical item selected for a current step of the ankle arthroplasty procedure. For instance, in the example of FIG. 38 where the surgeon may use talar impactor 17002, MR system 212 may select talar impactor 17002 as the selected surgical item. As discussed above, MR system 212 may display an animation, video, or text to describe how a particular step or steps are to be performed. For instance, MR system 212 may cause visualization device 213 to display a diagram or animation showing how talar implant 16902 is to be installed. As also discussed above, MR system 212 may display a virtual checklist, with each item on the checklist items corresponding to an item in a checklist of steps of an orthopedic surgery. For instance, MR system 212 may display a virtual checklist having a checklist item specifying a current step, or sequence of steps, of installing the talar implant.

With continued reference to FIG. 20, the surgeon may perform a bearing installation process (15020). The surgeon may install a bearing between tibial implant 16602 and talar implant 16902. For instance, as shown in FIG. 41, the surgeon may install bearing 17102 between tibial implant 16602 and talar implant 16902. As discussed above, MR system 212 may display an animation, video, or text to describe how a particular step or steps are to be performed. For instance, MR system 212 may cause visualization device 213 to display a diagram or animation showing how bearing 17102 is to be installed. As also discussed above, MR system 212 may display a virtual checklist, with each item on the checklist items corresponding to an item in a checklist of steps of an orthopedic surgery. For instance, MR system 212 may display a virtual checklist having a checklist item specifying a current step, or sequence of steps, of installing the bearing.

Subsequently, in the example of FIG. 20, the surgeon may perform a wound closure process (15022). During the wound closure process, the surgeon may reconnect tissues severed during the incision process in order to close the wound in the patient's ankle. As discussed above, MR system 212 may display an animation, video, or text to describe how a particular step or steps are to be performed. For instance, MR system 212 may cause visualization device 213 to display a diagram or animation showing how the wound is to be closed. As also discussed above, MR system 212 may display a virtual checklist, with each item on the checklist items corresponding to an item in a checklist of steps of an orthopedic surgery. For instance, MR system 212 may display a virtual checklist having a checklist item specifying a current step, or sequence of steps, of closing the wound.

FIG. 42 is a flowchart illustrating example technique for recommending ancillary steps for inclusion in a surgical plan, in accordance with one or more techniques of this disclosure. The technique of FIG. 42 may be performed by one or more processors of a computing system, such as one or more processors of orthopedic surgical system 100 of FIG. 1. For purposes of explanation, the technique of FIG. 42 will be described as being performed by preoperative surgical system 202. However, the technique of FIG. 42 may be performed by other components of orthopedic surgical system 100, such as processors 210.

Preoperative surgical system 202 may obtain values of a plurality of parameters of a particular patient on which an arthroplasty procedure is to be performed (4302). The plurality of parameters may be various data points about the patient. Example parameters include, but are not limited to, images of a joint of the patient (e.g., MRI/CT images), morphological measurements of the joint (e.g., determined based on the images), demographic information (e.g., patient age, patient activities, patient gender, patient body mass index (BMI), etc.), an objective of the patient, etc. As discussed above, different surgical procedures may include different standard sets of steps. The standard set of steps for a surgical procedure may be the steps that must be performed in order to achieve completion of the surgical procedure. As one example, where the surgical procedure is a shoulder arthroplasty, the standard set of steps may include those shown in FIG. 11 (e.g., glenoid preparation, glenoid implantation, humeral preparation, and humeral implantation). As another example, where the surgical procedure is an ankle arthroplasty, the standard set of steps may include those shown in FIG. 20 (e.g., tibia preparation, tibia resection, talus preparation, talus resection, tibia implantation, and talus implantation).

Preoperative surgical system 202 may select, based on the values of the plurality of parameters, one or more ancillary steps of a plurality of ancillary steps for inclusion in the arthroplasty procedure (4304). For instance, preoperative surgical system 202 may evaluate a large quantity of ancillary steps available for performance during the arthroplasty procedure. As noted above, the plurality of ancillary steps may not be included in the standard set of steps.

In some examples, preoperative surgical system 202 may utilize a machine learning model to select the ancillary steps. For instance, preoperative surgical system 202 may apply the values of the plurality of parameters as an input vector to a machine learning model to generate an output vector, determine, based on the output vector, the one or more ancillary steps. In some examples, the machine learning model may be a NN. For instance, the machine learning model may be a NN specific to the type of arthroplasty procedure and output layer neurons of the neural network may each correspond to an ancillary step of the plurality of ancillary steps. Further details of the use of machine learning are discussed above with reference to FIGS. 17-19.

Preoperative surgical system 202 may output, for the particular patient, a surgical plan including the standard set of steps and the selected one or more ancillary steps (4306). Preoperative surgical system 202 may output the surgical plan in any of a wide variety of modalities. For instance, preoperative surgical system 202 may output a graphical representation of at least one of the selected one or more ancillary steps, a textual description of at least one of the selected one or more ancillary steps, and/or a combined graphical and textual description of at least one of the selected one or more ancillary steps.

In some examples, preoperative surgical system 202 may output a representation (e.g., graphical, textual, or both) of how the at least one ancillary step will affect an outcome of the particular patient. As one example, preoperative surgical system 202 may output a representation of how much range of motion a patient will gain if the at least one ancillary step is performed. As another example, preoperative surgical system 202 may output a representation of how performance of the at least one ancillary step will affect the postoperative healing and recovery process.

The surgical plan output by preoperative surgical system 202 may include a subset of the plurality of ancillary steps. For instance, the surgical plan output by preoperative surgical system 202 may not include non-selected ancillary steps of the plurality of ancillary steps.

As discussed above, an MR system may provide virtual guidance to assist a surgeon in performance of a surgical procedure. In accordance with one or more techniques of this disclosure, in addition to or in place of providing virtual guidance for performance of standard steps of an arthroplasty, a mixed reality system (e.g., MR system 212) may provide virtual guidance for performance of ancillary steps of an arthroplasty. For instance, during performance of the arthroplasty on the particular patient, MR system 212 may display, via a visualization device and overlaid on a portion of the joint, a virtual guide that guides performance of at least one of the selected one or more one or more ancillary steps. In this way, the quality of less frequently performed ancillary steps may be improved.

In some examples, preoperative surgical system 202 may output a recommendation that the arthroplasty procedure be separated into two or more separate procedures. For instance, based on the values of the parameters, preoperative surgical system 202 may determine that patient outcome will be more favorable if the arthroplasty procedure is divided into two surgeries that are temporally distinct (e.g., performed on different days, which could be a week or more apart). As one specific example, fixing the hindfoot and midfoot that are severely malrotated (in multiple planes, often) may require fusing them and having the new bony construct be stable before performing additional surgical steps. If the total ankle surgery were performed immediately afterward, the new bony construct (fusion) may not be strong enough to support the ankle implants. As such, in examples where the hindfoot and midfoot are to be fused, preoperative surgical system 202 may determine to divide the arthroplasty procedure into two surgeries (e.g., a first surgery to at least perform the fixing of the hindfoot and midfoot and a second surgery to at least install implant components).

While the techniques been disclosed with respect to a limited number of examples, those skilled in the art, having the benefit of this disclosure, will appreciate numerous modifications and variations there from. For instance, it is contemplated that any reasonable combination of the described examples may be performed. It is intended that the appended claims cover such modifications and variations as fall within the true spirit and scope of the invention.

It is to be recognized that depending on the example, certain acts or events of any of the techniques described herein can be performed in a different sequence, may be added, merged, or left out altogether (e.g., not all described acts or events are necessary for the practice of the techniques). Moreover, in certain examples, acts or events may be performed concurrently, e.g., through multi-threaded processing, interrupt processing, or multiple processors, rather than sequentially.

In one or more examples, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium and executed by a hardware-based processing unit. Computer-readable media may include computer-readable storage media, which corresponds to a tangible medium such as data storage media, or communication media including any medium that facilitates transfer of a computer program from one place to another, e.g., according to a communication protocol. In this manner, computer-readable media generally may correspond to (1) tangible computer-readable storage media which is non-transitory or (2) a communication medium such as a signal or carrier wave. Data storage media may be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, code and/or data structures for implementation of the techniques described in this disclosure. A computer program product may include a computer-readable medium.

By way of example, and not limitation, such computer-readable storage media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage, or other magnetic storage devices, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Also, any connection is properly termed a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. It should be understood, however, that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transitory media, but are instead directed to non-transitory, tangible storage media. Disk and disc, as used herein, includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc, where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.

Operations described in this disclosure may be performed by one or more processors, which may be implemented as fixed-function processing circuits, programmable circuits, or combinations thereof, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Fixed-function circuits refer to circuits that provide particular functionality and are preset on the operations that can be performed. Programmable circuits refer to circuits that can programmed to perform various tasks and provide flexible functionality in the operations that can be performed. For instance, programmable circuits may execute instructions specified by software or firmware that cause the programmable circuits to operate in the manner defined by instructions of the software or firmware. Fixed-function circuits may execute software instructions (e.g., to receive parameters or output parameters), but the types of operations that the fixed-function circuits perform are generally immutable. Accordingly, the terms “processor” and “processing circuitry,” as used herein may refer to any of the foregoing structures or any other structure suitable for implementation of the techniques described herein.

Various examples have been described. These and other examples are within the scope of the following claims.