Section: New Results

Physics-based Deep Neural Network for Augmented Reality

Participants : Jean-Nicolas Brunet, Andrea Mendizabal, Antoine Petit, Nicolas Golse, Eric Vibert, Stéphane Cotin.

Figure 5. The U-Mesh framework allows for extremely fast simulations of soft tissues accounting for large non linear deformations.

We propose an approach combining a finite element method and a deep neural network to learn complex elastic deformations with the objective of providing augmented reality during hepatic surgery. Derived from the U-Net architecture, our network is built entirely from physics-based simulations of a preoperative segmentation of the organ (see figure 5). These simulations are performed using an immersed-boundary method, which offers several numerical and practical benefits, such as not requiring boundary-conforming volume elements. We perform a quantitative assessment of the method using synthetic and ex vivo patient data. Results show that the network is capable of solving the deformed state of the organ using only a sparse partial surface displacement data and achieve similar accuracy as a FEM solution, while being about 100x faster. When applied to an ex vivo liver example, we achieve the registration in only $3ms$ with a mean target registration error (TRE) of $2.9mm$. This results were presented at MICCAI 2019 [22].