Keywords
Computer Science and Digital Science
- A2.5. Software engineering
- A3.1.1. Modeling, representation
- A3.1.4. Uncertain data
- A3.2.2. Knowledge extraction, cleaning
- A5.1. Human-Computer Interaction
- A5.3.4. Registration
- A5.4.4. 3D and spatio-temporal reconstruction
- A5.4.5. Object tracking and motion analysis
- A5.6. Virtual reality, augmented reality
- A6.1.1. Continuous Modeling (PDE, ODE)
- A6.1.2. Stochastic Modeling
- A6.1.5. Multiphysics modeling
- A6.2.3. Probabilistic methods
- A6.2.4. Statistical methods
- A6.2.5. Numerical Linear Algebra
- A6.2.8. Computational geometry and meshes
- A6.3.1. Inverse problems
- A6.3.2. Data assimilation
- A6.3.3. Data processing
- A6.3.4. Model reduction
- A9.2. Machine learning
- A9.10. Hybrid approaches for AI
Other Research Topics and Application Domains
- B1.2. Neuroscience and cognitive science
- B2.2.6. Neurodegenerative diseases
- B2.4. Therapies
- B2.4.3. Surgery
- B2.6. Biological and medical imaging
- B2.7. Medical devices
- B2.7.1. Surgical devices
1 Team members, visitors, external collaborators
Research Scientists
- Stephane Cotin [Team leader, INRIA, Senior Researcher, HDR]
- Hadrien Courtecuisse [CNRS, Researcher, HDR]
- Michel Duprez [INRIA, Researcher]
- Axel Hutt [INRIA, Senior Researcher, HDR]
PhD Students
- Paul Baksic [UNIV STRASBOURG, from Sep 2022]
- Sidaty El Hadramy [IHU STRASBOURG, from Mar 2022]
- Thuc Long Ha [UNIV STRASBOURG]
- François Lecomte [INRIA]
- Claire Martin [INRIA, from Oct 2022]
- Alban Odot [INRIA]
- Josephine Riedinger [INRIA]
- Valentina Scarponi [UNIV STRASBOURG]
- Killian Vuillemot [UNIV MONTPELLIER, from Oct 2022]
- Thomas Wahl [INRIA]
- Ziqiu Zeng [UNIV STRASBOURG]
- Nicola Zotto [UNIV STRASBOURG]
Technical Staff
- Paul Baksic [UNIV STRASBOURG, Engineer, from Jul 2022]
- Robin Enjalbert [INRIA, Engineer]
- Vincent Italiano [INRIA, Engineer, from Dec 2022]
Interns and Apprentices
- Chiara Cignolini [UNIV STRASBOURG, from Oct 2022]
Administrative Assistant
- Ouiza Herbi [INRIA]
Visiting Scientist
- Vanessa Lleras [UNIV MONTPELLIER, from Sep 2022]
External Collaborator
- Nazim Haouchine [BWH]
2 Overall objectives
2.1 Team Overview
The MIMESIS team develops numerical methods for computer-based training, surgical planning and computer-assisted interventions. The underlying objectives include patient-specific biophysical and electrophysiological modeling, novel numerical methods for real-time computation, data assimilation using Bayesian methods and more generally data-driven simulation. The aim is to significantly facilitate the development of digital twins and improve their predictive capabilities. This last topic is a transverse research theme which raises several challenges, related to the field of machine learning. To pursue these directions we have assembled a team with a multidisciplinary background, and have established close collaborations with academic and clinical partners. We also continue the development of the SOFA framework as a means to disseminate our results to the community.
2.2 Challenges
In a first research axis, our research aims at developing more advanced simulations, with stronger mathematical and biomechanical foundations, and their adaptation to a specific patient. We want, in particular, to develop more stable numerical methods that would, at the same time, be suited for the automatic generation of digital twins of organs. We chose to investigate immersed boundary methods that do not require an exact discretization of the domain. It led to the development of -FEM, a finite element method on domains defined by level-sets that provides better convergence while significantly simplifying the mesh generation process. In addition, we decided to investigate how Bayesian data assimilation methods can help learn patient-specific model parameters from sparse observations, as they typically occur during surgical interventions. Random processes play an important role in biological systems and need specific mathematical analysis. In neuroscience, stochastic fluctuations have been shown to control oscillatory brain states that may be induced by neurostimulation in clinical practice. To improve our understanding of nonlinear neural dynamics during neurostimulation and thus improve experimental neurostimulation protocols in single patients, we have extended the mathematical analysis of additive noise-induced system evolution. To achieve real-time computation using complex models based on nonlinear PDEs, we chose to investigate the use of deep learning techniques for accelerating physics-based simulations at a time when nearly nobody was looking into this. Given that our main field of application is computer-guided surgery, we also investigated the numerical aspects of contact problems. Such problems are very hard to solve in real time, as they involve non-smooth dynamics. Yet they are essential in many applications, as we will see below.
Our second research axis is derived from our application context and essentially consists of developing optimization and control methods for computer-assisted interventions. At the core of our activity is the hypothesis that data-driven simulation has the potential to bridge the gap between medical data (most often images) and clinical routine by updating pre-operative knowledge with the information available at the time of the procedure. We have been pioneers in the use of biomechanical models and real-time finite element simulation to perform augmented reality on deformable organs. In the field of non-rigid registration, we have demonstrated the benefit of our physics-based approaches as they bring not only an implicit regularization of the solution but also a plausible explanation of the deformation. We have continued this work with an emphasis on robustness to uncertainty and outliers in the information extracted in real-time from image data, and in real-time parameter estimation. This was done by combining Bayesian methods with physics-based algorithms to handle uncertainties in data-driven simulations. However, Bayesian or similar methods require performing a large set of simulations to sample the domain space even when using efficient methods such as a Reduced Order Unscented Kalman Filter. New directions of research are currently underway to address this challenge. We recently demonstrated the benefit of optimal control methods to perform non-rigid registration while ensuring a physical meaning of the result. Optimal control may be achieved in a closed feedback loop and we investigate its implementation in a neurostimulation setup. The objective of this loop is to control the spectral power density of a subject's brain activity. In a Kalman observer feedback model, brain observations are fed into a Kalman Filter which estimates instantaneous stimulation input to the brain in real-time. Another approach treats the brain as a linear time-invariant system and optimizes the brain stimulation input by direct feedback control based on observed brain signals. In the field of robotic control of flexible structures, such as needles during percutaneous procedures or catheters and guidewires during endovascular interventions, we are combining our expertise in real-time simulation of flexible devices, and soft tissues with efficient methods of contact and friction also developed in the team. These methods are based on the projected Gauss-Seidel algorithm and its variants, such as the non-smooth nonlinear conjugate gradient. They use fast linear solvers combined with CPU-GPU asynchronous preconditioners and exploit the sparsity of the system matrix to optimize the computation of the Schur-complement. We also recently started the development of reinforcement learning methods and differentiable simulations to address the challenge of autonomous robots.
3 Research program
3.1 Interactive computational models
3.1.1 Immersed Boundary Methods
For several years, we have investigated finite element methods that fall under the class of unfitted (also known as immersed boundary) methods. Because such methods do not require a discretization that strictly conforms to the domain boundary, they are particularly suited for the development of digital twins, as they facilitate the automatic generation of patient-specific simulations on complex geometries. Our current results have focused on the development, the numerical analysis and mathematical foundations of a new method called -FEM. The main advantage of our method is that it uses the classical Finite Element tools on unfitted meshes. We have also highlighted that the method significantly improves the convergence when compared to a similar, fitted, discretization of the domain. The benefit and applicability of the -FEM method have already been demonstrated on different problems: heat transfer, crack, interface between two material, and fluid structure interaction. We are now working on extending the method to 3D (non-linear) elasticity with both Neumann and Dirichlet boundary conditions.
3.1.2 Scientific Machine Learning
Various dimensionality reduction techniques have been proposed to speed up the simulation of non-linear solids. These techniques remain essentially linear transformations that may, in some cases, be insufficient to capture correctly the nonlinearity that can be found in biological tissues. In 2018, we proposed to address these limitations by developing physics-based deep neural networks, a supervised version of the Physically Informed Neural Networks (PINNs) approach. As preliminary work, we developed a neural network that learns the relationship between an input force and the resulting displacement field via a U-Net architecture 3. The choice of this U-net architecture is not random, as it matches the model reduction process through a neural network (with the latent space representing the reduced model). The predictions were about 100 times faster than our reference FEM computation while providing sufficient accuracy for our applications. This work was followed by several new ideas and publications. In 27 we proposed to solve elastic registration problems for augmented surgery using the same network architecture but trained to learn a mapping between a partial, noisy point cloud and the deformed state of an elastic body. In 28, we added a physics-based loss to a fully connected network to improve its prediction accuracy and to make it “physics-aware”. We also enforced the robustness of the solution by integrating the prediction of the network within a classical Newton Raphson algorithm. Our next steps concern the extension of our methods to the case of dynamic simulation, and also the development of neural network architectures that provide more genericity in the solution space (by being somewhat invariant to the meshing of the domain for instance).
3.1.3 Non-Smooth Mechanics
Many clinical interventions rely on mechanical interactions between a device and the anatomy. Surgery and interventional radiology are the two main areas where these interactions are essential and generally complex. During this evaluation period we have continued our activity on constraint modeling and simulation of contacts with friction. The complexity of the problem often leads to unstable numerical approaches, inaccuracy of the solution and slow computation times. Our latest works address the issue of computational performance through an asynchronous Cholesky decomposition and by exploiting matrix sparsity and time-consistency 16.
3.1.4 Order in random systems by stochastic perturbations
The brain is a complex system with several spatial and temporal scales. The microscopic scales are rather unstructured in space and activity observations show random fluctuations, whereas upper hierarchical levels at the mesoscopic and macroscopic scales exhibit more regular dynamics. Some years ago, we found in theoretical studies that additive random input on the microscopic scale to random networks tunes the systems stability and may induce stability change. Such so-called bifurcations induce ordered structures, being in space or time or in both. For instance, additive noise may induce coherence resonance in Erdös-Rényie neural networks which exhibit an oscillatory spatially coherent state with frequency of 40Hz. Since this oscillatory state evolves in the gamma frequency range in which visual information is processed, this theoretical result indicates that the brain may tune internal random fluctuations to induce coherence resonance and hence process visual information.
3.2 Optimization and control for Computer-Aided-Interventions
3.2.1 Optimal control and differentiable simulation
We recently started to investigate optimal control as a solution to several challenges in the field of computer-aided intervention. The first problem we addressed was the elastic intraoperative registration of a preoperative liver model and, as an extension, force estimation during robotic surgery. Both are formulated as optimal control problems where the unknown is the surface force distribution applied onto the organ. The deformation is computed using an hyperelastic model, and the optimization problem is solved using an adjoint method. This approach provides greater control over the set of admissible forces and leads to physically-consistent displacement fields. As a byproduct, it permits the recovery of the forces that led to the deformation, and can provide visual haptic feedback in medical robotics. We also rely on optimal control in other problems addressed in the team, such as autonomous endovascular navigation, where we have a similar control challenge. In this case the deformable structure is the catheter or guidewire rather than an organ. Finally, to address the relatively slow computation times that are typical of such optimization methods, we have started the development of differentiable algorithms. They exploit both GPU acceleration and the automatic differentiation tools from frameworks such as PyTorch. This significantly speeds up the computation of the adjoint. This approach is combined with the use of deep neural networks to predict the solution of the forward elastic problem in just a few milliseconds.
3.2.2 Data Assimilation
An essential step in the development of digital twins is the parametrization of the underlying model. A possibility is to rely on Bayesian inference. Model parameters are estimated using a reduced-order unscented Kalman filter based on observations taken, for instance, from laparoscopic image streams. As an example, estimating boundary conditions rather than using generic data from the literature improves the accuracy of the simulation by 75% when compared to methods integrating a priori knowledge as boundary conditions. Our approach significantly improves the accuracy during registration problems involving sparse and noisy observations. We also demonstrated that our data assimilation method can be used for the reconstruction of the spatiotemporal profile of soft tissue temperature evolution during laser irradiation.
3.2.3 Closed loop control for neurostimulation
Today typical clinical neurostimulation, such as transcranial electric or magnetic stimulation, applies stimulation protocols with pre-defined parameters, such as stimulation intensity, duration and inter-stimulus interval. However, such patient-unspecific open-loop stimulation protocols may not be effective for all patients. We propose to develop a closed-loop neurostimulation protocol that estimates the optimal stimulation intensity in real-time adaptively from observed brain activity. The new stimulation protocol aims to tune the brain's spectral power distribution in a pre-defined way which has been determined in previous experimental open-loop neurostimulation studies. Two PhD projects address different approaches. In one project, observed brain activity is fed into a Kalman observer model (implemented as an Ensemble Kalman Filter (ETKF) which estimates the optimal stimulation and is applied to the brain instantaneously. The nonlinear Kalman observer model defines the target spectral power distribution. In a different project, the closed-loop technique assumes linear brain dynamics and applies conventional feedback control techniques taking into account feedback delay and model errors. In contrast to the first project, which is rather heuristic, this second project applies mathematical analysis techniques to develop the optimal stimulation.
3.2.4 Control in medical robotics
Standard robotic solutions address the deformation problem by extracting a set of features from live images (visual servoing) and adjusting the pose/motion of the robot locally to compensate for the deformations. Nevertheless, visual servoing raises several limitations, particularly for needle insertion applications: 1) intraoperative images usually offer poor visibility of anatomical structures and extracting meaningful information in real-time is challenging; 2) when large deformations occur, the control law of the robot can be significantly modified, which is extremely difficult to relate with image-based displacements; 3) traditional controllers do not exploit biomechanical models capable of predicting the deformation of organs in real-time, which is essential for some control. To address this problem, we proposed a numerical solution to solve inverse Finite Elements simulations (iFE) and derive robotic commands to steer a needle in soft tissues. The method combines a non-rigid registration process to keep the model aligned with the actual organ. Then, we derive robotic commands by computing, from the FE simulation, the Jacobian linking the displacement of the base of the needle (controlled by the robot) and the corresponding displacement of the needle tip inside the tissue. We showed that the Jacobian helps account for complex nonlinear and discontinuous interactions (such as friction and contacts) and allows adapting the robot's behavior faster than waiting for image-based corrections. We recently extended the method with a shared control strategy to increase the safety, stability, and accuracy and the developed solution's acceptance. The primary motivation is to leave potentially dangerous decisions and actions to the practitioner. Other controls of the needle, e.g. to compensate for breathing motion, are performed automatically.
4 Application domains
4.1 Computer-based surgical training
Virtual training helps medical students to get familiar with surgical procedures before manipulation of real patients. The development of simulation used for medical training usually requires important computational power, since realistic behaviors are key to deliver a high-fidelity experience to the trainee. Further, the quality of interaction with the simulator (usually via visual and haptic rendering) is also of paramount importance. All these constraints make the development of training systems time-consuming, thus limiting the deployment of virtual simulators in standard medical curriculum.
Our activity in this area is twofold: we continue the development of fast and accurate numerical solutions for computing the interactions between a medical device and anatomical structures, and we investigate means of assessing trainee's performance and ways of providing valuable feedback in the form of a virtual coach.
4.2 Pre-operative planning
Beyond training, clinicians ask for innovative tools that can assist them in the pre-operative planning of an intervention. Using the patient information acquired before the operation, physics-based simulations allow to simulate the effect of therapy with no risk to the patient. The clinicians can thus virtually assess different strategies and select the optimal procedure. Compared to a training simulation, a planning system requires a high accuracy to ensure reliability. Constrained by the time elapsed between the preoperative acquisition and the intervention, the computation must also be efficient.
4.3 Intra-operative guidance
Besides the surgery training and planning, another major need from clinicians is surgical guidance. While the clinician is performing the operation, a guidance system provides enriched visual feedback. This is especially useful with the emergence of minimally invasive surgery (MIS) where the visual information is often strongly limited. It can be used for example to avoid critical areas such as vessels or to highlight the position of a tumor during its resection. In the MIS technique, the clinician does not interact with organs directly as in the open surgery, but manipulates instruments inserted through trocars placed in small incisions in the wall of the abdominal cavity. The surgeon can observe these instruments on a display showing a video stream captured by an endoscopic camera inserted through the navel. The main advantage of the method resides in reducing pain and time recovery, in addition to reducing bleeding and risks of infection. However, from a surgical standpoint, the procedure is quite complex since the field of view is considerably reduced and the direct manipulation of organs is not possible.
4.4 Feedback control of neural activity
In the medical treatment of mental disorders, pharmacological medication plays an important role. However, it is well-known that patients develop resistencies against the medication after some time and hence alternative treatment is mandatory. Besides psycho-cognitive treatment, neurostimulation plays a more and more important role. Today typical clinical neurostimulation, such as transcranial electric or magnetic stimulation, applies stimulation protocols with pre-defined parameters, such as stimulation intensity, duration and inter-stimulus interval. However, such patient-unspecific open-loop stimulation protocols may not be effective for all patients. Our research aims to develop a closed-loop neurostimulation protocol that estimates the optimal stimulation intensity in real-time adaptively based on observed brain activity. In this context, we work together with the laboratory INSERM 1114, the ICube team IMIS
4.5 Open source software
For many years, members of the team have considered essential to disseminate our research results (and the algorithms to produce them) in an open manner. The objective was to also develop a framework that could be used internally as a means to integrate our various contributions and facilitate validation and technology transfer. Many of our research results have since been released to the community as open source code, either through improvements of SOFA or as plugins of the framework.
SOFA is an efficient and accurate simulation framework written in C++, developed by our team and researchers from a couple of other Inria teams. It is one of the few open source frameworks for the simulation of heterogeneous mechanical systems. It is particularly aimed at real-time applications involving contacts. It is also often aimed at medical applications. Thanks to its high level of modularity, SOFA allows users to rapidly create complex simulations based on the large set of algorithms ranging from collision detection methods to volume rendering to finite element methods. It is structured around an open source core and many plugins, often open source as well. More than 30 researchers, students or engineers have contributed at various degrees to SOFA, for a total of about 1,500,000 lines of code. Today, the project has gone far beyond French borders.
5 Highlights of the year
5.1 Awards
We initiated a new startup project in 2021, named "Twinical", which aim is to develop digital twins for assisting the clinician in the planning and interventional stages. Twinical will develop medical software to optimize the surgical journey of patients diagnosed with liver cancer. The software provides surgeons with digital assistance both during the planning and execution phases of the surgery. During the intervention, the surgeon’s vision is digitally augmented thanks to the accurate, real-time, AI-powered simulation based on digital twins. This augmented reality provides important insights on the current anatomical structure of the liver and the tumor location, which is variable due to the deformations occurring during the tissue manipulation.
This project was awarded the “Grand Prix” from the 24th national innovation competition in June 2022.
5.2 Best presentation award
Our paper "SOniCS: Interfacing SOFA and FEniCS for advanced constitutive models" presented in August at the FEniCS conference received the best presentation award.
The goal of this work was to incorporate state-of-the-art code generation tools from the FEniCS Project into SOFA in order to increase SOFA’s capabilities in terms of soft tissue mechanics. To this end we have developed a new SOFA plugin named SOniCS. For adding a new material model in SOniCS, the user describes its strain energy density function using UFL (Unified Form Language) syntax. Then, using FFCx (FEniCSx Form Compiler) we generate the C code associated with the kernels corresponding to the automatically differentiated cell-local residual and stiffness forms. Finally, we assemble these kernels in SOFA into global tensors and solve the resulting non-linear system of equations. This makes it possible to straightforwardly implement complex material models such as the Holzapfel-Ogden anisotropic model into SOFA, and to use them alongside SOFA’s existing strong feature set in medical simulation.
6 New software and platforms
6.1 New software
6.1.1 DeepPhysX
-
Name:
DeepPhysX: interfacing AI with multi-physics simulation
-
Keywords:
Numerical simulations, Deep learning, Neural networks, Python
-
Functional Description:
The purpose of DeepPhysX framework is to provide an interface between deep learning algorithms and numerical simulations. It is a full Python project with two main pipelines, allowing both to train artificial neural networks with simulated data and to use trained neural networks as components of numerical simulations. DeepPhysX manages not only the production of synthetic data with multiple numerical simulations in multiprocessing but also the storage of the produced dataset. Additional tools are provided to visualize numerical simulations and to follow the evolution of training sessions.
-
News of the Year:
The release of the DeepPhysX project was done in May 2022, coming with examples, tutorials and a dedicated documentation. A proof of concept was implemented to handle data as a SQL database.
- URL:
-
Authors:
Robin Enjalbert, Alban Odot, Stephane Cotin
-
Contact:
Robin Enjalbert
7 New results
7.1 Digital twins of organs
Participants: Michel Duprez, Vanessa Lleras, Stéphane Cotin.
For several years, we have investigated finite element methods that falls under the class of unfitted (also known as immersed boundary) methods. Because such methods do not require a discretization that strictly conforms to the domain boundary, they are particularly suited for the development of digital twins, as they facilitate the automatic generation of patient-specific simulations on complex geometries. Our current results have focused on the development, the numerical analysis and mathematical foundations of a new method called -FEM 1. The main advantage of our method is that it uses the classical Finite Element tools on unfitted meshes. We have also highlighted that the method significantly improves the convergence when compared to a similar, fitted, discretization of the domain. The benefit and applicability of the -FEM method have already been demonstrated on different problems: heat transfer, crack, interface between two material, and fluid structure interaction 20, 25, 24. Our current activity on -FEM consists in an Open Source implementation in a SOFA plugin. This is done in relationship with another development activity that integrates automatic differentiation tools from FEniCS to quickly and efficiently add new constitutive models in our code base, through simple Python scripting. We are convinced of the impact of this work for both our research activity and the field in general. It also ties nicely with our work on deep learning for physics, as it offers an elegant alternative to graph networks to perform convolutions on unstructured meshes.7.2 Non-smooth mechanics
Participants: Ziqui Zeng, Hadrien Courtecuisse, Stéphane Cotin.
Non-smooth mechanics: many clinical interventions rely on mechanical interactions between a device and the anatomy. Surgery and interventional radiology are the two main areas where these interactions are essential and generally complex. During this evaluation period we have continued our activity on constraint modeling and simulation of contacts with friction. The complexity of the problem often leads to unstable numerical approaches, inaccuracy of the solution and slow computation times. Our latest work relies on the idea of solving the constrained problem as a Linear Complementarity Problem (LCP) through an iterative solver (Gauss Seidel in our case) and addresses the issue of computational performance. We use a preconditioner-based contact method relying on an asynchronous Cholesky decomposition. By exploiting the sparsity in assembled matrices, we propose a reduced and parallel computation scheme to address the expensive computation of the Schur complement. An additional numerical scheme that exploits the time consistency of contacts across multiple time steps makes it possible to obtain real-time simulations in the presence of hundreds of contacts 16. Our results have direct applications in CAI problems such as robotic control of needle insertion or endovascular navigation, but also in surgical training. They are also very relevant in Computer Graphics.7.3 Order in random systems by stochastic perturbations
Participants: Joséphine Riedinger, Thomas Wahl, Axel Hutt.
The brain is a complex system with several spatial and temporal scales. The microscopic scales are rather unstructured in space and activity observations show random fluctuations, whereas upper hierarchical levels at the mesoscopic and macroscopic scales exhibit more regular dynamics. In previous studies, we found in theoretical studies that additive random input on the microscopic scale to random networks tunes the systems stability and may induce stability changes 13. Such so-called bifurcations induce ordered structures, being in space or time or in both. In recent years, this additive noise-induced system evolution (ANISE) 11 has been shown to describe successfully the increase of the mammalian brains excitability observed under transcranial Direct Current Stimulation 14, 26, 4. ANISE also demonstrates how to bridge modeling scales by machine learning-based subgrid scale modeling, see also 12.7.4 Optimal control to limit epidemia
Participants: Michel Duprez.
There exists no efficient vaccin against arbovirus like Malaria, Dengue, Lymphatic Filariasis, Zika, Chikungunya, Yellow fever, and Japanese encephalitis. One way to limit the propagation of such diseases is to fight their vector which is the mosquito. The sterile insect technique (SIT) is a biological control technique that can be used either to eliminate or decay a wild mosquito population under a given threshold to reduce the nuisance or the epidemiological risk. In 6 we have studied the mathematical properties of the optimal release strategies in order to maximize the efficiency of this technique. In 22 we propose a model using a differential system that takes into account the variations of rainfall and temperature over time and study their impacts on sterile males releases strategies.7.5 Optimal control of computer-aided interventions
Participants: Stéphane Cotin, Guillaume Mestdagh.
We recently started to investigate optimal control as a solution to several challenges in the field of computer-aided intervention. The first problem we addressed was the elastic intraoperative registration of a preoperative liver model and, as an extension, force estimation during robotic surgery. Both are formulated as optimal control problems where the unknown is the surface force distribution applied onto the organ. The deformation is computed using an hyperelastic model, and the optimization problem is solved using an adjoint method. This approach provides greater control over the set of admissible forces and leads to physically-consistent displacement fields. As a byproduct, it permits the recovery of the forces that led to the deformation, and can provide visual haptic feedback in medical robotics 17, 21. We also rely on optimal control in other problems addressed in the team, such as autonomous endovascular navigation, where we have a similar control challenge. In this case the deformable structure is the catheter or guidewire rather than an organ. Finally, to address the relatively slow computation times that are typical of such optimization methods, we have started the development of differentiable algorithms. They exploit both GPU acceleration and the automatic differentiation tools from frameworks such as PyTorch. This significantly speeds up the computation of the adjoint. This approach is combined with the use of deep neural networks to predict the solution of the forward elastic problem in just a few milliseconds.7.6 Data Assimilation
Participants: Nava Schulmann, Stéphane Cotin.
Several research groups have proposed patient-specific liver simulations, yet very few have addressed the question of boundary conditions. Resulting mainly from ligaments attached to the liver, they are not visible in preoperative images, but play a key role in the computation of the deformation. We proposed to estimate both the location and stiffness of ligaments by using a combination of a statistical atlas, numerical simulation, and Bayesian inference. Model parameters are estimated using a reduced-order unscented Kalman filter based on observations taken from the laparoscopic image stream. Results show that our estimation of the boundary conditions improves the accuracy of the simulation by when compared to methods integrating a priori knowledge as boundary conditions. Our approach significantly improves the accuracy of the intraoperative liver registration. It also inherently handles noisy observations, a key feature in the context of augmented reality. We also demonstrated that our data assimilation method can be used for the reconstruction of the spatiotemporal profile of soft tissue temperature evolution during laser irradiation. The predictions of a physical model simulating the heat transfer in the tissue are associated with sparse temperature measurements, using an Unscented Kalman Filter. Using a joint state and parameter estimation approach, the filter corrects both the prediction of the simulation and the uncertain model parameters 15. The joint estimation allows recovering the temperature distribution in a liver tissue sample while using only a few measurements. This work opens new perspectives on the benefit of data-assimilation frameworks for laser therapy monitoring and our research continues in the context of the ERC LaserOptimal with Politecnico di Milano.7.7 Digital Addiction
Participants: Axel Hutt.
In 2020, the World Health Organization formally recognized addiction to digital technology (connected devices) as a worldwide problem, where excessive online activity and internet use lead to inability to manage time, energy, and attention during daytime and produce disturbed sleep patterns or insomnia during nighttime. The extent to which dysfunctional sleep is a consequence of altered motivation, memory function, mood, diet, and other lifestyle variables or results from excess of blue-light exposure when looking at digital device screens for long hours at day and night is one of many still unresolved questions. Our work 7 offers a narrative overview of some of the most recent literature on this topic. The analysis provides a conceptual basis for understanding digital addiction as one of the major reasons why people, and adolescents in particular, sleep less and less well in the digital age. It discusses definitions as well as mechanistic model accounts in context. Digital addiction is identified as functionally equivalent to all addictions, characterized by the compulsive, habitual, and uncontrolled use of digital devices and an excessively repeated engagement in a particular online behavior. Once the urge to be online has become uncontrollable, it is always accompanied by severe sleep loss, emotional distress, depression, and memory dysfunction. In extreme cases, it may lead to suicide. The syndrome has been linked to the known chronic effects of all drugs, producing disturbances in cellular and molecular mechanisms of the GABAergic and glutamatergic neurotransmitter systems. Dopamine and serotonin synaptic plasticity, essential for impulse control, memory, and sleep function, are measurably altered. The full spectrum of behavioral symptoms in digital addicts include eating disorders and withdrawal from outdoor and social life. Evidence pointing towards dysfunctional melatonin and vitamin D metabolism in digital addicts should be taken into account for carving out perspectives for treatment.8 Partnerships and cooperations
8.1 International initiatives
8.1.1 Participation in other International Programs
- Title: Real-World Human Cognition by Care Robots
-
Partner Institutions:
- MLMS / ICube, Strasbourg / France
- MIMESIS / INRIA, Strasbourg / France
- Coordinator: ETRI
- Date/Duration: 2021-2024
-
Additionnal info/keywords:
In this project, we propose a model-based learning framework to push the current limits of robot vision in human cognition in the wild, with specific objectives that can significantly contribute to realistic 4D human models and real-world robot vision. To achieve this goal we propose to develop a photo-realistic physics-based 4D human model, in a differentiable simulation framework.
8.2 International research visitors
8.2.1 Visits of international scientists
Other international visits to the team
Participants: Nora Hagmeyer.
-
Status:
PhD student
- Institute for Mathematics and Computer-Based Simulation, University of the Bundeswehr, Munich.
-
Dates:
October 10th – December 21st, 2022
-
Context of the visit:
The objective of this visit was to investigate the design of patient-specific devices (stents and angioplasty ballons) for endovascular treatments through the combined use of optimization techniques and numerical simulation of the interactions between the device and the anatomy.
8.3 European initiatives
8.3.1 H2020 projects
ERC LaserOptimal
-
Title:
Integrated solutions to achieve effective and selective laser ablations
-
Duration:
2018 - 2023
-
Partners:
- MIMESIS
- IHU Strasbourg
-
Coordinator:
Politecnico di Milano
-
Inria contact:
stephane.cotin@inria.fr
-
Summary:
The objective of this project is to develop integrated solutions to achieve effective and selective laser ablations. These procedures use a localized heat source to destroy tumors while spearing healthy surrounding tissues. This objective will be achieved by: a) enhancing the selective absorption of laser light through biocompatible nanoparticles; b) developing a digital twin of the patient’s anatomy and combining it with data assimilation techniques to visualize the treatment dynamics and control the laser settings in a closed-loop manner.
DRIVEN
-
Title:
Increasing the scientific excellence and innovation capacity in Data-Driven Simulation of the University of Luxembourg
-
Duration:
2018 - 2022
-
Partners:
- Inria, France
- University of Limerick, Ireland
- Luxembourg University, Luxembourg
- University of Texas, USA
-
Inria contact:
Sylvain Petitjean
-
Coordinator:
Stephane Bordas (Univ Luxembourg)
-
Summary:
DRIVEN is a project coordinated by the University of Luxembourg and aiming at boosting the scientific excellence and tech transfer capacity in data-driven simulation of the University of Luxembourg. Funded by the EU’s Horizon 2020 programme for research and innovation, the University of Luxembourg will benefit from top international partners: Inria (France), University of Limerick (Ireland) and University of Texas at Austin (USA).
8.4 National initiatives
ADT - DeepPhysX: Data-driven simulation
-
Title:
DeepPhysX
-
Duration:
2021 - 2022
-
Coordinator:
Stéphane Cotin
-
Inria contact:
stephane.cotin@inria.fr
-
Summary:
This project aims to develop new tools for real-time navigation and registration in image-guided surgery. It aims to integrate learning methods with numerical simulations in order to obtain robust, real-time predictions adapted to the patient. Robustness will be enforced through hybrid methods, such as a new Newton algorithm combining AI-based prediction and “classical” numerical steps. The development of this framework supports several Ph.D. projects, for both direct and inverse problems.
ADT - AI for Surgical Vision
-
Title:
AI for Surgical Vision
-
Duration:
2021 - 2024
-
Coordinator:
Stéphane Cotin
-
Partners:
- BOPA innovation chair
- Paul Brousse Hospital
-
Inria contact:
stephane.cotin@inria.fr
-
Summary:
The objective of the project is to develop and integrate computer vision algorithms capable of processing images from the operating room in real time into a clinically usable augmented reality prototype. This project reinforces the work carried out in partnership with the "Augmented Operating Room" (BOPA) innovation chair at Hospital Paul Brousse in Paris. The project is part of the Blok-Viz axis whose goal is to process video feeds of the operating field to extract relevant information on which AI algorithms dedicated to attention analysis, registration and augmented reality can be based.
ANR - SPERRY
-
Title:
SuPervisEd Robotic suRgerY - application to needle insertion.
-
Duration:
2018 - 2022
-
Coordinator:
Hadrien Courtecuisse (MIMESIS)
-
Partners:
- Inria Strasbourg / Nancy – Grand Est, MIMESIS team (France)
- ICube Laboratoire des sciences de l'Ingénieur, de l'Informatique et de l'Imagerie (France)
-
Inria contact:
hadrien.courtecuisse@inria.fr
-
Summary:
In this project, we proposed to develop new solutions for the control of medical robots performing percutaneous procedures. Using our expertise in real-time simulation and contact modeling, we have developed methods able to control the trajectory of a flexible needle such that it follows a predefined path even when the organ is moving or deforming. A working prototype has been developed to illustrate these results.
ANR - SPECULAR
-
Title:
Simulation of needle insertion with virtual reality and haptics.
-
Duration:
2021 - 2025
-
Coordinator:
Stephane Cotin (MIMESIS)
-
Partners:
- Inria Strasbourg / Nancy – Grand Est, MIMESIS team (France)
- Inria Lille - Nord Europe, DEFROST team (France)
- InfinityTech 3D (France)
-
Inria contact:
stephane.cotin@inria.fr
-
Summary:
The objective of this project is to develop a complete virtual reality training system for radiofrequency ablation. The research program includes the real-time simulation of the needle-organ interactions, realistic and immersive rendering of the operating room, medical image generation and haptic feedback. The results of this project will accelerate the training of these procedures and could change the standard of care which remains a surgery in many cases.
ANR - VATSOP
-
Title:
Images and models for computer guidance during Video Assisted Thoracic Surgery (VATS).
-
Duration:
2021 - 2025
-
Coordinator:
Jean-Louis Dillenseger (LTIS Rennes)
-
Partners:
- TIMC-IMAG Laboratory, Grenoble (France)
- Inria Strasbourg / Nancy – Grand Est, MIMESIS team (France)
- LTSI Laboratoire Traitement du Signal et de l'Image (Rennes, France)
-
Inria contact:
stephane.cotin@inria.fr
-
Summary:
Video-Assisted Thoracic Surgery (VATS) is a minimally-invasive thoracic procedure that has become prominent for the treatment of lung cancer. However, localizing the nodules during VATS is extremely challenging due to the technical difficulties inherent to endoscopic surgery, and due to the collapse of the lung during the procedure. The main objective of our project is to develop and integrate, in an operating room, an image-based guidance solution which would: 1) estimate the position of the patient’s nodule after pneumothorax from CBCT images; 2) propose an augmented reality-based guidance in the endoscopic view; 3) allow to follow this location in real-time on the endoscopic view, during manipulation of the lung. Partners of the project are TIMC Grenoble and LTSI Rennes.
ANR - TRECOS
-
Title:
New Trends in Control and Stabilization: Constraints and a Non-local Terms.
-
Duration:
2020 - 2024
-
Coordinator:
Sylvain Ervedoza (Institut de Mathématiques de Bordeaux, Laboratoire Traitement du Signal et de l'Image)
-
Partners:
- IMT Institut de Mathématiques de Toulouse (France)
- Inria Strasbourg / Nancy – Grand Est, MIMESIS team (France)
- IMB Institut de mathématiques de Bordeaux (France)
- LJLL Laboratoire Jacques-Louis Lions, Paris (France)
-
Inria contact:
michel.duprez@inria.fr
-
Summary:
The goal of this project is to develop new solutions in control theory for partial differential equations, motivated by models arising in ecology and biology. The project focuses on two aspects. The first one is related to the constraints required on the controls or on the controlled trajectories, for instance positivity constraints, which appears naturally when the state models a temperature. The second aspect concerns the questions of controllability and stabilization d of problems involving non-local operators, such as integral operators in space, in order to take into account phenomena depending on the total mass of the population for instance, or delay and memory terms, such as in visco-elastic fluids, often used to a model blood flows, or more generally models described by systems coupling hyperbolic and parabolic effects.
ANR - S-KELOID
-
Title:
Understanding Keloid Disorders: A multi-scale in vitro/in vivo/in silico approach towards digital twins of skin organoids on the chip.
-
Duration:
2021 - 2025
-
Coordinator:
Eftimie RALUCA and Stéphane BORDAS (Univ. Luxembourg)
-
Partners:
- Laboratoire de Mathématiques de Besançon (France)
- CHU de Besançon (France)
- FEMTO-ST, Besançon (France)
- Institut Mathématiques de Bourgogne, Dijon (France)
- Université du Luxembourg (Luxembourg)
- Inria Strasbourg / Nancy – Grand Est, MIMESIS team (France)
-
Inria contact:
michel.duprez@inria.fr
-
Summary:
Mathematical and numerical modeling approaches allow us to integrate pathological processes that occur across different scales: single cell, cell assembly and tissue. The S-Keloid project aims to investigate the role of mechanical and inflammatory environmental factors on cells associated with keloid disorders. From applying experimental tests at tissue-scale and using a multiscale approach, the mechanical stress fields will be integrated into the 3D mathematical model. Parameter identification, optimization and their use across multiple scales will ensure the realism of the models and the quantitative and qualitative predictions of the keloid disorder.
ANR - PhiFEM
-
Title:
φ-FEM : development of a Finite Element Method for the design of real-time digital twins in surgery
-
Duration:
2022 - 2026
-
Coordinator:
Michel Duprez (Inria)
-
Partners:
- MIMESIS team
- Laboratoire de Mathématiques de Besançon (France)
- Institut de Mathématiques Alexander Grothendieck, Montpellier (France)
- Institut de Recherche en Mathématiques Appliquées, Strasbourg (France)
- Univertsité du Luxembourg (Luxembourg)
-
Inria contact:
michel.duprez@inria.fr
-
Summary:
φ-FEM is a recently proposed finite element method for the efficient numerical solution of partial differential equations posed in domains of complex shapes, using simple regular meshes. The main goal of this project is to further develop φ-FEM turning it into a tool for efficient, patient-specific and real-time simulations of human organs. To reach this objective, we shall adapt φ-FEM to the equations appropriate to biomechanics, provide an efficient implementation for it allowing for the use of actual organ geometries, and finally combine it with convolution neural networks to make it real time after training. The ultimate, long-term, goal is thus to contribute to the construction of digital twins of organs able to guide the surgical act in real time using information acquired before the operation and to reduce the costs of a medical doctors’ training by working on visual organs. The innovation of φ-FEM lies in its ability to combine the ease of implementation of classical immersed boundary methods with the accuracy of more recent CutFEM/XFEM approaches. It incorporates, by its very construction, the popular description of geometry by Level Set functions, which can represent the real geometry with whatever accuracy desired which makes this approach numerically less expensive than classical finite element methods. The φ-FEM paradigm will also be used to develop efficient registration algorithms. Our results will be integrated into the open-source SOFA platform developed in the MIMESIS team to facilitate dissemination.
National Exploratory Action - A/D Drugs
-
Title:
A/D Drugs
-
Duration:
2020 - 2023
-
Coordinator:
Axel Hutt
-
Partners:
Didier Pinault (INSERM 1114)
-
Participants:
Axel Hutt, Joséphine Riedinger, Didier Pinault
-
Summary:
When it comes to treating mental disorders, the emergence of resistance to medication is a major problem. Replacing chemical medicine with digital medicine (neural stimulation) could be one way of getting around the problem. This is the concept behind "A/D Drugs" (PI : Axel Hutt) which will deploy a process of data assimilation and control in order to adapt stimulation to each patient. But this research is not without its risks – very little is known about the links between the effect of the chemical molecules and neurostimulation (www.inria.fr/en/ad-drugs-exploratory-action-aimed-optimising-neurostimulation).
8.5 Regional initiatives
HealthTech
-
Title:
Optimal auditory neurostimulation to alleviate visual deficits in Attention-Deficit/Hyperactivity Disorder (ADHD)
-
Duration:
2022 - 2023
-
Coordinator:
Axel Hutt
-
Partners:
Anne BONNEFOND (INSERM 1114)
-
Participants:
Gabriel ALVES CASTRO, Axel HUTT, Anne BONNEFOND
-
Summary:
Attention-deficit/hyperactivity disorder (ADHD) is a neuropsychiatric disorder and patients suffer from inattention, hyperactivity and impulsivity. Since pharmacological treatment may yield adverse cognitive side effects, non-pharmacological neurostimulation is suggested as an alternative. The project investigates the impact of auditory beat stimulation on visual attention in healthy subjects showing ADHD symptoms. The student will perform psycho- physical experiments in the presence of binaural beats or isochronic tones and will identify stimulation parameters that improve best visual attention in subjects.
9 Dissemination
9.1 Promoting scientific activities
9.1.1 Scientific events: organisation
Member of organizing committees
- Axel Hutt has been member of the Program Committee of the Organization for Computational Neuroscience (2021-2024).
- Members of the team participated to the organization of the online SOFA week (November 2022) with 230 registered participants and an average audience of 80+ people online throughout the day.
9.1.2 Scientific events: selection
Reviewer
- Axel Hutt has been reviewer of the Computational Neuroscience (CNS2022) conference.
9.1.3 Journal
Member of the editorial boards
- Axel Hutt is Chief Section Editor of Frontiers in Applied Mathematics and Statistics - Dynamical Systems.
- Axel Hutt has been Guest Associate Editor in Frontiers in Systems Neuroscience and Frontiers in Applied Mathematics and Statistics.
Reviewer - reviewing activities
- Axel Hutt has been Review Editor in Frontiers in Human Neuroscience and Frontiers in Computational Neuroscience.
- Axel Hutt has been Reviewer of manuscripts submitted to European Journal of Physics, Frontiers in Applied Mathematics and Statistics, Frontiers in Neuroscience, Applied Sciences, Brain Sciences, Algorithms.
9.1.4 Invited talks
- Michel Duprez presented his research results at the CANUM conference (June 2022, Evians).
- Michel Duprez gave a talk at ECCOMAS (June 2022, Oslo).
- Michel Duprez presented his research during the "Winter CMS" meeting (December 2022, Toronto).
- Axel Hutt gave a talk at the DSAI iCube - workshop in Strasbourg, May 2022.
- Axel Hutt was invited to a multidisciplinary workshop Computational and mathematical approaches for neurosciences at the Paris Brain Institute (Pitié-Salpêtrière hospital, Paris), June 2022.
- Axel Hutt gave a talk at the Minisymposium Mathematical modeling of psychiatric disorders and addiction as part of the ECMTB22 conference in Heidelberg / Germany, September 2022.
- Axel Hutt gave a talk at Advanced tools for data analysis in neuroscience in Strasbourg, September 2022.
- Axel Hutt was invited to the workshop Cortical and brain modeling to understand EEG and fMRI of the University of Michigan / USA, December 2022.
- Stéphane Cotin gave a talk during the BEST symposium, Strasbourg / France, August 2022.
- Stéphane Cotin gave an invited talk during the SMIT Conference, Oslo / Norway, May 2022.
9.1.5 Leadership within the scientific community
- Axel Hutt has co-organized the Frontiers Research Topics Modelling collective motion across scales published in Frontiers in Applied Mathematics and Statistics.
- Axel Hutt has organized the Frontiers Research Topics Insights in Dynamical Systems to be published in Frontiers in Applied Mathematics and Statistics.
- Axel Hutt has co-organized the Frontiers Research Topics Neuromodulation by Digital and Analogue Drugs in Consciousness Research to be published in Frontiers in Systems Neuroscience.
9.1.6 Scientific expertise
Axel Hutt was invited expert reviewer for two proposals in the grant program Equipe action (2021-2024) of the LabEx Persyval-Lab.
9.1.7 Research administration
Axel Hutt has co-organized the Summer School Advanced tools for data analysis in neuroscience in Strasbourg, September 2022.
9.2 Teaching - Supervision - Juries
9.2.1 Teaching
- License: Michel Duprez, Numerical analysis techniques, 17h+15h, L3, University of Strasbourg.
- Master: Michel Duprez, Incertitude quantification, 14h, M2, University of Strasbourg.
- License: Axel Hutt, Anglais, 2h, L3, University of Strasbourg.
9.2.2 Supervision
- Axel Hutt has supervised the Master II - student Gabriel Alves Castro. Title: Binaural beats improve visual attention.
- Axel Hutt supervises the PhD student Joséphine Riedinger (2020-2023). Title: Closed-loop transcranial electric stimulation of neural networks in a rat psychosis model.
- Axel Hutt supervises (with Michel Duprez) the PhD student Thomas Wahl (2021-2024). Title: Model-based closed-loop neurostimulation with application to schizophrenia.
- Hadrien Courtecuisse has supervised the PhD student Paul Baksic (defence in June 2022). Title: Supervised Robotic surgery, computer simulation for needle insertion.
- Hadrien Courtecuisse supervises the PhD student Ziqiu Zeng (2019-2023). Title: Real‐Time FE Simulation for Large‐Scale Problems Using Precondition‐Based Contact Resolution and Isolated DOFs Constraints.
- Michel Duprez supervises the PhD students Killian Vuillemot (October 2022-September 2025) together with Vanessa Lleras and Mohammadi Bijan. Title: Unfitted finite element method for the development of organ digital twins.
- Michel Duprez supervises the PhD student Valentina Scarponi (2021-2024) together with Florent Nageotte and Stéphane Cotin. Title: Autonomous Catheter Navigation.
- Stephane Cotin supervises the PhD student Sidaty El Hadramy (2022-2025) together with Benoit Gallix (ICube). Title: AI-enabled Intraoperative planning for liver surgery using ultrasound imaging.
- Stephane Cotin supervises the PhD student Alban Odot (2020-2023). Title: Data-driven computational biomechanics using deep neural networks.
- Stephane Cotin supervises the PhD student Nicola Zotto (2022-2025) together with Benoit Gallix (ICube). Title: Combining AI and biomechanics for computer-assisted interventions.
- Stephane Cotin supervises the PhD student Francois Lecomte (2021-2024) together with Jean-Louis Dillenseger (Univ Rennes). Title: Physics informed neural networks for organ shape prediction from interventional images. Application to Augmented Lung Surgery.
9.2.3 Juries
- Axel Hutt was Reviewer in PhD thesis jury of Rick Evertz (The relationship between alpha (8-13 Hz) and 1/fβ activity in spontaneous EEG during rest and dissociative gaseous anaesthesia), University of Swinburne / Australia, April 2022
- Stéphane Cotin was member of the PhD thesis jury of Saima Safdar (SlicerCBM: Framework for Seamless Solution of Partial Differential Equations and Visualisation of Results on Medical Images), University of Western Australia / Australia, December 2022
- Stéphane Cotin was member of the PhD thesis jury of Arnaud Mazier (Data-driven patient-specific breast modeling: a simple, automatized and robust computational pipeline), University of Luxembourg / Luxembourg, September 2022
- Stéphane Cotin was member of the PhD thesis jury of Guillaume Mestdagh (An optimal control formulation for organ registration in augmented surgery), University of Strasbourg / France, December 2022
- Stéphane Cotin was member of the PhD thesis jury of Paul Baksic (Commande robotique basée simulation pour l’assistance à la radiologie interventionnelle), University of Strasbourg, June 2022
- Axel Hutt was Reviewer for a national grant proposal for National Research and Development Agency (ANID) of Chile, January 2022
9.2.4 Interventions
Stéphane Cotin participated in two round tables during SantExpo 2022, a large audience healthcare forum (about 20,000 visitors).
10 Scientific production
10.1 Major publications
-
1
articleA new
-FEM approach for problems with natural boundary conditions.Numerical Methods for Partial Differential Equations2022 - 2 thesisAnatomical, Hemodynamic and Physiological Modeling in Liver Surgery.Université Paris-SaclaySeptember 2021
- 3 articleSimulation of hyperelastic materials in real-time using deep learning.Medical Image Analysis592019, 101569
- 4 articleMathematical Model Insights into EEG Origin under Transcranial Direct Current Stimulation (tDCS) in the Context of Psychosis.Journal of Clinical Medicine1172022, 1845
- 5 articleReal‐Time FE Simulation for Large‐Scale Problems Using Precondition‐Based Contact Resolution and Isolated DOFs Constraints.Computer Graphics Forum416June 2022, 418-434
10.2 Publications of the year
International journals
International peer-reviewed conferences
Scientific book chapters
Doctoral dissertations and habilitation theses
Reports & preprints
10.3 Cited publications
- 27 articlePhysics-based Deep Neural Network for Augmented Reality during Liver Surgery.Medical image computing and computer-assisted intervention : MICCAI ... International Conference on Medical Image Computing and Computer-Assisted Intervention1-82019
- 28 articleDeepPhysics: A physics aware deep learning framework for real‐time simulation.International Journal for Numerical Methods in Engineering12302 2022