EN FR
EN FR
TANGRAM - 2023
TANGRAM - 2023

2023Activity reportProject-TeamTANGRAM

RNSR: 202023684L
  • Research center Inria Centre at Université de Lorraine
  • In partnership with:Université de Lorraine, CNRS
  • Visual Registration with Physically Coherent Models
  • In collaboration with:Laboratoire lorrain de recherche en informatique et ses applications (LORIA)
  • Domain:Perception, Cognition and Interaction
  • Theme:Vision, perception and multimedia interpretation

Keywords

Computer Science and Digital Science

  • A5.3. Image processing and analysis
  • A5.4. Computer vision
  • A5.4.1. Object recognition
  • A5.4.5. Object tracking and motion analysis
  • A5.4.6. Object localization
  • A5.6. Virtual reality, augmented reality
  • A5.10.2. Perception

Other Research Topics and Application Domains

  • B2.6. Biological and medical imaging
  • B5.9. Industrial maintenance
  • B9.5.3. Physics

1 Team members, visitors, external collaborators

Research Scientists

  • Marie-Odile Berger [Team leader, INRIA, Senior Researcher, HDR]
  • Erwan Kerrien [INRIA, Researcher, HDR]

Faculty Members

  • Fabien Pierre [UL, Associate Professor]
  • Gilles Simon [UL, Associate Professor, HDR]
  • Frédéric Sur [UL, Professor, HDR]
  • Pierre-Fredéric Villard [UL, Associate Professor]
  • Brigitte Wrobel-Dautcourt [UL, Associate Professor]

PhD Students

  • Youssef Assis [UL, until Nov 2023]
  • Nathan Boulangeot [UL]
  • Abdelkarim Elassam [INRIA]
  • Radhouane Jilani [INRIA]
  • Nariman Khaledian [INRIA]
  • Hugo Leblond [UL, from Oct 2023]
  • Liang Liao [CHRU NANCY]
  • Nicolas Maignan [UL]

Technical Staff

  • Youssef Assis [INRIA, Engineer, from Nov 2023]
  • Romain Boisseau [INRIA, Engineer, until Aug 2023]

Interns and Apprentices

  • Gaetano Agazzotti [UL, until Mar 2023]

Administrative Assistant

  • Véronique Constant [INRIA]

External Collaborators

  • Cédric Demonceaux [UNIV BOURGOGNE, from Mar 2023, HDR]
  • Renato Martins [UNIV BOURGOGNE, from Mar 2023]

2 Overall objectives

Visual registration is a research topic with a rich history in computer vision. Though a plethora of methods have been developed and can be used for general situations, there are still many open problems which originate in the nature of the scene (poorly textured or specular environments), in the type of motion undergone by the object (tiny motions which hardly emerge from the noise floor, or in contrast, highly deformable objects) and in dissimilarities which may occur in the scene between the time the modeling stage occurs and the application time.

Registration is in practice tightly linked to the choice of the model which represents the scene and the desirable physical properties of the objects. Handling complex —possibly dynamic— scenes thus requires a tradeoff between physical realism of the model, convergence issues and robustness of the registration or tracking tasks.

Recent years have seen a surge in research at the intersection of image and deep learning which has impacted many topics of computer vision. Besides our continued exploration of modeling and registration with traditional approaches derived from signal processing, geometry, and robust estimation, one of the team's aims is to integrate machine learning methods, either as end-to-end methods or as components, into these 2D or 3D geometric tasks.

Targeted trans-disciplinary applications are mixed and augmented reality, computational photomechanics and minimally invasive medical interventions.

3 Research program

3.1 Localization and geometric reasoning with high level features

Our goal is to push forward vision-based scene understanding and localization through the joint use of learning-based methods with geometrical reasoning. Our hypothesis is that the use of intermediate representations instead or in addition to the classical point feature will lead to increased capacity in terms of scale and robustness to changing conditions. These intermediate representations can be concrete objects which are recognized and used directly in the global pose computation, in the continuity of our works on ellipsoid modeling of objects, or conceptual objects such as vanishing points (VP) or horizon lines that are of specific interest both for localization and modeling of urban or industrial scenes.

A first goal is to improve our method for localization from sets of ellipse/ellipsoid correspondences 3. Besides the need to have more accurate prediction of ellipses, another objective is to elaborate robust strategies and associated numerical schemes for refining the initial pose from a set of objects. This requires us to develop appropriate metrics for characterizing good reprojection of 3D objects onto 2D ones and study their impact on minimization issues in localization. Another goal is to define strategies to integrate into the localization procedure various features such as points, objects and VPs, which each bring information at different levels. We especially want to investigate how predictive uncertainty and explainability mechanisms can be used to select and weight these various features in the estimation process.

3.2 Building dedicated models

In this line of research, our goal is to build physically coherent models with a good accuracy vs. efficiency compromise despite the interactive time constraint set in some targeted applications. Though general purpose solutions exist for building models, such techniques are still greatly challenged in more complex cases when specific constraints on the shape or its deformation must be met. This is especially the case in medical imaging of thin deformable organs, such as the diaphragm, the mitral valve or blood vessels, but also for classical scene modeling where constraints, such as ellipsoidal abstraction of objects, must be introduced. The use of mechanical models has become increasingly important in the team's activities in medical imaging, especially for handling organs with large deformations. We want to push forward the development of such models with image-guided procedures or predictive simulation in view.

Facing difficulties of meshing complex geometries, especially thin ones, we want to promote mesh free methods such as implicit models. In the continuity of past works 4, automatic adaptation of node locations and sizes to the image will be investigated to improve compactness, and computational efficiency of implicit models. As the fidelity of a mechanical model is often impaired by approximations required to solve its dynamical system equations at interactive frame rates, a second objective is to take advantage of our implicit models to improve contact and deformation resolution.

Another topic of interest is the investigation of shape-aware methods either for shape segmentation or shape recognition, in order to be able to enforce global shape constraints or geometric shape priors in the output of CNNs. This topic is still addressed in the team in the context of localization from 3D ellipsoidal abstraction of objects 10. Two applications are especially targeted: (i) We aim at improving the detection of pathologies (e.g. brain aneurysms that are mostly located at vessel bifurcations), through adapted and guided sampling of input data during training, as well as through mechanisms inspired by visual attention modules. (ii) In the context of fluid structure simulation methods for patient-based mitral valve simulation, it often appears that the geometric segmented model leads to divergence of the numerical scheme. Our intention is to identify geometric conditions under which simulation works well and to incorporate them in the segmentation process.

3.3 Estimation and inverse problems

Most aforementioned tasks lead to image-based inverse, possibly ill-posed, problems. While some of them can be solved with well-established estimation techniques, others necessitate the design of new strategies. In this perspective, we consider in this research axis several fundamental aspects of estimation, common to our problems, such as sampling methods, traditional optimization methods, or end-to-end learning methods for pose estimation.

3.3.1 Optimization, variational calculus and numerical schemes

We are interested in non-convex optimization problems, especially those raised by variational calculus. While the convergence of numerical schemes is well established for convex problems, this is not always the case for non-convex functionals. Our aim is to continue the work already carried out in the biconvex framework 8, and extend it to primal-dual algorithms. We especially want to address energy minimization problems where the energy is convex with respect to each variable, but non-convex with respect to the pair of variables.

Another research topic is to investigate new neural architectures adapted to non-Euclidean data, and also to plug variational methods into deep learning approaches to regularize the results. The obtained theoretical results will be applied to image colorization, with the idea to reduce artefacts caused both by a lack of regularization and by the non-Euclidean structure of color information as perceived by the human visual system.

3.3.2 Machine learning for physical problems

We aim at continuing our efforts towards supervised and unsupervised learning for estimation problems. Concerning supervised learning, we intend to investigate further the opportunities offered by neural network estimation of displacement and strain fields in experimental mechanics that we have recently introduced with colleagues in mechanics and signal processing 1. Besides, we also aim at developing unsupervised learning in problems where a quantity has to be estimated over a spatio-temporal domain, which is a recent trend in several application domains. Neural networks are indeed universal approximators whose derivative can be exactly computed with the backpropagation algorithm, which is supposed to make them robust to acquisition noise.

4 Application domains

Applications on which our program is expected to have an impact are mixed reality, computational photomechanics and minimally invasive medical interventions. These fields correspond to areas where we have established trans-disciplinary collaborations with academic or industrial experts of the applicative fields. Common to these applications are the need for finely characterizing the acquisition context of vision-based applications and the need for accurate registration procedures. Another common point is the availability of a limited amount of data for characterizing the variability of the observed phenomena.

Mixed reality

Being able to perform reliable and accurate registration under large viewpoint variations, seasonal or lighting changes opens the way towards challenging mixed reality applications. Urban AR and industrial maintenance in large and cluttered environments are examples of application fields that would successfully capitalize on more robust localization solutions. Improved robustness of camera localization is especially expected for poorly textured, specular environments and in the presence of repeated patterns that are common in industrial contexts

Photomechanics

Photomechanics is the field of experimental mechanics which is dedicated to mechanical measurement from images. In particular, we are interested in contactless image-based methods for extensometry, that is the estimation of displacement and strain fields on the surface of materials subjected to different types of mechanical loads. Full-field extensometry is a challenging task since strains often have tiny values and result in gray level changes at the limit of the sensor noise floor. The economic stakes are high and concern for example the automotive and aeronautics industries, or civil engineering. In order for these methods to be adopted by industry, it is, however, necessary to quantify their metrological performance, which is limited by the registration process or by the image acquisition chain, and especially by sensor noise. This topic is the subject of a long-term trans-disciplinary collaboration with Institut Pascal (Clermont-Ferrand Université).

Minimally invasive medical interventions

The trend towards the design and performance of minimally invasive procedures will increase in the near future. But the benefit for the patient is at the expense of the surgeon who can only sense the surgical scene through intra-operative imaging. Commercial solutions now exist to teach this increasingly difficult surgical gesture with interactive simulation technologies. However, challenges remain to fill the gap between the learning environment, where qualitative correctness of the setup is sufficient, and the surgical theater, where accuracy and predictability are required. In this context, we aim at addressing the key problem of modeling the geometry and dynamics of deformable organs and surgical devices, in order to make progress towards a faithful 3D rendition of the surgical scene. To circumscribe practical and experimental difficulties, three specific applications will be addressed with our clinical partners: intra-operative guidance in interventional neuroradiology, augmented reality for laparoscopic liver surgery, and simulation of the mitral valve behaviour.

5 Highlights of the year

5.1 Awards

  • The TANGRAM team received the best paper presentation award at FIMH 2023 for the paper "Influence of Anisotropy on Fluid-Structure Interaction Simulations of Image-Based and Generic Mitral Valves" written by Nariman Khaledian, Pierre-Frédéric Villard and Marie-Odile Berger 20.
  • Youssef Assis received a STAR (STudent-Author Registration) award at MICCAI 2023 conference for his paper "Aneurysm Pose Estimation with Deep Learning" written with co-authors Liang Liao, Fabien Pierre, René Anxionnat and Erwan Kerrien 16.

6 New software, platforms, open data

6.1 New software

6.1.1 OA-SLAM

  • Name:
    Object -aided SLAM
  • Keywords:
    Localization, 3D reconstruction, Object detection
  • Scientific Description:
    Details on the method can be found in the paper published in ISMAR 2022 9.
  • Functional Description:
    OA-SLAM uses objects as landmarks to improve the relocalization capabilities of SLAM systems. OA-SLAM builds on the point-based ORB-SLAM2. It allows online reconstruction of 3D objects modeled as ellipsoids from their detections in 2D images. OA-SLAM dramatically improves the relocalization capabilities of SLAM.
  • News of the Year:
    The code was consolidated through its demonstration at CVPR 2023.
  • URL:
    https://­gitlab.­inria.­fr/­tangram/­oa-slam
  • Publication:
    hal-03837883
  • Contact:
    Gilles Simon
  • Participants:
    Matthieu Zins, Gilles Simon, Marie-Odile Berger

6.1.2 P1E

  • Name:
    Perspective-1-Ellipsoid
  • Keywords:
    Visual localization, Object detection
  • Functional Description:
    Pose computation from one correpondance ellipse/ellipsoid. This Matlab code implements the solution described in 12.
  • URL:
    https://­gitlab.­inria.­fr/­vgaudill/­p1e/
  • Publication:
    hal-04132261
  • Contact:
    Gilles Simon
  • Participants:
    Vincent Gaudilliere, Gilles Simon, Marie-Odile Berger

6.1.3 DeepAnePose

  • Name:
    Pose estimation of brain aneurysms using deep learning
  • Keywords:
    Deep learning, Anomaly detection, Medical imaging, Brain MRI, Brain aneurysm, Pose estimation
  • Functional Description:
    DeepAnePose is a deep convolution network for the detection and pose estimation of intracranial aneurysms from 3D TOF-MRI images. It is a YOLOv3-inspired anchor-free detection model in 3D, extended with a pose estimation head, coupled with an original strategy for small patch generation that combines data augmentation and data synthesis.
  • News of the Year:
    The code was finalized, consolidated, and issued through our publication at MICCAI 2023.
  • URL:
    https://­gitlab.­INRIA.­fr/­yassis/­DeepAnePose
  • Publications:
    hal-04207337, hal-03391884, hal-03897642
  • Contact:
    Erwan Kerrien
  • Participants:
    Youssef Assis, Erwan Kerrien
  • Partner:
    Loria

7 New results

7.1 Visual localization

Participants: Marie-Odile Berger, Romain Boisseau, Hugo Leblond, Gilles Simon.

For several years now, and following the track of research inited in 3, the team has been investigating the use of objects as landmarks for pose computation. Objects are here approximated by ellipsoids from their 2D detection in images as ellipses. Using generic object detectors allows handling any environment without accurate knowledge of its geometry.

This year we have investigated the fundamental problem of camera pose estimation from one ellipse-ellipsoid correspondence, referred to as the P1E problem. On the theoretical side, and except in the case of a spherical object, we have demonstrated in 12 that the solutions are a variety of dimension 1. We have also provided an effective way to compute the camera locus (i.e. set of solutions). This problem was addressed by Wokes and Palmer in 2010 23 in the particular case of a spheroid (specific ellipsoid having an axis of revolution) but was never considered for general ellipsoids. To the best of our knowledge, we are the first to propose a constructive solution to the P1E problem without resorting to any additional approximation nor prior knowledge. In this study, we also consider two particular cases of important practical interest: (i) computing the camera position when the orientation is known (ii) computing the orientation when the position is known. Besides the theoretical aspects, solving the P1E problem opens the way towards automatic positioning solutions in texture-less or low-textured environments, for instance leveraging several ellipse-ellipsoid correspondences. The accompagnying code of the method is available here.

Still in the context of object-based positionning, our objet aided SLAM method OA-SLAM 9 was selected in the demo session at CVPR 2023.

7.2 Vanishing point computation and applications

Participants: Marie-Odile Berger, Romain Boisseau, Abdelkarim Elassam, Gilles Simon.

For several years now, the Tangram team has been working on the automatic detection of vanishing points in an image, a preliminary step in the rectification of planar structures present in a picture, which is also useful for determining the intrinsic and orientation parameters of a camera. Traditionally considered as a problem of grouping line segments by meeting points, this task has been reconsidered in recent years in the light of deep learning techniques. The main difficulties lie in the inability to predict the number of vanishing points expected and the possibility of obtaining such points outside and sometimes far from the image. A first solution was to model the extraction of vanishing points as a problem of classifying their positions among intervals of coordinates along the horizon line. An a-contrario method made it possible to detect the peak scores obtained at the output of a CNN, each peak corresponding to a horizontal vanishing point.

In a paper published at ICIP 2023, we demonstrated that the performance of this method can be improved by simultaneously detecting the vertical structures associated with vanishing points, in the form of a map of discretized normal orientations, using the same classification as for vanishing points 18. The method makes it possible to detect additional vanishing points, corresponding to score peaks in the output of the previous CNN, which were however considered to be meaningless by the a-contrario decision scheme. Considered as a by-product of the method, the vertical regions associated with vanishing points are currently being investigated to match vanishing points between distant images, something that was difficult to envisage until now, vanishing points being abstract primitives not directly linked to photometry. This paves the way for robust vanishing point matching and relative pose computation between distant images.

In parallel with these activities, a collaboration with the historian Ludovic Balavoine (Université de Bordeaux Montaigne) was undertaken to analyse the vanishing points, using the probabilistic method presented in 7, of around a hundred paintings of Flemish primitive art. This study led to a revision of twentieth-century historiography, which predicted a late introduction of perspective in Flanders compared with Italy. A book resulting from this collaboration has recently been accepted for publication by the international publisher Brepols.

7.3 Handling non rigid deformation

Participants: Marie-Odile Berger, Nariman Khaledian, Pierre-Frédéric Villard.

7.3.1 Individual mitral valve modeling

Our previous work on the mitral valve includes segmenting the valve anatomy and simulating its closure using fluid-structure interaction (FSI) with a generic valve. This work allows us to improve the detection of blood leaks by creating a map of contact 5. This year we have focused on moving towards patient-based modelling and incorporating anisotropy into our model to get closer to real valve behaviour 20. Four valves extracted from medical images were tested to show that our biomechanical model can be scaled to work with real data. We also investigated the importance of the anisotropic directions by testing various fiber map strategies: random, parallel to the annulus and based on physical computation. Although the simulation result is affected by this choice, there is no strong evidence that it leads to better sealing properties. Our next step will be to collect experimental data, which can provide the missing ground truth, and to run simulations under the same conditions.

7.3.2 Image-based biomechanical simulation of the diaphragm during mechanical ventilation

  This ultimate goal of this project is to perform high-fidelity, real-time simulations of the respiratory function of a critical care patient. The focus is on the respiratory muscles, where the diaphragm is the main muscle. This involves as the first step a realistic tissue model, which allows us to model the passive deformation of the muscle. This year, we have been working on a model for active muscle contraction, which includes at least an approximate model for the muscle fiber bundles and their orientation. From a numerical perspective this corresponds to solving non-linear PDEs in thin structures. The methods used are localized radial basis function (RBF) methods. Last year, we implemented a linear elasticity solver using the RBF with the partition of unity method (PUM) method, which was used to deform a shell bunny model, a webbed cube and the diaphragm. We have done a FEM implementation for comparison. We are also continuing to work on RBF-PUM applied to elasticity problems. For next year, we aim to also consider models that are not linear. We will try different types of models to see how they perform. We would also like to make the first tests on muscle activation.

7.4 Interventional radiology

Participants: Youssef Assis, Radhouane Jilani, Erwan Kerrien, Liang Liao, Fabien Pierre, Pierre-Frédéric Villard.

7.4.1 Detection of brain aneurysms using deep learning

The YOLO-inspired anchor free deep neural network designed by Youssef Assis during his PhD thesis was further validated this year using two clinical experts to assess each aneurysm detection by our model, nnDetection 22 and nnUnet, on a database of 270 patients. The performance assessed using the induced new clinical ground truth (double blind annotation with consensus) was even improved, exceeding the experts' sensitivity. This model was extended this year to tackle the innovative task of aneurysm pose estimation 16. Median errors of 0.5 mm for the position and 12° for the orientation were deemed accurate enough for the display of a reformatted cut plane through the aneurysm that is clinically useful. This work received a STAR award (STudent-Author Registration) at the MICCAI 2023 conference. The corresponding code and ground truth annotations were made available in the DeepAnePose software. In the context of his PhD thesis, Liang Liao is leading an extensive data collection campaign to have 8 medical doctors annotate the database. The aim is to 1) further characterize the strengths and weaknesses of our model; 2) better assess the impact of our model to improve the physician's performance, depending on their expertise; and 3) study how multiple annotations can compensate for the lack of patient data, in order to tighten the statistical significance of the model comparison.

7.4.2 Predictive simulation of catheter navigation

Our main contribution to the PreSPIN ANR project consists in achieving smooth, interactive and predictive simulation of a catheter navigating in the brain vasculature. In the course of his PhD thesis work, Radhouane Jilani proposed a new collocation method to solve the boundary value problem resulting from the Cosserat formulation. This method relies on the approximation of the internal forces and moments of a 1D rod on a finite Chebyshev basis. This method was demonstrated to be more stable than previous methods, though a bit slower than the shooting method in the static case. Our method was extended to the dynamic case and experimented in various standard tests used in the robotics community 19. A further extension to manage contact with implicit surfaces is currently under development.

7.5 Image and signal processing

Participants: Gaetano Agazzotti, Nicolas Maignan, Fabien Pierre, Frédéric Sur.

7.5.1 Computational photomechanics

The work of this year concerns two aspects of displacement and strain field estimation over a material submitted to a mechanical load. Displacement estimation shows similarities with optical flow estimation. In order to process the surface of the material, either a contrasted marking is projected on it, giving a speckle pattern, or a periodic pattern (a checkerboard) is marked on the surface. We have customized a state-of-the-art convolutional neural network dedicated to optical flow estimation (for natural images) to reach better performance when processing speckle images. We have designed a lightweight CNN by reducing as much as possible the number of filters while keeping equivalent metrological performance to the original version (and almost equivalent to state-of-the-art standard digital image correlation estimation), in order to accelerate image processing on a power-efficient compact Graphics Processing Unit (GPU) 11. The ultimate goal is to develop smart-cameras dedicated to displacement and strain measurement. Another contribution of this year concerns the estimation of displacement and strain fields from a deformed checkerboard pattern, with windowed Fourier analysis. In this case, we have proved that the estimated displacement and strain fields are merely the convolution of the true fields with the analysis window. We have proposed and fine-tuned a deconvolution process to enhance the small and sharp details in displacement and strain maps 13.

7.5.2 Variational methods for image processing

During the internship of Gaetano Agazzotti, the combination of deep image priors and variational methods has been explored for single image colorization. The deep image prior model is able to produce realistic images while suffering from overfitting of the data that results in noisy images and blurring of the colorization results (the so-called halo effect). With a regularization of the result with coupled total variation, these effects can be avoided and controled 15.

The colorization of videos is mostly based on color transfer between frames. The state-of-the-art methods suffer from occlusions and dis-occlusions during the matching phase. Since such problems are generally solved through 3D knowledge on the scene, the PhD thesis of Nicolas Maignan aims, in particular, at incorporating the 3D information from a monocular video sequence in the colorization process.

7.6 Application of deep learning

Participants: Nathan Boulangeot, Mehdi Serdoun, Frédéric Sur.

7.6.1 Extensive searches for complex intermetallic catalysts

Material science is at the heart of many technological revolutions. The search for new materials is crucial to meet the challenges posed by climate change, or the growing global demand for energy and consumer goods. In this context, the discovery of new catalytic materials is a major challenge to improve the performance of industrial processes, for example in the context of the hydrogen economy, in order to provide energy sources with limited environmental impact. Although some theoretical models exist, catalyst research is still largely empirical and based on a trial-and-error approach. This is probably because the application of theoretical models requires particularly expensive calculations based on density functional theory (DFT). Recently, the materials community has been seizing on statistical learning tools to limit the cost of calculations and to accelerate the discovery of new catalysts. Together with Emilie Gaudry at Institut Jean-Lamour (IJL) at Nancy, we have developed a collaboration on this subject since September 2021 through Nathan Boulangeot's PhD thesis. The work of this year concerns models for infering the energy of the system and the forces applied to the atoms in the context of predicting the adsorption properties of Al13Co4 based on Gaussian process regression. In particular, we train a model to efficiently select adsorption configurations and capture the symmetry of the true adsorption with a minimal numer of DFT calculations.

7.6.2 Multivariate analysis of geochemical, physical and mineralogical signatures of uranium deposits in the Athabasca Basin (Saskatchewan, Canada)

We are engaged in the co-supervision (together with Julien Mercadier, GéoRessources) of the PhD thesis of Mehdi Serdoun, which is part of the GeoMin3D project funded by ANR and Orano Mining, since January 2022. The goal is to develop statistical learning models to analyze the large amount of data of diverse nature provided during the exploratory drillings in Athabasca basin, the largest known source of uranium. The ultimate goal is to develop new analysis tools to accelerate exploration and reduce its cost, in cooperation with the industrial actors.

8 Partnerships and cooperations

8.1 International initiatives

8.1.1 Associate Teams in the framework of an Inria International Lab or in the framework of an Inria International Program

CURATIVE
  • Title:
    CompUteR-based simulAtion Tool for mItral Valve rEpair
  • Duration:
    2021-2024
  • Coordinator:
    Pierre-Frédéric Villard
  • Partners:
    • Harvard University (États-Unis)
  • Participants:
    Marie-Odile Berger , Nariman Khaledian , Pierre-Fredéric Villard
  • Summary:
    The mitral valve of the heart ensures one-way flow of oxygenated blood from the left atrium to the left ventricle. However, many pathologies damage the valve anatomy producing undesired backflow, or regurgitation, decreasing cardiac efficiency and potentially leading to heart failure if left untreated. Such cases could be treated by surgical repair of the valve. However, it is technically difficult and outcomes are highly dependent upon the experience of the surgeon. One of the main difficulties of valve repair is that valve tissues must be surgically altered during open heart surgery such that the valve opens and closes effectively after the heart is closed and blood flow is restored. In order to do this successfully, the surgeon must essentially mentally predict the displacement and deformation of anatomically and biomechanically complex valve leaflets and supporting structures [11]. Even if patient-based mitral valve models have been recently used for scientific understanding of its complex physiology, the patient geometry is manually segmented on medical images. This task is long and cumbersome except if the valve has been artificially isolated in-vitro. There is a lack in the literature about the variety of metrics in both anatomy and biomechanics of the valve. In order to study mitral valve behavior or to prepare models for planning, it is necessary to have methods to extract the valve components i) on real clinical data ii) with minor user input and ii) that are mechanically valid.

8.2 International research visitors

8.2.1 Visits of international scientists

Other international visits to the team
Peter Hammer
  • Status
    researcher
  • Institution of origin:
    Harvard Medical School
  • Country:
    USA
  • Dates:
    5/06/2023 - 09/06/2023
  • Context of the visit:
    Visit within the context of the associated team CURATIVE. In order to perform mitral valve modeling with real patient's data, our discussions focussed on (i) future experiments in Boston to determine anisotropic constitutive law on porcine data (ii) the analysis of our first results with real patient's data.
  • Mobility program/type of mobility:
    research stay
Hao Gao
  • Status
    senior lecturer
  • Institution of origin:
    University of Glasgow
  • Country:
    Great Britain
  • Dates:
    2/12/2023 - 21/12/2023
  • Context of the visit:
    After meeting at the FIMH conference in June, Hao Gao agreed to visit our team. He is a well-known expert in cardiac modelling. We worked on sharing our knowledge of mitral valve modelling. He gave a seminar to the team. He helped us to install the Immersed Boundary Framework that he uses for his research and we started to use it with our own dataset (1 generic and 4 image-based valves). The idea is to continue working on the comparison of our two methods in 2024.
  • Mobility program/type of mobility:
    research stay funded by Inria

8.2.2 Visits to international teams

Research stays abroad
Pierre-Frédéric Villard
  • Visited institution: Uppsala University
  • Country:
    Sweden
  • Dates:
    01/04/2023 - 31/04/2023
  • Context of the visit:
    Visit within the context of the project INVIVE. The aim was first to continue the supervision of a medical student who started working last year on segmenting the diaphragm. One new dataset was segmented and added to our database. The aim was also to supervise the PhD student (Andreas Michael) on the mechanical modeling of the diaphragm.
  • Mobility program/type of mobility:
    research stay
Pierre-Frédéric Villard
  • Visited institution: Boston Robotics Lab
  • Country:
    USA
  • Dates:
    01/11/2023 - 30/11/2023
  • Context of the visit:
    Visit within the context of the associated team CURATIVE. The aim was mainly to perform experiments on a phantom valve closing in a fully controlled environment. The phantom valve was already scanned last year to have the 3D geometry. Ultrasound and stereoscopic images were acquired with various conditions as well as pressure and flowrate data. Pierre-Frédéric Villard participated in the Harvard Robotics lab life by attending and giving seminars.
  • Mobility program/type of mobility:
    research stay

8.3 European initiatives

8.3.1 Other european programs/initiatives

  • Title:
    Towards robust spatial scene understanding in dynamic environments using intermediate representations
  • Partners:
    DFKI Kaiserslautern
  • Coordinator:
    Marie-Odile Berger
  • Participants:
    M.-O. Berger, R. Boisseau, A. Elassam, G. Simon
  • Duration:
    2020-2023
  • Additionnal info/keywords:
    The aim of the MOVEON project is to push forward the state of the art in vision-based, spatio-temporal scene understanding by merging novel machine-learning approaches with geometrical reasoning. Deep-learning-based recognition and understanding of high-level concepts such as vanishing points or large object classes will serve as unitary building blocks for a spatio-temporal localization and environment reconstruction that will use geometric reasoning as underlying support. This research will lead to a novel generation of visual positioning systems that go beyond classical localization and mapping, which focuses currently only on point cloud reconstruction. In contrast, our aim is to allow for 6DoF positioning and global scene understanding in wild and dynamic environments (e.g. crowded streets) that scales up nicely with the size of the environment, and that can be used persistently over time by reusing consistent maps.

8.4 National initiatives

PEPR ICCARE
  • Gilles Simon is coordinator of the "Museum and Cultural Heritage" sector of the PEPR ICCARE (Cultural and Creative Industries: Action, Research, Experimentation).
ANR Arcé
  • Title
    :Colorisation automatique de vidéos
  • Coordinator:
    Fabien Pierre
  • Participants:
    F. Pierre, N. Maignan, F. Sur
  • Duration:
    2022-2026
  • Additionnal info/keywords:
    The Arcé project aims at proposing new methods for automatic, fast and perceptually satisfying video colorization. Image colorization methods based on deep learning based have encountered a great success in recent years. These techniques are fully automatic and very fast, but they have not been adopted by colorization industry. The reason is that they do not ensure the temporal coherence of the colorization, which is particularly disturbing for the viewer. The ultimate goal is the use of our work in audiovisual production studios.
ANR PRC PreSPIN
  • Title:
    Predictive Simulation for Planning Interventional Neuroradiology procedures
  • Partners:
    CReSTIC (Reims), Creatis (Lyon) and CIC-IT/CHRU Nancy
  • Coordinateur:
    Erwan Kerrien
  • Participants:
    Y. Assis, R. Jilani, E. Kerrien, P.-F Villard.
  • Duration:
    2020-2024
  • Additionnal info/keywords:
    This 4-year project is coordinated by E. Kerrien. It aims at improving the planning phase in the therapeutic management of cerebral ischemic strokes thanks to predictive simulation of both the therapeutic interventional gesture and post-interventional perfusion images. The consortium is set to address the challenges of geometrical and topological modeling of the full brain vasculature; physics-based simulation of interventional devices; simulation of MRI perfusion images; and clinical validation.

9 Dissemination

9.1 Promoting scientific activities

9.1.1 Scientific events: organisation

Member of the organizing committees
  • Erwan Kerrien was a member of the organizing committee for the Digital and Health Sciences day that brought together 100 numerical scientists and clinicians at the University Hospital in Nancy.

9.1.2 Scientific events: selection

Member of the conference program committees
  • Frédéric Sur was a program committee member for the International Conference on Computer Vision Theory and Applications (VISAPP 2023).
Reviewer
  • Marie-Odile Berger was a reviewer for IPCAI (International Conference on Information Processing in Computer-Assisted Interventions), IROS (International Conference on Intelligent Robots and Systems), ICRA (International Conference on Robotics and Automation) and for the French conference ORASIS.
  • Erwan Kerrien was a reviewer for MIDL (Medical Imaging with Deep Learning), MICCAI (International Conference on Medical Image Computing and Computer Assisted Interventions), IROS, and the French conference ORASIS.
  • Gilles Simon was a reviewer for IROS and the French conference ORASIS.
  • Pierre-Frédéric Villard was a reviewer for the Eurographics Workshop on Visual Computing for Biology and Medicine 2023 and the International Conference on Computer Graphics, Visualization, Computer Vision And Image Processing 2023.

9.1.3 Journal

Reviewer - reviewing activities
  • Erwan Kerrien was a reviewer for Medical Image Analysis and Medical Physics.
  • Fabien Pierre was a reviewer for SIAM imaging sciences
  • Gilles Simon was a reviewer for the International Journal of Computer Vision (IJCV).
  • Frédéric Sur was a reviewer for Experimental Mechanics, Optics and Lasers in Engineering, Measurement, and IEEE Transactions on Instrumentation and Measurement.

9.1.4 Invited talks

  • Abdelkadir Elassam gave a seminar in September 2023 in the one day workshop "Calcul visuel de pose en robotique et réalité-X" organized by GDR ISIS. Title: "Advancing Wide Baseline Scene Registration with Structural Vanishing Points".
  • Fabien Pierre gave a seminar at INSA Rouen in May 2023. Title "Couplage de méthodes variationnelles et CNN pour la colorisation d’images".
  • Gilles Simon gave a seminar to the Translitteræ research school at ENS Paris-Saclay. Title: “Les dispositifs optiques en peinture chez les maîtres anciens" on June 14th 2023.
  • Pierre-Frédéric Villard did a presentation to the Harvard Biorobotics Lab within the CURATIVE collaboration. Title: "New contributions in mitral valve simulation" on November 9th 2023.
  • Pierre-Frédéric Villard gave a seminar at the department of information technology of Uppsala University. Title: "Simulation of Mitral Closing with anisotropy" on May 17th 2023.
  • Pierre-Frédéric Villard gave an invited talk at the workshop "Simulation and Imaging for Mitral Regurgitation" on June 22nd 2023 in Lyon. Title: "Mitral Valve Modeling: From Medical Image Analysis to Fluid Structure Interaction"
  • Pierre-Frédéric Villard gave a keynote talk at the national conference " journée de la simulation numérique en mécanique" on July 12th in Metz. Title: "Modélisation basée patient de la fermeture de la valve mitrale avec de l'Interaction Fluide-Structure"

9.1.5 Scientific expertise

  • Marie-Odile Berger was a member of the committee that awards the annual AFRIF thesis prize. She was an expert for the Israel Science Foundation and for ANRT.
  • Erwan Kerrien was an expert for the ANR and the FC3R scientific interest group.

9.1.6 Research administration

  • Marie-Odile Berger is head of the INRIA COMIPERS PhD and postdoctoral recruitment committee. She was a member of the recruitment committee for an assistant professor position position at Université Savoie Mont-Blanc.
  • Gilles Simon was an elected member of the CNU (Conseil National des Universités)
  • Frédéric Sur was a member of the recruitment committee for two Assistant Professor positions at Université de Lorraine, and one recruitment committee for a Professor position at Université de Strasbourg.
  • Pierre-Frédéric Villard is an elected member of the Scientific Council of the Université de Lorraine

9.2 Teaching - Supervision - Juries

9.2.1 Teaching

The assistant professors of the TANGRAM team actively teach at Université de Lorraine with an annual number of around 200 teaching hours in computer sciences, some of them being accomplished in the field of image processing. INRIA researchers have punctual teaching activities in computer vision and shape recognition mainly in the computer science Master of Nancy and in several Engineering Schools near Nancy (ENSMN Nancy, SUPELEC Metz, ENSG). Our goal is to attract Master students with good skills in applied mathematics towards the field of computer vision.

The list of courses given by staff members is detailed below:

  • M.-O. Berger
    • Master : Shape recognition, 24 h, Université de Lorraine.
    • Master : Introduction to image processing, 12 h, ENSMN Nancy.
    • Master : Image processing for Geosciences, 12h, ENSG.
  • E. Kerrien
    • Master : Introduction to image processing, 15 h, ENSMN Nancy.
    • Licence : Initiation au développement, 80h, IUT St Dié-des-Vosges.
  • Fabien Pierre
    • Master: Introduction à l’apprentissage automatique, 14h, Mines Nancy.
    • Master: Vision artificielle et traitement des images, 12h, Polytech Nancy.
    • Licence: Introduction au traitement d’image, 30h, IUT Saint-Dié des Vosges.
    • Licence: Algorithmique et programmation, 87h, IUT Saint-Dié des Vosges
    • Licence: Culture scientifique et traitement de l’information, 69h, IUT Saint-Dié des Vosges
    • Licence: Programmation objet et évènementielle, 35h, IUT Saint-Dié des Vosges
    • Licence: Initiation à l’intelligence artificielle, 18h, IUT Saint-Dié des Vosges
  • G. Simon
    • Master: Augmented reality, 9 h, Télécom-Nancy.
    • Master: Augmented reality, 24h, M2 Informatique FST
    • Master: Visual data modeling, 12h, M1 Informatique FST
    • Master: Computer Vision, 12h, M1 Informatique FST
    • Licence pro: 3D modeling and augmented reality, 50h FST - CESS d’Epinal
    • Licence: Programming methodology, L1 informatique, 48h FST
  • F. Sur
    • Master: Introduction to machine learning, 40 h, Université de Lorraine (Mines Nancy).
    • Licence: Javascript programming, 100h, IUT Charlemagne
  • P.-F Villard
    • Master : Augmented and Virtual Reality, 16h, M2 Cognitive Sciences and Applications, Institut des Sciences du Digital, Université de Lorraine
    • Licence: Computer Graphics with webGL, 30h, IUT Saint-Dié des Vosges.
    • Licence: Virtual and Augmented Reality in Industrial Maintenance, 2h, Faculty of Science and Technology, Université de Lorraine
    • Licence: Web programming, 20h, IUT Saint-Dié des Vosges.
    • Licence: Graphical user interface programming, 30h, IUT Saint-Dié des Vosges.
    • Licence: Security and life privacy with internet, 2h, IUT Saint-Dié des Vosges.
    • Licence: Parallel programming, 18h, IUT Saint-Dié des Vosges.
    • Licence: Initiation to machine learning, 24h, IUT Saint-Dié des Vosges.
    • Licence: Initiation to cryptography, 12h, IUT Saint-Dié des Vosges.
  • B. Wrobel-Dautcourt
    • Licence: interfaces graphiques, 22h, FST

9.2.2 Supervision

  • PhD in progress: Abdelkarim Elassam, Robust visual localization using high level features, October 2020, Marie-Odile Berger, Gilles Simon.
  • PhD in progress: Nariman Khaledian, Toward a Functional Model of the Mitral Valve, October 2020, Marie-Odile Berger, Pierre-Frédéric Villard.
  • PhD in progress: Youssef Assis, Deep learning for the automated detection of brain aneurysms, November 2020, Erwan Kerrien, René Anxionnat (CHRU Nancy).
  • PhD in progress: Radhouane Jilani, Predictive simulation for interventional neuroradiology, October 2021, Erwan Kerrien, Pierre-Frédéric Villard.
  • PhD in progress: Liang Liao, Detection of cerebral aneurysms from MRI images using deep learning: deep neural network creation and its clinical evaluation, November 2022, René Anxionnat (CHRU Nancy) and Erwan Kerrien.
  • PhD in progress: Nathan Boulangeot, Extensive searches for complex intermetallic catalysts, October 2021, Émilie Gaudry (Institut Jean-Lamour), Frédéric Sur.
  • PhD in progress: Mehdi Serdoun, Multivariate analysis of mineralogical, geochemical and physical signatures, January 2022, Julien Mercadier (GéoRessources), Frédéric Sur.
  • PhD in progress: Nicolas Maignan, Image and video colorization, October 2022, Fabien Pierre, Frédéric Sur
  • PhD in progress: Hugo Leblond, November 2023, Analyse de scènes dynamiques à partir d’une représentation neuronale implicite (NeRF) basée sur des données LiDAR-caméra, Gilles Simon, Renato Martins

9.2.3 Juries

  • Marie-Odile Berger was president of the PhD committee of Prerak Srivastava (Université de Lorraine), Nuwan Herath Mudiyanselage (Université de Lorraine) and Yumin Du (ENPC). She was external reviewer for the PhD thesis of Clémentin Boittiaux (Université de Toulon). She was examiner of M. Tammazousti's HDR committee (CEA List).
  • Gilles Simon was a reviewer for the PhD thesis of Andrea Macario Barros (CEA List, Université de Paris Saclay).
  • Frédéric Sur was president of the PhD committee of Sandipana Dowerah (Université de Lorraine).
  • Erwan Kerrien was an external reviewer for the PhD theses of Jonas Lamy (Université de Lyon 2) and Aurélien de Turenne (Université de Rennes 1).

9.3 Popularization

9.3.1 Internal or external Inria responsibilities

  • Erwan Kerrien is Chargé de Mission for scientific mediation at Inria Nancy-Grand Est, and thereby is part of the Inria scientific mediation network. As such, he is a member of the steering committee of "la Maison pour la Science de Lorraine", and member of the IREM Lorraine (Institut de Recherche sur l’Enseignement des Mathématiques - Research Institute for Teaching Mathematics) steering council. He is also the local scientific referent for the "Chiche!" initiative.
  • Erwan Kerrien shares the local coordination of MATh.en.JEANS in the Lorraine area with Samuel Tapie from the IECL lab (mathematics).

9.3.2 Articles and contents

  • Frédéric Sur wrote a paper about his experience in setting up a new course on machine learning at École des Mines de Nancy 14.

9.3.3 Education

  • Pierre-Frédéric Villard was a mentor for the secondary school of Champigneulles (France) as a "Collège Pilote" of "La Main à la pâte" foundation. He gave a seminar on augmented and virtual realities to the pupils, he helped the teacher with preparing some activities with augmented and virtual reality technologies. Eventually, he is supervising master students to produce teaching applications with augmented reality technologies that will be used in secondary school classes.
  • Pierre-Frédéric Villard presented a workshop on automatic character recognition using deep learning technique at the "Fête de la Science" in St-Dié-des-Vosges.
  • Pierre-Frédéric Villard presented a workshop for high school teachers in computer science at Loria on April 5th on how to program a person recognition application.
  • Pierre-Frédéric Villard did two science cafés: "Ecodesign and responsible digital practices" Thursday on March 9 2023, at the amphitheater of IUT Saint-Dié (Saint Dié des Vosges) and "The other side of streaming", on April 13rd 2023, at Inria Grand Est (Nancy)
  • Pierre-Frédéric Villard and Erwan Kerrien did presentations to high school students in the context of the "Chiche!" initiative.
  • Erwan Kerrien was an associate researcher to a MATh.en.JEANS workshop within Henri Loritz high school in Nancy, Alfred Cytère secondary school in Rambervillers and Jean d'Arc technical school in Commercy.
  • Erwan Kerrien animated a workshop during the APMEP day (Maths teachers association in public education).

9.3.4 Interventions

  • Gilles Simon gave a talk as part of the CNRS "Visites insolites" outreach intiative.

10 Scientific production

10.1 Major publications

10.2 Publications of the year

International journals

International peer-reviewed conferences

Other scientific publications

10.3 Cited publications

  • 22 inproceedingsM.Michael Baumgartner, P. F.Paul F Jäger, F.Fabian Isensee and K. H.Klaus H Maier-Hein. nnDetection: a self-configuring method for medical object detection.Medical Image Computing and Computer Assisted Intervention--MICCAI 2021: 24th International Conference, Strasbourg, France, September 27--October 1, 2021, Proceedings, Part V 24Springer2021, 530--539back to text
  • 23 articleD. S.D. S. Wokes and P. L.P. L. Palmer. Perspective Reconstruction of a Spheroid from an Image Plane Ellipse.International Journal of Computer Vision9032010, 369--379back to text