3.1 Accelerated algorithms for solving high-dimensional continuous optimization problems

Variational problems requiring the estimation of a huge number of variables have now to be tackled, especially in the field of 3D reconstruction/restoration (e.g. $\ge {10}^9$ variables in 3D imaging). In addition to the curse of dimensionality, another difficulty to overcome is that the cost function usually reads as the sum of several loss/regularization terms, possibly composed with large-size linear operators. These terms can be nonsmooth and/or nonconvex, as they may serve to promote the sparsity of the sought solution in some suitable representation (e.g. a frame) or to fufill some physical constraints. In such a challenging context, there is a strong need for developing fast parallelized optimization agorithms for which sound theoretical guarantees of convergence can be established. We explore deterministic and stochastic approaches based on proximal tools, MM (Majorization-Minimization) strategies, and trust region methods. Because of the versatility of the methods that will be proposed, a wide range of applications in image recovery are considered: parallel MRI, breast tomosynthesis, 3D ultrasound imaging, and two-photon microscopy. For example, in breast tomosynthesis (collaboration with GE Healthcare), 3D breast images have to be reconstructed from a small number of X-ray projections with limited view angles. Our objective is to facilitate the clinical task by developing advanced reconstruction methods allowing micro-calcifications to be highlighted. In two-photon microscopy (collaboration with XLIM), our objective is to provide effective numerical solutions to improve the 3D resolution of the microscope, especially when cheap laser sources are used, with applications to muscle disease screening.

3.2 Optimization over graphs

Graphs and hypergraphs are rich data structures for capturing complex, possibly irregular, dependencies in multidimensional data. Coupled with Markov models, they constitute the backbones of many techniques used in computer vision. Optimization is omnipresent in graph processing. Firstly, it allows the structure of the underlying graph to be inferred from the observed data, when the former is hidden. Second, it permits to develop graphical models based on the prior definition of a meaningful cost function. This leads to powerful nonlinear estimates of variables corresponding to unknown weights on the vertices and/or the edges of the graph. Tasks such as partitioning the graph into subgraphs corresponding to different clusters (e.g., communities in social networks) or graph matching, can effectively be performed within this framework. Finally, graphs by themselves offer flexible structures for formulating and solving optimization problems in an efficient distributed manner. On all these topics, our group has acquired a long-term expertise 3 that we plan to further strengthen. In terms of applications, novel graph mining methods are proposed for gene regulatory and brain network analysis. For example, we plan to develop sophisticated methods for better understanding the gene regulatory network of various microscopic fungi, in order to improve the efficiency of the production of bio-fuels (collaboration with IFP Energies Nouvelles).

3.3 Toward more understandable deep learning

Nowadays, deep learning techniques efficiently solve supervised tasks in classification or regression by utilizing large amounts of labeled data and the powerful high level features that they learn by using the input data. Their good performance has caught the attention of the optimization community since currently these methods offer virtually no guarantee of convergence, stability or generalization. Deep neural networks are optimized through a computationally intensive engineering process via methods based on stochastic gradient descent. These methods are slow and they may not lead to relevant local minima. Thus, more efforts must be dedicated in order to improve the training of deep neural networks by proposing better optimization algorithms applicable to large-scale datasets. Beyond optimization, incorporating some structure in deep neural networks permits more advanced regularization than the current methods. This should reduce their complexity, as well as allow us to derive some bounds regarding generalization. For example, many signal processing models (e.g. those based on multiscale decompositions) exhibit some strong correspondence with deep learning architectures, yet they do not require as many parameters. One can thus think of introducing some supervision into these models in order to improve their performance on standard benchmarks. A better mathematical understanding of these methods permits to improve them, but also to propose some new models and representations for high-dimensional data. This is particularly interesting in settings such as the diagnosis or prevention of diseases from medical images, because they correspond to critical applications where the made decision is crucial and needs to be interpretable. One of the main applications of this work is to propose robust models for the prediction of the outcome of cancer immunotherapy treatments from multiple and complementary sources of information: images, gene expression data, patient profile, etc (collaboration with Institut Gustave Roussy).

4.1 Artificial intelligence fighting against COVID-19

Participants: Guillaume Chassagnon, Emilie Chouzenoux, Kavya Gupta, Yingping Li, Ana Neascu, Jean-Christophe Pesquet, Hugues Talbot, Matthieu Terris, Maria Vakalopoulou, Sagar Verma (Collaboration: Institut Gustave Roussy ; Kremlin-Bicetre APHP ; Owkin France ; Hopital Cochin Broca Hotel Dieu ; Therapanacea ; IIIT Delhi)

Coronavirus disease 2019 (COVID-19) emerged in 2019 and disseminated around the world rapidly. Computed tomography (CT) imaging has been proven to be an important tool for screening, disease quantification and staging. The latter is of extreme importance for organizational anticipation (availability of intensive care unit beds, patient management planning) as well as to accelerate drug development through rapid, reproducible and quantified assessment of treatment response. Even if currently there are no specific guidelines for the staging of the patients, CT together with some clinical and biological biomarkers are used. Through two collaborative projects with medical partners and French start-ups, we aim in 10, 22 to investigate and propose artificial intelligence tools that could help on the clinical practice.

Our work in 77, 76, in collaboration with IIIT Delhi, builds computational models for drug re-positioning for COVID-19 treatment. The aim is to assist clinicians with a tool for selecting prospective antiviral treatments. The main contributions are (i) a manually curated database publicly shared, comprising of existing associations between viruses and their corresponding antivirals along with a set of state-of-the-art computational drug re-positioning tools based on matrix completion, (ii) a novel multiple graph regularized matrix completion approach with assessed performance on antiviral repositioning.

4.2 Design of robust neural networks in safety critical industrial domains

Participants: Jean-Christophe Pesquet (Collaboration: P.L. Combettes, North Carolina State University; Audrey Repetti, Yves Wiaux, Heriot Watt University)

One of the main challenges faced today by companies like Thales or Schneider Electric designing advanced industrial systems, is to ensure the safety of new generations of products based on the use of neural networks. Since 2013, neural networks have been shown to be sensitive to adversarial perturbations. Deep neural networks can thus be fooled, in an intentional (security issue) or in undeliberate manner (safety issue), which raises a major robustness concern for safety-critical systems which need to be certified by an independent certification authority prior to any entry into production/operation. Techniques based on mathematical proofs of robustness are generally preferred by industrial safety experts since they enable a safe-by-design approach that is more efficient than a robustness verification activity done a posteriori with a necessarily bounded effort. Among the possible mathematical approaches, we focus on those relying upon the analysis of the Lipschitz properties of neural networks 78, 2, 62. Such properties play a fundamental role in the understanding of the internal mechanisms governing these complex nonlinear systems. Besides, they make few assumptions on the type of non-linearities used and are thus valid for a wide range of networks.

4.3 Radiology, Hyper-Progressive Disease and immunotherapy

In recent years, novel treatments for a variety of cancers have emerged, as a “fourth way”, beyond the well-established chemotherapy, surgery and radiotherapy. These treatments have been in development for decades under the heading of “immunotherapy” but have finally met with considerable success.

One defining feature of cancer cells is that they are considered as host cells by the patient's immune system, even when they are malignant, and so they are allowed to grow unchecked, whereas normally abnormal mutant cells are recognized and destroyed by the host.

Successful immunotherapy has focused on programmed cell death, also known as apoptosis. These approaches against some forms of cancer aim to block the ability of tumor cells to resist patient immune response by acting on the checkpoints of immune cells. These drugs are able to either block the PD-1 receptor present on the surface of the lymphocytes, or the PD-L1 and PD-L2 ligands expressed by the cancer cells. This can prevent the binding of both, blocking the immunomodulatory signal and allowing the killer T cells continue to be active against the tumor.

However, no therapy is yet universal. One aspect of these approaches is that some patient will respond (and their tumor will therefore not “progress”) and some not. “Progressor” patient, in oncology, are those who continue to see their tumor grow. It is important to use all available data as early as possible in the treatment to decide quickly if the patient is responding positively. One of our projects in this sections aims at doing this.

4.4 Sparse signal processing in chemistry

Participants: Marc Castella, Emilie Chouzenoux, Arthur Marmin, Jean-Christophe Pesquet (Collaboration: Laurent Duval, IFPEN, Rueil Malmaison)

Mass Spectrometry (MS) is a powerful tool used for robust, accurate, and sensitive detection and quantification of molecules of interest. Thanks to its sensibility and selectivity, MS is widely used in proteomics such anti-doping, metabolomics, medicine or structural biology. In particular, it has applications in clinical research, personalized medicine, diagnosis process and tumours profiling and pharmaceutical quality control. In an MS experiment, the raw signal arising from the molecule ionization in an ion beam is measured as a function of time via Fourier Transform-based measures such as Ion Cyclotron Resonance (FT-ICR) and Orbitrap. A spectral analysis step is then performed to improve the quality of data. The goal is then to determine from this observed pattern distribution the most probable chemical composition of the sample, through the determination of the monoisotopic mass, charge state and abundance of each present molecule. This amounts to solve a large scale signal estimation problem under specific sparsity constraints 13. Collaboration with Dr. L. Duval, Research Engineer at IFP Energies Nouvelles, France is on-going in this applicative context.

4.5 Image restoration for two-photon microscopy

Participants: Emilie Chouzenoux, Jean-Christophe Pesquet, Mathieu Chalvidal (Collaboration: Claire Lefort, XLIM, CNRS, Limoges)

Through an ongoing collaboration with physicists from XLIM laboratory (CNRS, Limoges, France), we propose advanced mathematical and computational solutions for multiphoton microscopy (MPM) 3D image restoration. This modality enjoys many benefits such as a decrease in phototoxicity and increase in penetration depth. However, blur and noise issues can be more severe than with standard confocal images. Our objective is to drastically improve the quality of the generated images and their resolution by improving the characterization of the PSF of the system 45 and compensating its effect. We consider the application of the improved MPM imaging tool to the microscopic analysis of muscle ultrastructure and composition, with the aim to help diagnosing muscle disorders including rare and orphan muscle pathologies.

4.6 Representation learning for biological networks

Participants: Fragkiskos Malliaros, Abdulkadir Çelikkanat (Collaboration: Duong Nguyen, UC San Diego)

Networks (or graphs) are ubiquitous in the domain of biology, as many biological systems can naturally be mapped to graph structures. Characteristic examples include protein-protein interaction and gene regulatory networks. To this extend, machine learning on graphs is an important task with many practical applications in network biology. For example, in the case on protein-protein interaction networks, predicting the function of a protein is a key task that assigns biochemical roles to proteins. The main challenge here is to find appropriate representations of the graph structure, in order to be easily exploited by machine learning models. The traditional approach to the problem was relying on the extraction of “hand-crafted" discriminating features that encode information about the graph, based on user-defined heuristics. Nevertheless, this approach has demonstrated severe limitations, as the learning process heavily depends on the manually extracted features. To this end, feature (or representation) learning techniques can be used to automatically learn to encode the graph structure into low-dimensional feature vectors – which can later be used in learning tasks. Our goal here is to develop a systematic framework for large-scale representation learning on biological graphs. Our approach takes advantage of the clustering structure of these networks, to further enhance the ability of the learned features to capture intrinsic structural properties.

4.7 Reconstruction approaches in medical imaging

Participants: Emilie Chouzenoux, Jean-Christophe Pesquet, Marion Savanier, Maissa Sghaier (collaboration S. Muller, C. Riddel, Y. Trousset, GE Healthcare)

The objective of these works, in collaboration with researchers from GE Healthcare, is to develop high quality reconstruction methodologies, for two modalities of medical imaging, namely digital breast tomosynthesis (DBT) and CT for interventional surgery.

The reconstruction of a volumetric image from DBT measurements is an ill-posed inverse problem, for which existing iterative regularized approaches can provide a good solution. However, the clinical task is somehow omitted in the derivation of those techniques, although it plays a primary role in the radiologist diagnosis. In this work, we address this issue by introducing a novel variational formulation for DBT reconstruction, tailored for a specific clinical task, namely the detection of microcalcifications. Our methods 80, 60 aim at simultaneously enhancing the detectability performance and enabling a high-quality restoration of the background breast tissues.

Discretizing and implementing tomographic forward and backward operations is a crucial step in the design of model-based iterative reconstruction algorithms in CT. The mathematical constraint of symmetry on the projector/backprojector pair prevents linear interpolation, which is a standard method in analytical reconstruction, to be used. Consequently, it often happens that these operators are approximated numerically, so that the adjoint property is no longer fulfilled. In the works 15, 59, we focus on the proximal gradient algorithm stability properties when such an adjoint mismatch arises. By making use of tools from convex analysis and fixed point theory, we establish conditions under which the algorithm can still converge to a fixed point. We provide bounds on the error between this point and the solution to the minimization problem. We illustrate the applicability of our theoretical results through numerical examples in the context of CT.

4.8 Inference of gene regulatory networks

Participants: Surabhi Jagtap, Fragkiskos Malliaros, Jean-Christophe Pesquet (collaboration A. Pirayre and L. Duval, IFPEN)

The discovery of novel gene regulatory processes improves the understanding of cell phenotypic responses to external stimuli for many biological applications, such as medicine, environment or biotechnologies. To this purpose, transcriptomic data are generated and analyzed from DNA microarrays or more recently RNAseq experiments. They consist in genetic expression level sequences obtained for all genes of a studied organism placed in dierent living conditions. From these data, gene regulation mechanisms can be recovered by revealing topological links encoded in graphs. In regulatory graphs, nodes correspond to genes. A link between two nodes is identified if a regulation relationship exists between the two corresponding genes. In our work, we propose to address this network inference problem with recently developed techniques pertaining to graph optimization. Given all the pairwise gene regulation information available, we propose to determine the presence of edges in the considered GRN by adopting an energy optimization formulation integrating additional constraints. Either biological (information about gene interactions) or structural (information about node connectivity) a priori are considered to restrict the space of possible solutions. Different priors lead to different properties of the global cost function, for which various optimization strategies, either discrete and continuous, can be applied.

4.9 Imaging biomarkers and characterization for chronic lung diseases

Participants: Guillaume Chassagnon, Maria Vakalopoulou (in collaboration with Marie-Pierre Revel and Nikos Paragios: AP-HP - Hopital Cochin Broca Hotel Dieu; Therapanacea)

Diagnosis and staging of chronic lung diseases is a major challenge for both patient care and approval of new treatments. Among imaging techniques, computed tomography (CT) is the gold standard for in vivo morphological assessment of lung parenchyma currently offering the highest spatial resolution in chronic lung diseases. Although CT is widely used its optimal use in clinical practice and as an endpoint in clinical trials remains controversial. Our goal is to develop quantitative imaging biomarkers that allow (i)severity assessment (based on the correlation to functional and clinical data) and (ii) monitoring the disease progression. In the current analysis we focus on scleroderma and cystic fibrosis as models for restrictive and obstructive lung disease, respectively. Two different approaches are investigated: disease assessment by deep convolutional neural networks and assessment of the regional lung elasticity through deformable registration. This work is in collaboration with the Department of Radiology, Cochin Hospital, Paris.

4.10 A generative model for heart left ventricle perfusion analysis

Participant: Hugues Talbot (Collaboration: L. Najman ESIEE Paris ; I. Vignon-Clementel, REO Team leader, Inria ; C. Taylor, Heartflow Inc.)

Cardio-vascular diseases continue to be the leading cause of mortality in the world. Understanding these diseases is a current, challenging and essential research project. The leading cause of heart malfunction are stenoses causing ischemia in the coronary vessels. Current CT and MRI technology can assess coronary diseases but are typically invasive, requiring risky catheterization and renal-toxic contrast agents injection. In collaboration with the REO team headed by Irène Vignon-Clementel, and Heartflow, a US based company, we have in the past contributed to Heartflow's major product, that replaces these physical exams with image-based exams only, limiting the use of contrast agents and in the cases that do not require a stent insertion, eliminating catheterisation. Heartflow is current the market leader in non-invasive coronary exams and the owner of most of the relevant IP in this domain.

Unfortunately, current imaging technology is unable to assess coronary disease along the full length of coronary vessels. CT is limited to a resolution of about 1mm, whereas coronary vessels can be much smaller, down to about 10 micrometers in diameter. To assess blood vessel health down to the smallest sizes, blood perfusion imaging technique throughout the heart muscle must be used instead. Perfusion imaging with PET or a Gamma camera, the current gold standard, is an invasive technology requiring the use of radioactive tracers. To avoid using these, a lower quality estimate of perfusion can be achieved using some ToF or injected gated MRI modalities.

We have investigated patient-specific vessel generation models together with porous model simulations in order to propose a direct model of perfusion imaging, based on the known patient data, computer flow dynamic simulations as well as experimental data consistent with known vessel and heart muscle physiology. The objective of this work is to both to provide a useful, complex forward model of perfusion image generation, and to solve the inverse problem of locating and assessing coronary diseases given a perfusion exam, even though the affected vessels may be too small to be imaged directly.

Continuing on our work from the period 2015-2019, this year we proposed a functional myocardial perfusion model consisting of the CT-derived segmented coronary vessels, a simulated vessel tree consisting of several thousands of terminal vessels, filling the myocardium in a patient-specific way, consistent with physiology data, physics-based and empirically-observed vessel growth rules, and a porous medium. We produced and validated a CFD code capable of simulating blood flow in all three coupled compartments, which allows us to simulate perfusion realistically.

This work was published in 33.

5.1 Footprint of research activities

The research carried out in OPIS aims at developing advanced techniques in the domain of data science for precision medicine. One of the main features of this research is to ensure that the proposed methods are not only efficient, but also grounded on sound mathematical foundations inherited from the areas of optimization and fixed point algorithms. In the biomedical domain, it appears indeed mandatory to guarantee the reliability and the explainability of the proposed approaches in their use by medical doctors or producers of medical imaging devices.

5.2 Impact of research results

OPIS participates in the design of innovative products developed by big companies working in the domain of medical imaging (GE Healthcare and Essilor) and several startups. Various application fields are targeted (breast cancer detection, surgical radiology, interventional surgery, coronary disease monitoring, vision correction, ...). In addition, OPIS has active collaborations with several hospitals, particularly Institut Gustave Roussy and hospitals from APHP in Paris. The purpose of these collaborations is to develop artificial intelligence tools aiding medical doctors in their practice. A large part of this research activity is oriented toward fighting against cancer using different kinds of data (CT scans, MRI, genomic data, microscopy images,...) OPIS is also currently involved in several projects for helping to better diagnose and cure COVID-19 infection.

6.1 Awards

• The article on “Influence Maximization using Influence and Susceptibility Embeddings” co-authored by G. Panagopoulos, F. Malliaros, and M. Vazirgiannis, received the best paper honorable mention award at the AAAI International Conference on Web and Social Media (ICWSM), 2020. The acceptance rate was $\sim 20\%$.
• T. Estienne took the 3rd place on the Learn2Reg competition 2020 of MICCAI conference.
• F. Malliaros received the Young Researcher Grant (JCJC) by the French National Research Agency (ANR).
• Our M.Sc. program in Data Sciences and Business Analytics (with ESSEC Business School) was ranked 3rd worldwide in the QS World University Rankings.

7.1 New platforms

8.1 Severity Prediction of Hospitalized COVID-19 Patients

Participants: Emilie Chouzenoux, Kavya Gupta, Yingping Li, Ana Neascu, Jean-Christophe Pesquet, Hugues Talbot, Matthieu Terris, Sagar Verma (Collaboration: Institut Gustave Roussy ; Kremlin-Bicetre APHP ; Owkin France)

The SARS-COV-2 pandemic has put pressure on intensive care units, so that identifying predictors of disease severity is a priority that we address in the work 22. We collect 58 clinical and biological variables, and chest CT scan data, from 1003 coronavirus-infected patients from two French hospitals. We train a deep learning model based on CT scans to predict severity. We then construct the multimodal AI-severity score that includes 5 clinical and biological variables (age, sex, oxygenation, urea, platelet) in addition to the deep learning model. We show that neural network analysis of CT-scans brings unique prognosis information, although it is correlated with other markers of severity (oxygenation, LDH, and CRP) explaining the measurable but limited 0.03 increase of AUC obtained when adding CT-scan information to clinical variables. Here, we show that when comparing AI-severity with 11 existing severity scores, we find significantly improved prognosis performance; AI-severity can therefore rapidly become a reference scoring approach.

8.2 Influence of the MRI field strength on radiomic features

Participants: Emilie Chouzenoux, Hugues Talbot (Collaboration: Institut Gustave Roussy)

The development and clinical adoption of quantitative imaging biomarkers (radiomics) has established the need for the identification of parameters altering radiomics reproducibility. The aim of the study 5 was to assess the impact of magnetic field strength on magnetic resonance imaging (MRI) radiomics features in neuroradiology clinical practice.

T1 3D SPGR sequence was acquired on two phantoms and 10 healthy volunteers with two clinical MR devices from the same manufacturer using two different magnetic fields (1.5 and 3T). Phantoms varied in terms of gadolinium concentrations and textural heterogeneity. 27 regions of interest were segmented (phantom: 21, volunteers: 6) using the LIFEX software. 34 features were analyzed.

In the phantom dataset, 10 (67%) out of 15 radiomics features were significantly different when measured at 1.5T or 3T (student's t-test, p < 0.05). Gray levels resampling, and pixel size also influence part of texture features. These findings were validated in healthy volunteers. Conclusions: According to daily used protocols for clinical examinations, radiomic features extracted on 1.5T should not be used interchangeably with 3T when evaluating texture features. Such confounding factor should be adjusted when adapting the results of a study to a different platform, or when designing a multicentric trial.

8.3 Artificial Intelligence Data Challenges Based on CT and MRI

Participants: Emilie Chouzenoux (Collaboration: Institut Gustave Roussy)

The second edition of the artificial intelligence (AI) data challenge was organized by the French Society of Radiology with the aim to: (i), work on relevant public health issues; (ii), build large, mul-ticentre, high quality databases; and (iii), include three-dimensional (3D) information and prognostic questions 23. Relevant clinical questions were proposed by French subspecialty colleges of radiology. Their feasibility was assessed by experts in the field of AI. A dedicated platform was set up for inclusion centers to safely upload their anonymized examinations in compliance with general data protection regulation. The quality of the database was checked by experts weekly with annotations performed by radiologists. Multidisciplinary teams competed between September 11 th and October 13 th 2019. Three questions were selected using different imaging and evaluation modalities, including: pulmonary nodule detection and classification from 3D computed tomography (CT), prediction of expanded disability status scale in multiple sclerosis using 3D magnetic resonance imaging (MRI) and segmentation of muscular surface for sarcopenia estimation from two-dimensional CT. A total of 4347 examinations were gathered of which only 6% were excluded. Three independent databases from 24 individual centers were created. A total of 143 participants were split into 20 multidisciplinary teams. Conclusion: Three data challenges with over 1200 general data protection regulation compliant CT or MRI examinations each were organized. Future challenges should be made with more complex situations combining histopathological or genetic information to resemble real life situations faced by radiologists in routine practice.

8.4 SPOQ lp over lq Regularization for Sparse Signal Recovery applied to Mass Spectrometry

Participants: Emilie Chouzenoux, Jean-Christophe Pesquet (Collaboration: Afef Cherni, Université Aix Marseille ; Laurent Duval, IFPEN)

Underdetermined or ill-posed inverse problems require additional information for sound solutions with tractable optimization algorithms. Sparsity yields consequent heuristics to that matter, with numerous applications in signal restoration, image recovery, or machine learning. Since the l0 count measure is barely tractable, many statistical or learning approaches have invested in computable proxies, such as the l1 norm. However, the latter does not exhibit the desirable property of scale invari-ance for sparse data. Generalizing the SOOT Euclidean/Taxicab l1/ l2 norm-ratio initially introduced for blind deconvolution, we propose in 13 SPOQ, a family of smoothed scale-invariant penalty functions. It consists of a Lipschitz-differentiable surrogate for p-over-q quasi-norm/norm ratios with $p\in ]0,2[$ and $q\ge 2$. This surrogate is embedded into a novel majorize-minimize trust-region approach, generalizing the variable metric forward-backward algorithm. For naturally sparse mass-spectrometry signals, we show that SPOQ significantly outperforms l0, l1, Cauchy, Welsch, and CEL0 penalties on several performance measures. Guidelines on SPOQ hyperparameters tuning are also provided, suggesting simple data-driven choices.

8.5 Deep Convolutional Transform-based Fusion Framework for Supervised and Unsupervised Data Processing

Participants: Emilie Chouzenoux (Collaboration: Giovanni Chierchia, ESIEE Paris ; Pooja Gupta, Jyoti Maggu, Angshul Majumdar, IIIT Delhi, India)

The work 19 proposes an unsupervised fusion framework based on deep convolutional transform learning. The great learning ability of convolutional filters for data analysis is well acknowledged. The success of convolutive features owes to the convolutional neural network (CNN). However, CNN cannot perform learning tasks in an unsupervised fashion. In a recent work, we show that such shortcoming can be addressed by adopting a convolutional transform learning (CTL) approach, where convolutional filters are learnt in an unsupervised fashion. The present paper aims at (i) proposing a deep version of CTL, (ii) proposing an unsupervised fusion formulation taking advantage of the proposed deep CTL representation , and (iii) developing a mathematically sounded optimization strategy for performing the learning task. We apply the proposed technique, named DeConFuse, on the problem of stock forecasting and trading. A comparison with state-of-the-art methods (based on CNN and long short-term memory network) shows the superiority of our method for performing a reliable feature extraction.

In the work 20, we propose a supervised multi-channel time-series learning framework for financial stock trading. Although many deep learning models have recently been proposed in this domain, most of them treat the stock trading time-series data as 2-D image data, whereas its true nature is 1-D time-series data. Since the stock trading systems are multi-channel data, many existing techniques treating them as 1-D time-series data are not suggestive of any technique to effectively fusion the information carried by the multiple channels. To contribute towards both of these shortcomings, we propose an end-to-end supervised learning framework inspired by the previously established (unsupervised) convolution transform learning framework. Our approach consists of processing the data channels through separate 1-D convolution layers, then fusing the outputs with a series of fully-connected layers, and finally applying a softmax classification layer. The peculiarity of our framework, that we call SuperDeConFuse (SDCF), is that we remove the nonlinear activation located between the multi-channel convolution layers and the fully-connected layers, as well as the one located between the latter and the output layer. We compensate for this removal by introducing a suitable regularization on the aforementioned layer outputs and filters during the training phase. Specifically, we apply a logarithm determinant regularization on the layer filters to break symmetry and force diversity in the learnt transforms, whereas we enforce the non-negativity constraint on the layer outputs to mitigate the issue of dead neurons. This results in the effective learning of a richer set of features and filters with respect to a standard convolutional neural network. Numerical experiments confirm that the proposed model yields considerably better results than state-of-the-art deep learning techniques for the real-world problem of stock trading.

8.6 Transformed Subspace Clustering

Participants: Emilie Chouzenoux (Collaboration: Giovanni Chierchia, ESIEE Paris ; Jyoti Maggu, Angshul Majumdar, IIIT Delhi, India)

Subspace clustering assumes that the data is separable into separate subspaces. Such a simple assumption, does not always hold. We assume that, even if the raw data is not separable into subspaces, one can learn a representation (transform coefficients) such that the learnt representation is separable into subspaces. To achieve the intended goal, we embed in the work 27, subspace clustering techniques (locally linear manifold clustering, sparse subspace clustering and low rank representation) into transform learning. The entire formulation is jointly learnt; giving rise to a new class of methods called transformed subspace clustering (TSC). In order to account for non-linearity, kernelized extensions of TSC are also proposed. To test the performance of the proposed techniques, benchmarking is performed on image clustering and document clustering datasets. Comparison with state-of-the-art clustering techniques shows that our formulation improves upon them. We then extend the approach in 26 by embedding a deep transform learning approach. The entire formulation is jointly learnt; giving rise to a new class of methods called deeply transformed subspace clustering (DTSC) with assessed performance on image clustering problems.

8.7 Deep Latent Factor Model for Collaborative Filtering

Participants: Emilie Chouzenoux (Collaboration: Aanchal Mongia, Neha Jhamb, Angshul Majumdar, IIIT Delhi, India)

Latent factor models have been used widely in collaborative filtering based recommender systems. In recent years, deep learning has been successful in solving a wide variety of machine learning problems. Motivated by the success of deep learning, we propose in 31 a deeper version of latent factor model. Experiments on benchmark datasets shows that our proposed technique significantly outperforms all state-of-the-art collaborative filtering techniques.

8.8 Convergence of Proximal Gradient Algorithm in the Presence of Adjoint Mismatch

Participants: Emilie Chouzenoux, Jean-Christophe Pesquet, Marion Savanier (Collaboration: Cyril Riddell, Yves Trousset, GE Healthcare)

We consider in 15 the proximal gradient algorithm for solving penalized least-squares minimization problems arising in data science. This first-order algorithm is attractive due to its flexibility and minimal memory requirements allowing to tackle large-scale minimization problems involving non-smooth penalties. However, for problems such as X-ray computed tomography, the applicability of the algorithm is dominated by the cost of applying the forward linear operator and its adjoint at each iteration. In practice, the adjoint operator is thus often replaced by an alternative operator with the aim to reduce the overall computation burden and potentially improve conditioning issues. In this paper, we propose to analyze the effect of such an adjoint mismatch on the convergence of the proximal gradient algorithm in an infinite-dimensional setting, thus generalizing the existing results on PGA. We derive conditions on the step-size and on the gradient of the smooth part of the objective function under which convergence of the algorithm to a fixed point is guaranteed. We also derive bounds on the error between this point and the solution to the original minimization problem. We illustrate our theoretical findings with two image reconstruction tasks in computed tomography.

8.9 Blind Kalman filtering for time series modeling and inference

Participants: Emilie Chouzenoux (Collaboration: Shalini Sharma, Angshul Majumdar, IIIT Delhi, India; Victor Elvira, University of Edinburgh)

Modeling and inference with multivariate sequences is central in a number of signal processing applications such as acoustics, social network analysis, biomedical, and finance, to name a few. The linear-Gaussian state-space model is a common way to describe a time series through the evolution of a hidden state, with the advantage of presenting a simple inference procedure due to the celebrated Kalman filter.

A fundamental question when analyzing multivariate sequences is the search for relationships between their entries (or the modeled hidden states), especially when the inherent structure is a non-fully connected graph. In such context, graphical modeling combined with parsimony constraints allows to limit the proliferation of parameters and enables a compact data representation which is easier to interpret by the experts. In the work 42, we propose a novel expectation-minimization algorithm for estimating the linear matrix operator in the state equation of a linear-Gaussian state-space model. Lasso regularization is included in the M-step, that we solved using a proximal splitting Douglas-Rachford algorithm. Numerical experiments illustrate the benefits of the proposed model and inference technique, named GraphEM, over competitors relying on Granger causality.

In the work 37, we address the problem of short-term load forecasting. We propose a blind Kalman filter algorithm associated to mini-batch processing strategy to allow on-the-fly forecasting. The experimental results show that the proposed method outperforms the state-of-the-art techniques, both on load profile estimation and peak load forecast problems.

8.10 Majorize-Minimize Adapted Metropolis-Hastings Algorithm

Participants: Emilie Chouzenoux, Jean-Christophe Pesquet (Collaboration: Yosra Marnissi, SAFRAN TECH ; Amel Benazza, Supcom Tunis)

The dimension and the complexity of inference problems have dramatically increased in statistical signal processing. It thus becomes mandatory to design improved proposal schemes in Metropolis-Hastings algorithms, providing large proposal transitions that are accepted with high probability. The proposal density should ideally provide an accurate approximation of the target density with a low computational cost. In the paper 30, the authors derive a novel Metropolis-Hastings proposal, inspired from Langevin dynamics, where the drift term is preconditioned by an adaptive matrix constructed through a Majorization-Minimization strategy. We propose several variants of low-complexity curvature metrics applicable to large scale problems. We demonstrate the geometric ergodicity of the resulting chain for the class of super-exponential distributions. The proposed method is shown to exhibit a good performance in two signal recovery examples.

8.11 A Proximal Interior Point Algorithm with Applications to Image Processing

Participants: Emilie Chouzenoux, Marie-Caroline Corbineau, Jean-Christophe Pesquet

In the work 14, we introduce a new proximal interior point algorithm (PIPA). This algorithm is able to handle convex optimization problems involving various constraints where the objective function is the sum of a Lipschitz differentiable term and a possibly nonsmooth one. Each iteration of PIPA involves the minimization of a merit function evaluated for decaying values of a logarithmic barrier parameter. This inner minimization is performed thanks to a finite number of subiterations of a variable metric forward-backward method employing a line search strategy. The convergence of this latter step as well as the convergence the global method itself are analyzed. The numerical efficiency of the proposed approach is demonstrated in two image processing applications.

8.12 Deep Unfolding of a Proximal Interior Point Method for Image Restoration

Participants: Emilie Chouzenoux, Marie-Caroline Corbineau, Jean-Christophe Pesquet (Collaboration: Carla Bertocchi, Marco Prato, Universita di Modena, Italy)

Variational methods are widely applied to ill-posed inverse problems for they have the ability to embed prior knowledge about the solution. However, the level of performance of these methods significantly depends on a set of parameters, which can be estimated through computationally expensive and time consuming methods. In contrast, deep learning offers very generic and efficient architectures, at the expense of explainability, since it is often used as a black-box, without any fine control over its output. Deep unfolding provides a convenient approach to combine variational-based and deep learning approaches. Starting from a variational formulation for image restoration, we developed in 1, iRestNet, a neural network architecture obtained by unfolding a proximal interior point algorithm. Hard constraints, encoding desirable properties for the restored image, are incorporated into the network thanks to a logarithmic barrier, while the barrier parameter, the stepsize, and the penalization weight are learned by the network. We derive explicit expressions for the gradient of the proximity operator for various choices of constraints, which allows training iRestNet with gradient descent and backpropagation. In addition, we provide theoretical results regarding the stability of the network for a common inverse problem example. Numerical experiments on image deblurring problems show that the proposed approach compares favorably with both state-of-the-art variational and machine learning methods in terms of image quality.

8.13 Proximal approaches for matrix optimization problems: Application to robust precision matrix estimation

Participants: Emilie Chouzenoux, Jean-Christophe Pesquet (Collaboration: Alessandro Benfenati, Universita di Milano)

In recent years, there has been a growing interest in mathematical models leading to the minimization, in a symmetric matrix space, of a Bregman divergence coupled with a regularization term. We address problems of this type within a general framework where the regularization term is split into two parts, one being a spectral function while the other is arbitrary. A Douglas–Rachford approach is proposed to address such problems, and a list of proximity operators is provided allowing us to consider various choices for the fit–to–data functional and for the regularization term. Based on our theoretical results, two novel approaches are proposed for the noisy graphical lasso problem, where a covariance or precision matrix has to be statistically estimated in presence of noise. The Douglas–Rachford approach directly applies to the estimation of the covariance matrix. When the precision matrix is sought, we solve a nonconvex optimization problem. More precisely, we propose a majorization–minimization approach building a sequence of convex surrogates and solving the inner optimization subproblems via the aforementioned Douglas–Rachford procedure. We establish conditions for the convergence of this iterative scheme. We illustrate the good numerical performance of the proposed approaches with respect to state–of–the–art approaches on synthetic and real-world datasets 8.

8.14 Graph Representation Learning with Exponential Family Distributions

Participants: Fragkiskos Malliaros, Abdulkadir Çelikkanat

Graph representation learning (GRL) methods aim to map each vertex into a low dimensional space by preserving both local and global structure of a given network. A widely applied graph representation learning paradigm is based on the combination of random walks for sampling context nodes and the traditional Skip-Gram model to capture center-context node relationships. We have emphasized on exponential family distributions to capture rich interaction patterns between nodes in random walk sequences. We have introduced the generic exponential family graph embedding model, that generalizes random walk-based network representation learning techniques to exponential family conditional distributions 40. We have studied three particular instances of this model, analyzing their properties and showing their relationship to existing unsupervised learning models. The experimental evaluation on real-world datasets has demonstrated that the proposed techniques outperform well-known baseline methods in two downstream machine learning tasks.

8.15 Influence Estimation and Maximization with Graph Representation Learning

Participants: Fragkiskos Malliaros (Collaboration: George Panagopoulos, Michalis Vazirgiannis, École Polytechnique)

Finding a set of users that can maximize the spread of information in a complex network is an important problem in social media analysis—being a critical part of several real-world applications such as viral marketing, political advertising, and epidemiology. Although influence maximization has been studied extensively in the past, the majority of works focus on the algorithmic aspect of the problem, overlooking several practical improvements that can be derived by data-driven observations or the inclusion of machine learning. Motivated by the recent criticism on diffusion models and the galloping advancements in influence learning, we have proposed IMINFECTOR (Influence Maximization with INFluencer vECTORs) 32, a method that uses representations learned from diffusion cascades to perform model-independent influence maximization (this is an extended version of the CELFIE model 57). The first part of the methodology is a multi-task neural network that learns embeddings of nodes that initiate cascades (influencer vectors) and embeddings of nodes that participate in them (susceptible vectors). The norm of an influencer vector captures a node's aptitude to initiate lengthy cascades and is used to reduce the number of candidate seeds. The combination of influencer and susceptible vectors form the diffusion probabilities between nodes. These are used to reformulate the computation of the influence spread and propose a greedy solution to influence maximization that retains the theoretical guarantees. We have applied our method in three sizable datasets and evaluate it using cascades from future time steps. IMINFECTOR's scalability and accuracy outperform various competitive algorithms and metrics from the diverse landscape of influence maximization.

8.16 Tools for Document Similarity via the Word Mover's Distance

Participants: Fragkiskos Malliaros (Collaboration: Konstantinos Skianis, Michalis Vazirgiannis, École Polytechnique; Nikolaos Tziortziotis, Jellyfish)

Measuring distance between documents has always been a key component in many natural language processing tasks, such as biomedical document classification, machine translation, question answering, and text generation. Nevertheless, the task poses various challenges, making it not trivial; whether two documents are similar or not, is not always clear and may vary from application to application. Word embeddings have opened a new path in creating novel approaches for addressing traditional problems in the natural language processing (NLP) domain. Nevertheless, using word embeddings to compare text documents remains a relatively unexplored topic—with Word Mover's Distance (WMD) being the prominent tool used so far. In the work 61, we present a variety of tools that can further improve the computation of distances between documents based on WMD. We have demonstrateed that, alternative stopwords, cross document-topic comparison, deep contextualized word vectors and convex metric learning, constitute powerful tools that can boost WMD.

8.17 Dynamic Monitoring of Medical Imaging Software Use with Recurrent Neural Networks

Participants: Fragkiskos Malliaros (Collaboration: Chloé Adam, Paul-Henry Cournède, MICS, CentraleSupélec; Antoine Aliotti, GE Healthcare)

User interaction with a software may be formalized as a sequence of actions. We have proposed two methods – based on different representations of input actions – to address two distinct industrial issues: next action prediction and software crash risk detection 4. Both methods take advantage of the recurrent structure of Long Short Term Memory neural networks to capture dependencies among our sequential data as well as their capacity to potentially handle different types of input representations for the same data. Given the history of user actions in the interface, our first method aims at predicting the next action. The proposed recurrent neural network outperforms state-of-the-art proactive user interface algorithms with standard one-hot vectors as inputs. Using the same data set, the second method aims at crash risk detection. To address this task, we have proposed to use feature vectors composed of actions with above average crash probabilities as inputs of the LSTM—with the idea to take advantage of its ability to learn relevant past information to detect crash patterns. We have observed that the method outperforms state-of-the-art sequence classification methods. Our approaches have been demonstrated on medical imaging software logs from ten different hospitals worldwide, though they might be applied to various user interfaces in a wide range of applications.

8.18 Robustness of Neural Networks

Participants: Ana Neacsu, Kavya Gupta, Fragkiskos Malliaros, Jean-Christophe Pesquet (Collaboration: Patrick Combettes, North Caroline State University; Corneliu Burileanu, Polithenica University of Bucharest)

Obtaining sharp Lipschitz constants for feed-forward neural networks is essential to assess their robustness in the face of perturbations of their inputs. In 16, we derive such constants in the context of a general layered network model involving compositions of nonexpansive averaged operators and affine operators. By exploiting this architecture, our analysis finely captures the interactions between the layers, yielding tighter Lipschitz constants than those resulting from the product of individual bounds for groups of layers. The proposed framework is shown to cover in particular many practical instances encountered in feed-forward neural networks. Our Lipschitz constant estimates are further improved in the case of structures employing scalar nonlinear functions, which include standard convolutional networks as special cases.

In [hal-03140282], we show how to practically build positive neural networks with a given Lipschitz constant bound. An application to automatic gesture recognition based on EMG signals is demonstrated. In 53, this robust design method is extended to a new class of neural networks which constitute an intermediate solution between fully connected networks and convolutive ones. An application to audio denoising is presented.

8.19 Prediction of Time Series with Neural Networks

Participants: Marc Castella, Jean-Christophe Pesquet, Sagar Verma

Neural networks offer new perspectives in terms of prediction of time series. We are especially interested in signals related to electrical motors, which are the most important source of mechanical energy in the industrial world. Their modeling traditionally relies on a physics-based approach, which aims at taking their complex internal dynamics into account. In 65, we explore the feasibility of modeling the dynamics of an electrical motor by following a data-driven approach, which uses only its inputs and outputs and does not make any assumption on its internal behaviour. We propose a novel encoder-decoder architecture which benefits from recurrent skip connections. We also propose a novel loss function that takes into account the complexity of electrical motor quantities and helps in avoiding model bias. We show that the proposed architecture can achieve a good learning performance on our high-frequency high-variance datasets. Two datasets are considered: the first one is generated using a simulator based on the physics of an induction motor and the second one is recorded from an industrial electrical motor. We benchmark our solution using variants of traditional neural networks like feedforward, convolutional, and recurrent networks. We evaluate various design choices of our architecture and compare it to the baselines. We show the domain adaptation capability of our model to learn dynamics just from simulated data by testing it on the raw sensor data. We finally show the effect of signal complexity on the proposed method ability to model temporal dynamics.

The work in 64 focuses on the quantitative analysis of deep neural networks used in data-driven modeling of induction motor dynamics. With the availability of a large amount of data generated by industrial sensor networks, it is now possible to train deep neural networks. Recently researchers have started exploring the usage of such networks for physics modeling, online control, monitoring, and fault prediction in induction motor operations. We consider the problem of estimating speed and torque from currents and voltages of an induction motor. Neural networks provide quite good performance for this task when analysed from a machine learning perspective using standard metrics. We show, however, that there are some caveats in using machine learning metrics to analyze a neural network model when applied to induction motor problems. Given the mission-critical nature of induction motor operations, the performance of neural networks has to be validated from an electrical engineering point of view. To this end, we evaluate several traditional neural network architectures and recent state of the art architectures on dynamic and quasi-static benchmarks using electrical engineering metrics.

8.20 Learning Maximally Monotone Operators for Image Recovery

Participant: Jean-Christophe Pesquet (Collaboration: Audrey Repetti, Matthieu Terris, Yves Wiaux, Heriot Watt University, UK)

In the work 78, the authors introduce a new paradigm for solving regularized variational problems. These are typically formulated to address ill-posed inverse problems encountered in signal and image processing. The objective function is traditionally defined by adding a regularization function to a data fit term, which is subsequently minimized by using iterative optimization algorithms. Recently, several works have proposed to replace the operator related to the regularization by a more sophisticated denoiser. These approaches, known as plug-and-play (PnP) methods, have shown excellent performance. Although it has been noticed that, under nonexpansiveness assumptions on the denoisers, the convergence of the resulting algorithm is guaranteed, little is known about characterizing the asymptotically delivered solution. In the current article, we propose to address this limitation. More specifically, instead of employing a functional regularization, we perform an operator regularization, where a maximally monotone operator (MMO) is learned in a supervised manner. This formulation is flexible as it allows the solution to be characterized through a broad range of variational inequalities, and it includes convex regularizations as special cases. From an algorithmic standpoint, the proposed approach consists in replacing the resolvent of the MMO by a neural network (NN). We provide a universal approximation theorem proving that nonexpansive NNs provide suitable models for the resolvent of a wide class of MMOs. The proposed approach thus provides a sound theoretical framework for analyzing the asymptotic behavior of first-order PnP algorithms. In addition, we propose a numerical strategy to train NNs corresponding to resolvents of MMOs. We apply our approach to image restoration problems and demonstrate its validity in terms of both convergence and quality.

8.21 Sparse signal reconstruction for nonlinear models via piecewise rational optimization

Participants: Marc Castella, Arthur Marmin, Jean-Christophe Pesquet (Collaboration: Laurent Duval, IFPEN)

In the work 28, the authors propose a method to reconstruct sparse signals degraded by a nonlinear distortion and acquired at a limited sampling rate. Our method formulates the reconstruction problem as a nonconvex minimization of the sum of a data fitting term and a penalization term. In contrast with most previous works which settle for approximated local solutions, we seek for a global solution to the obtained challenging nonconvex problem. Our global approach relies on the so-called Lasserre relaxation of polynomial optimization. We here specifically include in our approach the case of piecewise rational functions, which makes it possible to address a wide class of nonconvex exact and continuous relaxations of the l0 penalization function. Additionally, we study the complexity of the optimization problem. It is shown how to use the structure of the problem to lighten the computational burden efficiently. Finally, numerical simulations illustrate the benefits of our method in terms of both global optimality and signal reconstruction.

8.22 Deep Neural Network Structures Solving Variational Inequalities

Participant: Jean-Christophe Pesquet (Collaboration: Patrick Louis Combettes, North Carolina University, USA)

Motivated by structures that appear in deep neural networks, we investigate nonlinear composite models alternating proximity and affine operators defined on different spaces. We first show that a wide range of activation operators used in neural networks are actually proximity operators. We then establish conditions for the averagedness of the proposed composite constructs and investigate their asymptotic properties. It is shown that the limit of the resulting process solves a variational inequality which, in general, does not derive from a minimization problem 2.

8.23 Global Optimization for Recovery of Clipped Signals Corrupted with Poisson-Gaussian Noise

Participant: Marc Castella, Arthur Marmin, Jean-Christophe Pesquet (Collaboration: Anna Jezierska, Polish Academy of Sciences, Warsaw, Poland)

In the work 29, the authors study a variational formulation for reconstructing nonlinearly distorted signals corrupted with a Poisson-Gaussian noise. In this situation, the data fidelity term consists of a sum of a weighted least squares term and a logarithmic one. Both of them are precomposed by a nonlinearity, modelling a clipping effect, which is assumed to be rational. A regularization term, being a piecewise rational approximation of the 0 function provides a suitable sparsity measure with respect to a preset linear operator. We propose a global optimization approach for such a problem. More specifically, it is first transformed into a generalized moment problem by introducing some auxiliary variables. Then, a hierarchy of semidefinite programming relaxations is built. Numerical examples show the good performance of the proposed approach.

8.24 Coupled Feature Learning for Multimodal Medical Image Fusion

Participant: Nora Ouzir (Collaboration: F. Veshki, S. Vorobyov, and E. Ollila)

Multimodal image fusion aims to combine relevant information from images acquired with different sensors. In medical imaging, fused images play an essential role in both standard and automated diagnosis. In 83, the authors propose a novel image fusion method based on coupled dictionary learning. Unlike many current medical fusion methods, the proposed approach does not suffer from intensity attenuation nor loss of critical information. The images are decomposed into coupled and independent components obtained using sparse representations with identical supports and a Pearson correlation constraint. An alternating minimization algorithm is designed to address the resulting optimization problem. Experiments are conducted using various multimodal inputs, such as in vivo MR-CT and MR-PET images.

8.25 Robust Registration of Multi-modal Medical Images Using Huber's Criterion

Participant: Nora Ouzir (Collaboration: E. Ollila and S. Vorobyov)

Registration of multi-modal medical images is an essential pre-processing step, for example, for fusion or image guided-interventions. However, the alignment process is prone to high variability in tissue appearance between modalities, in addition to local intensity variations and artefacts. The work in 56 introduces a robust approach that mitigates the undesirable effect of such variability. We propose a novel method using Huber's criterion, which enables a jointly convex estimation of the motions and the associated scale parameters. The problem is formulated as a complex 2D transformation estimation and a novel dictionary learning-based data fidelity term is proposed.

8.26 Shaping for PET Image Analysis

Participant: Hugues Talbot (Collaboration: Eloïse Grossiord, Laurent Najman, ESIEE Paris ; Benoit Naegel, iCube, Strasbourg ; Nicolas Passat, CRESTIC, Reims)

Component-trees constitute an efficient data structure for hierarchical image modeling. In particular they are relevant for processing and analyzing images where the structures of interest correspond either to local maxima or local minima of intensity. This is indeed the case of functional data in medical imaging. This motivates the use of component-tree-based approaches for analyzing Positron Emission Tomography (PET) images in the context of oncology. In this article 18, we present a simple, yet efficient, methodological framework for PET image analysis based on component-trees. More precisely, we show that the second-order paradigm of shaping, that broadly consists of computing the component-tree of a component-tree, provides a relevant way of generalizing the threshold-based strategies classically used by medical practitioners for handling PET images. In addition, it also allows to embed relevant priors regarding the sought cancer lesions.

8.27 Myocardial Perfusion Simulation for Coronary Artery Disease

Participant: Hugues Talbot (Collaboration: L. Najman ESIEE Paris ; I. Vignon-Clementel, REO Team leader, Inria ; C. Taylor, Heartflow Inc.)

Patient-specific models of blood flow are being used clinically to diagnose and plan treatment for coronary artery disease. A remaining challenge is bridging scales from flow in arteries to the micro-circulation supplying the myocardium. Previously proposed models are descriptive rather than predictive and have not been applied to human data. In this article 33, we develop a multiscale patient-specific model enabling blood flow simulation from large coronary arteries to myocardial tissue. Patient vasculatures are segmented from coronary computed tomography angiography data and extended from the image-based model down to the arteriole level using a space-filling forest of synthetic trees. Blood flow is modeled by coupling a 1D model of the coronary arteries to a single-compartment Darcy myocardium model. Simulated results on five patients with non-obstructive coronary artery disease compare overall well to $H_2^{15}O$ PET exam data for both resting and hyperemic conditions. Results on a patient with severe obstructive disease link coronary artery narrowing with impaired myocardial blood flow, demonstrating the model's ability to predict myocardial regions with perfusion deficit. This is the first report of a computational model for simulating blood flow from the epicardial coronary arteries to the left ventricle myocardium applied to and validated on human data.

8.28 In-vivo Quantification of Skin Ageing

Participant: Hugues Talbot (Collaboration: Laboratoires Clarins)

Reflectance confocal microscopy (RCM) is a powerful tool to visualize the skin layers at cellular resolution up to a depth of 200$\mu$m. A semi-quantitative score of skin aging from RCM images has been previously published, requiring visual assessment of the images by experienced dermatologists. In this article 34, we propose new computer-based methods to automatically quantify the skin aging process on RCM image.

8.29 Hyperprogressive Disease During Immunotherapy

Participant: Baptiste Kas, Hugues Talbot (Collaboration: Institut Gustave Roussy)

Hyperprogressive disease (HPD) is an aggressive pattern of progression reported for patients treated with programmed cell death 1 (PD-1)/programmed cell death 1 ligand (PD-L1) inhibitors as a single agent in several studies. However, the use of different definitions of HPD introduces the risk of describing different tumoral behaviors. In this paper 21, we assess the accuracy of each HPD definition to identify the frequency of HPD and the association with poorer outcomes of immune-checkpoint inhibitor (ICI) treatment in patients with advanced non-small cell lung cancer (NSCLC) and to provide an optimized and homogenized definition based on all previous criteria for identifying HPD. The findings of this retrospective cohort study of patients with NSCLC suggest that the previous five definitions of HPD were not associated with the same tumor behavior. A new definition, based on a more physical model of tumor growth rate, appeared to be associated with the characteristics expected with HPD (increase of the tumor kinetics and poor survival).

8.30 Bio-particle Sizing

Participant: Hugues Talbot (Collaboration: Nanyang Technological University, Singapore)

High accuracy measurement of size is essential in physical and biomedical sciences. Various sizing techniques have been widely used in sorting colloidal materials, analyzing bioparticles and monitoring the qualities of food and atmosphere. Most imaging-free methods such as light scattering measure the averaged size of particles and have difficulties in determining non-spherical particles. Imaging acquisition using camera is capable of observing individual nanoparticles in real time, but the accuracy is compromised by the image defocusing and instrumental calibration. In this work, a machine learning-based pipeline is developed to facilitate a high accuracy imaging-based particle sizing. In this article 25, we propose a pipeline consisting of an image segmentation module for cell identification and a machine learning model for accurate pixel-to-size conversion. The results manifest a significantly improved accuracy, showing great potential for a wide range of applications in environmental sensing, biomedical diagnostical, and material characterization.

8.31 Position Estimation of 3D Fluorescent Spherical Beads

Participant: Hugues Talbot (Collaboration: Alessandro Benfenati, Universita di Milano, Francesco Bonacci, Laboratoire Navier, Tarik Bourouina, ESIEE Paris)

Particle estimation is a classical problem arising in many science fields, such as biophysics, fluid mechanics and biomedical imaging. Many interesting applications in these areas involve 3D imaging data: In this work 7, we present a technique to estimate the 3D coordinates of the center of spherical particles. This procedure has its core in the processing of the images of the scanned volume: It firstly applies denoising techniques to each frame of the scanned volume and then provides an estimation of both the center and the profile of the 2D intersections of the particles with the frames, by coupling the usage of Total Variation functional and of a regularized weighted Least Squares fit. Then, the 2D information is used to retrieve the 3D coordinates using geometrical properties. The experiments provide evidence that image denoising has a large impact on the performance of the particle tracking procedures, since they strongly depend on the quality of the initial acquisition. This work shows that the choice of tailored image denoising technique for Poisson noise leads to a better estimation of the particle positions

8.32 An Elastica-driven Digital Curve Evolution Model for Image Segmentation

Participant: Hugues Talbot (Collaboration: University Grenoble-Alpes Savoie Mont-Blanc)

Geometric priors have been shown to be useful in image segmentation to regularize the results. For example, the classical Mumford-Shah functional uses region perimeter as prior. This has inspired much research in the last few decades, with classical approaches like the Rudin-Osher-Fatemi and most graph-cut formulations, which all use a weighted or binary perimeter prior. It has been observed that this prior is not suitable in many applications, for example for segmenting thin objects or some textures, which may have high perimeter/surface ratio.Mumford observed that an interesting prior for natural objects is the Euler elastical model, which involves the squared curvature. In other areas of science, researchers have noticed that some physical binarization processes, like emulsion unmixing, can be well-approximated by curvature-related flow like the Willmore flow. However, curvature-related flows are not easy to compute because curvature is difficult to estimate accurately, and the underlying optimization processes are not convex. In this article 6, we propose to formulate a digital flow that approximates an Elastica-related flow using a multigrid-convergent curvature estimator, within a discrete variational framework. We also present an application of this model as a post-processing step to a segmentation framework

8.33 Segmentation of White Matter in Thick 3d Confocal Images of Eebrafish

Participant: Hugues Talbot (Collaboration: Paris-Saclay TEFOR institute, CNRS)

Tissue clearing methods have boosted the microscopic observations of thick samples such as wholemount mouse or zebrafish. However, even with the best tissue clearing methods, specimens are not completely transparent and light attenuation increases with depth, reducing signal output and signal-to-noise ratio. In addition, since tissue clearing and microscopic acquisition techniques have become faster, automated image analysis is now an issue. In this context, mounting specimens at large scale often leads to imperfectly aligned or oriented samples, which makes relying on predefined, sample-independent parameters to correct signal attenuation impossible. In this article 24, we propose a sample-dependent method for contrast correction. It relies on segmenting the sample, and estimating sample depth isosurfaces that serve as reference for the correction. We segment the brain white matter of zebrafish larvae. We show that this correction allows a better stitching of opposite sides of each larva, in order to image the entire larva with a high signal-to-noise ratio throughout. We also show that our proposed contrast correction method makes it possible to better recognize the deep structures of the brain by comparing manual vs. automated segmentations. This is expected to improve image observations and analyses in high-content methods where signal loss in the samples is significant.

8.34 Multiband Astronomical Source Detection

Participant: Hugues Talbot (ERC SUNDIAL ITN Grant, collaboration with 7 other institutes in Europe)

Component-graphs provide powerful and complex structures for multi-band image processing. In this article 55, we propose a multiband astronomical source detection framework with the component-graphs relying on a new set of component attributes. We propose two modules to differentiate nodes belong to distinct objects and to detect partial object nodes. Experiments demonstrate an improved capacity at detecting faint objects on a multi-band astronomical dataset.

8.35 Superresolution and OCR

Participant: Hugues Talbot (Collaboration: Artefact)

Recently, many deep learning methods have been used to handle single image super-resolution (SISR) tasks and often achieve state-of-the-art performance. From a visual point of view, the results look convincing. Yet, does it mean that those techniques are reliable and robust enough to be implemented in real business cases to enhance the performance of other computer vision tasks? In this article 58, we investigate the use of SISR to construct higher-resolution images of real receipt photos sent by the customers of a company and evaluate its impact on the performance of an OCR task (receipt information retrieval). Using built-in task-based performance evaluation methods, we show that the use of SISR can significantly improve OCR performance in the case where recognition was poor in low-resolution, but can also deteriorate the performance for receipts that were already successfully recognized. As a conclusion, we provide recommendations on how to best use SISR in a production environment.

8.36 Artificial Intelligence for COVID-19 Disease Quantification

Participants: Maria Vakalopoulou (Collaboration: AP-HP - Hopital Cochin Broca Hotel Dieu; Therapanacea)

Coronavirus disease 2019 (COVID-19) emerged in 2019 and disseminated around the world rapidly. Computed tomography (CT) imaging has been proven to be an important tool for screening, disease quantification and staging. The latter is of extreme importance for organizational anticipation (availability of intensive care unit beds, patient management planning) as well as to accelerate drug development through rapid, reproducible and quantified assessment of treatment response. Even if currently there are no specific guidelines for the staging of the patients, CT together with some clinical and biological biomarkers are used. In 10, we collected a multi-center cohort and we investigated the use of medical imaging and artificial intelligence for disease quantification, staging and outcome prediction. Our approach relies on automatic deep learning-based disease quantification using an ensemble of architectures, and a data-driven consensus for the staging and outcome prediction of the patients fusing imaging biomarkers with clinical and biological attributes. Highly promising results on multiple external/independent evaluation cohorts as well as comparisons with expert human readers demonstrate the potentials of our approach.

8.37 Artificial Intelligence Applications for Thoracic imaging

Participants: Maria Vakalopoulou (Collaboration: AP-HP - Hopital Cochin Broca Hotel Dieu; Therapanacea)

Artificial intelligence is a hot topic in medical imaging. The development of deep learning methods and in particular the use of convolutional neural networks (CNNs), have led to substantial performance gain over the classic machine learning techniques. Multiple usages are currently being evaluated, especially for thoracic imaging, such as such as lung nodule evaluation, tuberculosis or pneumonia detection or quantification of diffuse lung diseases. Chest radiography is a near perfect domain for the development of deep learning algorithms for automatic interpretation,requiring large annotated datasets, in view of the high number of procedures and increasing data availability. Current algorithms are able to detect up to 14 common anomalies, when present as isolated findings. Chest computed tomography is another major field of application for artificial intelligence, especially in the perspective of large scale lung cancer screening. It is important for radiologists to apprehend, contribute actively and lead this new era of radiology powered by artificial intelligence. Such a perspective requires understanding new terms and concepts associated with machine learning. The objective of this study 11 is to provide useful definitions for understanding the methods used and their possibilities, and report current and future developments for thoracic imaging.Prospective validation of AI tools will be required before reaching routine clinical implementation.

Finally, in 74, we develop a deep learning algorithm for the automatic assessment of the extent of systemic sclerosis (SSc)-related interstitial lung disease (ILD) on chest CT images. In particular, a multicomponent deep neural network (AtlasNet) was trained on 6888 fully annotated CT images ($80\%$ for training and $20\%$ for validation) from 17 patients with no, mild, or severe lung disease. The model was tested on a dataset of 400 images from another 20 patients, independently partially annotated by three radiologist readers. The ILD contours from the three readers and the deep learning neural network were compared by using the Dice similarity coefficient (DSC). The correlation between disease extent obtained from the deep learning algorithm and that obtained by using pulmonary function tests (PFTs) was then evaluated in the remaining 171 patients and in an external validation dataset of 31 patients based on the analysis of all slices of the chest CT scan. The Spearman rank correlation coefficient ($\rho$) was calculated to evaluate the correlation between disease extent and PFT results.

8.38 Learning-based Registration of Medical Imaging

Participants: Théo Estienne, Enzo Battistella, Marvin Lerousseau, Roger Sun, Hugue Talbot, Maria Vakalopoulou (Collaboration: Institut Gustave Roussy; Therapanacea)

Image registration and segmentation are the two most studied problems in medical image analysis. Deep learning algorithms have recently gained a lot of attention due to their success and state-of-the-art results in variety of problems and communities. In 17, we propose a novel, efficient, and multi-task algorithm that addresses the problems of image registration and brain tumor segmentation jointly. Our method exploits the dependencies between these tasks through a natural coupling of their interdependencies during inference. In particular, the similarity constraints are relaxed within the tumor regions using an efficient and relatively simple formulation. We evaluated the performance of our formulation both quantitatively and qualitatively for registration and segmentation problems on two publicly available datasets (BraTS 2018 and OASIS 3), reporting competitive results with other recent state-of-the-art methods. Moreover, our proposed framework reports significant amelioration (p $<0.005$) for the registration performance inside the tumor locations, providing a generic method that does not need any predefined conditions (e.g. absence of abnormalities) about the volumes to be registered.

Moreover, in 74, we summarize our work presented on Learn2Reg challenge 2020. The main contributions of our work rely on (i) a symmetric formulation, predicting the transformations from source to target and from target to source simultaneously, enforcing the trained representations to be similar and (ii) integration of variety of publicly available datasets used both for pretraining and for augmenting segmentation labels. Our method reports a mean dice of $0.64$ for task 3 and $0.85$ for task 4 on the test sets, taking third place on the challenge.

8.39 Radiomics and Pathomics using Advanced Machine and Deep Learning Approaches

Participants: Théo Estienne, Enzo Battistella, Marvin Lerousseau, Roger Sun, Hugues Talbot, Maria Vakalopoulou (Collaboration: Institut Gustave Roussy; University of Lyon; Therapanacea)

Combining radiotherapy (RT) with immuno-oncology (IO) therapy (IORT) may enhance IO-induced antitumor response. Quantitative imaging biomarkers can be used to provide prognosis, predict tumor response in a non-invasive fashion and improve patient selection for IORT. A biologically inspired CD8 T-cells-associated radiomics signature has been developed on previous cohorts. In 38 we evaluated whether this CD8 radiomic signature is associated with lesion response, whether it may help to assess disease spatial heterogeneity for predicting outcomes of patients treated with IORT. We also evaluated differences between irradiated and non-irradiated lesions.

Histopathological image segmentation is a challenging and important topic in medical imaging with tremendous potential impact in clinical practice. State of the art methods rely on hand-crafted annotations which hinder clinical translation since histology suffers from significant variations between cancer phenotypes. In 46, we propose a weakly supervised framework for whole slide imaging segmentation that relies on standard clinical annotations, available in most medical systems. In particular, we exploit a multiple instance learning scheme for training models. The proposed framework has been evaluated on multi-locations and multi-centric public data from The Cancer Genome Atlas and the PatchCamelyon dataset. Promising results when compared with experts' annotations demonstrate the potentials of the presented approach.

Segmentation and accurate localization of nuclei in histopathological images is a very challenging problem, with most existing approaches adopting a supervised strategy. These methods usually rely on manual annotations that require a lot of time and effort from medical experts. In 68, we present a self-supervised approach for segmentation of nuclei for whole slide histopathology images. Our method works on the assumption that the size and texture of nuclei can determine the magnification at which a patch is extracted. We show that the identification of the magnification level for tiles can generate a preliminary self-supervision signal to locate nuclei. We further show that by appropriately constraining our model it is possible to retrieve meaningful segmentation maps as an auxiliary output to the primary magnification identification task. Our experiments show that with standard post-processing, our method can outperform other unsupervised nuclei segmentation approaches and report similar performance with supervised ones on the publicly available MoNuSeg dataset.

Finally, a novel approach to combine medical imaging with additional clinical information is presented in 79. In particular, we investigate the use of recent advances in deep learning and propose an end-to-end trainable multi-instance convolutional neural network within a mixture-of-experts formulation that combines information from two types of data-images and clinical attributes-for the diagnosis of lymphocytosis. The convolutional network learns to extract meaningful features from images of blood cells using an embedding level approach and aggregates them. Moreover, the mixture-of-experts model combines information from these images as well as clinical attributes to form an end-to-end trainable pipeline for diagnosis of lymphocytosis. Our results demonstrate that even the convolutional network by itself is able to discover meaningful associations between the images and the diagnosis, indicating the presence of important unexploited information in the images. The mixture-of-experts formulation is shown to be more robust while maintaining performance via. a repeatability study to assess the effect of variability in data acquisition on the predictions. The proposed methods are compared with different methods from literature based both on conventional handcrafted features and machine learning, and on recent deep learning models based on attention mechanisms. Our method reports a balanced accuracy of $85.41\%$ and outperforms the handcrafted feature-based and attention-based approaches as well that of biologists which scored $79.44\%$, $82.89\%$ and $77.07\%$ respectively. These results give insights on the potentials of the applicability of the proposed method in clinical practice.

9.1 Bilateral contracts with industry

• PhD Contract with IFP Energies Nouvelles
  • Project title: Polynomial optimization for sparse signal recovery
  • Duration: 2018-2020
  • Leaders: M. Castella and J.-C. Pesquet
• PhD Contract with IFP Energies Nouvelles
  • Project title: Seismic signal analysis by using neural networks
  • Duration: 2019-2022
  • Leaders: A. Fraysse and J.-C. Pesquet
• PhD Contract with Thales Group
  • Project title: Neural network solutions for safety of complex systems
  • Duration: 2019-2022
  • Responsible: J.-C. Pesquet and F. Malliaros
• PhD Contract with General Electric Healthcare
  • Project title: Minimally invasive assessement of coronary disease
  • Duration: 2018-2021
  • Leader: Hugues Talbot
• PhD Contract with General Electric Healthcare
  • Project title: Optimization methods for breast tomosynthesis
  • Duration: 2017-2020
  • Leader: J.-C. Pesquet and E. Chouzenoux
• PhD Contract with General Electric Healthcare
  • Project title: Reconstruction 3D interventionnelle
  • Duration: 2019-2022
  • Leader: J.-C. Pesquet and E. Chouzenoux
• PhD Contract with IFP Energies nouvelles
  • Project title: Graph-based learning from integrated multi-omics and multi-species data
  • Duration: 2019-2022
  • Leader: F. Malliaros and J.-C. Pesquet
• Contract with Schneider Electric
  • Project title: Neural network modeling of electrical motors
  • Duration: 2019
  • Leader: J.-C. Pesquet
• Contract with SNCF
  • Project title: SNCF Platipus: Examining the potential of machine learning algorithms in the analysis of scouring reports of aquatic foundations.
  • Duration: 2019-2020
  • Leader: F. Malliaros, M. Vakalopoulou.
• Contract with ESSILOR
  • Project title: Software development for photo-refraction analysis
  • Duration: 2020-2021
  • Leader: E. Chouzenoux, J.C. Pesquet

10.1 International initiatives

10.2 International research visitors

10.3 European initiatives

10.4 National initiatives

10.5 Regional initiatives

• DATAIA UltraBioLearn (2019-2022). The project aims to research machine learning approaches for medical applications, in particular by leveraging semi-supervised learning using generative, graphbased and certifiable networks, in the context of predicting patient response to cancer treatments. Responsible: H. Talbot, F. Malliaros (N. Lassau, Institut Gustave Roussy).

11.1 Promoting scientific activities

11.2 Teaching - Supervision - Juries

11.3 Popularization

12.1 Major publications

• 1 article CarlaC. Bertocchi, EmilieE. Chouzenoux, Marie-CarolineM.-C. Corbineau, Jean-ChristopheJ.-C. Pesquet and MarcoM. Prato. Deep Unfolding of a Proximal Interior Point Method for Image Restoration Inverse Problems 2020
  HAL DOI back to text
• 2 article Patrick LP. Combettes and Jean-ChristopheJ.-C. Pesquet. Deep Neural Network Structures Solving Variational Inequalities * Set-Valued and Variational Analysis February 2020
  HAL DOI back to text back to text
• 3 article FragkiskosF. Malliaros, ChristosC. Giatsidis, Apostolos NA. Papadopoulos and MichalisM. Vazirgiannis. The Core Decomposition of Networks: Theory, Algorithms and Applications The VLDB Journal 2019
  HAL back to text

12.2 Publications of the year

12.3 Cited publications