6.1.1 Cyanure

• Name: Cyanure: An Open-Source Toolbox for Empirical Risk Minimization
• Keyword: Machine learning
• Functional Description: Cyanure is an open-source C++ software package with a Python interface. The goal of Arsenic is to provide state-of-the-art solvers for learning linear models, based on stochastic variance-reduced stochastic optimization with acceleration mechanisms and Quasi-Newton principles. Arsenic can handle a large variety of loss functions (logistic, square, squared hinge, multinomial logistic) and regularization functions (l2, l1, elastic-net, fused Lasso, multi-task group Lasso). It provides a simple Python API, which is very close to that of scikit-learn, which should be extended to other languages such as R or Matlab in a near future.
• Release Contributions: version initiale
• URL: http://thoth.inrialpes.fr/people/mairal/arsenic/welcome.html
• Author: Julien Mairal
• Contact: Julien Mairal
• Participant: Julien Mairal

6.1.2 graph-ckn

• Name: Convolutional Kernel Networks for Graph-Structured Data
• Keyword: Machine learning
• Functional Description: This is an open-source software package that reproduces the results of the ICML paper "Convolutional Kernel Networks for Graph-Structured Data". It addresses classification and regression problems when data points are graphs.
• URL: https://github.com/claying/GCKN
• Author: Dexiong Chen
• Contact: Julien Mairal

6.1.3 non-local-sparse-coding

• Name: Fully Trainable and Interpretable Non-Local Sparse Models for Image Restoration.
• Keyword: Image processing
• Functional Description: This is an open-source implementation of the ECCV 2020 paper " Fully Trainable and Interpretable Non-Local Sparse Models for Image Restoration.", which reproduces their results.
• URL: https://github.com/bruno-31/groupsc
• Contact: Julien Mairal

Discrete Point Flow Networks for Efficient Point Cloud Generation

Participants: Roman Klokov, Edmond Boyer, Jakob Verbeek.

Generative models have proven effective at modeling 3D shapes and their statistical variations. In this paper 19, we investigate their application to point clouds, a 3D shape representation widely used in computer vision for which, however, only few generative models have yet been proposed. We introduce a latent variable model, depicted in Figure 1, that builds on normalizing flows with affine coupling layers to generate 3D point clouds of an arbitrary size given a latent shape representation. To evaluate its benefits for shape modeling we apply this model for generation, autoencoding, and single-view shape reconstruction tasks. We improve over recent GAN-based models in terms of most metrics that assess generation and autoencoding. Compared to recent work based on continuous flows, our model offers a significant speedup in both training and inference times for similar or better performance. For single-view shape reconstruction we also obtain results on par with state-of-the-art voxel, point cloud, and mesh-based methods.

Unsupervised Learning of Visual Features by Contrasting Cluster Assignments

Participants: Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, Armand Joulin.

Unsupervised image representations have significantly reduced the gap with supervised pretraining, notably with the recent achievements of contrastive learning methods. These contrastive methods typically work online and rely on a large number of explicit pairwise feature comparisons, which is computationally challenging. In this paper 7, we propose an online algorithm, SwAV, that takes advantage of contrastive methods without requiring to compute pairwise comparisons. Specifically, our method simultaneously clusters the data while enforcing consistency between cluster assignments produced for different augmentations (or “views”) of the same image, instead of comparing features directly as in contrastive learning. Simply put, we use a “swapped” prediction mechanism where we predict the code of a view from the representation of another view. Our method can be trained with large and small batches and can scale to unlimited amounts of data. Compared to previous contrastive methods, our method is more memory efficient since it does not require a large memory bank or a special momentum network. In addition, we also propose a new data augmentation strategy, multi-crop, that uses a mix of views with different resolutions in place of two full-resolution views, without increasing the memory or compute requirements. We validate our findings by achieving $75.3\%$ top-1 accuracy on ImageNet with ResNet-50, as well as surpassing supervised pretraining on all the considered transfer tasks.

Synthetic Humans for Action Recognition from Unseen Viewpoints

Participants: Cordelia Schmid.

Although synthetic training data has been shown to be beneficial for tasks such as human pose estimation, its use for RGB human action recognition is relatively unexplored. Our goal in this paper 38 is to answer the question whether synthetic humans can improve the performance of human action recognition, with a particular focus on the generalization to unseen viewpoints. We make use of the recent advances in monocular 3D human body reconstruction from real action sequences to automatically render synthetic training videos for the action labels. We make the following contributions: (i) we investigate the extent of variations and augmentations that are beneficial to improving performance at the new viewpoints. We consider changes in body shape and clothing for individuals, as well as more action relevant augmentations such as non-uniform frame sampling, and interpolating between the motion of individuals performing the same action; (ii) We introduce a new data generation methodology, SURREACT, that allows training of spatio-temporal CNNs for action classification; (iii) We substantially improve the state-of-the-art action recognition performance on the NTU RGB+D and UESTC standard human action multi-view benchmarks; Finally, (iv) we extend the augmentation approach to in-the-wild videos from a subset of the Kinetics dataset to investigate the case when only one-shot training data

Distilled Hierarchical Neural Ensembles with Adaptive Inference Cost

Participants: Adria Ruiz.

Deep neural networks form the basis of state-of-the-art models across a variety of application domains. Moreover, networks that are able to dynamically adapt the computational cost of inference are important in scenarios where the amount of computation or input data varies over time. In this paper 36, we propose Hierarchical Neural Ensembles (HNE), a novel framework to embed an ensemble of multiple networks by sharing intermediate layers using a hierarchical structure. In HNE we control the inference cost by evaluating only a subset of models, which are organized in a nested manner. Our second contribution is a novel co-distillation method to boost the performance of ensemble predictions with low inference cost. This approach leverages the nested structure of our ensembles, to optimally allocate accuracy and diversity across the ensemble members. Comprehensive experiments over the CIFAR and ImageNet datasets confirm the effectiveness of HNE in building deep networks with adaptive inference cost for image classification.

Leveraging Photometric Consistency over Time for Sparsely Supervised Hand-Object Reconstruction

Participants: Cordelia Schmid.

Modeling hand-object manipulations is essential for understanding how humans interact with their environment. While of practical importance, estimating the pose of hands and objects during interactions is challenging due to the large mutual occlusions that occur during manipulation. Recent efforts have been directed towards fully-supervised methods that require large amounts of labeled training samples. Collecting 3D ground-truth data for hand-object interactions, however, is costly, tedious, and error-prone. To overcome this challenge we present in 17 a method to leverage photometric consistency across time when annotations are only available for a sparse subset of frames in a video. Our model is trained end-to-end on color images to jointly reconstruct hands and objects in 3D by inferring their poses. Given our estimated reconstructions, we differentiably render the optical flow between pairs of adjacent images and use it within the network to warp one frame to another. We then apply a self-supervised photometric loss that relies on the visual consistency between nearby images. We achieve state-of-the-art results on 3D hand-object reconstruction benchmarks and demonstrate that our approach allows us to improve the pose estimation accuracy by leveraging information from neighboring frames in low-data regimes.

Selecting Relevant Features from a Multi-domain Representation for Few-shot Classification

Participants: Nikita Dvornik, Cordelia Schmid, Julien Mairal.

Learning Obstacle Representations for Neural Motion Planning

Participants: Cordelia Schmid, Ricardo Garcia.

Motion planning and obstacle avoidance is a key challenge in robotics applications. While previous work succeeds to provide excellent solutions for known environments, sensor-based motion planning in new and dynamic environments remains difficult. In this work 26 we address sensor-based motion planning from a learning perspective. Motivated by recent advances in visual recognition, we argue the importance of learning appropriate representations for motion planning. We propose a new obstacle representation based on the PointNet architecture and train it jointly with policies for obstacle avoidance. We experimentally evaluate our approach for rigid body motion planning in challenging environments and demonstrate significant improvements of the state of the art in terms of accuracy and efficiency.

TAO: A Large-Scale Benchmark for Tracking Any Object

Participants: Cordelia Schmid.

For many years, multi-object tracking benchmarks have focused on a handful of categories. Motivated primarily by surveillance and self-driving applications, these datasets provide tracks for people, vehicles, and animals, ignoring the vast majority of objects in the world. By contrast, in the related field of object detection, the introduction of large-scale, diverse datasets (e.g., COCO) have fostered significant progress in developing highly robust solutions. To bridge this gap, we introduce in 10 a similarly diverse dataset for Tracking Any Object (TAO)4 . It consists of 2,907 high resolution videos, captured in diverse environments, which are half a minute long on average. Importantly, we adopt a bottom-up approach for discovering a large vocabulary of 833 categories, an order of magnitude more than prior tracking benchmarks. To this end, we ask annotators to label objects that move at any point in the video, and give names to them post factum. Our vocabulary is both significantly larger and qualitatively different from existing tracking datasets. To ensure scalability of annotation, we employ a federated approach that focuses manual effort on labeling tracks for those relevant objects in a video (e.g., those that move). We perform an extensive evaluation of state-of-the-art trackers and make a number of important discoveries regarding large-vocabulary tracking in an open world. In particular, we show that existing single- and multi-object trackers struggle when applied to this scenario in the wild, and that detection-based, multi-object trackers are in fact competitive with user-initialized ones. We hope that our dataset and analysis will boost further progress in the tracking community.

Learning to combine primitive skills: A step towards versatile robotic manipulation

Participants: Cordelia Schmid.

Manipulation tasks such as preparing a meal or assembling furniture remain highly challenging for robotics and vision. Traditional task and motion planning (TAMP) methods can solve complex tasks but require full state observability and are not adapted to dynamic scene changes. Recent learning methods can operate directly on visual inputs but typically require many demonstrations and/or task-specific reward engineering. In this work 27, we aim to overcome previous limitations and propose a reinforcement learning (RL) approach to task planning that learns to combine primitive skills. First, compared to previous learning methods, our approach requires neither intermediate rewards nor complete task demonstrations during training. Second, we demonstrate the versatility of our vision-based task planning in challenging settings with temporary occlusions and dynamic scene changes. Third, we propose an efficient training of basic skills from few synthetic demonstrations by exploring recent CNN architectures and data augmentation. Notably, while all of our policies are learned on visual inputs in simulated environments, we demonstrate the successful transfer and high success rates when applying such policies to manipulation tasks on a real UR5 robotic arm.

Radioactive Data: Tracing Through Training

Participants: Alexandre Sablayrolles, Cordelia Schmid.

Data tracing determines whether particular data samples have been used to train a model. In 25, we propose a new technique, radioactive data, that makes imperceptible changes to these samples such that any model trained on them will bear an identifiable mark. Given a trained model, our technique detects the use of radioactive data and provides a level of confidence (p-value). Experiments on large-scale benchmarks (Imagenet), with standard architectures (Resnet-18, VGG-16, Densenet-121) and training procedures, show that we detect radioactive data with high confidence (p <0.0001) when only 1% of the data used to train a model is radioactive. Our radioactive mark is resilient to strong data augmentations and variations of the model architecture. As a result, it offers a much higher signal-to-noise ratio than data poisoning and backdoor methods.

Multi-modal Transformer for Video Retrieval

Participants: Valentin Gabeur, Karteek Alahari, Cordelia Schmid.

The task of retrieving video content relevant to natural language queries plays a critical role in effectively handling internet-scale datasets. Most of the existing methods for this caption-to-video retrieval problem do not fully exploit cross-modal cues present in video. Furthermore, they aggregate per-frame visual features with limited or no temporal information. In this paper 16, we present a multi-modal transformer to jointly encode the different modalities in video, which allows each of them to attend to the others. The transformer architecture is also leveraged to encode and model the temporal information. On the natural language side, we investigate the best practices to jointly optimize the language embedding together with the multi-modal transformer. This novel framework (Fig. 10) allows us to establish state-of-the-art results for video retrieval on three datasets. More details are available at http://thoth.inrialpes.fr/research/MMT

Memory-Augmented Reinforcement Learning for Image-Goal Navigation

Participants: Lina Mezghani, Karteek Alahari.

In this work 34, we address the problem of image-goal navigation in the context of visually-realistic 3D environments. This task involves navigating to a location indicated by a target image in a previously unseen environment. Earlier attempts, including RL-based and SLAM-based approaches, have either shown poor generalization performance, or are heavily reliant on pose/depth sensors. We present a novel method, shown in Figure 11, that leverages a cross-episode memory to learn to navigate. We first train a state-embedding network in a self-supervised fashion, and then use it to embed previously-visited states into an agent’s memory. In order to avoid overfitting, we propose to use data augmentation on the RGB input during training. We validate our approach through extensive evaluations, showing that our data-augmented memory-based model establishes a new state of the art on the image-goal navigation task in the challenging Gibson dataset. We obtain this competitive performance from RGB input only, without access to additional sensors such as position or depth.

Meta-Learning with Shared Amortized Variational Inference

Participants: Ekaterina Iakovleva, Jakob Verbeek, Karteek Alahari.

In this paper 18, we propose a novel amortized variational inference scheme for an empirical Bayes meta-learning model, where model parameters are treated as latent variables. We learn the prior distribution over model parameters conditioned on limited training data using a variational autoencoder approach. Our framework, presented on Figure 12, proposes sharing the same amortized inference network between the conditional prior and variational posterior distributions over the model parameters. While the posterior leverages both the labeled support and query data, the conditional prior is based only on the labeled support data. We show that in earlier work, relying on Monte-Carlo approximation, the conditional prior collapses to a Dirac delta function. In contrast, our variational approach prevents this collapse and preserves uncertainty over the model parameters. We evaluate our approach on the miniImageNet, CIFAR-FS and FC100 datasets, and present results demonstrating its advantages over previous work.

Context Aware Group Activity Recognition

Participants: Karteek Alahari.

This paper 9 addresses the task of group activity recognition in multi-person videos. Existing approaches decompose this task into feature learning and relational reasoning. Despite showing progress, these methods only rely on appearance features for people and overlook the available contextual information, which can play an important role in group activity understanding. In this work, we focus on the feature learning aspect and propose a two-stream architecture that not only considers person-level appearance features, but also makes use of contextual information present in videos for group activity recognition. In particular, we propose to use two types of contextual information beneficial for two different scenarios: pose context and scene context that provide crucial cues for group activity understanding. We combine appearance and contextual features to encode each person with an enriched representation. Finally, these combined features are used in relational reasoning for predicting group activities. We evaluate our method on two benchmarks, Volleyball and Collective Activity and show that joint modeling of contextual information with appearance features benefits in group activity understanding.

Concept generalization in visual representation learning

Participants: Mert Bulent Sariyildiz, Karteek Alahari.

In this work 37, we study the concept generalization, i.e., the extent to which models trained on a set of (seen) visual concepts can be used to recognize a new set of (unseen) concepts. Although this paradigm is a popular way of evaluating visual representations, the choice of which unseen concepts to use is usually made arbitrarily, and independently from the seen concepts used to train representations, thus ignoring any semantic relationships between the two. We argue that semantic relationships between seen and unseen concepts affect generalization performance and propose ImageNet-CoG (see Figure 13), a novel benchmark on the ImageNet dataset that enables measuring concept generalization in a principled way. Our benchmark leverages expert knowledge that comes from WordNet in order to define a sequence of unseen ImageNet concept sets that are semantically more and more distant from the ImageNet-1K subset, a ubiquitous training set. We analyze a number of such models from supervised, semi-supervised and self-supervised approaches under the prism of concept generalization, and show how our benchmark is able to uncover a number of interesting insights.

Estimate Sequences for Stochastic Composite Optimization: Variance Reduction, Acceleration, and Robustness to Noise

Participants: Andrei Kulunchakov, Julien Mairal.

In 3, we propose a unified view of gradient-based algorithms for stochastic convex composite optimization. By extending the concept of estimate sequence introduced by Nesterov, we interpret a large class of stochastic optimization methods as procedures that iteratively minimize a surrogate of the objective. This point of view covers stochastic gradient descent (SGD), the variance-reduction approaches SAGA, SVRG, MISO, their proximal variants, and has several advantages: (i) we provide a simple generic proof of convergence for all of the aforementioned methods; (ii) we naturally obtain new algorithms with the same guarantees; (iii) we derive generic strategies to make these algorithms robust to stochastic noise, which is useful when data is corrupted by small random perturbations. Finally, we show that this viewpoint is useful to obtain accelerated algorithms. A comparison with different approaches is shown in Figure 14.

Screening Data Points in Empirical Risk Minimization via Ellipsoidal Regions and Safe Loss Functions

Participants: Grégoire Mialon, Alexandre d'Aspremont, Julien Mairal.

In this work 24, we design simple screening tests to automatically discard data samples in empirical risk minimization without losing optimization guarantees. We derive loss functions that produce dual objectives with a sparse solution. We also show how to regularize convex losses to ensure such a dual sparsity-inducing property, as can be seen in Figure 15, and propose a general method to design screening tests for classification or regression based on ellipsoidal approximations of the optimal set. In addition to producing computational gains, our approach also allows us to compress a dataset into a subset of representative points.

Counterfactual Learning of Stochastic Policies with Continuous Actions

Participants: Houssam Zenati, Alberto Bietti, Matthieu Martin, Eustache Diemert, Julien Mairal.

Counterfactual reasoning from logged data has become increasingly important for many applications such as web advertising or healthcare. In this paper, we address the problem of counterfactual learning of stochastic policies with continuous actions, which raises difficult challenges about (i) data modelization, (ii) optimization, and (iii) evaluation on real data.

First, we introduce a modeling strategy based on a joint kernel embedding of contexts and actions, which overcomes the shortcomings of previous discretization strategies as shown in Fig. 16. Second, we empirically show that the optimization aspect of counterfactual learning is more important than previously thought, and we demonstrate the benefits of proximal point algorithms and differentiable estimators. Finally, we propose an evaluation protocol for offline policies in real-world logged systems, which is challenging since policies cannot be replayed on test data, and we release a new large-scale dataset along with multiple synthetic, yet realistic, evaluation setups.

Tight Nonparametric Convergence Rates for Stochastic Gradient Descent under the Noiseless Linear Model

Participants: Raphaël Berthier, Francis Bach, Pierre Gaillard.

In the context of statistical supervised learning, the noiseless linear model assumes that there exists a deterministic linear relation $Y=\langle {\theta }_{*},X,\rangle$ between the random output $Y$ and the random feature vector $\varphi \left(U\right)$, a potentially non-linear transformation of the inputs U. In 5, we analyze the convergence of single-pass, fixed-step-size stochastic gradient descent on the least-square risk under this model. The convergence of the iterates to the optimum ${\theta }_{*}$ and the decay of the generalization error follow polynomial convergence rates with exponents that both depend on the regularities of the optimum ${\theta }_{*}$ and of the feature vectors $\varphi \left(u\right)$. We interpret our result in the reproducing kernel Hilbert space framework. As a special case, we analyze an online algorithm for estimating a real function on the unit interval from the noiseless observation of its value at randomly sampled points; the convergence depends on the Sobolev smoothness of the function and of a chosen kernel. Figure 17 illustrates the convergence of the algorithm to the optimum (blue dots) and the tightness of our analysis (orange line). Finally, we apply our analysis beyond the supervised learning setting to obtain convergence rates for the averaging process (a.k.a. gossip algorithm) on a graph depending on its spectral dimension.

A Trainable Optimal Transport Embedding for Feature Aggregation and its Relationship to Attention

Participants: Grégoire Mialon*, Dexiong Chen*, Alexandre d'Aspremont, Julien Mairal.

This work 23 addresses the problem of learning on sets of features, motivated by the need of performing pooling operations in long biological sequences of varying sizes, with long-range dependencies, and possibly few labeled data. To address this challenging task, we introduce a parametrized representation of fixed size, which embeds and then aggregates elements from a given input set according to the optimal transport plan between the set and a trainable reference, see Figure 18. Our approach scales to large datasets and allows end-to-end training of the reference, while also providing a simple unsupervised learning mechanism with small computational cost. Our aggregation technique admits two useful interpretations: it may be seen as a mechanism related to attention layers in neural networks, or it may be seen as a scalable surrogate of a classical optimal transport-based kernel. We experimentally demonstrate the effectiveness of our approach on biological sequences, achieving state-of-the-art results for the protein fold recognition task and detection of chromatin profiles, and, as a proof of concept, we show promising results for processing natural language sequences. We provide an open-source implementation of our embedding that can be used alone or as a module in larger learning models.

Convolutional Kernel Networks for Graph-Structured Data

Participants: Dexiong Chen, Laurent Jacob, Julien Mairal.

In this paper 8, we introduce a family of multilayer graph kernels and establish new links between graph convolutional neural networks and kernel methods. Our approach generalizes convolutional kernel networks to graph-structured data, by representing graphs as a sequence of kernel feature maps, where each node carries information about local graph substructures. Figure 19 illustrates the construction of the kernel feature map from layer $j$ to $j+1$. On the one hand, the kernel point of view offers an unsupervised, expressive, and easy-to-regularize data representation, which is useful when limited samples are available. On the other hand, our model can also be trained end-to-end on large-scale data, leading to new types of graph convolutional neural networks. We show that our method achieves competitive performance on several graph classification benchmarks, while offering simple model interpretation.

Fully Trainable and Interpretable Non-Local Sparse Models for Image Restoration

Participants: Bruno Lecouat, Jean Ponce, Julien Mairal.

In this work 21 we propose a novel differentiable relaxation of joint sparsity that exploits both principles and leads to a general framework for image restoration which is (1) trainable end to end, (2) fully interpretable, and (3) much more compact than competing deep learning architectures. We apply this approach to denoising, blind denoising, jpeg deblocking, and demosaicking, see Figure 20, and show that, with as few as 100K parameters, its performance on several standard benchmarks is on par or better than state-of-the-art methods that may have an order of magnitude or more parameters.

A Flexible Framework for Designing Trainable Priors with Adaptive Smoothing and Game Encoding

Participants: Bruno Lecouat, Jean Ponce, Julien Mairal.

In this paper20, we introduce a general framework for designing and training neural network layers whose forward passes can be interpreted as solving non-smooth convex optimization problems, and whose architectures are derived from an optimization algorithm. We focus on convex games, solved by local agents represented by the nodes of a graph and interacting through regularization functions. This approach is appealing for solving imaging problems, as it allows the use of classical image priors within deep models that are trainable end to end. The priors used in this presentation include variants of total variation, Laplacian regularization, bilateral filtering, sparse coding on learned dictionaries, and non-local self similarities, see Figure 21. Our models are fully interpretable as well as parameter and data efficient. Our experiments demonstrate their effectiveness on a large diversity of tasks ranging from image denoising and compressed sensing for fMRI to dense stereo matching.

Depth-adaptive transformer

Participants: Maha Elbayad.

State of the art sequence-to-sequence models for large scale tasks perform a fixed number of computations for each input sequence regardless of whether it is easy or hard to process. In 13, we train Transformer models which can make output predictions at different stages of the network and we investigate different ways to predict how much computation is required for a particular sequence. Unlike dynamic computation in Universal Transformers, which applies the same set of layers iteratively, we apply different layers at every step to adjust both the amount of computation as well as the model capacity. On IWSLT German-English translation our approach matches the accuracy of a well-tuned baseline Transformer while using less than a quarter of the decoder layers.

Efficient Wait-k Models for Simultaneous Machine Translation

Participants: Maha Elbayad.

Online Versus Offline NMT Quality: An In-depth Analysis on English–German and German–English

Participants: Maha Elbayad.

We conduct in this work 15 an evaluation study comparing offline and online neural machine translation architectures. Two sequence-to-sequence models: convolutional Pervasive Attention (Elbayad et al., 2018) and attention-based Transformer (Vaswani et al., 2017) are considered. We investigate, for both architectures, the impact of online decoding constraints on the translation quality through a carefully designed human evaluation on English-German and German-English language pairs, the latter being particularly sensitive to latency constraints. The evaluation results allow us to identify the strengths and shortcomings of each model when we shift to the online setup.

Hyperspectral Images Super-Resolution via Learning High-Order Coupled Tensor Ring Representation

Participants: Jocelyn Chanussot.

Cross-Attention in Coupled Unmixing Nets for Unsupervised Hyperspectral Super-Resolution

Participants: Jocelyn Chanussot.

The recent advancement of deep learning techniques has made great progress on hyperspectral image super-resolution (HSI-SR). Yet the development of unsupervised deep networks remains challenging for this task. To this end, we propose in 28 a novel coupled unmixing network with a cross-attention mechanism, CUCaNet for short, to enhance the spatial resolution of HSI by means of higher-spatial-resolution multispectral image (MSI). Inspired by coupled spectral unmixing, a two-stream convolutional autoencoder framework is taken as backbone to jointly decompose MS and HS data into a spectrally meaningful basis and corresponding coefficients. CUCaNet is capable of adaptively learning spectral and spatial response functions from HS-MS correspondences by enforcing reasonable consistency assumptions on the networks. Moreover, a cross-attention module is devised to yield more effective spatial-spectral information transfer in networks. Extensive experiments are conducted on three widely used HS-MS datasets in comparison with state-of-the-art HSI-SR models, demonstrating the superiority of the CUCaNet in the HSI-SR application.

X-ModalNet: A semi-supervised deep cross-modal network for classification of remote sensing data

Participants: Jocelyn Chanussot.

This paper 2 addresses the problem of semi-supervised transfer learning with limited cross-modality data in remote sensing. A large amount of multi-modal earth observation images, such as multispectral imagery (MSI) or synthetic aperture radar (SAR) data, are openly available on a global scale, enabling parsing global urban scenes through remote sensing imagery. However, their ability in identifying materials (pixel-wise classification) remains limited, due to the noisy collection environment and poor discriminative information as well as limited number of well-annotated training images. To this end, we propose a novel cross-modal deep-learning framework, called X-ModalNet, with three well-designed modules: the self-adversarial module, the interactive learning module, and the label propagation module, by learning to transfer more discriminative information from a small-scale hyperspectral image (HSI) into the classification task using a large-scale MSI or SAR data. Significantly, X-ModalNet generalizes well, owing to propagating labels on an updatable graph constructed by high-level features on the top of the network, yielding semi-supervised cross-modality learning. We evaluate X-ModalNet on two multi-modal remote sensing datasets (HSI-MSI and HSI-SAR) and achieve a significant improvement in comparison with several state-of-the-art methods.

9.1.1 Inria International Labs

9.2.1 FP7 & H2020 Projects

9.3.1 ANR Project DeepInFrance

Participants: Jakob Verbeek, Adria Ruiz Ovejero.

9.3.2 ANR Project AVENUE

Participants: Karteek Alahari, D. Khuê Lê-Huu.

9.4.1 3IA MIAI chair: Towards More Data Efficiency in Machine Learning

Participants: Julien Mairal, Karteek Alahari, Massih-Reza Amini, Margot Selosse.

Training deep neural networks when the amount of annotated data is small or in the presence of adversarial perturbations is challenging. More precisely, for convolutional neural networks, it is possible to engineer visually imperceptible perturbations that can lead to arbitrarily different model predictions. Such a robustness issue is related to the problem of regularization and to the ability to generalizing with few training examples. Our objective is to develop theoretically-grounded approaches that will solve the data efficiency issues of such huge-dimensional models. The principal investigator is Julien Mairal.

Member of the conference program committees

• J. Mairal: area chair for AISTATS 2020, ECCV 2020, NeurIPS 2020, ICLR 2021, AISTATS 2021 and ICML 2021.
• K. Alahari: area chair for CVPR 2020, CVPR 2021, ECCV 2020, ICCV 2021.
• P. Gaillard: area chair for COLT 2020.

Reviewer

The permanent members, postdocs and senior PhD students of the team reviewed numerous papers for international conferences in artificial intelligence, computer vision and machine learning, including AISTATS, CVPR, ECCV, ICML, ICLR, NeurIPS in 2020.

10.1.1 Journal

10.1.2 Invited talks

• J. Mairal: DataSig seminar from the Alan Turing Institute/Oxford University/UCL, 2020 (online).
• J. Mairal: Keynote speaker at ICT Innovation, Skopje, 2020 (online).
• J. Mairal: MalGA seminar at the University of Genova, 2020 (online).
• J. Mairal: seminar given at ATOS, Grenoble, 2020 (online).
• J. Mairal: seminar given at NaverLabs, Grenoble, 2020 (online).
• P. Gaillard: Talk given at the Valpred Workshop, Aussois, 2020
• P. Gaillard: Seminar given at the Potsdamer Research Seminar, Potsdam, 2020 (online)
• P. Gaillard: Seminar given at the Statify Research Seminar, Grenoble, 2020
• D. Chen: seminar given at DFKZ Heidelberg, 2020 (online).
• D. Chen: Seminar given at TUM Munich, 2020 (online).

10.1.3 Leadership within the scientific community

• J. Mairal: panel member of the New in AI workshop at NeurIPS 2020
• J. Mairal: has become ELLIS fellow
• K. Alahari: Panel discussion on computer vision for India, Vaibhav Summit, Govt. of India Initiative, Online, 2020

10.1.4 Scientific expertise

• K. Alahari: Reviewer for ERC grants, 2020

10.1.5 Research administration

• J. Mairal: jury member for assistant professor position at ENS Lyon.

10.2.1 Teaching

• Master: J. Mairal, Kernel methods for statistical learning, 17h eqTD, M2, Ecole Normale Supérieure, Cachan, France.
• Master: J. Mairal, Advanced Learning Models, 27h eqTD, M2, UGA, Grenoble.
• Master: J. Mairal, Kernel methods for statistical learning, African Masters of Machine Intelligence, Kigali, Rwanda. (online)
• Master: K. Alahari, Machine Learning for Computer Vision and Audio Processing, 6.75h eqTD, M2, UGA, Grenoble
• Master: K. Alahari, Understanding Big Visual Data, 13.5h eqTD, M2, Grenoble INP
• Master: K. Alahari, Graphical Models Inference and Learning, 18h eqTD, M2, CentraleSupelec, Paris
• Master: K. Alahari, Introduction to computer vision, 9h eqTD, M1, ENS Paris
• License: Pierre-Louis Guhur, chargé de cours sur un module de 30h d'enseignement à l'UGA auprès des licences pro MMI.
• Master: P. Gaillard, Sequential Learning, 27h eqTD, M2, Ecole Normale Supérieure, Cachan, France.
• Master: P. Gaillard, Advanced Machine Learning, 9h eqTD, M2, Telecom ParisTech, Saclay, France
• License: P. Gaillard, Introduction to Machine Learning, 15h eqTD, L3, Ecole Normale Supérieure, Paris, France
• Master: J. Chanussot, 16h Lectures on hyperspectral remote sensing, M2 level, PHELMA / ENSE3
• License: J. Chanussot, 48h Lectures on signal processing, L3 level, ENSE3
• Master: G. Mialon, Cours de Projets Informatiques, M2 MASH at Université Paris-Dauphine.
• License: H. Leterme, Analyse pour l'ingénieur (1ère année): animation d'un groupe de TD, M1, 18.5h, UGA Grenoble

10.2.2 Supervision (PhD defenses)

• PhD: Dexiong Chen, Modélisation de données structurées avec des machines profondes à noyaux et des applications en biologie computationnelle , Univ. Grenoble Alpes, 15/12/2020, director: Julien Mairal.
• PhD: Daan Wynen,Une représentation archétypale de style artistique : résumer et manipuler des styles artistiques d'une façon interprétable , Univ. Grenoble Alpes, 9/12/2020, directors: Cordelia Schmid and Julien Mairal.
• PhD: Andrei Kulunchakov, Optimisation stochastique pour l'apprentissage machine à grande échelle : réduction de la variance et accélération, Univ. Grenoble Alpes, 3/12/2020, directors: Anatoli Juditsky and Julien Mairal.
• PhD: Thomas Lucas, Modèles génératifs profonds : sur-généralisation et abandon de mode, Univ. Grenoble Alpes, 25/9/2020, directors: Karteek Alahari and Jakob Verbeek.

10.2.3 Juries

• J. Mairal: Jury member for the PhD thesis of Yaroslav Averyanov, Université de Lille
• J. Mairal: Jury member for the PhD thesis of Dmitry Grishcnenko, Université Grenoble-Alpes
• J. Mairal: Jury member for the PhD thesis of Boris Muzellec, ENSAE
• J. Mairal: Reviewer for the PhD thesis of Vincent Prost, Université d'Evry
• J. Mairal: Jury member for the PhD thesis of Pierre Laforgue, Telecom ParisTech
• J. Mairal: Reviewer for the PhD thesis of Leonard Berrada, Oxford University
• J. Mairal: member of the comité de suivi de thèse of Olga PERMIAKOVA, Université Grenoble Alpes
• J. Mairal: member of the comité de suivi de thèse of Tayeb ZARROUK, Université Grenoble Alpes
• K. Alahari: Jury member of the thesis proposal committee of Allison Del Giorno, Carnegie Mellon University
• K. Alahari: member of the comité de suivi de thèse of Abid ALI, Université Côte d'Azur
• K. Alahari: member of the comité de suivi de thèse of Miguel SOLINAS, Université Grenoble Alpes

International journals

• 1 articlePierreP. Gratier, JérômeJ. Pety, EmericE. Bron, AntoineA. Roueff, Jan H.J. Orkisz, MaryvonneM. Gerin, VictorV. De Souza Magalhaes, MathildeM. Gaudel, MaximeM. Vono, SébastienS. Bardeau, JocelynJ. Chanussot, PierreP. Chainais, Javier R.J. Goicoechea, Viviana V.V. Guzmán, AnnieA. Hughes, JouniJ. Kainulainen, DavidD. Languignon, JacquesJ. Le Bourlot, FranckF. Le Petit, FrançoisF. Levrier, HarveyH. Liszt, NicolasN. Peretto, EvelyneE. Roueff and AlbrechtA. Sievers. Quantitative inference of the H2 column densities from 3 mm molecular emission: case study towards Orion BAstronomy and Astrophysics - A&A645January 2021January 2021, A27
  HALDOI
• 2 articleDanfengD. Hong, NaotoN. Yokoya, Gui-SongG.-S. Xia, JocelynJ. Chanussot and Xiao xiangX. Zhu. X-ModalNet: A semi-supervised deep cross-modal network for classification of remote sensing dataISPRS Journal of Photogrammetry and Remote Sensing167September 2020, 12-23
  HALDOIback to text
• 3 articleAndreiA. Kulunchakov and JulienJ. Mairal. Estimate Sequences for Stochastic Composite Optimization: Variance Reduction, Acceleration, and Robustness to NoiseJournal of Machine Learning Research21155July 2020, 1-52
  HALback to text
• 4 articleYangY. Xu, ZebinZ. Wu, JocelynJ. Chanussot and ZhihuiZ. Wei. Hyperspectral Images Super-Resolution via Learning High-Order Coupled Tensor Ring RepresentationIEEE Transactions on Neural Networks and Learning Systems3111November 2020, 4747-4760
  HALDOIback to text

International peer-reviewed conferences

• 5 inproceedingsRaphaëlR. Berthier, FrancisF. Bach and PierreP. Gaillard. Tight Nonparametric Convergence Rates for Stochastic Gradient Descent under the Noiseless Linear ModelNeurIPS '20 - 34th International Conference on Neural Information Processing Systems33Advances in Neural Information Processing SystemsVancouver, Canadahttps://proceedings.neurips.cc/paper/2020/hash/1b33d16fc562464579b7199ca3114982-Abstract.htmlDecember 2020, 2576--2586
  HALback to text
• 6 inproceedingsAlbertoA. Bietti and FrancisF. Bach. Deep Equals Shallow for ReLU Networks in Kernel RegimesICLR 2021 - International Conference on Learning RepresentationsVirtual, AustriaMay 2021, 1-22
  HAL
• 7 inproceedingsMathildeM. Caron, IshanI. Misra, JulienJ. Mairal, PriyaP. Goyal, PiotrP. Bojanowski and ArmandA. Joulin. Unsupervised Learning of Visual Features by Contrasting Cluster Assignments34th Conference on Neural Information Processing Systems, NeurIPS'2033Advances in Neural Information Processing SystemsVirtual-only, United Stateshttps://proceedings.neurips.cc/paper/2020/hash/70feb62b69f16e0238f741fab228fec2-Abstract.htmlDecember 2020, 9912-9924
  HALback to text
• 8 inproceedingsDexiongD. Chen, LaurentL. Jacob and JulienJ. Mairal. Convolutional Kernel Networks for Graph-Structured DataProceedings of the37thInternational Conference on MachineLearningICML 2020 - 37th International Conference on Machine Learning119Proceedings of Machine Learning ResearchVienna, Austriahttp://proceedings.mlr.press/v119/chen20h.htmlJuly 2020, 1576-1586
  HALback to text
• 9 inproceedingsAvijitA. Dasgupta, C. V.C. Jawahar and KarteekK. Alahari. Context Aware Group Activity RecognitionICPR 2020 - International Conference on Pattern RecognitionMilan (Virtual), ItalyJanuary 2021, 1-8
  HALback to text
• 10 inproceedingsAchalA. Dave, TarashaT. Khurana, PavelP. Tokmakov, CordeliaC. Schmid and DevaD. Ramanan. TAO: A Large-Scale Benchmark for Tracking Any ObjectECCV 2020 - European Conference on Computer Vision12350Lecture Notes in Computer ScienceGlasgow / Virtual, United KingdomOctober 2020, 436-454
  HALDOIback to text
• 11 inproceedingsNikitaN. Dvornik, CordeliaC. Schmid and JulienJ. Mairal. Selecting Relevant Features from a Multi-domain Representation for Few-shot ClassificationECCV 2020 - European Conference on Computer Vision12355Lecture Notes in Computer ScienceGlasgow / Virtual, United KingdomNovember 2020, 769-786
  HALDOIback to text
• 12 inproceedingsMahaM. Elbayad, LaurentL. Besacier and JakobJ. Verbeek. Efficient Wait-k Models for Simultaneous Machine TranslationInterspeech 2020 - Conference of the International Speech Communication AssociationShangai (Virtual Conf), ChinaOctober 2020, 1461--1465
  HALDOIback to text
• 13 inproceedingsMahaM. Elbayad, JiataoJ. Gu, EdouardE. Grave and MichaelM. Auli. Depth-adaptive TransformerICLR 2020 - Eighth International Conference on Learning RepresentationsAddis Ababa, Ethiopiahttps://iclr.cc/December 2019, 1-14
  HALback to text
• 14 inproceedingsMahaM. Elbayad, HaH. Nguyen, FethiF. Bougares, NataliaN. Tomashenko, AntoineA. Caubrière, BenjaminB. Lecouteux, YannickY. Estève and LaurentL. Besacier. ON-TRAC Consortium for End-to-End and Simultaneous Speech Translation Challenge Tasks at IWSLT 2020Proceedings of the 17th International Conference on Spoken Language TranslationSeattle, WA, United StatesJuly 2020, 35–43
  HALDOI
• 15 inproceedingsMahaM. Elbayad, MichaelM. Ustaszewski, EmmanuelleE. Esperança-Rodier, FrancisF. Brunet Manquat, JakobJ. Verbeek and LaurentL. Besacier. Online Versus Offline NMT Quality: An In-depth Analysis on English–German and German–EnglishCOLING 2020 - 28th International Conference on Computational LinguisticsVirtual, SpainDecember 2020, 5047–5058
  HALDOIback to text
• 16 inproceedingsValentinV. Gabeur, ChenC. Sun, KarteekK. Alahari and CordeliaC. Schmid. Multi-modal Transformer for Video RetrievalEuropean Conference on Computer Vision (ECCV)12349Lecture Notes in Computer ScienceGlasgow, United KingdomOctober 2020, 214-229
  HALDOIback to text
• 17 inproceedingsYanaY. Hasson, BugraB. Tekin, FedericaF. Bogo, IvanI. Laptev, MarcM. Pollefeys and CordeliaC. Schmid. Leveraging Photometric Consistency over Time for Sparsely Supervised Hand-Object ReconstructionCVPR 2020 - IEEE Conference on Computer Vision and Pattern RecognitionSeattle / Virtual, United StatesJune 2020, 568-577
  HALDOIback to text
• 18 inproceedingsEkaterinaE. Iakovleva, JakobJ. Verbeek and KarteekK. Alahari. Meta-Learning with Shared Amortized Variational InferenceICML 2020 - 37th International Conference on Machine Learning119Proceedings of Machine Learning ResearchVienna (Online), Austriahttp://proceedings.mlr.press/v119/iakovleva20a.htmlJuly 2020, 4572-4582
  HALback to text
• 19 inproceedingsRomanR. Klokov, EdmondE. Boyer and JakobJ. Verbeek. Discrete Point Flow Networks for Efficient Point Cloud Generation16th European Conference on Computer Vision - ECCV 202012368Lecture Notes in Computer ScienceGlasgow, United KingdomAugust 2020, 694-710
  HALDOIback to text
• 20 inproceedingsBrunoB. Lecouat, JeanJ. Ponce and JulienJ. Mairal. A Flexible Framework for Designing Trainable Priors with Adaptive Smoothing and Game EncodingNeurIPS '20 - 34th International Conference on Neural Information Processing Systems33Advances in Neural Information Processing SystemsVancouver, Francehttps://proceedings.neurips.cc/paper/2020/hash/b4edda67f0f57e218a8e766927e3e5c5-Abstract.htmlOctober 2020, 15664-15675
  HALback to text
• 21 inproceedingsBrunoB. Lecouat, JeanJ. Ponce and JulienJ. Mairal. Fully Trainable and Interpretable Non-Local Sparse Models for Image RestorationECCV 2020 - European Conference on Computer Vision12367Lecture Notes in Computer ScienceGlasgow / Virtual, United KingdomNovember 2020, 238-254
  HALDOIback to text
• 22 inproceedingsXiaotianX. Li, ShuzheS. Wang, YiY. Zhao, JakobJ. Verbeek and JuhoJ. Kannala. Hierarchical Scene Coordinate Classification and Regression for Visual LocalizationCVPR 2020 - IEEE Conference on Computer Vision and Pattern RecognitionSeattle, United StatesAugust 2020, 11980-11989
  HALDOI
• 23 inproceedings GrégoireG. Mialon, DexiongD. Chen, AlexandreA. D'Aspremont and JulienJ. Mairal. A Trainable Optimal Transport Embedding for Feature Aggregation and its Relationship to Attention ICLR 2021 - The Ninth International Conference on Learning Representations Virtual, France May 2021
  HAL back to text
• 24 inproceedingsGrégoireG. Mialon, AlexandreA. D'Aspremont and JulienJ. Mairal. Screening Data Points in Empirical Risk Minimization via Ellipsoidal Regions and Safe Loss FunctionsAISTATS 2020 - 23rd International Conference on Artificial Intelligence and Statistics108Proceedings of Machine Learning ResearchPalermo / Virtual, Italyhttp://proceedings.mlr.press/v108/mialon20a.htmlAugust 2020, 3610-3620
  HALback to text
• 25 inproceedingsAlexandreA. Sablayrolles, MatthijsM. Douze, CordeliaC. Schmid and HervéH. Jégou. Radioactive Data: Tracing Through TrainingICML 2020 - Thirty-seventh International Conference on Machine Learning119Proceedings of Machine Learning ResearchVienna / Virtual, AustriaJuly 2020, 8326-8335
  HALback to text
• 26 inproceedings RobinR. Strudel, RicardoR. Garcia, JustinJ. Carpentier, Jean-PaulJ.-P. Laumond, IvanI. Laptev and CordeliaC. Schmid. Learning Obstacle Representations for Neural Motion Planning CoRL 2020 - Conference on Robot Learning Cambridge MA / Virtual, United States November 2020
  HAL back to text
• 27 inproceedings RobinR. Strudel, AlexanderA. Pashevich, IgorI. Kalevatykh, IvanI. Laptev, JosefJ. Sivic and CordeliaC. Schmid. Learning to combine primitive skills: A step towards versatile robotic manipulation ICRA 2020 - IEEE International Conference on Robotics and Automation Paris / Virtuel, France May 2020
  HAL DOI back to text
• 28 inproceedingsJingJ. Yao, DanfengD. Hong, JocelynJ. Chanussot, DeyuD. Meng, XiaoxiangX. Zhu and ZongbenZ. Xu. Cross-Attention in Coupled Unmixing Nets for Unsupervised Hyperspectral Super-ResolutionEuropean Conference on Computer Vision (ECCV)ECCV 2020 - 16th European Conference on Computer Vision12374Lecture Notes i Computer ScienceGlasgow, United KingdomOctober 2020, 208-224
  HALDOIback to text

Doctoral dissertations and habilitation theses

• 29 thesis ThomasT. Lucas. Deep generative models : over-generalisation and mode-dropping Université Grenoble Alpes [2020-....] September 2020
  HAL
• 30 thesis DaanD. Wynen. An Archetypal Representation of Artistic Style : Summarizing and manipulating artistic style in an interpretable manner Université Grenoble Alpes [2020-....] December 2020
  HAL

Reports & preprints

• 31 misc RaphaëlR. Berthier, FrancisF. Bach, NicolasN. Flammarion, PierreP. Gaillard and AdrienA. Taylor. A Continuized View on Nesterov Acceleration February 2021
  HAL
• 32 misc MathildeM. Caron, AriA. Morcos, PiotrP. Bojanowski, JulienJ. Mairal and ArmandA. Joulin. Pruning Convolutional Neural Networks with Self-Supervision June 2020
  HAL
• 33 misc Anatoli B.A. Juditsky, AndreiA. Kulunchakov and HlibH. Tsyntseus. Sparse recovery by reduced variance stochastic approximation March 2021
  HAL
• 34 misc LinaL. Mezghani, SainbayarS. Sukhbaatar, ThibautT. Lavril, OleksandrO. Maksymets, DhruvD. Batra, PiotrP. Bojanowski and KarteekK. Alahari. Memory-Augmented Reinforcement Learning for Image-Goal Navigation January 2021
  HAL back to text
• 35 misc AnantA. Raj, PierreP. Gaillard and ChristopheC. Saad. Non-stationary Online Regression November 2020
  HAL
• 36 misc AdriàA. Ruiz and JakobJ. Verbeek. Distilled Hierarchical Neural Ensembles with Adaptive Inference Cost March 2020
  HAL back to text
• 37 misc Mert BulentM. Sariyildiz, YannisY. Kalantidis, DianeD. Larlus and KarteekK. Alahari. Concept Generalization in Visual Representation Learning 2020
  HAL back to text
• 38 misc GülG. Varol, IvanI. Laptev, CordeliaC. Schmid and AndrewA. Zisserman. Synthetic Humans for Action Recognition from Unseen Viewpoints January 2020
  HAL back to text
• 39 misc HoussamH. Zenati, AlbertoA. Bietti, MatthieuM. Martin, EustacheE. Diemert and JulienJ. Mairal. Counterfactual Learning of Continuous Stochastic Policies June 2020
  HAL