Exemplar application: deep probabilistic sequential modeling

We have investigated a latent-variable generative model called mixture of dynamical variational autoencoders (MixDVAE) to model the dynamics of a system composed of multiple moving sources. A DVAE model is pre-trained on a single-source dataset to capture the source dynamics. Then, multiple instances of the pre-trained DVAE model are integrated into a multi-source mixture model with a discrete observation-to-source assignment latent variable. The posterior distributions of both the discrete observation-to-source assignment variable and the continuous DVAE variables representing the sources content/position are estimated using the variational expectation-maximization algorithm, leading to multi-source trajectories estimation. We illustrated the versatility of the proposed MixDVAE model on two tasks: a computer vision task, namely multi-object tracking, and an audio processing task, namely single-channel audio source separation. Consequently, this mixture models allows to mix different non-linear source models within the maximum likelihood umbrella and combine the model with other probabilistic models as well.

Show image description

MixDVAE overall diagram.

Exemplar application: expression-preserving face frontalization

Face frontalization consists of synthesizing a frontally-viewed face from an arbitrarily-viewed one. We proposed a frontalization methodology that preserves non-rigid facial deformations in order to boost the performance of visually assisted speech communication. The method alternates between the estimation of (i) the rigid transformation (scale, rotation, and translation) and (ii) the non-rigid deformation between an arbitrarily-viewed face and a face model. The method has two important merits: it can deal with non-Gaussian errors in the data and it incorporates a dynamical face deformation model. For that purpose, we used the generalized Student t-distribution in combination with a linear dynamic system in order to account for both rigid head motions and time-varying facial deformations caused by speech production. We proposed to use the zero-mean normalized cross-correlation (ZNCC) score to evaluate the ability of the method to preserve facial expressions. We showed that the method, when incorporated into deep learning pipelines, namely lip reading and speech enhancement, improves word recognition and speech intelligibility scores by a considerable margin.

Show image description

Some results of the proposed expression-preserving face frontalization method.

Exemplar application: transfering poilicies via successor feature representations

Transfer in Reinforcement Learning aims to improve learning performance on target tasks using knowledge from experienced source tasks. Successor Representations (SR) and their extension Successor Features (SF) are prominent transfer mechanisms in domains where reward functions change between tasks. They reevaluate the expected return of previously learned policies in a new target task to transfer their knowledge. The SF framework extended SR by linearly decomposing rewards into successor features and a reward weight vector allowing their application in high-dimensional tasks. But this came with the cost of having a linear relationship between reward functions and successor features, limiting its application to tasks where such a linear relationship exists. We proposed a novel formulation of SR based on learning the cumulative discounted probability of successor features, called Successor Feature Representations (SFR). Crucially, SFR allows to reevaluate the expected return of policies for general reward functions. We introduced different SFR variations, prove its convergence, and provide a guarantee on its transfer performance. Experimental evaluations based on SFR with function approximation demonstrate its advantage over SF not only for general reward functions, but also in the case of linearly decomposable reward functions.

Anand Ballou

defended his PhD on “Meta-reinforcement learning for social robotics.” Social robots aim to assist, cooperate, take part in collaboration tasks and engage with human users. This requires the robotics system to interact within an unknown environment. The current predominant way to tackle this challenge relies on machine learning tools and algorithms. Machine learning approaches allow the robot to learn information from its environment. A popular machine-learning approach for social robotics is reinforcement learning. Reinforcement learning (RL) is a framework for solving decision-making problems in which an agent learns to interact with its environment through a trial-and-error process. By interacting with its environment, the agent will receive feedback that will either reward or penalize the taken action. The goal of reinforcement learning algorithms is to learn an action selection strategy (policy), that will maximize the reward the agent will receive. RL is well suited to solve decision problems such as robot social interaction environments, where labeled datasets are usually unavailable. However, applying reinforcement learning directly to a social environment is challenging due to several factors. One such factor is the diversity of environments a social agent will face when deployed, and the necessity for the agent to adapt to different user preferences. However, it is notoriously challenging for reinforcement learning agents to adapt to environments with different dynamics or reward functions. To help tackle this issue, this thesis investigates two ways to improve reinforcement learning agents' adaptability. In the first part, we study the usability of meta-reinforcement learning approaches in a social robotics context. In this respect, we provide improvements to a state-of-the-art meta-reinforcement learning algorithm to generate more diverse behaviors. In the second part, we study how to directly use user feedback for fast adaptation to new unknown user preferences. To accomplish this, we propose to integrate meta-reinforcement learning in the classical preference-based learning framework to build a robust and time-saving algorithm for preference-based learning in the context of social robotics. We moreover benchmark our approach on a suite of social reinforcement learning environments, which allow us to test our algorithm on various social tasks with different settings and complex user preferences.

Xiaoyu Lin

defended her PhD on “Deep latent-variable generative models for multimedia processing.” Deep probabilistic generative models hold a crucial position within the realm of machine learning research. They serve as powerful tools for comprehending complex real-world data, such as image, audio, and text, by modeling their underlying distributions. This capability further enables the generation of new data samples. Moreover, these models can be utilized to discover hidden structures and the intrinsic factors of variation within data. The data representations that are learned through this process can be leveraged across a spectrum of downstream prediction tasks, thereby enhancing the decision-making process. Another research direction involves leveraging the flexibility and robust generalization ability of deep probabilistic generative models for solving intricate scientific and engineering problems. Though supervised deep learning methods applied to sophisticatedly designed neural architectures have achieved state-of-the-art performance across various domains, their practical application to real-world situations remains constrained. These limitations arise from the necessity of extensive volumes of annotated data for training and a shortfall in model interpretability. In this PhD work, we explore an alternative approach using deep probabilistic generative models within an unsupervised or weakly supervised framework to overcome these hurdles. Specifically, the proposed approach involves initially pre-training a deep probabilistic generative model with natural or synthetic signals to embed prior knowledge about the complex data patterns. Subsequently, this pre-trained model is integrated into an extended latent variable generative model (LVGM) to address the specific practical problem. Our research focuses on a specific type of deep probabilistic generative model designed for sequential data, referred to as dynamical variational auto-encoders (DVAEs). DVAEs are a family of deep latent variable models extended from the variational auto-encoder (VAE) for sequential data modeling. They leverage a sequence of latent vectors to depict the intricate temporal dependencies within the sequential observed data. By integrating DVAEs within a LVGM, we address a range of audio and visual tasks, namely multi-object tracking, single-channel audio source separation, and speech enhancement. The solutions are derived based on variational inference methods. Additionally, we also investigate a novel architecture, HiT-DVAE, which incorporates the Transformer architecture within the probabilistic framework of DVAEs. HiT-DVAE and its variant, LigHT-DVAE, both demonstrate excellent performance in speech modeling through robust sequential data handling. The findings from our experiments confirm the potential of deep probabilistic generative models to address real-world problems with limited labeled data, offering scalable and interpretable solutions. Furthermore, the introduction of HiT-DVAE represents a significant advancement in the field, combining the strengths of Transformer architectures with probabilistic modeling for enhanced sequential data analysis. These works not only contribute to the theoretical understanding of deep generative models, but also demonstrate their practical applicability across various domains, laying the groundwork for future innovations in machine learning.

7.1.1 xi_learning

• Name:
  Successor Feature Representation Learning
• Keywords:
  Reinforcement learning, Transfer Learning
• Functional Description:
  Transfer in Reinforcement Learning aims to improve learning performance on target tasks using knowledge from experienced source tasks. Successor features (SF) are a prominent transfer mechanism in domains where the reward function changes between tasks. They reevaluate the expected return of previously learned policies in a new target task and to transfer their knowledge. A limiting factor of the SF framework is its assumption that rewards linearly decompose into successor features and a reward weight vector. We propose a novel SF mechanism, ξ-learning, based on learning the cumulative discounted probability of successor features. Crucially, ξ-learning allows to reevaluate the expected return of policies for general reward functions. We introduce two ξ-learning variations, prove its convergence, and provide a guarantee on its transfer performance. Experimental evaluations based on ξ-learning with function approximation demonstrate the prominent advantage of ξ-learning over available mechanisms not only for general reward functions, but also in the case of linearly decomposable reward functions.
• URL:
  https://­gitlab.­inria.­fr/­robotlearn/­sfr_learning
• Contact:
  Chris Reinke

7.1.2 Social MPC

• Keyword:
  Navigation
• Functional Description:

  A library for controlling a social robot. This library allows a non-holonomic robot to navigate in a crowded environment using model predictive control and social force models. This library has been developed for the SPRING project that has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 871245.

  The main components of this library are: - A module to determine optimal positioning of a robot in a group, using methods from the litterature. - A navigation component to compute optimal paths - The main module, implementing a model predictive controller using the Jax library to determine optimal commands to steer the robot

• URL:
  https://­gitlab.­inria.­fr/­spring/­wp6_robot_behavior/­robot_behavior
• Contact:
  Alex Auternaud

7.1.3 2D Social Simulator

• Keywords:
  Simulator, Robotics
• Functional Description:

  A python based simulator using Box2D allowing a robot to interact with people. This software enables: - The configuration of a scene with physical obstacles and people populating a room - The simulation of the motion of a robot in this space - Social force models for the behaviour of people, groups between themselves and in reaction to the motion of the robot

  Rendering is done using PyGame and is optional (headless mode is possible).

  A gym environment is provided for reinforcement learning.

• URL:
  https://­gitlab.­inria.­fr/­spring/­wp6_robot_behavior/­multiparty_interaction_simulator
• Contact:
  Alex Auternaud

7.1.4 dvae-speech

• Name:
  dynamic variational auto-encoder for speech re-synthesis
• Keywords:
  Variational Autoencoder, Deep learning, Pytorch, Speech Synthesis
• Functional Description:
  It can be considered a library for speech community, to use different dynamic VAE models for speech re-synthesis (potentially for other speech application)
• URL:
  https://­github.­com/­XiaoyuBIE1994/­DVAE-speech
• Publication:
  hal-02926215
• Contact:
  Xavier Alameda Pineda

7.1.5 exputils

• Name:
  experiment utilities
• Keywords:
  Python, Toolbox, Computer Science
• Functional Description:

  Experiment Utilities (exputils) contains various tools for the management of scientific experiments and their experimental data. It is especially designed to handle experimental repetitions, including to run different repetitions, to effectively store and load data for them, and to visualize their results.

  Main features: Easy definition of default configurations using nested python dictionaries. Setup of experimental configuration parameters using an ODF file. Running of experiments and their repetitions in parallel. Logging of experimental data (numpy, json). Loading and filtering of experimental data. Interactive Jupyter widgets to load, select and plot data as line, box and bar plots.

• URL:
  https://­gitlab.­inria.­fr/­creinke/­exputils
• Contact:
  Chris Reinke

7.1.6 MixDVAE

• Name:
  Source code for the article "Mixture of Dynamical Variational Autoencoders for Multi-Source Trajectory Modeling and Separation"
• Keywords:
  Variational Autoencoder, Finite mixture
• Functional Description:
  In this paper, we propose a latent-variable generative model called mixture of dynamical variational autoencoders (MixDVAE) to model the dynamics of a system composed of multiple moving sources. A DVAE model is pre-trained on a single-source dataset to capture the source dynamics. Then, multiple instances of the pre-trained DVAE model are integrated into a multi-source mixture model with a discrete observation-to-source assignment latent variable. The posterior distributions of both the discrete observation-to-source assignment variable and the continuous DVAE variables representing the sources content/position are estimated using a variational expectation-maximization algorithm, leading to multi-source trajectories estimation. We illustrate the versatility of the proposed MixDVAE model on two tasks: a computer vision task, namely multi-object tracking, and an audio processing task, namely single-channel audio source separation. Experimental results show that the proposed method works well on these two tasks, and outperforms several baseline methods.
• URL:
  https://­gitlab.­inria.­fr/­robotlearn/­dvae-umot-release
• Contact:
  Xiaoyu Lin

7.1.7 Light-DVAE

• Keyword:
  Variational Autoencoder
• Functional Description:
  The dynamical variational autoencoders (DVAEs) are a family of latent-variable deep generative models that extends the VAE to model a sequence of observed data and a corresponding sequence of latent vectors. In almost all the DVAEs of the literature, the temporal dependencies within each sequence and across the two sequences are modeled with recurrent neural networks. In this paper, we propose to model speech signals with the Hierarchical Transformer DVAE (HiT-DVAE), which is a DVAE with two levels of latent variable (sequence-wise and frame-wise) and in which the temporal dependencies are implemented with the Transformer architecture. We show that HiT-DVAE outperforms several other DVAEs for speech spectrogram modeling, while enabling a simpler training procedure, revealing its high potential for downstream low-level speech processing tasks such as speech enhancement.
• URL:
  https://­gitlab.­inria.­fr/­robotlearn/­light-dvae
• Contact:
  Xiaoyu Lin

7.1.8 DDGM-SE

• Keywords:
  Speech processing, Generative Models
• Functional Description:
  This work builds on a previous work on unsupervised speech enhancement using a dynamical variational autoencoder (DVAE) as the clean speech model and non-negative matrix factorization (NMF) as the noise model. We propose to replace the NMF noise model with a deep dynamical generative model (DDGM) depending either on the DVAE latent variables, or on the noisy observations, or on both. This DDGM can be trained in three configurations: noise-agnostic, noise-dependent and noise adaptation after noise-dependent training. Experimental results show that the proposed method achieves competitive performance compared to state-of-the-art unsupervised speech enhancement methods, while the noise-dependent training configuration yields a much more time-efficient inference process.
• URL:
  https://­gitlab.­inria.­fr/­robotlearn/­ToBeProcessed/­ddgm_se
• Contact:
  Xiaoyu Lin

8.1.1 A Multimodal Dynamical Variational Autoencoder for Audiovisual Speech Representation Learning

Participants: Samir Sadok, Laurent Girin, Xavier Alameda-Pineda.

High-dimensional data such as natural images or speech signals exhibit some form of regularity, preventing their dimensions from varying independently. This suggests that there exists a lower dimensional latent representation from which the high-dimensional observed data were generated. Uncovering the hidden explanatory features of complex data is the goal of representation learning, and deep latent variable generative models have emerged as promising unsupervised approaches. In particular, the variational autoencoder (VAE) which is equipped with both a generative and an inference model allows for the analysis, transformation, and generation of various types of data. Over the past few years, the VAE has been extended to deal with data that are either multimodal or dynamical (i.e., sequential). In this paper, we present a multimodal and dynamical VAE (MDVAE) applied to unsupervised audiovisual speech representation learning. The latent space is structured to dissociate the latent dynamical factors that are shared between the modalities from those that are specific to each modality. A static latent variable is also introduced to encode the information that is constant over time within an audiovisual speech sequence. The model is trained in an unsupervised manner on an audiovisual emotional speech dataset, in two stages. In the first stage, a vector quantized VAE (VQ-VAE) is learned independently for each modality, without temporal modeling. The second stage consists in learning the MDVAE model on the intermediate representation of the VQ-VAEs before quantization. The disentanglement between static versus dynamical and modality-specific versus modality-common information occurs during this second training stage. Extensive experiments are conducted to investigate how audiovisual speech latent factors are encoded in the latent space of MDVAE. These experiments include manipulating audiovisual speech, audiovisual facial image denoising, and audiovisual speech emotion recognition. The results show that MDVAE effectively combines the audio and visual information in its latent space. They also show that the learned static representation of audiovisual speech can be used for emotion recognition with few labeled data, and with better accuracy compared with unimodal baselines and a state-of-the-art supervised model based on an audiovisual transformer architecture.

8.1.2 Unsupervised performance analysis of 3D face alignment with a statistically robust confidence test

Participants: Xavier Alameda-Pineda, Radu Horaud.

This paper addresses the problem of analyzing the performance of 3D face alignment (3DFA), or facial landmark localization. This task is usually supervised, based on annotated datasets. Nevertheless, in the particular case of 3DFA, the annotation process is rarely error-free, which strongly biases the results. Alternatively, unsupervised performance analysis (UPA) is investigated. The core ingredient of the proposed methodology is the robust estimation of the rigid transformation between predicted landmarks and model landmarks. It is shown that the rigid mapping thus computed is affected neither by non-rigid facial deformations, due to variabilities in expression and in identity, nor by landmark localization errors, due to various perturbations. The guiding idea is to apply the estimated rotation, translation and scale to a set of predicted landmarks in order to map them onto a mathematical home for the shape embedded in these landmarks (including possible errors). UPA proceeds as follows: (i) 3D landmarks are extracted from a 2D face using the 3DFA method under investigation; (ii) these landmarks are rigidly mapped onto a canonical (frontal) pose, and (iii) a statistically-robust confidence score is computed for each landmark. This allows to assess whether the mapped landmarks lie inside (inliers) or outside (outliers) a confidence volume. An experimental evaluation protocol, that uses publicly available datasets and several 3DFA software packages associated with published articles, is described in detail. The results show that the proposed analysis is consistent with supervised metrics and that it can be used to measure the accuracy of both predicted landmarks and of automatically annotated 3DFA datasets, to detect errors and to eliminate them. Source code and supplemental materials for this paper are publicly available at https://­team.­inria.­fr/­robotlearn/­upa3dfa/.

8.1.3 MEGA: Masked Generative Autoencoder for Human Mesh Recovery

Participants: Xavier Alameda Pineda.

Human Mesh Recovery (HMR) from a single RGB image is a highly ambiguous problem, as similar 2D projections can correspond to multiple 3D interpretations. Nevertheless, most HMR methods overlook this ambiguity and make a single prediction without accounting for the associated uncertainty. A few approaches generate a distribution of human meshes, enabling the sampling of multiple predictions; however, none of them is competitive with the latest single-output model when making a single prediction. This work proposes a new approach based on masked generative modeling. By tokenizing the human pose and shape, we formulate the HMR task as generating a sequence of discrete tokens conditioned on an input image. We introduce MEGA, a MaskEd Generative Autoencoder trained to recover human meshes from images and partial human mesh token sequences. Given an image, our flexible generation scheme allows us to predict a single human mesh in deterministic mode or to generate multiple human meshes in stochastic mode. MEGA enables us to propose multiple outputs and to evaluate the uncertainty of the predictions. Experiments on in-the-wild benchmarks show that MEGA achieves state-of-the-art performance in deterministic and stochastic modes, outperforming single-output and multi-output approaches.

8.2.1 Autoregressive GAN for Semantic Unconditional Head Motion Generation

Participants: Louis Airale, Stéphane Lathuilière, Xavier Alameda Pineda.

Over the past years, semantic segmentation, as many other tasks in computer vision, benefited from the progress in deep neural networks, resulting in significantly improved performance. However, deep architectures trained with gradient-based techniques suffer from catastrophic forgetting, which is the tendency to forget previously learned knowledge while learning new tasks. Aiming at devising strategies to counteract this effect, incremental learning approaches have gained popularity over the past years. However, the first incremental learning methods for semantic segmentation appeared only recently. While effective, these approaches do not account for a crucial aspect in pixel-level dense prediction problems, i.e. the role of attention mechanisms. To fill this gap, in this paper we introduce a novel attentive feature distillation approach to mitigate catastrophic forgetting while accounting for semantic spatial- and channel-level dependencies. Furthermore, we propose a continual attentive fusion structure, which takes advantage of the attention learned from the new and the old tasks while learning features for the new task. Finally, we also introduce a novel strategy to account for the background class in the distillation loss, thus preventing biased predictions. We demonstrate the effectiveness of our approach with an extensive evaluation on Pascal-VOC 2012 and ADE20K, setting a new state of the art.

8.2.2 Semi-supervised learning made simple with self-supervised clustering

Participants: Pietro Astolfi, Elisa Ricci, Xavier Alameda-Pineda.

Self-supervised models have been shown to produce comparable or better visual representations than their supervised counterparts when trained offline on unlabeled data at scale. However, their efficacy is catastrophically reduced in a Continual Learning (CL) scenario where data is presented to the model sequentially. In this paper, we show that self-supervised loss functions can be seamlessly converted into distillation mechanisms for CL by adding a predictor network that maps the current state of the representations to their past state. This enables us to devise a framework for Continual self-supervised visual representation Learning that (i) significantly improves the quality of the learned representations, (ii) is compatible with several state-of-the-art self-supervised objectives, and (iii) needs little to no hyperparameter tuning. We demonstrate the effectiveness of our approach empirically by training six popular self-supervised models in various CL settings.

8.2.3 Motion-DVAE: Unsupervised learning for fast human motion denoising

Participants: Xavier Alameda-Pineda.

In this work, we address the task of unconditional head motion generation to animate still human faces in a low-dimensional semantic space from a single reference pose. Different from traditional audio-conditioned talking head generation that seldom puts emphasis on realistic head motions, we devise a GAN-based architecture that learns to synthesize rich head motion sequences over long duration while maintaining low error-accumulation levels. In particular, the autoregressive generation of incremental outputs ensures smooth trajectories, while a multi-scale discriminator on input pairs drives generation toward better handling of high- and low-frequency signals and less mode collapse. We experimentally demonstrate the relevance of the proposed method and show its superiority compared to models that attained state-of-the-art performances on similar tasks.

8.2.4 Robust Audio-Visual Contrastive Learning for Proposal-based Self-supervised Sound Source Localization in Videos

Participants: Xavier Alameda-Pineda.

By observing a scene and listening to corresponding audio cues, humans can easily recognize where the sound is. To achieve such cross-modal perception on machines, existing methods take advantage of the maps obtained by interpolation operations to localize the sound source. As semantic object-level localization is more attractive for prospective practical applications, we argue that these map-based methods only offer a coarse-grained and indirect description of the sound source. Additionally, these methods utilize a single audio-visual tuple at a time during selfsupervised learning, causing the model to lose the crucial chance to reason about the data distribution of large-scale audio-visual samples. Although the introduction of Audio-Visual Contrastive Learning (AVCL) can effectively alleviate this issue, the contrastive set constructed by randomly sampling is based on the assumption that the audio and visual segments from all other videos are not semantically related. Since the resulting contrastive set contains a large number of faulty negatives, we believe that this assumption is rough. In this paper, we advocate a novel proposal-based solution that directly localizes the semantic object-level sound source, without any manual annotations. The Global Response Map (GRM) is incorporated as an unsupervised spatial constraint to filter those instances corresponding to a large number of sound-unrelated regions. As a result, our proposal-based Sound Source Localization (SSL) can be cast into a simpler Multiple Instance Learning (MIL) problem. To overcome the limitation of random sampling in AVCL, we propose a novel Active Contrastive Set Mining (ACSM) to mine the contrastive sets with informative and diverse negatives for robust AVCL. Our approaches achieve state-of-the-art (SOTA) performance when compared to several baselines on multiple SSL datasets with diverse scenarios.

8.2.5 A weighted-variance variational autoencoder model for speech enhancement

Participants: Xavier Alameda-Pineda.

We address speech enhancement based on variational autoencoders, which involves learning a speech prior distribution in the timefrequency (TF) domain. A zero-mean complex-valued Gaussian distribution is usually assumed for the generative model, where the speech information is encoded in the variance as a function of a latent variable. In contrast to this commonly used approach, we propose a weighted variance generative model, where the contribution of each spectrogram time-frame in parameter learning is weighted. We impose a Gamma prior distribution on the weights, which would effectively lead to a Student's t-distribution instead of Gaussian for speech generative modeling. We develop efficient training and speech enhancement algorithms based on the proposed generative model. Our experimental results on spectrogram auto-encoding and speech enhancement demonstrate the effectiveness and robustness of the proposed approach compared to the standard unweighted variance model.

8.2.6 Lost and found: Overcoming detector failures in online multi-object tracking

Participants: Xavier Alameda-Pineda.

Multi-object tracking (MOT) endeavors to precisely estimate the positions and identities of multiple objects over time. The prevailing approach, tracking-by-detection (TbD), first detects objects and then links detections, resulting in a simple yet effective method. However, contemporary detectors may occasionally miss some objects in certain frames, causing trackers to cease tracking prematurely. To tackle this issue, we propose BUSCA, meaning ‘to search’, a versatile framework compatible with any online TbD system, enhancing its ability to persistently track those objects missed by the detector, primarily due to occlusions. Remarkably, this is accomplished without modifying past tracking results or accessing future frames, i.e., in a fully online manner. BUSCA generates proposals based on neighboring tracks, motion, and learned tokens. Utilizing a decision Transformer that integrates multimodal visual and spatiotemporal information, it addresses the object-proposal association as a multi-choice question-answering task. BUSCA is trained independently of the underlying tracker, solely on synthetic data, without requiring fine-tuning. Through BUSCA, we showcase consistent performance enhancements across five different trackers and establish a new state-of-the-art baseline across three different benchmarks. Code available at: https://­github.­com/­lorenzovaquero/­BUSCA.

8.3.1 Navigating the Practical Pitfalls of Reinforcement Learning for Social Robot Navigation

Participants: Xavier Alameda-Pineda.

Navigation is one of the essential tasks in order forrobots to be deployed in environments shared with humans.The problem becomes increasingly complex when taking inconsideration that the robot's behaviour should be suitable tohumans. This is referred to as social navigation and it is acognitive task that us humans pay little attention to as it comesnaturally. Since crafting a model of the environment dynamicsthat faithfully characterises how humans navigate seems animpossible task, we look on the side of learning-based approachesand especially reinforcement learning. In this paper we areinterested in drawing conclusions on the vast number of designchoices when training a navigation agent using reinforcementlearning. To make this educated decisions, we offer a short surveyon recent papers addressing the social navigation problem usinglearning-based algorithms. Additionally, we take note of what worked best in our testing.

8.3.2 Socially Pertinent Robots in Gerontological Healthcare

Participants: Soraya Arias, Nicolas Turro, Alex Auternaud, Chris Reinke, Victor Sanchez, Xavier Alameda-Pineda.

Despite the many recent achievements in developing and deploying social robotics, there are still many underexplored environments and applications for which systematic evaluation of such systems by end-users is necessary. While several robotic platforms have been used in gerontological healthcare, the question of whether or not a social interactive robot with multi-modal conversational capabilities will be useful and accepted in real-life facilities is yet to be answered. This paper is an attempt to partially answer this question, via two waves of experiments with patients and companions in a day-care gerontological facility in Paris with a full-sized humanoid robot endowed with social and conversational interaction capabilities. The software architecture, developed during the H2020 SPRING project, together with the experimental protocol, allowed us to evaluate the acceptability (AES) and usability (SUS) with more than 60 end-users. Overall, the users are receptive to this technology, especially when the robot perception and action skills are robust to environmental clutter and flexible to handle a plethora of different interactions.

9.2.1 H2020 projects

9.3.1 ANR JCJC Project ML3RI

Participants: Xiaoyu Lin, Xavier Alameda-Pineda.

Starting on March 1st 2020 and finalising on February 28th 2024, ML3RI is an ANR JCJC that has been awarded to Xavier Alameda-Pineda. Multi-person robot interaction in the wild (i.e. unconstrained and using only the robot's resources) is nowadays unachievable because of the lack of suitable machine perception and decision-taking models. Multi-Modal Multi-person Low-Level Learning models for Robot Interaction (ML3RI) has the ambition to develop the capacity to understand and react to low-level behavioral cues, which is crucial for autonomous robot communication. The main scientific impact of ML3RI is to develop new learning methods and algorithms, thus opening the door to study multi-party conversations with robots. In addition, the project supports open and reproducible research.

Website: https://project.inria.fr/ml3ri/

9.3.2 ANR MIAI Chair

Participants: Xiaoyu Bie, Anand Ballou, Radu Horaud, Xavier Alameda-Pineda.

The overall goal of the MIAI chair “Audio-visual machine perception & interaction for robots” it to enable socially-aware robot behavior for interactions with humans. Emphasis on unsupervised and weakly supervised learning with audio-visual data, Bayesian inference, deep learning, and reinforcement learning. Challenging proof-of-concept demonstrators. We aim to develop robots that explore populated spaces, understand human behavior, engage multimodal dialog with several users, etc. These tasks require audio and visual cues (e.g. clean speech signals, eye-gaze, head-gaze, facial expressions, lip movements, head movements, hand and body gestures) to be robustly retrieved from the raw sensor data. These features cannot be reliably extracted with a static robot that listens, looks and communicates with people from a distance, because of acoustic reverberation and noise, overlapping audio sources, bad lighting, limited image resolution, narrow camera field of view, visual occlusions, etc. We will investigate audio and visual perception and communication, e.g. face-to-face dialog: the robot should learn how to collect clean data (e.g. frontal faces, signals with high speech-to-noise ratios) and how to react appropriately to human verbal and non-verbal solicitations. We plan to demonstrate these skills with a companion robot that assists and entertains the elderly in healthcare facilities.

Website: https://project.inria.fr/avbot/

10.1.1 Scientific events: organisation

10.1.2 Scientific events: selection

10.1.3 Journal

10.1.4 Invited talks

: Xavier Alameda-Pineda was invited to give two invited talks at CEA-LIST Days and at Valeo.ai, and was one of the Keynote Speakers at the French joint conference RFIAP/CAp 2024 in Lille.

10.1.5 Leadership within the scientific community

: Xavier Alameda-Pineda is the co-founder and Chair of the SIGMM European Chapter.

10.2.1 Teaching

In 2024, Xavier Alameda-Pineda was involved and responsible in teaching one course at Masters 2 level: Advanced Machine Learning, Applications to Vision, Audio and Text - at Master of Science in Informatics at Grenoble.

10.2.2 Supervision

Xavier Alameda-Pineda (co-)supervised Xiaoyu Lin (defended), Gaétan Lepage, Jordan Cosio, and Jean-Eudes Adylo.

10.2.3 Juries

Xavier Alameda-Pineda was a PhD reviewer for the Thesis of Hakim Benkirane, and was an examiner and appointed committee chair of the PhD of Ilyass Moummad.

10.3.1 Productions (articles, videos, podcasts, serious games, ...)

: We have produced a video for the H2020 SPRING project: https://­www.­youtube.­com/­watch?v=JHDaQmK5HOg&t=10s&pp=ygUUc3ByaW5nIHByb2plY3QgaDIwMjA

International journals

• 16 articleL.Louis Airale, X.Xavier Alameda-Pineda, S.Stéphane Lathuilière and D.Dominique Vaufreydaz. Autoregressive GAN for Semantic Unconditional Head Motion Generation.ACM Transactions on Multimedia Computing, Communications and Applications2024, 1-11HALDOI
• 17 articleX.Xiaoyu Lin, L.Laurent Girin and X.Xavier Alameda-Pineda. Mixture of Dynamical Variational Autoencoders for Multi-Source Trajectory Modeling and Separation.Transactions on Machine Learning Research Journal2024, 1-19HALback to text
• 18 articleM.Mostafa Sadeghi, X.Xavier Alameda-Pineda and R.Radu Horaud. Unsupervised Performance Analysis of 3D Face Alignment with a Statistically Robust Confidence Test.Neurocomputing564January 2024, 1-16HALDOI
• 19 articleS.Samir Sadok, S.Simon Leglaive, L.Laurent Girin, X.Xavier Alameda-Pineda and R.Renaud Séguier. A Multimodal Dynamical Variational Autoencoder for Audiovisual Speech Representation Learning.Neural Networks172April 2024, 106120HALDOI

International peer-reviewed conferences

• 20 inproceedingsG.Guénolé Fiche, S.Simon Leglaive, X.Xavier Alameda-Pineda, A.Antonio Agudo and F.Francesc Moreno-Noguer. VQ-HPS: Human Pose and Shape Estimation in a Vector-Quantized Latent Space.Proceedings of the 18th European Conference on Computer Vision ECCV 2024ECCV 2024 - 18th European Conference on Computer VisionMilan, ItalyMay 2024, 1-21HAL
• 21 inproceedingsA.Ali Golmakani, M.Mostafa Sadeghi, X.Xavier Alameda-Pineda and R.Romain Serizel. A weighted-variance variational autoencoder model for speech enhancement.ICASSP 2024 - International Conference on Acoustics Speech and Signal ProcessingSeoul (Korea), South Korea2024, 1-5HAL
• 22 inproceedingsL.Lorenzo Vaquero, Y.Yihong Xu, X.Xavier Alameda-Pineda, V. M.Victor M. Brea and M.Manuel Mucientes. Lost and Found: Overcoming Detector Failures in Online Multi-Object Tracking.ECCV 24 - 18th European Conference on Computer VisionMilan (Italie), ItalyJuly 2024, 1-30HAL

Conferences without proceedings

• 23 inproceedingsX.Xavier Alameda-Pineda. Learning for Companion Robots: Preparation and Adaptation.CAp - RFIAP 2024 - Joint Conférences sur l'Apprentissage automatique and Reconnaissance des Formes, Image, Apprentissage et PerceptionLille, France2024HAL
• 24 inproceedingsJ.-E.Jean-Eudes Ayilo, M.Mostafa Sadeghi, R.Romain Serizel and X.Xavier Alameda-Pineda. Diffusion-based Unsupervised Audio-visual Speech Enhancement.International Conference on Acoustics Speech and Signal Processing (ICASSP)Hyderabad, IndiaApril 2025HAL
• 25 inproceedingsD.Dhimiter Pikuli, J.Jordan Cosio, X.Xavier Alameda-Pineda, T.Thierry Fraichard and P.-B.Pierre-Brice Wieber. Navigating the Practical Pitfalls of Reinforcement Learning for Social Robot Navigation.RSS 2024 - Workshop “Unsolved Problems in Social Robot NavigationDelft / Netherlands, NetherlandsJuly 2024, 1-9HAL

Reports & preprints

• 26 reportX.Xavier Alameda-Pineda, A.Angus Addlesee, D. H.Daniel Hernández García, C.Chris Reinke, S.Soraya Arias, F.Federica Arrigoni, A.Alex Auternaud, L.Lauriane Blavette, C.Cigdem Beyan, L. G.Luis Gomez Camara, O.Ohad Cohen, A.Alessandro Conti, S.Sébastien Dacunha, C.Christian Dondrup, Y.Yoav Ellinson, F.Francesco Ferro, S.Sharon Gannot, F.Florian Gras, N.Nancie Gunson, R.Radu Horaud, M.Moreno d'Incà, I.Imad Kimouche, S.Séverin Lemaignan, O.Oliver Lemon, C.Cyril Liotard, L.Luca Marchionni, M.Mordehay Moradi, T.Tomas Pajdla, M.Maribel Pino, M.Michal Polic, M.Matthieu Py, A.Ariel Rado, B.Bin Ren, E.Elisa Ricci, A.-S.Anne-Sophie Rigaud, P.Paolo Rota, M.Marta Romeo, N.Nicu Sebe, W.Weronika Sieińska, P.Pinchas Tandeitnik, F.Francesco Tonini, N.Nicolas Turro, T.Timothée Wintz and Y.Yanchao Yu. Socially Pertinent Robots in Gerontological Healthcare.InriaApril 2024, 1-21HAL