3.1 Deep probabilistic models

A large number of perception and interaction processes require temporal modeling. Consider for example the task of extracting a clean speech signal from visual and audio data. Both modalities live in high-dimensional observation spaces and one challenge is to extract low-dimensional embeddings that encode information in a compact way and to update it over time. These high-dimensional to low-dimensional mappings are nonlinear in the general case. Moreover, audio and visual data are corrupted by various perturbations, e.g. by the presence of background noise which is mixed up with the speech signal uttered by a person of interest, or by head movements that overlap with lip movements. Finally, for robotics applications, the available data is scarce, and datasets captured in other settings can only serve as proxies, thus requiring either adaptation  114 or the use of unsupervised models  51. Therefore, the problem is manyfold: to extract low-dimensional compact representations from high-dimensional inputs, to disregard useless data in order to retain information that is relevant for the task at hand, to update and maintain reliable information over time, and to do so in without (or with very few) annotated data from the robot.

This class of problems can be addressed in the framework of state-space models (SSMs). In their most general form, SSMs are stochastic nonlinear systems with latent variables. Such a system is composed of a state equation, that describes the dynamics of the latent (or state) variables, and $M$ observation equations (an observation equation for each sensorial modality $m$) that predict observations from the state of the system, namely:

where the latent vector $𝐱\in {ℝ}^L$ evolves according to a nonlinear stationary Markov dynamic model driven by the observed control variable $𝐮$ and corrupted by the noise $𝐯$. Similarly, the observed vectors ${𝐲}^m\in {ℝ}^{D_m}$ are modeled with nonlinear stationary functions of the current state and current input, affected by noise ${𝐰}^m$. Models of this kind have been examined for decades and their complexity increases from linear-Gaussian models to nonlinear and non-Gaussian ones. Interestingly, they can also be viewed in the framework of probabilistic graphical models to represent the conditional dependencies between the variables. The objective of an SSM is to infer the sequence of latent variables by computing the posterior distribution of the latent variable, conditioned by the sequence of observations, $p\left({𝐱}_t\right|{𝐲}_{1:t})$.

When both $f$ and $g$ are linear and when the noise processes $𝐯$ and $𝐰$ are both Gaussian, this becomes a linear dynamical system (LDS), also well known as the Kalman filter (KF), which is usually solved in the framework of probabilistic latent variable models. Things become more complex when both $f$ and $g$ are nonlinear, as the integrals required by the evaluation of the posterior become intractable. Several methods were proposed to deal with nonlinear SSMs, e.g. Bayesian tracking with particle filters, the extended Kalman filter (EKF), and the unscented Kalman filter (UKF).

Outcomes of nonlinear and non-Gaussian Bayesian trackers based on sampling were reviewed and discussed  49, most notably the problems of degeneracy, choice of importance density, and resampling. The basic idea of EKF is to linearize the equations using a first-order Taylor expansion and to apply the standard KF to the linearized model. The additional error due to linearization is not taken into account which may lead to sub-optimal performance. Rather than approximating a nonlinear dynamical system with a linear one, UKF specifies the state distribution using a minimal set of deterministically selected sample points. The sample points, when propagated through the true nonlinear system, capture the posterior state distribution accurately up to the third-order Taylor expansion. An expectation-maximization (EM) algorithm was proposed in 103 that alternates between an extended Kalman smoother which estimates an approximate posterior distribution (E-step), and nonlinear regression using a Gaussian radial basis function network to approximate $f$ and $g$ (M-step).

An alternative to nonlinear SSMs is to consider $K$ different linear dynamic regimes and to introduce an additional discrete variable, a switch, that can take one out of $K$ values – the switching Kalman filter (SKF). The drawback of SKFs is the exponential increase of the number of mixture components of the posterior distribution over time, namely $K^t$, hence an approximate posterior must be evaluated at each time step, e.g. the generalized pseudo-Bayes of order 2 (GPB2) algorithm 98.

A similar type of intractability (exponential increase of the number of mixture components of the posterior distribution) appears in the case when SSMs are used to track several objects and when there are several possible observations that are likely to be associated with each object. In such cases, additional discrete hidden variables are necessary, namely a variable that associates the $i$-th observation ${𝐲}_{it}$ with the $j$-th object ${𝐱}_{jt}$ at time $t$. Let these variables be denoted with $Z\in \mathbb{N}$, e.g. $Z_{it}=j$ means that observation $i$ at $t$ is assigned to object $j$. The number of mixture components of the posterior distribution after $t$ time steps is $N^{Mt}$, where $N$ is the number of state variables (objects to be tracked) and $M$ is the number of observed variables. Problems like these can be solved in the framework of Bayesian variational inference. We developed a general framework for variational multiple object tracking and proposed several tractable variational expectation-maximization algorithms (VEM) for visual, audio, and audio-visual multiple-object tracking, 50, 87, 52, 3.

Very recently, there has been strong interest into building SSMs in the framework of deep neural networks (DNNs). This is a very promising topic of research for several reasons. It allows the representation of arbitrary nonlinear state and observation functions, $f$ and $g$, using a plethora of feedforward and recurrent neural network architectures and hence to develop practical discriminative and generative deep filters, without the limitations of linear-Gaussian models that have been the state-of-the-art for several decades. In its general form, an RNN replaces eq. (1) with (for simplicity, we consider a single modality and hence we omit the modality index $m$):

where $𝐖$, $𝐔$, $𝐕$ are hidden-to-hidden, input-to-hidden and hidden-to-output weight matrices, $𝐛$, $𝐜$ are bias vectors, and $f$, $g$ are activation functions. The discriminative formulation allows end-to-end learning using a loss and simple and scalable stochastic gradient descent methods, thus exploiting the power of deep neural networks to represent data. It opens the door to devising SSMs with high-dimensional observation spaces. Nevertheless, while discriminative recurrent neural network (RNN) learning is well understood and efficient training methods are available, they are strongly dependent on the availability of large corpora of annotated data. In some cases, data annotation can be done relatively easily, e.g. adding various noise types to speech signals, in many other cases, augmenting the data with the corresponding ground-truth annotations is cumbersome.

One can build on the analogy between SSMs, i.e. (1) and RNNs, i.e. (2). Roughly speaking, ${𝐮}_t$ is the network input, ${𝐱}_t$ is the internal (or hidden) state and ${𝐲}_t$ is the output. A large number of combinations of feedforward and recurrent network architectures are possible in order to build the two functions. These combinations must be carefully studied as there is no universal solution to solve all the problems raised by processing complex audio and visual data. For example, the back-propagation Kalman filter (BKF)  74 combines a feedforward convolutional neural network (CNN) that transforms the input into into a low-dimensional vector which serves then as input for an RNN, improperly named Kalman filter.

In addition to require large amounts of annotated data for training their parameters, RNNs suffer from another main drawback: they are deterministic. Therefore, it is not possible to learn, exploit and track over time the uncertainty associated with the underlying temporal processes. Moreover, it is unclear how to use such models in unsupervised settings where the test data might be scarce and from a slighly different statistical distribution (typical case in robotic applications). Recently, there has been a burgeoning literature that addresses these issues, at the cross-roads of deep recurrent neural networks and probabilistic models. We recently released an extensive and comprehensive review of these model and methods and proposed several promising research avenues  8. In more detail, we proposed a novel class of models that may well be viewed as an umbrella for several methodologies that were recently proposed in the literature, we unified the notations, and we identified a number of promising research lines. We termed this class of models Dynamic Variational Autoencoders (DVAE). In one sentence, this means that we aim at modeling recurrent processes and the associated uncertainty by means of deep neural networks and probabilistic models. We name the larger family of all these methods as Deep Probabilistic Models (DPMs), which form a backbone among the methodological foundations of RobotLearn.

Learning DPMs is challenging from the theoretical, methodological and computational points of view. Indeed, the problem of learning, for instance, deep generative Bayesian filters in the framework of nonlinear and non-Gaussian SSMs remains intractable and approximate solutions, that are both optimal from a theoretical point of view and efficient from a computational point of view, remain to be proposed. We plan to investigate both discriminative and generative deep recurrent Bayesian networks and to apply them to audio, visual and audio-visual processing tasks.

Exemplar application: audio-visual speech enhancement

Speech enhancement is the task of filtering a noisy speech signal, e.g. speech corrupted by the ambient acoustics. In the recent past we have developed a handful of methods to address this task in challenging scenarios (e.g. high reverberation or very low signal-to-noise ratios).

We first proposed an architecture based on LSTMs to perform spectral-noise estimation 96 and speech enhancement 95. The idea of the latter is to map speech signals into the spectral domain using the short-time Fourier transform (STFT) and hence to represent audio signals in a time-frequency space. The input of the proposed LSTM-based narrow-band filter is a noisy signal while the target used for network training is a noise-free signal. This discriminative deep filter formulation yields excellent results when applied to speech enhancement. Since the filter processes the STFT input frequency-wise (hence the name narrow band) it is generalizable to other types of temporal data. For example we can use this same concept to process human gestures and facial expressions over time.

In order to capture and exploit the uncertainty, we also exploited variational auto-encoders (VAEs)  77 which are feed-forward encoder-decoder latent variable networks, that have recently gained an immense popularity. We developed a VAE-based speech enhancement method which learns a speech model. At test time, this pre-trained speech model is combined with a nonnegative matrix factorization (NMF) noise model whose parameters are estimated from an observed noise-corrupted speech signal 84, 85. This formulation has two distinctive features: (i) there is no need to learn in the presence of various noise types, since the VAE network learns a clean-speech model, and (ii) pairs of noisy- and clean-speech signals are not necessary for training, as it is the case with discriminative approaches. Currently the use of NMF techniques limits the representation power of the noise signal. More powerful models, such as DVAEs could also be used within the same general-purpose formulation.

We have also started to investigate the extension of unimodal (audio) VAE-based speech enhancement method to multimodal (audio-visual) speech enhancement. It is well established that audio and visual data convey complementary information for the processing of speech. In particular the two modalities are affected by completely different sources of noise. Indeed, audio-speech is contaminated by additive noise due to the presence of other audio sources, while visual-speech is contaminated by occlusions and by head movements. Currently, audio-visual processing methods assume clean visual information and it is absolutely not clear how to deal with noisy visual data in the framework of speech processing.

Along this line of research, we have proposed an audio-visual VAE that is trained using synchronized audio-speech and visual-speech data, thus yielding an audio-visual prior model for speech. At test time, the approach follows the same idea as in the case of audio speech enhancement: NMF for audio-noise estimation and speech reconstruction 108. Very recently we have started to develop the concept of mixture of variational auto-encoders (MVAE) which is an attempt to put the two modalities on an equal footing 104, 105, as well as their temporal extension  106. The central idea is to consider an audio encoder and a visual encoder that are jointly trained with a shared decoder. The general architecture of proposed MVAE formulation is shown on Figure 1. As is the case with VAEs, this leads to an intractable posterior distribution and we resort to variational inference to devise a tractable solver.

The proposed mixture VAE architecture for learning a speech generative model using audio and visual information (from ).

Research directions

We will investigate the following topics on deep probabilistic models:

• Discriminative deep filters. We plan to build on our recent work on discriminative deep filtering for speech enhancement 95, in order to address challenging problems associated with the temporal modeling and data fusion for robot perception and action. In particular we plan to devise novel algorithms that enable the robotic platform to, for instance, robustly track the visual focus of attention, or appropriately react to its changes. Such tasks require end-to-end learning, from the detection of facial and body landmarks to the prediction of their trajectories and activity recognition. In particular, we will address the task of characterizing temporal patterns of behavior in flexible settings, e.g. users not facing the camera. For example, lip reading for speech enhancement and speech recognition must be performed in unconstrained settings, e.g. in the presence of rigid head motions or when the user's face is partially occluded. Discriminative deep filters will also be investigated, within the framework of reinforcement learning, to devise optimal action policies exploiting sequential multi-modal data.

• Generative deep recurrent neural networks. Most of the VAE-based methods in the literature are tailored to use uni-modal data. VAE models for multimodal data are merely available and we are among the first to propose an audio-visual VAE model for speech enhancement 108. Nevertheless, the proposed framework treats the two modalities unevenly. We started to investigate the use of mixture models in an attempt to put the two modalities on an equal footing 104, 105, 106. However, this is a long term endeavor since it raises many difficult questions from both theoretical and algorithmic points of view. Indeed, while the concept of noisy speech is well formalized in the audio signal processing domain, it is not understood in the computer vision domain. We plan to thoroughly address the combination of generative deep networks with robust mixture modeling, using for instance heavy-tailed Student-t distributions, and coping with the added complexity by means of variational approximations. Eventually, we will consider combinations of VAEs with sequential models such as for instance RNNs, and with attention-based architectures such as transformers  66. Ideally, we will work towards devising generic methodologies spaning a wide variety of temporal models. As already mentioned, we started to investigate this problem in the framework of our work on speech enhancement 83, which may be viewed either as a recurrent VAE or, more generally, as a non-linear DNN-based formulation of SSMs. We will apply this kind of deep generative/recurrent architectures to other problems that are encountered in audio-visual perception and we will propose case-by-case tractable and efficient solvers.

3.2 Human behavior understanding

Interactions between a robot and a group of people require human behavior understanding (HBU) methods. Consider for example the tasks of detecting eye-gaze and head-gaze and of tracking the gaze directions associated with a group of participants. This means that, in addition to gaze detection and gaze tracking, it is important to detect persons and to track them as well. Additionally, it is important to extract segments of speech, to associate these segments with persons and hence to be able to determine over time who looks to whom and who is the speaker and who are the listeners. The temporal and spatial fusion of visual and audio cues stands at the basis of understanding social roles and of building a multimodal conversational model.

Performing HBU tasks in complex, cluttered and noisy environments is challenging for several reasons: participants come in an out of the camera field of view, their photometric features, e.g. facial texture, clothing, orientation with respect to the camera, etc., vary drastically, even over short periods of time, people look at an object of interest (a person entering the room, a speaking person, a TV/computer screen, a wall painting, etc.) by turning their heads away from the camera, hence facial image analysis is difficult, small head movements are often associated with speech which perturbs both lip reading and head-gaze tracking, etc. Clearly, understanding multi-person human-robot interaction is complex because the person-to-person and person-to-object, in addition to person-to-robot, interactions must explicitly be taken into account.

We propose to perform audio-visual HBU by taking explicitly into account the complementary nature of these two modalities. Differently from one current trend in AV learning  45, 57, 69, we opt for unsupervised probabilitic methods that can (i) assign observations to persons without supervision, (ii) be combined with various probabilistic noise models and (iii) and fuse various cues depending on their availability in time (i.e. handle missing data). Indeed, in face-to-face communication, the robot must choose with who it should engage dialog, e.g. based on proximity, eye gaze, head movements, lip movements, facial expressions, etc., in addition to speech. Unlike in the single-user human-robot interaction case, it is crucial to associate temporal segments of speech to participants, referred to as speech diarization. Under such scenarios, speech signals are perturbed by noise, reverberation and competing audio sources, hence speech localization and speech enhancement methods must be used in conjunction with speech recognition. The relationship with natural language understanding and spoken dialog, while very relevant, falls outside the team's expertise. This relationship will be investigated in collaboration with the Interaction Lab at Heriot-Watt University (lead by Prof. Oliver Lemon), a partner of H2020 SPRING project and with the Laboratoire d'Intelligence Artificielle at Université d'Avignon (professor Fabrice Lefèvre), partner of ANR $\mu$Dialbot project.

As already explained (see Section 3.1) we have recently investigated various aspects of dynamic HBU, namely multiple-person tracking based on visual 50, audio 87, 52, or audio-visual information 3, head-pose estimation 64, eye-gaze tracking 98, e.g. Fig. 2, and audio-visual diarization 71. Our recent work has relied on Gaussian mixture regression 60, on dynamic Bayesian networks  100 and on their variational approximations, e.g. 3. Such probabilistic and statistical formulations provide robust, powerful and flexible unsupervised learning techniques for HBU. In parallel, there has been strong interest in using deep learning techniques for HBU, e.g. person detection, person tracking, facial expression recognition, etc. Nevertheless, deep neural networks still have difficulties in capturing motion information directly from image sequences. For example, human activity detection, tracking and recognition use pre-computed optical flow to compute motion information. Most of the work has focused on HBU at a single person level and less effort has been devoted into developing deep learning methods for studying group activities and behavior, in particular in the context of interaction.

This figure shows some examples of eye-gaze detection and tracking obtained with the method proposed in . The algorithm infers both eye-gaze (green arrows) and visual focus of attention (blue circles) from head-gaze (red arrows). A side-effect of this inference is the detection of people looking at each other (dashed blue line).

A comprehensive analysis of groups of people should rely on combining Bayesian modeling with deep neural networks. Indeed, this enables us to sum up the flexibility of the former with the representative and discriminative power of the latter. We plan to combine deep generative networks (see Section 3.1) trained for person tracking with person descriptors based on deep discriminative learning. Fully generative strategies will also be investigated, possibly exploiting features pre-trained in discriminative settings, thus exploiting large-scale annotated datasets available for certain tasks. Indeed, current state of the art provides DNN architectures well suited for learning embeddings of images and of image primitives. However, these embeddings are learned off-line using very large training datasets to guarantee data variability and generality. It is however necessary to perform some kind of adaptation to the distribution of the particular data at hand, e.g. collected with robot sensors. If these data are available in advance, off-line adaptation can be done, otherwise the adaptation needs to be performed on-line or at run time. Such strategies will be useful given the particular experimental conditions of practical human-robot interaction scenarios.

On-line learning based on deep neural networks is far from being well understood. We plan to thoroughly study the incorporation of on-line learning into both Bayesian and discriminative deep networks. In the practical case of interaction, real-time processing is crucial. Therefore, a compromise must be found between the size of the network, its discriminative power and the computational cost of the learning and prediction algorithms. Clearly, there is no single solution given the large variety of problems and scenarios that are encountered in practice.

Exemplar application: multi-person facial landmark tracking

The problem of facial landmark tracking of mulitple persons can be formulated as a two-stage problem, namely, first we track each face, and second we extract facial landmarks from each tracked face, e.g. 107, and this in a robust manner. We recently proposed solutions to perform on-line multi-person tracking and started to explore how to robustly extract landmarks.

We proposed an on-line method to concurrently track a variable number of people and update the appearance model of each person 62, in order to make it more robust to changes in illumination, pose, etc. Such appearance models must yield extremely discriminative descriptors, such that two observed faces are unambiguously assigned to two different persons being tracked. This means that observation-to-person assignments, computed during the tracking itself, must be used to fine tune the (possibly deep) representation. This fine tuning needs to be carefully investigated and properly incorporated into the probabilistic tracker. Indeed, if enough data is available, the network could be fine-tuned, as in  5. Otherwise, the representation could be updated without fine-tuning the network  62.

From these tracks, one could attempt to analyse facial expressions based on, e.g. facial landmarks. Even if several facial landmark extraction algorithms exist in the literature, how to properly separate rigid head motions (such as turning the head or simply moving) from non-rigid face movements, i.e. expressions, within the current models is unclear. Indeed, the analysis of facial expressions is a difficult task by its own, even if rigid head movements have been subtracted out. We started investigating how to assess the quality of the extracted landmarks and we plan to use these assessments to design and train architectures implementing SSMs for robust facial landmark extraction and tracking. Such architectures could be easily used for other tasks such as human gesture recognition, robust body landmark estimation, facial expression recognition or speech activity estimation.

More generally, these examples are instances of the problem of on-line discriminative learning. Generally, discriminative learning uses deterministic targets/labels for learning, such as the ones produced by manual annotation of large amounts of data. In the on-line case we do not have the luxury of manual annotation. We must therefore rely on less reliable labels. In other words, we must compute and maintain over time a measure of label reliability.

Consider again the problem of tracking $N$ persons over time and let $M$ be the number of observations at each time step. Let a latent discrete variable $Z$ denote the association between an observation and a person. At each time step $t$, each observation $i$ is assigned to each person $j$ with probability $p(Z_{it}=j)$. Therefore, one can replace deterministic labels with their probability distribution function. Case-by-case analysis must be carefully carried out in order to choose the proper network and learning strategy.

Research directions

Our research plan on human behavior understanding is summarized as follows:

• Deep visual descriptors. One of the most important ingredients of HBU is to learn visual representations of humans using deep discriminative networks. This process comprises detecting people and body parts in images and then extracting 2D or 3D landmarks. We plan to combine body landmark detectors and facial landmark detectors, based on feedforward architectures, with landmark tracking based on recurrent neural networks. The advantage is twofold: to eliminate noise, outliers and artefacts, which are inherent to any imaging process, and to build spatio-temporal representations for higher-level processes such as action and gesture recognition. While the task of noise filtering can be carried out using existing techniques, the task of removing outliers and artefacts is more difficult. Based on our recent work on robust deep regression, we plan to develop robust deep learning methods to extract body and facial landmarks. In addition to the Gaussian-uniform mixture used in 82, we plan to investigate the Student t-distribution and its variants as it has interesting statistical properties, such as robustness due to their so-called heavy tail. Moreover, we plan to combine deep learning methods with robust rigid registration methods in order to distinguish between rigid and non-rigid motion and to separate them. This research will combine robust probability distributions with deep learning and hence will lead to novel algorithms for robustly detecting landmarks and tracking them over time. Simultaneously, we will address the problem of assessing the quality of the landmarks without systematic recourse to annotated datasets.

• Deep audio descriptors. We will also investigate methods for extracting descriptors from audio signals. These descriptors must be free of noise and reverberation. While there are many noise filtering and dereverberation methods available, they are not necessarily well adapted to the tasks involved in live interaction between a robot and a group of people. In particular, they often treat the case of a static acoustic scene: both the sources and the microphones remain fixed. This represents a strong limitation and the existing methods must be extended to deal with dynamic acoustic scenes, e.g. 78. Based on our recent work 88, we plan to develop deep audio descriptors that are robust against noise and reverberation. We will train these descriptors to help the tasks of speech enhancement and speech dereverberation in order to facilitate down-stream tasks such as speech-source localization and speech recognition. Moreover, we plan to develop a speaker recognition method that can operate in a complex acoustic environment. As done in computer vision for person re-identification  68, recent works adapt the embedding network to an unknown domain. Adversarial strategies to further increase the performance have also been proposed  75, and we have contributed for person re-identification  5. How to exploit these strategies with a continuous flow of observations acquired by a robotic platform remains to be investigated.

3.3 Learning and control for social robots

Traditionally, research on human-robot interaction focused on single-person scenarios also called dyadic interactions. However, over the past decade several studies were devoted to various aspects of multi-party interactions, meaning situations in which a robot interacts with a group of two or more people  109. This line of research is much more challenging because of two main reasons. First, the behavioral cues of each individual and of the group need to be faithfully extracted (and assigned to each individual). Second, the behavioral dynamics of groups of people can be pushed by the presence of the robot towards competition  56 or even bullying  55. This is why some studies restrict the experimental conditions to very controlled collaborative scenarios, often lead by the robot, such as quiz-like game playing  113 or very specific robot roles  67. Intuitively, constraining the scenario also reduces the gesture variabilty and the overall interaction dynamics, leading to methods and algorithms with questionable generalisation to free and natural social multi-party interactions.

Whenever a robot participates in such multi-party interactions, it must perform social actions. Such robot social actions are typically associated with the need to perceive a person or a group of persons in an optimal way as well as to take appropriate decisions such as to safely move towards a selected group, to pop into a conversation or to answer a question. Therefore, one can distinguish between two types of robot social actions: (i) physical actions which correspond to synthesizing appropriate motions using the robot actuators (motors), possibly within a sensorimotor loop, so as to enhance perception and maintain a natural interaction and (ii) spoken actions which correspond to synthesizing appropriate speech utterances by a spoken dialog system. In RobotLearn we will focus on the former, and integrate the latter via collaborations with research groups having with established expertise in speech technologies.

For example, robust speech communication requires clean speech signals. Nevertheless, clean speech could be retrieved by the robot in several ways and based on different strategies. The first strategy is that the robot stays still and performs audio signal processing in order to reconstruct clean speech signals from noisy ones, e.g. in the presence of reverberation and of competing audio sources. The second strategy consists of moving towards a speaking person in order to face her/him directly and to optimize the quality of the audio signals gathered with the onboard microphones. Therefore, apparently simple speech communication tasks between a robot and a person involve a complex analysis in order to take appropriate decisions: Is the room noisy? Are there many people in the robot's field of view? How far are they? Are they looking at the robot? Is speech enhancement sufficient, or should the robot move towards a person in order to reduce the effects of room reverberation and of ambient noise? Clearly, robot perception and robot action are intimately interleaved, and the robot actions should be selected on the premise that social behavior counts.

In this regard we face three problems. First, given the complexity of the environment and the inherent limitations of the robot's perception capabilities, e.g. limited camera field of view, cluttered spaces, complex acoustic conditions, etc., the robot will only have access to a partial representation of the environment, and up to a certain degree of accuracy. Second, for learning purposes, there is no easy way to annotate which are the best actions the robot must choose given a situation: supervised methods are therefore not an option. Third, since the robot cannot learn from scratch by random exploration in a new environment, standard model-free RL approaches cannot be used. Some sort of previous knowledge on the environment or a similar one should be exploited. Finally, given that the robot moves within a populated environment, it is desirable to have the capability to enforce certain constrains, thus limiting the range of possible robot actions.

Building algorithms to endow robots with autonomous decision taking is not straightforward. Two relatively distinct paradigms are available the literature. First, one can devise customized strategies based on techniques such as robot motion planning combined with sensor-based robot control. These techniques lack generalization, in particular when the robot acts in complex, dynamic and unconstrained environments. Second, one can let the robot devise its own strategies based on reinforcement learning (RL) – a machine learning paradigm in which “agents" learn by themselves by trial and error to achieve successful strategies 110. It is very difficult, however, to enforce any kind of soft- or hard-constraint within this framework. We will showcase these two scientific streams with one group of techniques for each one: model predictive control (MPC) and Q-learning, deep Q-networks (DQNs), more precisely. These two techniques are promising. Moreover, they are well documented in the robotics and machine learning. Nevertheless, combining them is extremely challenging.

MPC is a generic framework which allows the incorporation of constraints in the process of robot decision-taking. More formally MPC requires (i) a transition function ${𝐱}_{t+1}=f({𝐱}_t,{𝐮}_t)$, i.e. generalization of (1), (ii) a correction function $e({𝐱}_t,{𝐮}_t)\in ℝ$ and (iii) an optional constraint function $h$. The MPC problem is formally stated as an optimisation problem  79:

where $T$ is the time horizon considered in the optimisation problem.

Often, one can devise efficient solvers to find the optimal control sequence ${𝐮}_0^{*},...,{𝐮}_{T-1}^{*}$. As discussed before, the advantage of MPC is the possibility to include constrains, modeled through $h$. Such constraints can be used to enforce safety or other must-comply rules, the scenario at hand may require. Even if it is technically possible to learn the transition function $f$, this has high computational cost. Therefore, one limitation of MPC is the common assumption that the transition function $f$ is completely known. In purely geometric tasks, this makes sense, since one can have a fairly accurate model of how the perception of the objects present in the evolves with the robot actions. However, it is much more complex to model how the behavior of people (from their body pose to their high-level global behavior) will change due to the robot actions. One may then rather learn the transition function.

Alternatively, an appealing framework for learning robot behavior is DQN. 99 As any RL method, DQN is based on rewards, evaluated at each time step $t$ and after taking an action ${𝐮}_t$ at state ${𝐱}_t$, $r_t=r({𝐱}_t,{𝐮}_t)$. The aim is to learn the optimal action policy$\pi$, i.e. the one that maximises the expected accumulated reward: ${\overline{r}}_t={\sum }_{\tau =0}^{\infty }{\gamma }^{\tau }{r}_{t+\tau }$, where $0\le \gamma <1$ is a discount factor. To do so, DQN uses the so-called $Q$ function, which is defined for a certain action policy $\pi$ at a state-action $(𝐱,𝐮)$ pair, as the expected accumulated reward when following policy $\pi$:

where the expectation is taken over the future state distribution, using $f$, therefore the latter becomes a stochastic mapping rather than a deterministic one, and the action distribution, using $\pi$. Implicitly, this means that the function $Q$ models jointly the effect of the transition function $f$ and of the policy action $\pi$. Thus, once $Q$ is learned, the effects of $f$ and $\pi$ cannot be disentangled.

It can be shown that the optimal Q function satisfies the following Bellman equation:

In DQN, the Q function is approximated by a deep neural network, which is learned by stochastic gradient descent based on the Bellman equation. While DQN has been successfully applied to various control problems, in particular computer games, it suffers from various drawbacks. First, DQN exhibits high performance when the set of actions is discrete, as opposed to continuous actions much more suitable in robotics. Second, and this is common to the majority of RL approaches, DQN requires lots of trajectories (sequences of state-action pairs) for training. These are usually obtained through computer simulations, raising a question that remains widely open: how to efficiently simulate social interactions that follow a data distribution that the agent will face in the real world? Third, by design, RL (and hence DQN) cannot be trained in the presence of constraints. Certainly, one can discourage certain robot behaviors by designing large negative rewards of some state-action pairs, but this does not guarantee that the robot will never execute such state-action pairs.

Summarizing, on the one hand we have sensor-based robot control techniques, such as MPC, that require a faithful representation of the transition function $f$ so as to compute the optimal action trajectory, and do not allow learning. On the other hand we have learning-based techniques that allow to learn the transition function $f$ (together with the optimal policy function), but they cannot be coupled with hard-constraints. Our scenario is complex enough to require learning (part of) the transition function, and at the same time we would like to enforce constraints when controling the robot.

Exemplar application: audio-visual robot gaze control

Recently, we applied DQN to the problem of controlling the gaze of a robotic head using audio and visual information 80, 81. In summary, the robot learns by itself how to turn its head towards a group of speaking faces. The DQN-like architecture is based on a long short-time memory (LSTM) network that takes as input a sequence of states ${𝐱}_{t-k},\cdots {𝐱}_t$ (namely motor positions, person detection and sound-source detection and localization) and which predicts a $Q$-value for each possible action (stay still, look up, look down, look left and look right).

In order to speed up training in real time, we proposed to simulate the pose of people in the scene using standard pose-estimation datasets that contain ground-truth pose. We combined the poses of different people thanks to a set of hand-crafted rules. Additionally we emulated the output of a sound localisation algorithm that would provide the direction of the most prominent active sources. The reward given to the agent would be the number of faces found in the field of view, plus an extra reward if the speaking face was within the field of view. In this way the robot learned actions that maximise the number of people within the field of view. In addition, the robot satisfactorily learned to look at a speaking person when found that person belonged to a group of people, and to look around (explore) when none of the participants were within the field of view.

While this application may seem very simple, one must understand that simulating such data in a realistic manner is not straightforward. In addition, lots of simulations were required before fine-tuning the DQN with real-world data: the pre-training phase was very intense for such a simple task. Thus, scaling up such a simulation to more complex scenarios, e.g. where one has to take into account conversational and group dynamics, remains an open question. Other strategies allowing better generalization, such as meta RL, would be highly desirable.

Research directions

• Constrained RL. Naturally one may be tempted to combine MPC and DQN, but this is unfortunately not possible. Indeed, DQN cannot disentangle the policy $\pi$ from the environment $f$, and MPC requires an explicit expression for $f$ to solve the associated optimisation problem, their direct combination is not possible. We will investigate two directions. First, to devise methodologies able to efficiently learn the transition function $f$, to later on use it within the MPC framework. Second, to design learning methodologies that are combined with MPC, so that the actions taken within the learning process satisfy the required constraints. A few combinations of RL and MPC for robot navigation in human-free scenarios  101, 76, as well as MPC variants driven by datahave recently appeared in the literature. How to adapt this recent trend to dynamic complex environments such as a multi-party conversational situation is still to be investigated. Additionally, the use of audio-visual fusion in this context needs to be explored deeply, and this also holds for the second research line.

• Meta RL. An additional challenge, independent from the learning and control combination foreseen, is the data distribution gap between the simulations and the real-world. Meta-learning, or the ability to learn how to learn, can provide partial answers to this problem. Indeed, developing machine learning methods able to understand how the learning is achieved can be used to extend this learning to a new task and speed up the learning process on the new task. Recent developments proposed meta-learning strategies specifically conceived for reinforcement learning, leading to Meta-RL methods. One promising trend in Meta-RL is to have a probabilistic formulation involving SSMs and VAEs, i.e. hence sharing the methodology based on dynamical variational autoencoders described before. Very importantly, we are not aware of any studies able to combine Meta-RL with MPC to handle the constraints, and within a unified formulation. From a methodological perspective, this is an important challenge we face in the next few years.

5.1 Impact of research results

Our line of research on developing unsupervised learning methods exploiting audio-visual data to understand social scenes and to learn to interact within is very interesting and challenging, and has large economical and societal impact. Economical impact since the auditory and visual sensors are the most common one, and we can find (many of) them in almost every smartphone in the market. Beyond telephones, manufacturers designing new systems meant for human use, should take into account the need for verbal interaction, and hence for audio-visual perception. A clear example of this potential is the transfer of our technology to a real robotic platform, for evaluation within a day-care hospital (DCH). This is possible thanks to the H2020 SPRING EU project, that assesses the interest of social robotics in the non-medical phases of a regular day for elder patients in a DCH. We are evaluating the performance of our methods for AV speaker tracking, AV speech enhancement, and AV sound source separation, for future technology transfer to the robot manufacturer. This is the first step toward a robot that can be part of the social environment of the DCH, helping to reduce patient and companion stress, at the same time as being a useful tool for the medical personnel. We are confident that developing robust AV perception and action capabilities for robots and autonomous systems, will make them more suitable for environments populated with humans.

6.1 Awards

Xavier Alameda-Pineda received the IEEE Transactions on Multimedia Outstanding Associate Editor Award.

6.2 Commitments

Since June 2021, Radu Horaud has been committed with Inria’s Direction des Partenariats Européens (DPE) to assist researchers to prepare and submit ERC projects.

6.3 Events

7.1 Deep Probabilistic Models

7.2 Human Behavior Understanding

7.3 Learning and Control for Social Robots

7.4 Utilities

7.5 New software

8.1 Deep Probabilistic Models

8.2 Human Behavior Understanding

8.3 Learning and Control for Social Robots

9.1 International initiatives

Nothing to report.

9.2 International research visitors

9.3 European initiatives

9.4 National initiatives

9.5 Regional initiatives

Nothing to report.

10.1 Promoting scientific activities

10.2 Teaching - Supervision - Juries

10.3 Popularization

Xavier Alameda-Pineda participated to a pod-cast from Interstices “Quels usages pour les robots sociaux ?” Link.

11.1 Major publications

• 1 articleL.Louis Airale, D.Dominique Vaufreydaz and X.Xavier Alameda-Pineda. SocialInteractionGAN: Multi-person Interaction Sequence Generation.IEEE Transactions on Affective ComputingMay 2022
  HALDOI
• 2 articleY.Yutong Ban, X.Xavier Alameda-Pineda, C.Christine Evers and R.Radu Horaud. Tracking Multiple Audio Sources with the Von Mises Distribution and Variational EM.IEEE Signal Processing Letters266June 2019, 798 - 802
  HALDOI
• 3 articleY.Yutong Ban, X.Xavier Alameda-Pineda, L.Laurent Girin and R.Radu Horaud. Variational Bayesian Inference for Audio-Visual Tracking of Multiple Speakers.IEEE Transactions on Pattern Analysis and Machine Intelligence435May 2021, 1761-1776
  HALDOIback to textback to textback to textback to text
• 4 articleX.Xiaoyu Bie, S.Simon Leglaive, X.Xavier Alameda-Pineda and L.Laurent Girin. Unsupervised Speech Enhancement using Dynamical Variational Autoencoders.IEEE/ACM Transactions on Audio, Speech and Language Processing30September 2022, 2993 - 3007
  HALDOI
• 5 inproceedingsG.Guillaume Delorme, Y.Yihong Xu, S.Stéphane Lathuilière, R.Radu Horaud and X.Xavier Alameda-Pineda. CANU-ReID: A Conditional Adversarial Network for Unsupervised person Re-IDentification.ICPR 2020 - 25th International Conference on Pattern RecognitionMilano, ItalyIEEE2021, 4428-4435
  HALDOIback to textback to text
• 6 articleG.Georgios Evangelidis and R.Radu Horaud. Joint Alignment of Multiple Point Sets with Batch and Incremental Expectation-Maximization.IEEE Transactions on Pattern Analysis and Machine Intelligence406June 2018, 1397 - 1410
  HALDOI
• 7 articleI.Israel Gebru, S.Sileye Ba, X.Xiaofei Li and R.Radu Horaud. Audio-Visual Speaker Diarization Based on Spatiotemporal Bayesian Fusion.IEEE Transactions on Pattern Analysis and Machine Intelligence405July 2018, 1086 - 1099
  HALDOI
• 8 articleL.Laurent Girin, S.Simon Leglaive, X.Xiaoyu Bie, J.Julien Diard, T.Thomas Hueber and X.Xavier Alameda-Pineda. Dynamical Variational Autoencoders: A Comprehensive Review.Foundations and Trends in Machine Learning151-2December 2021, 1-175
  HALDOIback to textback to text
• 9 articleZ.Zhiqi Kang, M.Mostafa Sadeghi, R.Radu Horaud and X.Xavier Alameda-Pineda. Expression-preserving face frontalization improves visually assisted speech processing.International Journal of Computer VisionJanuary 2023
  HALDOI
• 10 articleS.Stéphane Lathuilière, B.Benoît Massé, P.Pablo Mesejo and R.Radu Horaud. Neural Network Based Reinforcement Learning for Audio-Visual Gaze Control in Human-Robot Interaction.Pattern Recognition Letters118February 2019, 61-71
  HALDOI
• 11 articleS.Stéphane Lathuilière, P.Pablo Mesejo, X.Xavier Alameda-Pineda and R.Radu Horaud. A Comprehensive Analysis of Deep Regression.IEEE Transactions on Pattern Analysis and Machine Intelligence429September 2020, 2065-2081
  HALDOI
• 12 articleX.Xiaofei Li, Y.Yutong Ban, L.Laurent Girin, X.Xavier Alameda-Pineda and R.Radu Horaud. Online Localization and Tracking of Multiple Moving Speakers in Reverberant Environments.IEEE Journal of Selected Topics in Signal Processing131March 2019, 88-103
  HALDOI
• 13 articleX.Xiaofei Li, S.Sharon Gannot, L.Laurent Girin and R.Radu Horaud. Multichannel Identification and Nonnegative Equalization for Dereverberation and Noise Reduction based on Convolutive Transfer Function.IEEE/ACM Transactions on Audio, Speech and Language Processing2610May 2018, 1755-1768
  HALDOI
• 14 articleX.Xiaofei Li, L.Laurent Girin, S.Sharon Gannot and R.Radu Horaud. Multichannel Speech Separation and Enhancement Using the Convolutive Transfer Function.IEEE/ACM Transactions on Audio, Speech and Language Processing273March 2019, 645-659
  HALDOI
• 15 articleX.Xiaofei Li, S.Simon Leglaive, L.Laurent Girin and R.Radu Horaud. Audio-noise Power Spectral Density Estimation Using Long Short-term Memory.IEEE Signal Processing Letters266June 2019, 918-922
  HALDOI
• 16 articleB.Benoît Massé, S.Silèye Ba and R.Radu Horaud. Tracking Gaze and Visual Focus of Attention of People Involved in Social Interaction.IEEE Transactions on Pattern Analysis and Machine Intelligence4011November 2018, 2711 - 2724
  HALDOI
• 17 articleM.Mostafa Sadeghi, S.Simon Leglaive, X.Xavier Alameda-Pineda, L.Laurent Girin and R.Radu Horaud. Audio-Visual Speech Enhancement Using Conditional Variational Auto-Encoders.IEEE/ACM Transactions on Audio, Speech and Language Processing28May 2020, 1788-1800
  HALDOI
• 18 articleA.Aliaksandr Siarohin, G.Gloria Zen, C.Cveta Majtanovic, X.Xavier Alameda-Pineda, E.Elisa Ricci and N.Nicu Sebe. Increasing Image Memorability with Neural Style Transfer.ACM Transactions on Multimedia Computing, Communications and Applications152June 2019
  HALDOI
• 19 articleD.Dan Xu, X.Xavier Alameda-Pineda, W.Wanli Ouyang, E.Elisa Ricci, X.Xiaogang Wang and N.Nicu Sebe. Probabilistic Graph Attention Network with Conditional Kernels for Pixel-Wise Prediction.IEEE Transactions on Pattern Analysis and Machine Intelligence445May 2022, 2673-2688
  HALDOI
• 20 articleY.Yihong Xu, Y.Yutong Ban, G.Guillaume Delorme, C.Chuang Gan, D.Daniela Rus and X.Xavier Alameda-Pineda. TransCenter: Transformers With Dense Representations for Multiple-Object Tracking.IEEE Transactions on Pattern Analysis and Machine IntelligenceNovember 2022, 1-16
  HALDOI
• 21 articleG.Guanglei Yang, E.Enrico Fini, D.Dan Xu, P.Paolo Rota, M.Mingli Ding, M.Moin Nabi, X.Xavier Alameda-Pineda and E.Elisa Ricci. Uncertainty-aware Contrastive Distillation for Incremental Semantic Segmentation.IEEE Transactions on Pattern Analysis and Machine IntelligenceMarch 2022, 1-14
  HALDOI
• 22 articleG.Guanglei Yang, E.Enrico Fini, D.Dan Xu, P.Paolo Rota, M.Mingli Ding, H.Hao Tang, X.Xavier Alameda-Pineda and E.Elisa Ricci. Continual Attentive Fusion for Incremental Learning in Semantic Segmentation.IEEE Transactions on MultimediaApril 2022
  HALDOI

11.2 Publications of the year

11.3 Cited publications

• 44 articleJ.Jordan Abdi, A.Ahmed Al-Hindawi, T.Tiffany Ng and M. P.Marcela P Vizcaychipi. Scoping review on the use of socially assistive robot technology in elderly care.BMJ open822018, e018815
  back to text
• 45 inproceedingsT.Triantafyllos Afouras, A.Andrew Owens, J. S.Joon Son Chung and A.Andrew Zisserman. Self-supervised learning of audio-visual objects from video.Computer Vision--ECCV 2020: 16th European Conference, Glasgow, UK, August 23--28, 2020, Proceedings, Part XVIII 16Springer2020, 208--224
  back to text
• 46 inproceedingsX.Xavier Alameda-Pineda, S.Soraya Arias, Y.Yutong Ban, G.Guillaume Delorme, L.Laurent Girin, R.Radu Horaud, X.Xiaofei Li, B.Bastien Mourgue and G.Guillaume Sarrazin. Audio-Visual Variational Fusion for Multi-Person Tracking with Robots.ACMMM 2019 - 27th ACM International Conference on MultimediaNice, FranceACM PressOctober 2019, 1059-1061
  HALDOIback to text
• 47 articleX.Xavier Alameda-Pineda and R.Radu Horaud. A Geometric Approach to Sound Source Localization from Time-Delay Estimates.IEEE Transactions on Audio, Speech and Language Processing226June 2014, 1082-1095
  HALDOIback to text
• 48 articleX.Xavier Alameda-Pineda and R.Radu Horaud. Vision-Guided Robot Hearing.The International Journal of Robotics Research344-5April 2015, 437-456
  HALDOIback to text
• 49 articleM. S.M Sanjeev Arulampalam, S.Simon Maskell, N.Neil Gordon and T.Tim Clapp. A tutorial on particle filters for online nonlinear/non-Gaussian Bayesian tracking.IEEE Transactions on signal processing5022002, 174--188
  back to text
• 50 articleS.Sileye Ba, X.Xavier Alameda-Pineda, A.Alessio Xompero and R.Radu Horaud. An On-line Variational Bayesian Model for Multi-Person Tracking from Cluttered Scenes.Computer Vision and Image Understanding153December 2016, 64--76
  HALDOIback to textback to textback to text
• 51 inproceedingsY.Yutong Ban, X.Xavier Alameda-Pineda, F.Fabien Badeig, S.Sileye Ba and R.Radu Horaud. Tracking a Varying Number of People with a Visually-Controlled Robotic Head.IEEE/RSJ International Conference on Intelligent Robots and SystemsVancouver, CanadaIEEESeptember 2017, 4144-4151
  HALDOIback to text
• 52 articleY.Yutong Ban, X.Xavier Alameda-Pineda, C.Christine Evers and R.Radu Horaud. Tracking Multiple Audio Sources with the Von Mises Distribution and Variational EM.IEEE Signal Processing Letters266June 2019, 798 - 802
  HALDOIback to textback to textback to text
• 53 articleT. W.Timothy W Bickmore, L. M.Laura M Pfeifer and M. K.Michael K Paasche-Orlow. Using computer agents to explain medical documents to patients with low health literacy.Patient education and counseling7532009, 315--320
  back to text
• 54 articleJ.Joost Broekens, M.Marcel Heerink, H.Henk Rosendal and others. Assistive social robots in elderly care: a review.Gerontechnology822009, 94--103
  back to text
• 55 inproceedingsD.Drażen Bršċić, H.Hiroyuki Kidokoro, Y.Yoshitaka Suehiro and T.Takayuki Kanda. Escaping from children's abuse of social robots.Proceedings of the tenth annual acm/ieee international conference on human-robot interaction2015, 59--66
  back to text
• 56 inproceedingsW.-L.Wan-Ling Chang, J. P.Jeremy P White, J.Joohyun Park, A.Anna Holm and S.Selma Šabanović. The effect of group size on people's attitudes and cooperative behaviors toward robots in interactive gameplay.RO-MAN International Symposium on Robot and Human Interactive CommunicationIEEE2012, 845--850
  back to text
• 57 inproceedingsC.Changan Chen, U.Unnat Jain, C.Carl Schissler, S. V.Sebastia Vicenc Amengual Gari, Z.Ziad Al-Halah, V. K.Vamsi Krishna Ithapu, P.Philip Robinson and K.Kristen Grauman. Soundspaces: Audio-visual navigation in 3d environments.Computer Vision--ECCV 2020: 16th European Conference, Glasgow, UK, August 23--28, 2020, Proceedings, Part VI 16Springer2020, 17--36
  back to text
• 58 articleA.Antoine Deleforge, F.Florence Forbes, S.Silèye Ba and R.Radu Horaud. Hyper-Spectral Image Analysis with Partially-Latent Regression and Spatial Markov Dependencies.IEEE Journal of Selected Topics in Signal Processing96September 2015, 1037-1048
  HALDOIback to text
• 59 articleA.Antoine Deleforge, F.Florence Forbes and R.Radu Horaud. Acoustic Space Learning for Sound-Source Separation and Localization on Binaural Manifolds.International Journal of Neural Systems251February 2015, 21p
  HALDOIback to text
• 60 articleA.Antoine Deleforge, F.Florence Forbes and R.Radu Horaud. High-Dimensional Regression with Gaussian Mixtures and Partially-Latent Response Variables.Statistics and Computing255September 2015, 893-911
  HALDOIback to textback to text
• 61 articleA.Antoine Deleforge, R.Radu Horaud, Y. Y.Yoav Y. Schechner and L.Laurent Girin. Co-Localization of Audio Sources in Images Using Binaural Features and Locally-Linear Regression.IEEE Transactions on Audio, Speech and Language Processing234April 2015, 718-731
  HALDOIback to text
• 62 inproceedingsG.Guillaume Delorme, Y.Yutong Ban, G.Guillaume Sarrazin and X.Xavier Alameda-Pineda. ODANet: Online Deep Appearance Network for Identity-Consistent Multi-Person Tracking.ICPR 2021 - 25th International Conference on Pattern Recognition / WorkshopsMilano / Virtual, ItalyJanuary 2021
  HALback to textback to text
• 63 inproceedingsV.Vincent Drouard, S.Silèye Ba, G.Georgios Evangelidis, A.Antoine Deleforge and R.Radu Horaud. Head Pose Estimation via Probabilistic High-Dimensional Regression.IEEE International Conference on Image Processing, ICIP 2015Proceedings of the IEEE International Conference on Image ProcessingQuebec City, QC, CanadaIEEESeptember 2015, 4624-4628
  HALDOIback to text
• 64 articleV.Vincent Drouard, R.Radu Horaud, A.Antoine Deleforge, S.Sileye Ba and G.Georgios Evangelidis. Robust Head-Pose Estimation Based on Partially-Latent Mixture of Linear Regressions.IEEE Transactions on Image Processing263March 2017, 1428 - 1440
  HALDOIback to textback to text
• 65 articleG.Georgios Evangelidis and R.Radu Horaud. Joint Alignment of Multiple Point Sets with Batch and Incremental Expectation-Maximization.IEEE Transactions on Pattern Analysis and Machine Intelligence406June 2018, 1397 - 1410
  HALDOIback to text
• 66 articleL.Le Fang, T.Tao Zeng, C.Chaochun Liu, L.Liefeng Bo, W.Wen Dong and C.Changyou Chen. Transformer-based Conditional Variational Autoencoder for Controllable Story Generation.arXiv preprint arXiv:2101.008282021
  back to text
• 67 articleM. E.Mary Ellen Foster, A.Andre Gaschler and M.Manuel Giuliani. Automatically classifying user engagement for dynamic multi-party human--robot interaction.International Journal of Social Robotics952017, 659--674
  back to text
• 68 inproceedingsY.Yang Fu, Y.Yunchao Wei, G.Guanshuo Wang, Y.Yuqian Zhou, H.Honghui Shi and T. S.Thomas S Huang. Self-similarity grouping: A simple unsupervised cross domain adaptation approach for person re-identification.Proceedings of the IEEE/CVF International Conference on Computer Vision2019, 6112--6121
  back to text
• 69 inproceedingsR.Ruohan Gao and K.Kristen Grauman. Visualvoice: Audio-visual speech separation with cross-modal consistency.IEEE/CVF CVPR2021
  back to text
• 70 articleI. D.Israel Dejene Gebru, X.Xavier Alameda-Pineda, F.Florence Forbes and R.Radu Horaud. EM Algorithms for Weighted-Data Clustering with Application to Audio-Visual Scene Analysis.IEEE Transactions on Pattern Analysis and Machine Intelligence3812December 2016, 2402 - 2415
  HALDOIback to text
• 71 articleI.Israel Gebru, S.Sileye Ba, X.Xiaofei Li and R.Radu Horaud. Audio-Visual Speaker Diarization Based on Spatiotemporal Bayesian Fusion.IEEE Transactions on Pattern Analysis and Machine Intelligence405July 2018, 1086 - 1099
  HALDOIback to textback to text
• 72 articleL.Laurent Girin, T.Thomas Hueber and X.Xavier Alameda-Pineda. Extending the Cascaded Gaussian Mixture Regression Framework for Cross-Speaker Acoustic-Articulatory Mapping.IEEE/ACM Transactions on Audio, Speech and Language Processing253March 2017, 662-673
  HALDOIback to text
• 73 articleM.Matthew Gombolay, X. J.Xi Jessie Yang, B.Bradley Hayes, N.Nicole Seo, Z.Zixi Liu, S.Samir Wadhwania, T.Tania Yu, N.Neel Shah, T.Toni Golen and J.Julie Shah. Robotic assistance in the coordination of patient care.The International Journal of Robotics Research37102018, 1300--1316
  back to text
• 74 inproceedingsT.Tuomas Haarnoja, A.Anurag Ajay, S.Sergey Levine and P.Pieter Abbeel. Backprop kf: Learning discriminative deterministic state estimators.Advances in neural information processing systems2016, 4376--4384
  back to text
• 75 articleJ.Jaesung Huh, H. S.Hee Soo Heo, J.Jingu Kang, S.Shinji Watanabe and J. S.Joon Son Chung. Augmentation adversarial training for self-supervised speaker recognition.arXiv preprint arXiv:2007.120852020
  back to text
• 76 inproceedingsN.Napat Karnchanachari, M. I.Miguel Iglesia Valls, D.David Hoeller and M.Marco Hutter. Practical Reinforcement Learning For MPC: Learning from sparse objectives in under an hour on a real robot.Learning for Dynamics and ControlPMLR2020, 211--224
  back to text
• 77 articleD. P.Diederik P Kingma and M.Max Welling. Auto-encoding variational bayes.arXiv preprint arXiv:1312.61142013
  back to text
• 78 articleD.Dionyssos Kounades-Bastian, L.Laurent Girin, X.Xavier Alameda-Pineda, S.Sharon Gannot and R.Radu Horaud. A Variational EM Algorithm for the Separation of Time-Varying Convolutive Audio Mixtures.IEEE/ACM Transactions on Audio, Speech and Language Processing248August 2016, 1408-1423
  HALDOIback to textback to text
• 79 inproceedingsJ.Jory Lafaye, C.Cyrille Collette and P.-B.Pierre-Brice Wieber. Model predictive control for tilt recovery of an omnidirectional wheeled humanoid robot.International conference on robotics and automation (ICRA)IEEE2015, 5134--5139
  back to text
• 80 inproceedingsS.Stéphane Lathuilière, B.Benoit Massé, P.Pablo Mesejo and R.Radu Horaud. Deep Reinforcement Learning for Audio-Visual Gaze Control.IROS 2018 - IEEE/RSJ International Conference on Intelligent Robots and SystemsMadrid, SpainIEEEOctober 2018, 1555-1562
  HALDOIback to textback to text
• 81 articleS.Stéphane Lathuilière, B.Benoît Massé, P.Pablo Mesejo and R.Radu Horaud. Neural Network Based Reinforcement Learning for Audio-Visual Gaze Control in Human-Robot Interaction.Pattern Recognition Letters118February 2019, 61-71
  HALDOIback to textback to text
• 82 inproceedingsS.Stéphane Lathuilière, P.Pablo Mesejo, X.Xavier Alameda-Pineda and R.Radu Horaud. DeepGUM: Learning Deep Robust Regression with a Gaussian-Uniform Mixture Model.ECCV 2018 - European Conference on Computer Vision11209Lecture Notes in Computer ScienceMunich, GermanySpringerSeptember 2018, 205-221
  HALDOIback to text
• 83 inproceedingsS.Simon Leglaive, X.Xavier Alameda-Pineda, L.Laurent Girin and R.Radu Horaud. A Recurrent Variational Autoencoder for Speech Enhancement.IEEE International Conference on Acoustics, Speech and Signal ProcessingIEEEBarcelone, SpainMay 2020, 1-7
  HALDOIback to text
• 84 inproceedingsS.Simon Leglaive, L.Laurent Girin and R.Radu Horaud. A variance modeling framework based on variational autoencoders for speech enhancement.EEE International Workshop on Machine Learning for Signal Processing (MLSP)Aalborg, DenmarkIEEESeptember 2018, 1-6
  HALDOIback to text
• 85 inproceedingsS.Simon Leglaive, L.Laurent Girin and R.Radu Horaud. Semi-supervised multichannel speech enhancement with variational autoencoders and non-negative matrix factorization.ICASSP 2019 - IEEE International Conference on Acoustics, Speech and Signal ProcessingBrighton, United KingdomIEEEMay 2019, 101-105
  HALDOIback to text
• 86 inproceedingsX.Xiaofei Li, Y.Yutong Ban, L.Laurent Girin, X.Xavier Alameda-Pineda and R.Radu Horaud. A Cascaded Multiple-Speaker Localization and Tracking System.Proceedings of the LOCATA Challenge Workshop - a satellite event of IWAENC 2018Tokyo, JapanSeptember 2018, 1-5
  HALback to text
• 87 articleX.Xiaofei Li, Y.Yutong Ban, L.Laurent Girin, X.Xavier Alameda-Pineda and R.Radu Horaud. Online Localization and Tracking of Multiple Moving Speakers in Reverberant Environments.IEEE Journal of Selected Topics in Signal Processing131March 2019, 88-103
  HALDOIback to textback to textback to textback to text
• 88 articleX.Xiaofei Li, S.Sharon Gannot, L.Laurent Girin and R.Radu Horaud. Multichannel Identification and Nonnegative Equalization for Dereverberation and Noise Reduction based on Convolutive Transfer Function.IEEE/ACM Transactions on Audio, Speech and Language Processing2610May 2018, 1755-1768
  HALDOIback to textback to text
• 89 inproceedingsX.Xiaofei Li, L.Laurent Girin, F.Fabien Badeig and R.Radu Horaud. Reverberant Sound Localization with a Robot Head Based on Direct-Path Relative Transfer Function.IEEE/RSJ International Conference on Intelligent Robots and SystemsIEEEDaejeon, South KoreaIEEEOctober 2016, 2819-2826
  HALDOIback to text
• 90 articleX.Xiaofei Li, L.Laurent Girin, S.Sharon Gannot and R.Radu Horaud. Multichannel Online Dereverberation based on Spectral Magnitude Inverse Filtering.IEEE/ACM Transactions on Audio, Speech and Language Processing279May 2019, 1365-1377
  HALDOIback to text
• 91 articleX.Xiaofei Li, L.Laurent Girin, S.Sharon Gannot and R.Radu Horaud. Multichannel Speech Separation and Enhancement Using the Convolutive Transfer Function.IEEE/ACM Transactions on Audio, Speech and Language Processing273March 2019, 645-659
  HALDOIback to text
• 92 articleX.Xiaofei Li, L.Laurent Girin and R.Radu Horaud. Expectation-Maximization for Speech Source Separation using Convolutive Transfer Function.CAAI Transactions on Intelligent Technologies41March 2019, 47 - 53
  HALDOIback to text
• 93 articleX.Xiaofei Li, L.Laurent Girin, R.Radu Horaud and S.Sharon Gannot. Estimation of the Direct-Path Relative Transfer Function for Supervised Sound-Source Localization.IEEE/ACM Transactions on Audio, Speech and Language Processing2411November 2016, 2171 - 2186
  HALDOIback to text
• 94 articleX.Xiaofei Li, L.Laurent Girin, R.Radu Horaud and S.Sharon Gannot. Multiple-Speaker Localization Based on Direct-Path Features and Likelihood Maximization with Spatial Sparsity Regularization.IEEE/ACM Transactions on Audio, Speech and Language Processing251016 pages, 4 figures, 4 tablesOctober 2017, 1997 - 2012
  HALDOIback to text
• 95 unpublishedX.Xiaofei Li and R.Radu Horaud. Narrow-band Deep Filtering for Multichannel Speech Enhancement.September 2020, working paper or preprint
  HALback to textback to text
• 96 articleX.Xiaofei Li, S.Simon Leglaive, L.Laurent Girin and R.Radu Horaud. Audio-noise Power Spectral Density Estimation Using Long Short-term Memory.IEEE Signal Processing Letters266June 2019, 918-922
  HALDOIback to text
• 97 articleJ. A.Jordan A Mann, B. A.Bruce A MacDonald, I.-H.I-Han Kuo, X.Xingyan Li and E.Elizabeth Broadbent. People respond better to robots than computer tablets delivering healthcare instructions.Computers in Human Behavior432015, 112--117
  back to text
• 98 articleB.Benoît Massé, S.Silèye Ba and R.Radu Horaud. Tracking Gaze and Visual Focus of Attention of People Involved in Social Interaction.IEEE Transactions on Pattern Analysis and Machine Intelligence4011November 2018, 2711 - 2724
  HALDOIback to textback to textback to textback to text
• 99 articleV.Volodymyr Mnih, K.Koray Kavukcuoglu, D.David Silver, A. A.Andrei A Rusu, J.Joel Veness, M. G.Marc G Bellemare, A.Alex Graves, M.Martin Riedmiller, A. K.Andreas K Fidjeland, G.Georg Ostrovski and others. Human-level control through deep reinforcement learning.nature51875402015, 529--533
  back to text
• 100 bookK. P.Kevin P Murphy. Machine learning: a probabilistic perspective.MIT press2012
  back to text
• 101 inproceedingsA.Anusha Nagabandi, G.Gregory Kahn, R. S.Ronald S Fearing and S.Sergey Levine. Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning.IEEE International Conference on Robotics and Automation (ICRA)IEEE2018, 7559--7566
  back to text
• 102 articleM.Maribel Pino, M.Mélodie Boulay, F.François Jouen and A. S.Anne Sophie Rigaud. “Are we ready for robots that care for us?” Attitudes and opinions of older adults toward socially assistive robots.Frontiers in aging neuroscience72015, 141
  back to text
• 103 articleS.Sam Roweis and Z.Zoubin Ghahramani. Learning nonlinear dynamical systems using the expectation-maximization algorithm.Kalman filtering and neural networks62001, 175--220
  back to text
• 104 articleM.Mostafa Sadeghi and X.Xavier Alameda-Pineda. Mixture of Inference Networks for VAE-based Audio-visual Speech Enhancement.IEEE Transactions on Signal ProcessingMarch 2021
  HALback to textback to textback to text
• 105 inproceedingsM.Mostafa Sadeghi and X.Xavier Alameda-Pineda. Robust Unsupervised Audio-visual Speech Enhancement Using a Mixture of Variational Autoencoders.IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)Barcelona, SpainMay 2020
  HALDOIback to textback to text
• 106 inproceedingsM.Mostafa Sadeghi and X.Xavier Alameda-Pineda. Switching Variational Auto-Encoders for Noise-Agnostic Audio-visual Speech Enhancement.IEEE International Conference on Acoustics, Speech and Signal ProcessingToronto, CanadaIEEEJune 2021, 1-5
  HALback to textback to text
• 107 unpublishedM.Mostafa Sadeghi, S.Sylvain Guy, A.Adrien Raison, X.Xavier Alameda-Pineda and R.Radu Horaud. Unsupervised Performance Analysis of 3D Face Alignment.October 2020, Submitted to Computer Vision and Image Understanding
  HALback to text
• 108 articleM.Mostafa Sadeghi, S.Simon Leglaive, X.Xavier Alameda-Pineda, L.Laurent Girin and R.Radu Horaud. Audio-Visual Speech Enhancement Using Conditional Variational Auto-Encoders.IEEE/ACM Transactions on Audio, Speech and Language Processing28May 2020, 1788-1800
  HALDOIback to textback to textback to text
• 109 articleS.Sarah Sebo, B.Brett Stoll, B.Brian Scassellati and M. F.Malte F Jung. Robots in groups and teams: a literature review.Proceedings of the ACM on Human-Computer Interaction4CSCW22020, 1--36
  back to text
• 110 bookR. S.Richard S Sutton and A. G.Andrew G Barto. Reinforcement learning: An introduction.MIT press2018
  back to text
• 111 articleY.-H.Ya-Huei Wu, V.Victoria Cristancho-Lacroix, C.Christine Fassert, V.Véronique Faucounau, J.Jocelyne de Rotrou and A.-S.Anne-Sophie Rigaud. The attitudes and perceptions of older adults with mild cognitive impairment toward an assistive robot.Journal of Applied Gerontology3512016, 3--17
  back to text
• 112 articleY.-H.Ya-Huei Wu, J.Jérémy Wrobel, M.Mélanie Cornuet, H.Hélène Kerhervé, S.Souad Damnée and A.-S.Anne-Sophie Rigaud. Acceptance of an assistive robot in older adults: a mixed-method study of human--robot interaction over a 1-month period in the Living Lab setting.Clinical interventions in aging92014, 801
  back to text
• 113 articleM.Mateusz Żarkowski. Multi-party turn-taking in repeated human--robot interactions: an interdisciplinary evaluation.International Journal of Social Robotics1152019, 693--707
  back to text
• 114 articleJ.Jingwei Zhang, L.Lei Tai, P.Peng Yun, Y.Yufeng Xiong, M.Ming Liu, J.Joschka Boedecker and W.Wolfram Burgard. Vr-goggles for robots: Real-to-sim domain adaptation for visual control.IEEE Robotics and Automation Letters422019, 1148--1155
  back to text