3.2.1 Axis A1 - NextGen Virtual Characters

3.2.2 Axis A2 - Populated Dynamic Worlds

3.2.3 Axis A3 - New Immersive Experiences

7.1.1 AvatarReady

• Name:
  A unified platform for the next generation of our virtual selves in digital worlds
• Keywords:
  Avatars, Virtual reality, Augmented reality, Motion capture, 3D animation, Embodiment
• Scientific Description:
  AvatarReady is an open-source tool (AGPL) written in C#, providing a plugin for the Unity 3D software to facilitate the use of humanoid avatars for mixed reality applications. Due to the current complexity of semi-automatically configuring avatars coming from different origins, and using different interaction techniques and devices, AvatarReady aggregates several industrial solutions and results from the academic state of the art to propose a simple and fast way to use humanoid avatars in mixed reality in a seamless way. For example, it is possible to automatically configure avatars from different libraries (e.g., rocketbox, character creator, mixamo), as well as to easily use different avatar control methods (e.g., motion capture, inverse kinematics). AvatarReady is also organized in a modular way so that scientific advances can be progressively integrated into the framework. AvatarReady is furthermore accompanied by a utility to generate ready-to-use avatar packages that can be used on the fly, as well as a website to display them and offer them for download to users.
• Functional Description:
  AvatarReady is a Unity tool to facilitate the configuration and use of humanoid avatars for mixed reality applications. It comes with a utility to generate ready-to-use avatar packages and a website to display them and offer them for download.
• URL:
  https://­gitlab.­inria.­fr/­avatar/­avatarreadypackage
• Contact:
  Ludovic Hoyet

7.1.2 PyNimation

• Keywords:
  Moving bodies, 3D animation, Synthetic human
• Scientific Description:
  PyNimation is a python-based open-source (AGPL) software for editing motion capture data which was initiated because of a lack of open-source software enabling to process different types of motion capture data in a unified way, which typically forces animation pipelines to rely on several commercial software. For instance, motions are captured with a software, retargeted using another one, then edited using a third one, etc. The goal of Pynimation is therefore to bridge the gap in the animation pipeline between motion capture software and final game engines, by handling in a unified way different types of motion capture data, providing standard and novel motion editing solutions, and exporting motion capture data to be compatible with common 3D game engines (e.g., Unity, Unreal). Its goal is also simultaneously to provide support to our research efforts in this area, and it is therefore used, maintained, and extended to progressively include novel motion editing features, as well as to integrate the results of our research projects. At a short term, our goal is to further extend its capabilities and to share it more largely with the animation/research community.
• Functional Description:

  PyNimation is a framework for editing, visualizing and studying skeletal 3D animations, it was more particularly designed to process motion capture data. It stems from the wish to utilize Python’s data science capabilities and ease of use for human motion research.

  In its version 1.0, Pynimation offers the following functionalities, which aim to evolve with the development of the tool : - Import / Export of FBX, BVH, and MVNX animation file formats - Access and modification of skeletal joint transformations, as well as a certain number of functionalities to manipulate these transformations - Basic features for human motion animation (under development, but including e.g. different methods of inverse kinematics, editing filters, etc.). - Interactive visualisation in OpenGL for animations and objects, including the possibility to animate skinned meshes

• URL:
  https://­gitlab.­inria.­fr/­lhoyet/­pynimation
• Contact:
  Ludovic Hoyet

Full Body Motion Capture of Single Individuals Following External Perturbations from Different Directions

• Contributors:
  Thomas Chatagnon, Charles Pontonnier, Anne-Hélène Olivier, Ludovic Hoyet and Julien Pettré
• Description:
  This dataset is composed of C3D files corresponding to full body motion of participants undergoing external perturbation at shoulder height with different sensory conditions. The temporal force profiles of the perturbations are also available. The following experiment received ethical approval from an ethics committee and all participants signed an informed consent form relative to the processing of their data. The experiments were carried on 21 healthy young adults (10 females, 11 males). All were between 20 and 38 yo with a mean age of 27.2 (std: 4.2). Mean mass was 70.2 (std: 12.1) kg and height was 1.74 (std: 0.08) m. Participants motion was recorded using 45 reflective markers and a 23 Qualisys camera system (200Hz). The markers were placed on participants following standardised anatomical landmarks. The output signal of the force sensor was processed using a Butterworth low pass filter with a 5Hz cutoff frequency without phase shift. The force sensor was synchronised with the motion capture software. Three reflective markers were also placed along the pole in order to retrieve the exact direction of the perturbations.
• Dataset PID:
  https://doi.org/10.5281/zenodo.10512652
• Project link:
  https://zenodo.org/records/10512652
• Publications:
  5
• Contact:
  julien.pettre@inria.fr
• Release contributions:

8.1.1 Human Motion Prediction under Unexpected Perturbation

Participants: Julien Pettré [contact].

Show image description

Overview of our model

We investigate a new task in human motion prediction 17, which is predicting motions under unexpected physical perturbation potentially involving multiple people. Compared with existing research, this task involves predicting less controlled, unpremeditated and pure reactive motions in response to external impact and how such motions can propagate through people. It brings new challenges such as data scarcity and predicting complex interactions. To this end, we propose a new method capitalizing differentiable physics and deep neural networks as illustrated in 3, leading to an explicit Latent Differentiable Physics (LDP) model. Through experiments, we demonstrate that LDP has high data efficiency, outstanding prediction accuracy, strong generalizability and good explainability. Since there is no similar research, a comprehensive comparison with 11 adapted baselines from several relevant domains is conducted, showing LDP outperforming existing research both quantitatively and qualitatively, improving prediction accuracy by as much as 70%, and demonstrating significantly stronger generalization.

8.1.2 Standing balance recovery strategies of young adults in a densely populated environment following external perturbations

Participants: Thomas Chatagnon, Ludovic Hoyet, Anne-Hélène Olivier, Julien Pettré [contact].

In this work 5, performed in collaboration with the MimeTIC Inria team and Forschungszentrum Jülich, we investigated the recovery strategies used by young adults to maintain standing balance following external force-controlled perturbations in densely populated group formations. In particular, the moment of step initiation as well and the characteristics of the first recovery steps and hand-raising were studied here. The experimental data considered in this work are part of a larger dataset relying on a new experimental paradigm inspired by Feldmann and Adrian (2023). In this experiment, 20 participants (8 females, 12 males, 24.8$\pm$3.7 y.o.) equipped with motion capture suits were asked to stand in tightly packed formation before receiving a force-controlled perturbation. In total, four group configurations and two interpersonal distancing conditions have been investigated here. The standing balance recovery strategies observed in this dense groups experiment were then compared with the observed behaviour of single individuals following external perturbations. Results suggest that the moment of initiation for recovery steps was affected by the initial interpersonal distancing conditions. The first recovery steps within the studied dense groups were observed to be slower, smaller and more dispersed than those of single individuals for comparable level of perturbation intensity. However, the relationship between the average speed of first recovery steps and the length of these steps remained similar to the one of single individuals. This suggests that the first recovery step duration remained almost constant during both the dense groups experiment and the experiment with single individuals. Finally, we observed a significant occurrence of participants raising their hands, as physical interactions played an important role in this dense groups experiment. This behaviour was mainly observed to be initiated before recovery steps.

8.1.3 Temporal segmentation of motion propagation in response to an external impulse

Participants: Thomas Chatagnon, Julien Pettré [contact].

Show image description

Overview of the phases of impulse propagation for the blue person in the middle: (0) no external impact (i) receiving an impulse, (ii) receiving and passing on an impulse, and (iii) passing on an impulse. The black dots represent a contact.

In high-density crowds, local motion can propagate, amplify, and lead to macroscopic phenomena, including ”density waves”. These density waves only occur when individuals interact, and impulses are transferred to neighbours. How this impulse is passed on by the human body and which effects this has on individuals is still not fully understood. To further investigate this, experiments focusing on the propagation of a push were conducted within the EU-funded project CrowdDNA, in collaboration with Forschungszentrum Jülich 6. In the experiments the crowd is greatly simplified by five people lining up in a row (Figure 4). The rearmost person in the row was pushed forward in a controlled manner with a punching bag. The intensity of the push, the initial distance between participants and the initial arm posture were varied. Collected data included side view and top view video recordings, head trajectories, 3D motion using motion capturing (MoCap) suits as well as pressure measured at the punching bag. With a hybrid tracking algorithm, the MoCap data are combined with the head trajectories to allow an analysis of the motion of each limb in relation to other persons. The observed motion of the body in response to the push can be divided into three phases. These are (i) receiving an impulse, (ii) receiving and passing on an impulse, and (iii) passing on an impulse. Using the 3D MoCap data, we can identify the start and end times of each phase. To determine when a push is passed on, the forward motion of the person in front has to be considered. The projection of the center of mass relative to the initial position of the feet is a measure of the extent to which a person is displaced from the rest position. Specifying the timing of these phases is particularly important to distinguish between different types of physical interactions. Our results contribute to the development and validation of a pedestrian model for identifying risks due to motion propagation in dense crowds.

8.1.4 A survey on realistic virtual humans in animation: what is realism and how to evaluate it?

Participants: Ludovic Hoyet, Anne-Hélène Olivier [contact], Rim Rekik Dit Nekhili, Katja Zibrek.

Show image description

The central goal of this survey is to explore the aspects of animation realism of virtual humans. We approach this topic by reviewing existing definitions of realism, provide a taxonomy of animation features that influence realism, and discuss existing evaluation methods: objective, subjective and hybrid approaches. Two examples of subjective approaches are depicted in the image above: (left) a user rating realism with a self-report measure, and (right) a user reacting to the virtual human while being immersed in a VR environment).

Generating realistic animated virtual humans is a problem that has been extensively studied with many applications in different types of virtual environments. However, the creation process of such realistic animations is challenging, especially because of the number and variety of influencing factors, that should then be identified and evaluated. In this survey 10, performed in collaboration with the Morpheo Inria team, we attempt to provide a clearer understanding of how the multiple factors that have been studied in the literature impact the level of realism of animated virtual humans (see Figure 5), by providing a survey of studies assessing their realism. This includes a review of features that have been manipulated to increase the realism of virtual humans, as well as evaluation approaches that have been developed. As the challenges of evaluating animated virtual humans in a way that agrees with human perception are still active research problems, this survey further identifies important open problems and directions for future research.

8.1.5 Collision avoidance behaviours of young adult walkers: Influence of a virtual pedestrian’s age-related appearance and gait profile

Participants: Ludovic Hoyet, Anne-Hélène Olivier [contact].

Show image description

Virtual environment from the viewpoint of the participant’s starting position (A), real-world set-up (B), and the Young Adult (YA) and Older Adult (OA) virtual pedestrians (C).

Populating virtual words with realistic pedestrian behaviours is challenging but essential to ensure natural interactions with users. In pedestrian encounters, successful collision avoidances require adapting speed and/or locomotor trajectory based on situational- and personal-characteristics. Personal characteristics such as the age of a pedestrian can easily be observed, but whether it affects avoidance behaviours is unknown. This is however an important question to ensure realism and variety of the simulations. The purpose of the current study 13, performed in collaboration with the University of Waterloo and the Wilfrid Laurier University (Canada), was to examine the influence of a virtual pedestrian’s (VP) age-related appearance and gait profile on avoidance behaviours during a circumvention task. We expected that older adult (OA) gait characteristics and/or appearance would result in more cautious behaviours. Young adults (YA; n=17, 5 females; 23.6$\pm$2.7yrs) navigated a virtual street using HMD. Individuals walked 8m towards a goal, while avoiding an approaching VP (Figure 6) who would approach and steer towards the participant’s left, right, or continue straight, while exhibiting different age-related appearances and gait profiles: 1. YA appearance, YA gait; 2. OA appearance, OA gait; 3. OA appearance, YA gait; and 4. YA appearance, OA gait. Statistical analyses revealed significant effects of VP appearance and gait profile on clearance distance at the time of crossing. Clearance was larger when the VP resembled an OA (M=.79m$\pm$.23) versus YA (M=.76m$\pm$.19), and when the VP walked like an OA (M=.80m$\pm$.23) versus YA (M=.76m$\pm$.20). Larger clearance distances observed with OA characteristics may be due to societal norms associated with the principle of parental respect as well as a cautious strategy for any potential instability (wavering) in balance commonly observed in OA. This research sheds light on how age-related cues influence pedestrian interactions, with implications for the design of populated virtual environments.

8.1.6 Entropy and Speed: Effects of Obstacle Motion Properties on Avoidance Behavior in Virtual Environment

Participants: Ludovic Hoyet, Yuliya Patotskaya, Julien Pettré, Katja Zibrek [contact].

Show image description

Participants traversed a virtual room, avoiding a moving cylinder with varying speed and entropy.

Avoiding moving obstacles in immersive environments requires adjustments in the walking trajectory and depends on the type of obstacle movement. Previous research studied the impact of speed and direction of motion but not much is known about how predictability of motion impacts human circumvention in terms of distance to the obstacle (proximity). In this paper 15, we investigate how participants navigate through VR collision avoidance scenarios with obstacles of varying motion characteristics in terms of speed and predictability (Figure 7). We introduce a novel concept of creating unpredictable motion using entropy calculations. We anticipated that higher entropy would increase the proximity distance which we measured with several metrics related to the distance from the obstacle and centre of the scene. We found a significant influence of motion speed and predictability on proximity-related metrics, with participants exhibiting a tendency to maintain larger distances in scenarios where obstacle speed and entropy were higher. We also outline two decision-making strategies for avoidance behaviour and investigate the factors that influence individuals' selection of one strategy over the other.

8.1.7 Correspondence-free online human motion retargeting

Participants: Anne-Hélène Olivier [contact], Rim Rekik Dit Nekhili.

Show image description

Given an untracked motion performed by a source subject (top) and a target body shape (bottom left), our method animates the target with the source motion, preserving temporal correspondences of the output motion (bottom right).

In this work 16, performed in collaboration with the Morpheo Inria team, we present a data-driven framework for unsupervised human motion retargeting that animates a target subject with the motion of a source subject (Figure 8). Our method is correspondence-free, requiring neither spatial correspondences between the source and target shapes nor temporal correspondences between different frames of the source motion. This allows to animate a target shape with arbitrary sequences of humans in motion, possibly captured using 4D acquisition platforms or consumer devices. Our method unifies the advantages of two existing lines of work, namely skeletal motion retargeting, which leverages long-term temporal context, and surface-based retargeting, which preserves surface details, by combining a geometry-aware deformation model with a skeleton-aware motion transfer approach. This allows to take into account long-term temporal context while accounting for surface details. During inference, our method runs online, i.e. input can be processed in a serial way, and retargeting is performed in a single forward pass per frame. Experiments show that including long-term temporal context during training improves the method's accuracy for skeletal motion and detail preservation. Furthermore, our method generalizes to unobserved motions and body shapes. We demonstrate that our method achieves state-of-the-art results on two test datasets and that it can be used to animate human models with the output of a multi-view acquisition platform.

8.2.1 Collaborative services for Crowd Safety systems across the Edge-Cloud Computing Continuum

Participants: Julien Pettré [contact].

Show image description

The ideas of digital twins focus on automatically controlling the state of data-driven and computational models by comparing the current state, derived from historical and real-time data, with the ideal or desired state gained from simulation.

Crowd or mass gatherings at various venues such as entertainment events or transportation systems are faced by individuals on a daily basis. Crowd management for largescale planned events is essential, and depends on behavioural and mathematical modelling to understand how crowds move in different scenarios. In this paper, we propose the creation of a set of collaborative and services, the CrowdMesh platform, to allow for processing, analysis, and sharing of crowd monitoring datasets constructed over the Edge-Cloud Continuum (see Figure 9). Comprehensive tools and best practices are needed to ensure that analysis pipelines are robust, and that results can be replicated 12. This work was performed in collaboration with the Stack Inria team.

8.2.2 Resolving Collisions in Dense 3D Crowd Animations

Participants: Ludovic Hoyet, Anne-Hélène Olivier, Julien Pettré [contact].

Show image description

Our physics-based method resolves the collisions in highly-dense 3D crowds, enabling the synthesis of 3D crowd animations where characters realistically interact and push each other, as shown in this indoor concert scene. Through a rigorous perceptual study, we demonstrate that resolving collisions is needed to generate 3D dense crowds that move in a natural and convincing way, while traditional animation methods do not model or correct such contacts between 3D characters.

In this work 7, performed in collaboration with the Universidad Rey Juan Carlos (Spain), we propose a novel contact-aware method to synthesize highly-dense 3D crowds of animated characters. Existing methods animate crowds by, first, computing the 2D global motion approximating subjects as 2D particles and, then, introducing individual character motions without considering their surroundings. This creates the illusion of a 3D crowd, but, with density, characters frequently intersect each other since character-to-character contact is not modeled. We tackle this issue and propose a general method that considers any crowd animation and resolves existing residual collisions. To this end, we take a physics-based approach to model contacts between articulated characters. This enables the real-time synthesis of 3D high-density crowds with dozens of individuals that do not intersect each other, producing an unprecedented level of physical correctness in animations (Figure 10). Under the hood, we model each individual using a parametric human body incorporating a set of 3D proxies to approximate their volume. We then build a large system of articulated rigid bodies, and use an efficient physics-based approach to solve for individual body poses that do not collide with each other while maintaining the overall motion of the crowd. We first validate our approach objectively and quantitatively. We then explore relations between physical correctness and perceived realism based on an extensive user study that evaluates the relevance of solving contacts in dense crowds. Results demonstrate that our approach outperforms existing methods for crowd animation in terms of geometric accuracy and overall realism.

8.3.1 With or Without You: Effect of Contextual and Responsive Crowds on VR-based Crowd Motion Capture

Participants: Marc Christie, Ludovic Hoyet, Julien Pettré [contact], Tairan Yin.

Show image description

Illustration of our proposed 3R paradigm (Replace-Record-Replay), where a single user (red outline) is initially immersed into a simulated contextual crowd whose autonomous agents (green outline) are successively replaced by the user’s captured data (blue outline). The consistency of the user’s behavior in this 3R scenario is compared with 4R (Replace-Record-Replay-Responsive), where the blue agents are made locally responsive.

While data is vital to better understand and model interactions within human crowds, capturing real crowd motions is extremely challenging. Virtual Reality (VR) demonstrated its potential to help, by immersing users into either simulated virtual crowds based on autonomous agents, or within motion-capture-based crowds. In the latter case, users' own captured motion can be used to progressively extend the size of the crowd, a paradigm called Record-and-Replay (2R). However, both approaches demonstrated several limitations which impact the quality of the acquired crowd data. In this work 11, we propose the new concept of contextual crowds to leverage both crowd simulation and the 2R paradigm towards more consistent crowd data. We evaluate two different strategies to implement it, namely a Replace-Record-Replay (3R) paradigm where users are initially immersed into a simulated crowd whose agents are successively replaced by the user's captured-data, and a Replace-Record-Replay-Responsive (4R) paradigm where the pre-recorded agents are additionally endowed with responsive capabilities (see Figure 11). These two paradigms are evaluated through two real-world-based scenarios replicated in VR. Our results suggest that the behaviors observed in VR users with surrounding agents from the beginning of the recording process are made much more natural, enabling 3R or 4R paradigms to improve the consistency of captured crowd datasets.

8.3.2 Virtual Crowds Rheology: Evaluating the Effect of Character Representation on User Locomotion in Crowds

Participants: Ludovic Hoyet, Jordan Martin, Julien Pettré [contact].

Show image description

Participant (yellow outline) moving through virtual crowds. Top-left: enclosing elliptic cylinder at $1.0a/m^2$ (where $a$ stands for agents); Top-middle: puppets at $1.0a/m^2$; Top-right: realistic characters at $1.0a/m^2$; Bottom-left: enclosing elliptic cylinder at $1.5a/m^2$; Bottom-middle: puppets at $0.75a/m^2$; Bottom-right: realistic characters at $2a/m^2$.

Crowd data is a crucial element in the modeling of collective behaviors, and opens the way to simulation for their study or prediction. Given the difficulty of acquiring such data, virtual reality is useful for simplifying experimental processes and opening up new experimental opportunities. This comes at the cost of the need to assess the biases introduced by the use of this technology. This work 9 is part of this effort, and investigates the effect of the graphical representation of a crowd on the behavior of a user immersed within (Figure 12), in collaboration with the firefighter's department of the Laboratoire Central de la Prefecture de Police de Paris. More specifically, we inspect the virtual navigation through virtual crowds, in terms of travel speeds and local navigation choices as a function of the visual representation of the virtual agents that make up the crowd (simple geometric model, anthropomorphic model or realistic model). Through an experiment in which we ask a user to navigate virtual crowds of varying densities, we show that the effect of the visual representation is limited, but that an anthropomorphic representation offers the best trade-off between computational complexity and ecological validity, even though a more realistic representation can be preferred when user behaviour is studied in more details. Our work leads to clear recommendations on the design of immersive simulations for the study of crowd behavior.

8.3.3 Wheelchair Proxemics: interpersonal behaviour between pedestrians and power wheelchair drivers in real and virtual environments

Participants: Emilie Leblong, Anne-Hélène Olivier [contact].

Show image description

we are interested in evaluating collision avoidance strategies (b) including speed, side of avoidance, and shape-to-shape distance between a pedestrian and a power wheelchair user to improve the design of power wheelchair navigation simulators. We designed 2 experiments where one of the 2 persons involved in the interaction was static and the other was moving to a given goal while avoiding the collision. We also aim to compare avoidance behavior in real (a) and virtual (c) conditions.

Immersive environments provide opportunities to learn and transfer skills to real life. This opens up new areas of application, such as rehabilitation, where people with neurological disabilities can learn to drive a power wheelchair (PWC) through the development of immersive simulators. To expose these specific users to daily-life study interaction situations, it is important to ensure realistic interactions with the virtual humans that populate the simulated environment, as PWC users should learn to drive and navigate under everyday conditions. While non-verbal pedestrian-pedestrian interactions have been extensively studied, understanding pedestrian-PWC user interactions during locomotion is still an open research area. Our study 14, performed in collaboration with the Rainbow Inria team, aimed to investigate the regulation of interpersonal distance (i.e., proxemics) between a pedestrian and a PWC user in real and virtual situations (Figure 13). We designed 2 experiments in which 1) participants had to reach a goal by walking (respectively driving a PWC) and avoid a static PWC confederate (respectively a standing confederate) and 2) participants had to walk to a goal and avoid a static confederate seated on a PWC in real and virtual conditions. Our results showed that interpersonal distances were significantly different whether the pedestrian avoided the PWC user or vice versa. We also showed an influence of the orientation of the person to be avoided. We discuss these findings with respect to pedestrian-pedestrian interactions, as well as their implications for the design of virtual humans interacting with PWC users for rehabilitation applications. In particular, we proposed a proof of concept by adapting existing microscopic crowd simulation algorithms to consider the specificity of pedestrian-PWC user interactions.

8.3.4 DepthLight: a Single Image Lighting Pipeline for Seamless Integration of Virtual Objects into Real Scenes

Participants: Samuel Boivin [contact], Marc Christie [contact], Raphaël Manus.

Show image description

Two qualitative examples showcasing DepthLight's capability to generate spatial-aware lighting effects from a single low dynamic range and limited field-of-view image (LDR LFOV). We integrated two 3D rabbits – half purely diffuse material, and half specular material in each input image. DepthLight reconstructs an entire emissive texture mesh representation of the scene that re-casts spatialized lighting on virtual objects and enables seamless integration of synthetic objects in real images for VFX applications whatever their location in the scene. This approach enables photorealistic relighting of objects, capturing intricate lighting variations and occlusion effects that traditional environment map representations fail to achieve.

We present DepthLight, a novel method to estimate spatial lighting for photorealistic Visual Effects (VFX) using a single image. Previous approaches either relied on estimated or captured environment maps that fail to account for localized lighting effects or use simplified representations of estimated light positions that do not fully capture the complexity of the lighting process. DepthLight addresses these limitations by using a single LDR image with a limited field of view (LFOV) as an input to build an emissive texture mesh, producing a simple and lightweight 3D representation for photorealistic object relighting. Our approach includes a two-step HDR environment map estimation process. First, an LDR panorama is generated around the input image using a photorealistic diffusion-based inpainting technique. Then, a LDR to HDR network reconstructs the HDR panorama. A deep-learning depth estimation technique is exploited on the photorealistic LDR panorama to finally build an emissive mesh representation which provides spatial lighting information and that is used to relight virtual objects and compose the final image. This flexible pipeline can be easily integrated into different VFX production workflows. In our experiments, DepthLight shows that virtual objects are seamlessly integrated into real scenes with a visually plausible simulation of the lighting. We compared our results to the ground truth lighting using Unreal Engine, as well as to state-of-the-art approaches that use pure HDRI lighting techniques. Finally, we validated our approach using well-established image quality assessment metrics.

8.3.5 Cinematographic Camera Diffusion Model

Participants: Marc Christie [contact].

Designing effective camera trajectories in virtual 3D environments is a challenging task even for experienced animators. Despite an elaborate film grammar, forged through years of experience, that enables the specification of camera motions through cinematographic properties (framing, shots sizes, angles, motions), there are endless possibilities in deciding how to place and move cameras with characters. Dealing with these possibilities is part of the complexity of the problem. While numerous techniques have been proposed in the literature (optimization-based solving, encoding of empirical rules, learning from real examples,...), the results either lack variety or ease of control. In this work 8, we propose a cinematographic camera diffusion model using a transformer-based architecture to handle temporality and exploit the stochasticity of diffusion models to generate diverse and qualitative trajectories conditioned by high-level textual descriptions. We extend the work by integrating keyframing constraints and the ability to blend naturally between motions using latent interpolation, in a way to augment the degree of control of the designers. We demonstrate the strengths of this text-to-camera motion approach through qualitative and quantitative experiments and gather feedback from professional artists. The code and data are available at https://github.com/jianghd1996/Camera-control.

8.3.6 SPARK: Self-supervised Personalized Real-time Monocular Face Capture

Participants: Marc Christie [contact], Kelian Baert.

Feedforward monocular face capture methods seek to reconstruct posed faces from a single image of a person. Current state of the art approaches have the ability to regress parametric 3D face models in real-time across a wide range of identities, lighting conditions and poses by leveraging large image datasets of human faces. These methods however suffer from clear limitations in that the underlying parametric face model only provides a coarse estimation of the face shape, thereby limiting their practical applicability in tasks that require precise 3D reconstruction (aging, face swapping, digital make-up,...). In this work , we propose a method for high-precision 3D face capture taking advantage of a collection of unconstrained videos of a subject as prior information. Our proposal builds on a two stage approach. We start with the reconstruction of a detailed 3D face avatar of the person, capturing both precise geometry and appearance from a collection of videos. We then use the encoder from a pre-trained monocular face reconstruction method, substituting its decoder with our personalized model, and proceed with transfer learning on the video collection. Using our pre-estimated image formation model, we obtain a more precise self-supervision objective, enabling improved expression and pose alignment. This results in a trained encoder capable of efficiently regressing pose and expression parameters in real-time from previously unseen images, which combined with our personalized geometry model yields more accurate and high fidelity mesh inference. Through extensive qualitative and quantitative evaluation, we showcase the superiority of our final model as compared to state-of-the-art baselines, and demonstrate its generalization ability to unseen pose, expression and lighting.

8.3.7 AKiRa: Augmentation Kit on Rays for optical video generation

Participants: Marc Christie [contact], Robin Courant.

Recent advances in text-conditioned video diffusion have greatly improved video quality. However, these methods offer limited or sometimes no control to users on camera aspects, including dynamic camera motion, zoom, distorted lens and focus shifts. These motion and optical aspects are crucial for adding controllability and cinematic elements to generation frameworks, ultimately resulting in visual content that draws focus, enhances mood, and guides emotions according to filmmakers' controls. In this work , we aim to close the gap between controllable video generation and camera optics. To achieve this, we propose AKiRa (Augmentation Kit on Rays), a novel augmentation framework that builds and trains a camera adapter with a complex camera model over an existing video generation backbone. It enables fine-tuned control over camera motion as well as complex optical parameters (focal length, distortion, aperture) to achieve cinematic effects such as zoom, fisheye effect, and bokeh. Extensive experiments demonstrate AKiRa's effectiveness in combining and composing camera optics while outperforming all state-of-the-art methods. This work sets a new landmark in controlled and optically enhanced video generation, paving the way for future optical video generation methods.

8.3.8 E.T. the Exceptional Trajectories: Text-to-camera-trajectory generation with character awareness

Participants: Marc Christie [contact], Robin Courant.

Stories and emotions in movies emerge through the effect of well-thought-out directing decisions, in particular camera placement and movement over time. Crafting compelling camera trajectories remains a complex iterative process, even for skilful artists. To tackle this, in this work, we propose a dataset called the Exceptional Trajectories (E.T.) with camera trajectories along with character information and textual captions encompassing descriptions of both camera and character. To our knowledge, this is the first dataset of its kind. To show the potential applications of the E.T. dataset, we propose a diffusion-based approach, named Director, which generates complex camera trajectories from textual captions that describe the relation and synchronisation between the camera and characters. To ensure robust and accurate evaluations, we train on the E.T. dataset CLaTr, a Contrastive Language-Trajectory embedding for evaluation metrics. We posit that our proposed dataset and method significantly advance the democratization of cinematography, making it more accessible to common users.

8.3.9 Plausible and Diverse Human Hand Grasping Motion Generation

Participants: Marc Christie [contact], Xiaoyuan Wang.

Techniques to grasp targeted objects in realistic and diverse ways find many applications in computer graphics, robotics and VR. This studygenerates diverse grasping motions while keeping plausible final grasps for human hands. We first build on a Transformer-based VAE to encode diverse reaching motions into a latent representation noted as GMF and then train an MLP-based cVAE to learn the grasping affordance of targeted objects. Finally, through learning a denoising process, we condition GMF with affordance to generate grasping motions for the targeted object. We identify improvements in our results, and will further address them in future work.

Cifre InterDigital - Deep-based semantic representation of avatars for virtual reality

Participants: Ludovic Hoyet [contact], Philippe De Clermont Gallerande.

The overall objective of the PhD thesis of Philippe De Clermont Gallerande, which started in February 2023, is to explore novel approaches (including both full body and facial elements) to enable both full body and facial encoding and decoding for multi-user immersive experiences. Objective is also to enable the evaluation of the quality of experience. More specifically, one of the focus is to propose solutions to represent digital characters (avatars) with semantic-based approaches in a context of multi-user immersive telepresence, that are compact, plausible and simulatenously resiliant to data perturbation caused by streaming. This PhD is conducted within the context of the joint laboratory Nemo.ai between Inria and InterDigital, and more specifically within the Ys.ai project which is dedicated to exploring novel research questions and applications in Virtual Reality. This work is also conducted in collaboration between the two Inria teams Hybrid and Virtus, as well as with the Interactive Media team of InterDigital.

Cifre InterDigital - Facial features from high quality cinema footage

Participants: Marc Christie [contact], Kelian Baert.

The overall objective of the PhD thesis of Kelian Baert funded by VFX company Mikros Image is to explore novel representations to extract facial features from high quality cinema footage, and provide intuitive techniques to perform facial editing including shape, appearance and animation. More precisely, we search to improve the controllability of learning-based techniques techniques for editing photo-realistic faces in video sequences, aimed at the visual effects for cinema. The aim is to accelerate post-production processes on faces by enabling an artist to finely control different characteristics over time. There are numerous applications: rejuvenation and aging, make-up/tattooing, strong modifications morphology (adding a 3rd eye, for example), replacing an understudy with the actor's face by the actor's face, adjustments to the actor's acting. The PhD will rely on a threefold approach: transfer of features from real to synthetic, editing in the synthetic domain on simplified representations, and then to transfer the contents back to photorealistic sequences using GAN/Diffusion-based models to ensure visual quality.

LCPP (PhD contract) - Immersive crowd simulation for the study and design of public spaces

Participants: Julien Pettré [contact], Ludovic Hoyet, Jordan Martin.

The overall objective of the PhD thesis of Jordan Martin, started in November 2022, is to explore the use of Virtual Reality to better design and assess public spaces. The VirtUs team specialises in the simulation, animation and immersion of virtual crowds. The aim of this thesis is to explore new ways of analysing crowd behaviours in real environments using new virtual reality technologies that allow users to be directly immersed in digital replicas of these situations. More specifically, Jordan explores the technical conditions of Virtual Reality experiments that lead to collecting realistic data. These conditions revolve around the visual representations of virtual humans, as well as display conditions.

Unreal MegaGrant

Participants: Marc Christie [contact], Anthony Mirabile.

The objective of the Unreal Megagrant (70k€) is to initiate the development of Augmented Reality techniques as animation authoring tools. While we have demonstrated the benefits of VR as a relevant authoring tool for artists, we would like to explore with this funding the animation capacities of augmented reality techniques. Underlying challenges are pertained to the capacity of estimating precisely hand poses for fine animation tasks, but also exploring means of high-level authoring tools such as gesturing techniques.

10.1.1 Inria associate team not involved in an IIL or an international program

10.2.1 Visits of international scientists

10.3.1 Horizon Europe

10.3.2 H2020 projects

Défi Ys.AI

Participants: Ludovic Hoyet, Philippe De Clermont Gallerande.

With the recent annoucements about massive investments on the Metaverses, which are seen as the future of the social and professional immersive communication for the emergent AI-based e-society, there is a need for the development of dedicated metaverse technologies and associated representation formats. In this context, the objective of this joint project between Inria and InterDigital is to focus on the representation formats of digital avatars and their behavior in a digital and responsive environment. In particular, the primary challenge tackled in this project consists in solving the uncanny valley effect to provide users with a natural and lifelike social interaction between real and virtual actors, leading to full engagement in those future metaverse experiences.

ANR Animation Conductor

Participants: Marc Christie, Ludovic Hoyet.

Duration: Oct 2023 - Oct 2027 Team funding: 286k€ Partners: LiX in Polytechnique and Dada Animation company

The fundamental idea of the ANR Animation Conductor project is to (i) express simultaneous multimodal inputs as high-level animation principles into a motion characteristics space, (ii) exploit spatial and temporal characteristics of the input signals to edit existing animations using learning techniques inspired by style transfer, (iii) combine the style transfer techniques with authoring constraints such as physics-based and (iv) co-design interactive tools with creative artists to exploit them in industrial pipelines. More precisely, we first aim at providing new ways of understanding on which part of an animation a “conductor” (i.e. animator or supervisor) is working, to build a correlation between different input signals and output animation curves by designing a dataset with experienced animators. Secondly, we aim at designing new computational models to efficiently modify 3D animations from mimics through the use of individual or combined input modalities - namely records of voices and sounds, video records of body parts gestures, and 3D space-time acquisition from lightweight VR worn or mounted systems, while not requiring full MOCAP infrastructure. To this end, we propose to develop novel methods able to extract the spatial and temporal authoring potential of these modalities into a motion characteristics space, as well as exploring new ways to leverage the use of combined modalities to ease 3D animation control, inspired from style-transfer techniques in animation, and to compose with authoring constraints. This project targets direct applications and prototypes, starting with our open-source one, within French animation studios for the refinement of existing animations for shape and character subparts.

11.1.1 Scientific events: organisation

• Julien Pettré, Katja Zibrek, Anne-Hélène Olivier: Co-Organizers of the Symposium " Stepping forward with immersive technology: virtual characters in applications", EWIC 2024, Naples, Italy
• Julien Pettré, Katja Zibrek, Anne-Hélène Olivier: Co-organizers of the Workshop "Virtual Humans and Crowds in Immersive Environments", IEEE Virtual Reality 2024, Orlando, USA
• Anne-Hélène Olivier: Chair of the steering committee of the Symposium on Applied Perception (SAP)

11.1.2 Scientific events: selection

11.1.3 Journal

11.1.4 Invited talks

• Katja Zibrek: Perception of virtual humans in VR: The importance of animation, University of Bonn, Germany, 22.6.2024
• Katja Zibrek: Perception of virtual humans in VR, IVA Workshop Invited talk, Glasgow, UK, 19.9.2024
• Katja Zibrek: Perception in Virtual Reality, NYU Abu Dhabi, UAE (remote talk), 1.10.2024
• Anne-Hélène Olivier and Julien Pettré: Du mouvement individuel au mouvement collectif: études expérimentales et modèles d'interactions . Colloque Le mouvement humain, des origines aux olympiades , Collège de France, July 1-2, 2024, Paris.
• Anne-Hélène Olivier: Social agents and non-verbal interactions: realism of virtual agents’ motion and validity of immersive interactions. XRIA Workshop, Santiago de Compostela, Spain, October 20, 2024.
• Ludovic Hoyet and Anne-Hélène Olivier: Populating virtual worlds: Leveraging VR to create the next generation of immersive crowd. Workshop 3D Move, Grenoble, France, December 2, 2024.
• Julien Pettré: Modeling and simulation of human crowds: the emergence of machine learning models and the question of training data. Seminar at the University of Wuppertal, Germany (december 2024)
• Julien Pettré: Modeling and simulation of human crowds: the emergence of machine learning models and the question of training data. Seminar at the conference : Collective Motion of Animals and Robots, Cargese, France (may 2024)
• Julien Pettré: Que diable font-ils au Hellfest. Talk at the Arts, Culture et Patrimoine Seminar, IRISA, France

11.1.5 Leadership within the scientific community

• Anne-Hélène Olivier: Chair of the steering committee of the Symposium on Applied Perception (SAP)

11.1.6 Scientific expertise

• Anne-Hélène Olivier: ERC Starting Grants 2024, AAP 2024 - Emergence Occitanie, ANR Envergure Amorçage 2024, Bourgogne Franche-Comté

11.1.7 Research administration

• Ludovic Hoyet is in charge of the “Virtual Reality, Virtual Humans, Interactions and Robotics” department of the IRISA laboratory (Institut de Recherche en Informatique et Systèmes Aléatoires). This department includes four Inria teams (Hybrid, MimeTIC, Rainbow, Virtus) and is aligned with one of the strategic objectives of the Inria Rennes centre described in the Inria COP (i.e., “humans-robots-virtual worlds interactions”).

11.2.1 Teaching

• Master: Marc Christie, Head of Master 2 Ingénierie Logicielle (45 students), University of Rennes, France
• Master: Marc Christie, "Multimedia Mobile", Master 2, leader of the module, 32h (IL) + 32h (Miage), Computer Science, University of Rennes, France
• Master: Marc Christie, "Projet Industriel Transverse", Master 2, 32h, leader of the module, Computer Science, University of Rennes, France
• Master: Marc Christie, "Modelistion Animation Rendu", Master 2, 16h, leader of the module, Computer Science, University of Rennes, France
• Master: Marc Christie, "Web Engineering", Master 1, 16h, leader of the module, Computer Science, University of Rennes, France
• Master: Marc Christie, "Advanced Computer Graphics", Master 1, 10h, leader of the module, Computer Science, ENS, France
• Master: Marc Christie, "Motion for Animation and Robotics", Master 2 SIF, Computer Science, France
• Master : Ludovic Hoyet, Motion Analysis and Gesture Recognition, 12h, INSA Rennes, France
• Master : Ludovic Hoyet, Computer Graphics, 8h, Ecole Normale Supérieure de Rennes, France
• Master : Ludovic Hoyet, Réalité Virtuelle pour l'Analyse Ergonomique, Master Ingénierie et Ergonomie des Activités Physique, 21h, University Rennes 2, France
• Master : Aline Hufschmitt, Using Unreal Engine for military training simulation creation, 36h, Académie Militaire de Saint-Cyr Coëtquidan, France
• Master : Aline Hufschmitt, Multi-user LAN simulation under Unreal Engine, 24h, Académie Militaire de Saint-Cyr Coëtquidan, France
• Master : Aline Hufschmitt, Artificial intelligence for simulating AMI/ENI behavior under Unreal Engine, 30h, Académie Militaire de Saint-Cyr Coëtquidan, France
• Master : Aline Hufschmitt, Artificial intelligence and Cyber Defense, 30h, Académie Militaire de Saint-Cyr Coëtquidan, France
• Master : Aline Hufschmitt, Adrenaline Rush Project, 30h, Académie Militaire de Saint-Cyr Coëtquidan, France
• Licence : Aline Hufschmitt, C programming, 40h, Académie Militaire de Saint-Cyr Coëtquidan, France
• Master : Anne-Hélène Olivier, co-leader of the APPCM Master (60 students) "Activités Physiques et Pathologies Chroniques et Motrices", STAPS, University Rennes2, France
• Master : Anne-Hélène Olivier, "Recueil et traitement des données", 26H, Master 1 and 2 APPCM/IEAP/EOPS, University Rennes2, France
• Master : Anne-Hélène Olivier, "Evaluation fonctionnelle des pathologies motrices", 3H Master 2 APPCM, University Rennes2, France
• Master : Anne-Hélène Olivier, "Méthodologie de la recherche et accompagnement de stage", 15H, Master 1 and 2 APPCM, University Rennes2, France
• Licence : Anne-Hélène Olivier, "Analyse cinématique du mouvement", 100H , Licence 1, University Rennes 2, France
• Licence : Anne-Hélène Olivier, "Biomécanique spécifique aux APA", 20H , Licence 3, University Rennes 2, France
• Licence : Anne-Hélène Olivier, "Biomécanique de l'avance en âge", 12H , Licence 3, University Rennes 2, France
• Master : Katja Zibrek, supervision of students' internship abroad program, Ecole supérieure d'ingénieurs de Rennes, 9h, University Rennes 2, France

11.2.2 Supervision

• PhD defended (beginning Sept. 2020, defended Feb. 2024): Agathe Bilhaut, Stratégies perceptivo-motrices durant la locomotion des patients douloureux chroniques : nouvelles méthodes d'analyse et de suivi, Armel Crétual, Anne-Hélène Olivier, Mathieu Ménard (Institut Ostéopathie Rennes, M2S)
• PhD defended (beginning Nov. 2020, defended April. 2024): Tairan Yin, Création de scènes peuplées dynamiques pour la réalité virtuelle, Marc Christie, Marie-Paule Cani (LIX), Ludovic Hoyet, Julien Pettré.
• PhD defended (beginning Oct. 2020, defended Sept. 2024): Jean-Baptiste Bordier, Innovative interfaces for the creation of character animations, Marc Christie
• defended (beginning Jan. 2022, defended Dec. 2024): Yulia Patotskaya, Appealing Character Design for Embodied Virtual Reality, Ludovic Hoyet, Julien Pettré, Katja Zibrek.
• PhD in progress (beginning Oct. 2020): Emilie Leblong, Prise en compte des interactions sociales dans un simulateur de conduite de fauteuil roulant électrique en réalité virtuelle : favoriser l'apprentissage pour une mobilité inclusive, Anne-Hélène Olivier, Marie Babel (Rainbow team)
• PhD in progress (beginning Nov. 2021): Rim Rekik Dit Nekhili, Learning and evaluating 3D human motion synthesis, Anne-Hélène Olivier, Stefanie Wuhrer (Morpheo team).
• PhD in progress (beginning Oct. 2021): Xiaoyuan Wang, Realistic planning of hand motions, with Marc Christie, Adnane Boukhayma (MimeTIC team).
• PhD in progress (beginning Oct. 2022): Jordan Martin, Simulation immersive de foule pour l’étude et l’aménagement de lieux destinés à accueillir du public, Ludovic Hoyet, Jean-Luc Paillat, Julien Pettré, Etienne Pinsard.
• PhD in progress (beginning Oct. 2022): Alexis Jensen, Simulation de foule dense par modélisation dynamique, Julien Pettré, Charles Pontonnier (MimeTIC).
• PhD in progress (beginning Feb. 2023): Philippe De Clermont Gallerande (CIFRE InterDigital), Deep-based semantic representation of avatars for virtual reality, Ferran Argelaguet (Hybrid team), Quentin Avril (InterDigital), Philippe-Henri Gosselin (InterDigital), Ludovic Hoyet.
• PhD in progress (beginning Feb. 2023): Tony Wolff, Creating socially reactive virtual characters for enhanced social interactions in Virtual Reality, Ludovic Hoyet, Anne-Hélène Olivier, Julien Pettré, Katja Zibrek.
• PhD in progress (beginning Sept. 2023): Celine Finet, Peuplement de scène par simulation de foule basée apprentissage, Julien Pettré.
• PhD in progress (beginning Oct. 2023): Kelian Baert, Face Editing for Digital Visual Effects in Film Production, with Marc Christie, Adnane Boukhayma
• PhD in progress (beginning Oct. 2023): Arthur Audrain, Creating digital tools to mitigate harassement in the Metaverse, with Katja Zibrek and Ferran Argelaguet

11.2.3 Juries

International journals

• 5 articleT.Thomas Chatagnon, S.Sina Feldmann, J.Juliane Adrian, A.-H.Anne-Hélène Olivier, C.Charles Pontonnier, L.Ludovic Hoyet and J.Julien Pettré. Standing Balance Recovery Strategies of Young Adults in a Densely Populated Environment Following External Perturbations.Safety Science2024, 1-21HALback to textback to text
• 6 articleS.Sina Feldmann, T.Thomas Chatagnon, J.Juliane Adrian, J.Julien Pettré and A.Armin Seyfried. Temporal segmentation of motion propagation in response to an external impulse.Safety Science1752024, 106512-106512HALDOIback to text
• 7 articleG.Gonzalo Gomez-Nogales, M.Melania Prieto-Martin, C.Cristian Romero, M.Marc Comino-Trinidad, P.Pablo Ramon-Prieto, A.-H.Anne-Hélène Olivier, L.Ludovic Hoyet, M. A.Miguel A Otaduy, J.Julien Pettré and D.Dan Casas. Resolving Collisions in Dense 3D Crowd Animations.ACM Transactions on GraphicsAugust 2024, 1-14HALDOIback to text
• 8 articleH.Hongda Jiang, X.Xi Wang, M.Marc Christie, L.Libin Liu and B.Baoquan Chen. Cinematographic Camera Diffusion Model.Computer Graphics Forum4322024, 1-14HALDOIback to text
• 9 articleJ.Jordan Martin, L.Ludovic Hoyet, E.Etienne Pinsard, J.-L.Jean-Luc Paillat and J.Julien Pettré. Virtual Crowds Rheology: Evaluating the Effect of Character Representation on User Locomotion in Crowds.IEEE Transactions on Visualization and Computer GraphicsSeptember 2024, 1-12HALDOIback to text
• 10 articleR.Rim Rekik, S.Stefanie Wuhrer, L.Ludovic Hoyet, K.Katja Zibrek and A.-H.Anne-Hélène Olivier. A Survey on Realistic Virtual Human Animations: Definitions, Features and Evaluations.Computer Graphics Forum432May 2024, e15064:1-21HALDOIback to text
• 11 articleT.Tairan Yin, L.Ludovic Hoyet, M.Marc Christie, M.-P.Marie-Paule Cani and J.Julien Pettré. With or Without You: Effect of Contextual and Responsive Crowds on VR-based Crowd Motion Capture.IEEE Transactions on Visualization and Computer Graphics2024, 1-11HALDOIback to text

International peer-reviewed conferences

• 12 inproceedingsD.Daniel Balouek and J.Julien Pettré. Collaborative services for Crowd Safety systems across the Edge-Cloud Computing Continuum.CCGridW 2024 - IEEE 24th International Symposium on Cluster, Cloud and Internet Computing WorkshopsPhiladelphia, FranceIEEEMay 2024, 92 - 97HALDOIback to text
• 13 inproceedingsS.Sheryl Bourgaize, M.Michael Cinelli, P.Pierre Raimbaud, L.Ludovic Hoyet and A.-H.Anne-Hélène Olivier. Collision avoidance behaviours of young adult walkers: Influence of a virtual pedestrian's age-related appearance and gait profile.SAP 2024 - ACM Symposium on Applied PerceptionDublin, IrelandAugust 2024, 1-10HALDOIback to text
• 14 inproceedingsE.Emilie Leblong, F.Fabien Grzeskowiak, S.Sebastien Thomas, L.Louise Devigne, M.Marie Babel and A.-H.Anne-Hélène Olivier. Wheelchair Proxemics: interpersonal behaviour between pedestrians and power wheelchair drivers in real and virtual environments.VRST 2024 - 30th ACM Symposium on Virtual Reality Software and TechnologyTrier, GermanyACM2024, 1-12HALDOIback to text
• 15 inproceedingsY.Yuliya Patotskaya, L.Ludovic Hoyet, K.Katja Zibrek and J.Julien Pettre. Entropy and Speed: Effects of Obstacle Motion Properties on Avoidance Behavior in Virtual Environment.SAP 2024 - ACM Symposium on Applied PerceptionDublin, IrelandAugust 2024, 1-13HALDOIback to text
• 16 inproceedingsR.Rim Rekik, M.Mathieu Marsot, A.-H.Anne-Hélène Olivier, J.-S.Jean-Sébastien Franco and S.Stefanie Wuhrer. Correspondence-free online human motion retargeting.International Conference on 3D Vision (3DV)Davos, SwitzerlandIEEEMarch 2024, 707-716HALDOIback to text
• 17 inproceedingsJ.Jiangbei Yue, B.Baiyi Li, A.Armin Seyfried, J.Julien Pettré and H.He Wang. Human Motion Prediction Under Unexpected Perturbation.CVPR 2024 - IEEE/CVF Conference on Computer Vision and Pattern RecognitionSeattle, United StatesIEEEMarch 2024, 1501-1511HALDOIback to text

Doctoral dissertations and habilitation theses

• 18 thesisT.Tairan Yin. The One-Man-Crowd : towards single-user capture of collective motions using virtual reality.Université de RennesApril 2024HAL