Keywords
Computer Science and Digital Science
- A5.4. Computer vision
- A5.5. Computer graphics
- A5.5.1. Geometrical modeling
- A5.5.3. Computational photography
- A5.5.4. Animation
- A5.6. Virtual reality, augmented reality
- A9.1. Knowledge
- A9.2. Machine learning
- A9.3. Signal analysis
Other Research Topics and Application Domains
- B2. Health
- B2.2. Physiology and diseases
- B5.7. 3D printing
- B9.1. Education
- B9.2.2. Cinema, Television
- B9.2.3. Video games
- B9.2.4. Theater
- B9.6.6. Archeology, History
- B9.6.10. Digital humanities
1 Team members, visitors, external collaborators
Research Scientists
- Remi Ronfard [Team leader, Inria, Senior Researcher, from Jul 2020, HDR]
- Melina Skouras [Inria, Researcher, from Jul 2020]
Faculty Members
- Stefanie Hahmann [Institut polytechnique de Grenoble, Professor, from Jul 2020, HDR]
- Olivier Palombi [Univ Grenoble Alpes, Professor, from Jul 2020, HDR]
Post-Doctoral Fellow
- Nachwa Abou Bakr [Inria, from Oct 2020]
PhD Students
- Sandrine Barbois [University Hospital Grenoble Alpes, 38043, Grenoble, France, from Jul 2020]
- Thomas Buffet [Inria, from Jul 2020]
- Pierre Casati [Inria, Jul 2020]
- Qianqian Fu [Univ Grenoble Alpes, from Jul 2020]
- Vaishnavi Ameya Murukutla [Univ Grenoble Alpes, from Jul 2020]
- Emmanuel Rodriguez [Inria, from Oct 2020]
- Manon Vialle [Inria, From Oct 2020]
Technical Staff
- Remi Colin De Verdiere [Inria, Engineer, from Jul 2020 until Sep 2020]
- Pierre Ecormier-Nocca [Inria, Engineer, from Oct 2020]
Interns and Apprentices
- Johana Marku [Inria, Jul 2020]
- Antoine Saget [Univ Grenoble Alpes, from Jul 2020 until Aug 2020]
Administrative Assistants
- Geraldine Christin [Inria, from Dec 2020]
- Marion Ponsot [Inria, from Jul 2020 until Oct 2020]
2 Overall objectives
ANIMA focuses on developing computer tools for authoring and directing animated movies, interactive games and mixed-reality applications, using virtual sets, actors, cameras and lights. This raises several scientific challenges. Firstly, we need to build a representation of the story that the user/director has in mind, and this requires dedicated user interfaces for communicating the story. Secondly, we need to offer tools for authoring the necessary shapes and motions for communicating the story visually, and this requires a combination of high-level geometric, physical and semantic models that can be manipulated in real-time under the user’s artistic control. Thirdly, we need to offer tools for directing the story, and this requires new interaction models for controlling the virtual actors and cameras to communicate the desired story while maintaining the coherence of the story world.
2.1 Understanding stories
Stories can come in many forms. An anatomy lesson is a story. A cooking recipe is a story. A geological sketch is a story. Many paintings and sculptures are stories. Stories can be told with words, but also with drawings and gestures. For the purpose of creating animated story worlds, we are particularly interested in communicating the story with words in the form of a screenplay or with pictures in the form of a storyboard. We also foresee the possibility of communicating the story in space using spatial gestures. The first scientific challenge for the ANIMA team is to propose new computational models and representations for screenplays and storyboards, and practical methods for parsing and interpreting screenplays and storyboards from multimodal user input. To do this, we reverse engineer existing screenplays and storyboards, which are well suited for generating animation in traditional formats. We also explore new representations for communicating stories with a combination of speech commands, 3D sketches and 3D gestures, which promise to be more suited for communicating stories in new media including virtual reality, augmented reality and mixed reality.
2.2 Authoring story worlds
Telling stories visually creates additional challenges not found in traditional, text-based storytelling. Even the simplest story requires a large vocabulary of shapes and animations to be told visually. This is a major bottleneck for all narrative animation synthesis systems. The second scientific challenge for the ANIMA team is to propose methods for quickly authoring shapes and animations that can be used to tell stories visually. We devise new methods for generating shapes and shape families, understanding their functions, styles, material properties and affordances, authoring animations for a large repertoire of actions, and printing and fabricating articulated and deformable shapes suitable for creating physical story worlds with tangible interaction.
2.3 Directing story worlds
Lastly, we develop methods for controlling virtual actors and cameras in virtual worlds and editing them into movies in a variety of situations ranging from 2D and 3D professional animation, to virtual reality movies and real-time video games. Starting from the well-established tradition of the storyboard, we create new tools for directing movies in 3D animation, where the user is really the director, and the computer is in charge of its technical execution using a library of film idioms. We also explore new areas, including the automatic generation of storyboards from movie scripts for use by domain experts, rather than graphic artists.
3 Research program
ANIMA is a multi-disciplinary team with a common research interest in story understanding, storyworld authoring and story world directing. We plan to confront those emerging topics from mul-tiple perspectives, by organizing the team into four research themes, where we have the necessary experience and expertise to make significant contributions. Each research theme will examine thesame challenges of understanding stories, authoring story worlds, and directing story worlds, fromits own perspective and with its own research agenda.
The four research themes pursued by ANIMA are (i) the geometry of story worlds; (ii) the physics of story worlds; (iii) the semantics of story worlds; and (iv) the aesthetics of story worlds.
In each theme, significant advances in the state of the art are needed to propose computational models of stories, and build the necessary tools for translating stories to 3D graphics and anima-tion.
3.1 Geometric modeling of story worlds
Scientist in charge: Stefanie Hahmann
Other participants: Rémi Ronfard, Mélina Skouras
We aim to create intuitive tools for designing 3D shapes and animations which can be used to populate interactive, animated story worlds, rather than inert and static virtual worlds. In many different application scenarios such as preparing a product design review, teaching human anatomy with a MOOC, composing a theatre play, directing a movie, showing a sports event, 3D shapes must be modeled for the specific requirements of the animation and interaction scenarios (stories) of the application.
We will need to invent novel shape modelling methods to support the necessary affordances for interaction and maintain consistency and plausibility of the shape appearances and behaviors during animation and interaction. Compared to our previous work, we will therefore focus increasingly on designing shapes and motions simultaneously, rather than separately, based on the requirements of the stories to be told.
Previous work in the IMAGINE team has emphasized the usefulness of space-time constructions for sketching and sculpting animation both in 2D and 3D. Future work in the ANIMA team will further develop this line of research, with the long-term goal of choreographing complex multi-character animation and providing full authorial and directorial control to the user.
3.1.1 Space-time modeling
The first new direction of research in this axis will be an investigation of space-time geometric modeling, i.e. the simultaneous creation of shapes and their motions. This is in continuity with our previous work on "responsive shapes", i.e. making 3D shapes respond in an intuitive way during both design and animation.
3.1.2 Spatial interaction
A second new direction of research of the ANIMA team will be the extension of sketching and sculpting tools to the case of spatial 3D interaction using virtual reality headsets, sensors and trackers.
Even though 3D modeling can be regarded as an ideal application for Virtual Reality, it is known to suffer from the lack of control for freehand drawing. Our insight is to exploit the expressiveness of hand (controller) motion and simple geometric primitives in order to form an approximated 3D shape. The goal is not to generate a final well shaped product, but to provide a 3D sketching tool for creating early design shapes, kind of scaffolds, and for rough idea exploration. Standard 3D modeling systems can then take over to generate more complex shape details.
Research directions to be explored include (i) direct interaction using VR; (ii) applications to form a 3D shape from rough design ideas; (iii) applications to modify existing objects during design review sessions; and (iv) provide tools to ease communications about imagined shapes.
3.2 Physical modeling of story worlds
Scientist in charge: Mélina Skouras
Other participants: Stefanie Hahmann, Rémi Ronfard
When authoring and directing story worlds, physics is important to obtain believable and realistic behaviors, e.g. to determine how a garment should deform when a character moves, or how the branches of a tree bend when the wind start to blow. In practice, while deformation rules could be defined a priori (e.g. procedurally), relying on physics-based simulation is more efficient in many cases as this means that we do not need to think in advance about all possible scenarii. In ANIMA, we want to go a step further. Not only do we want to be able to predict how the shape of deformable objects will change, but we also want to be able to control their deformation. In short, we are interested in solving inverse problems where we adjust some parameters of the simulation, yet to be defined so that the output of the simulation matches what the user wants.
By optimizing design parameters, we can get realistic results on input scenarii, but we can also extrapolate to new settings. For example, solving inverse problems corresponding to static cases can be useful to obtain realistic behaviors when looking at dynamics. E.g. if we can optimize the cloth material and the shape of the patterns of a dress such that it matches what an artist designed for the first frame of an animation, then we can use the same parameters for the rest of the animation. Of course, matching dynamics is also one of our goals.
Compared to more traditional approaches, this raises several challenges. It is not clear what the best way is for the user to specify constraints, i.e. how to define what she wants (we do not necessarily want to specify the positions of all nodes of the meshes for all frames, for example). We want the shape to deform according to physical laws, but also according to what the user specified, which means that the objectives may conflict and that the problem can be over-constrained or under-constrained.
Physics may not be satisfied exactly in all story worlds i.e. input may be cartoonish, for example. In such cases, we may need to adapt the laws of physics or even to invent new ones. In computational fabrication, the designer may want to design an object that cannot be fabricated using traditional materials for example. But in this case, we cannot cheat with the physics. One idea is to extend the range of things that we can do by creating new materials (meta-materials), creating 3D shapes from flat patterns, increasing the extensibility of materials, etc.
To achieve these goals, we will need to find effective metrics (how to define objective functions that we can minimize); develop efficient models (that can be inverted); find suitable parameterizations; and develop efficient numerical optimization schemes (that can account for our specific constraints).
3.2.1 Computational design of articulated and deformable objects
We would like to extend sketch-based modeling to the design of physical objects, where material and geometric properties both contribute to the desired behaviors. Our goal in this task will be to provide efficient and easy-to-use physics-aware design tools. Instead of using a single 3D idealized model as input, we would like to use sketches, photos, videos together with semantic annotations relating to materials and motions. This will require the conceptualization of physical storyboards. This implies controlling the matter and includes the computational design of meso-scale materials that can be locally assigned to the objects; the optimization of the assignment of these materials such that the objects behave as expected; the optimization of the actuation of the object (related to the point below). Furthermore, the design of the meta-materials/objects can take into account other properties in addition to the mechanical aspects. Aesthetics, in particular, might be important.
3.2.2 Physical storyboarding
Story-boards in the context of physical animation can be seen as a concept to explain how an object/character is supposed to move or to be used (a way to describe the high-level objective). Furthermore, they can be used to represent the same object from different views, in different scales, even at different times and in different situations, to better communicate the desired behavior. Finally, they can be used to represent different objects behaving "similarly".
Using storyboards as an input to physical animation raises several scientific challenges. If one shape is to be optimized; we need to make sure that the deformed shape can be reached (i.e. that there is a continuous path from the initial shape to the final shape) - e.g. deployable structures. We will need to explore different types of inputs: full target animations, key-frames, annotations (arrows), curves, multi-modal inputs. Other types of high-level goals, which implies that the object should be moving/deforming in a certain way (to be optimized), e.g locomotion, dressing-up a character.
3.3 Semantic modeling of story worlds
Scientist in charge: Oliver Palombi
Other participants: Rémi Ronfard, Nicolas Szilas
Beyond geometry and physics, we aim at representing the semantics of story worlds. We use ontologies to organize story worlds into entities described by well defined concepts and relations between them. Especially important to us is the ability to "depict" story world objects and their properties during the design process 8 while their geometric and material properties are not yet defined. Another important aspect of this research axis is to make it possible to quickly create interactive 3D scenes and movies by assembling existing geometric objects and animations. This requires a conceptual model for semantic annotations, and high level query languages where the result of a semantic query can be a 3D scene or 3D movie.
One important application area for this research axis will be the teaching of human anatomy. The Phd thesis of Ameya Murukutla focuses on automatic generation of augmented reality lessons and exercises for teaching anatomy to medical students and sports students using the prose storyboard language which we introduced during Vineet Gandhi's PhD thesis 22. By specializing to this particular area, we are hoping to obtain a formal validation of the proposed methods before we attempt to generalize them to other domains such as interactive storytelling and computer games.
3.3.1 Story world ontologies
We will extend our previous work on ontology modeling of anatomy 21, 24 in two main directions. Firstly, we will add procedural representations of anatomic functions that make it possible to create animations. This requires work in semantic modeling of 3D processes, including anatomic functions in the teaching of anatomy. This needs to be generalized to actor skills and object affordances in the more general setting of role playing games and storytelling. Secondly, we will generalize the approach to other storytelling domains. We are starting to design an ontology of dramatic functions, entities and 3D models. In storytelling, dramatic functions are actions and events. Dramatic entities are places, characters and objects of the story. 3D models are virtual actors, sets and props, together with their necessary skills and affordances. In both cases, story world generation is the problem of linking 3D models with semantic entities and functions, in such a way that a semantic query (in natural language or pseudo natural language) can be used to create a 3D scene or 3D animation.
3.3.2 Story world scenarios
While our research team is primarily focused on providing authoring and directing tools to artists, there are cases where we also would like to propose methods for generating 3D content automatically. The main motivation for this research direction is virtual reality, where artists attempt to create story worlds that respond to the audience actions. An important application is the emerging field of immersive augmented reality theatre 16, 25, 19, 20, 15, 10, 17. In those cases, new research work must be devoted to create plausible interactions between human and virtual actors based on an executable representation of a shared scenario.
3.4 Aesthetic modeling of story worlds
Scientist in charge:Rémi Ronfard
Other participants: Stefanie Hahmann, Mélina Skouras, François Garnier
Data-driven methods for shape modeling and animation are becoming increasingly popular in computer graphics, due to the recent success of deep learning methods. In the context of the ANIMA team, we are particularly interested in methods that can help us capture artistic styles from examples and transfer them to new content. This has important implications in authoring and directing story worlds because it is important to offer artistic control to the author or director, and to maintain a stylistic coherence while generating new content. Ideally, we would like to learn models of our user's authoring and directing styles, and create contents that matches those styles.
3.4.1 Learning and transferring shape styles
We want to better understand shape aesthetics and styles, with the long-term goal of creating complex 3D scenes with a large number of shapes with consistent styles. We will also investigate methods for style transfer, allowing to re-use existing shapes in novel situations by adapting their style and aesthetics 18.
In the past exhaustive research has been done on aesthetic shape design in the sense of fairness, visual pleasing shapes using e.g. bending energy minimization and visual continuity. Note, that these aspects are still a challenge in motion design (see next section). In shape design, we now go one step further by focusing on style. Whereas fairness is general, style is more related to application contexts, which we would like to formalize.
3.4.2 Learning and transferring motion styles
While the aesthetics of spatial curves and surfaces has been extensively studied in the past, resulting in a large vocabulary of spline curves and surfaces with suitable control parameters, the aesthetics of temporal curves and surfaces is still poorly understood. Fundamental work is needed to better understand which geometric features are important in the perception of the aesthetic qualities of motions and to design interpolation methods that preserve them. Furthermore, we would like to transfer the learned motion styles to new animations. This is a very challenging problem, which we started to investigate in previous work in the limited domains of audiovisual speech animation 7 and hand puppeteering 14.
3.4.3 Learning and transfering film styles
In recent years, we have proposed new methods for automatically composing cinematographic shots in live action video 13 or 3D animation 11 and to edit them together into aesthetically pleasing movies 12. In future work, we plan to apply similar techniques for the new use case of immersive virtual reality. This raises interesting new issues because spatial and temporal discontinuities must be computed in real time in reaction to the user's movements. We have established a strong collaboration with the Spatial Media team at ENSADLAB to investigate those issues. We also plan to transfer the styles of famous movie directors to the generated movies by learning generative models of their composition and film editing styles, therefore extending the previous work of Thomas 23 from photographic style to cinematographic style. The pioneering work of Cutting and colleagues 9 used a valuable collection of 150 movies covering the years 1935 to 2010, mostly from classical Hollywood cinema. A more diverse dataset including European cinema in different genres and styles will be a valuable contribution to the field. Towards this goal, we are building a dataset of movie scenes aligned with their screenplays and storyboards.
4 Application domains
The research goals of the ANIMA team are applicable to many application domains which use computer graphics and are in demand of more intuitive and accessible authoring and directing tools for creating animated story worlds. This includes arts and entertainment, education and industrial design.
Arts and entertainement
Animated story worlds are central to the industries of 3D animation and video games, which are very strong in France. Designing 3D shapes and motions from storyboards is a worthwhile research goal for those industries, where it is expected to reduce production costs while at the same time increasing artistic control, which are two critical issues in those domains. Furthermore, story is becoming increasingly important in video games and new authoring and directing tools are needed for creating credible interactive story worlds, which is a challenge to many video game companies. Traditional live action cinematography is another application domain where the ANIMA team is hoping to have an impact with its research in storyboarding, virtual cinematography and film editing.
Performance art, including dance and theater, is an emergent application domain with a strong need for dedicated authoring and directing tools allowing to incorporate advanced computer graphics in live performances. This is a challenging application domain, where computer-generated scenography and animation need to interact with human actors in real-time. As a result, we are hoping that the theater stage becomes an experimental playground for our most exploratory research themes. To promote this new application domain, we are organizing the first international workshop on computer theater in Grenoble in February 2020, under the name Journées d’Informatique Théâtrale (JIT). The workshop will assemble theater researchers, artists and computer engineers whose practice incorporates computer graphics as a means of expression and/or a creative tool. With this workshop, our goal is to create a new research discipline that could be termed “computer theater”, following the model of computer music, which is now a well established discipline.
Education
Teaching of Anatomy is a suitable domain for research. As professor of Anatomy, Olivier Palombi gives us the opportunity to experiment in the field. The formalization of anatomical knowledge in our ontology called My Corporis Fabrica (MyCF) is already operational. One challenge for us is to formalize the way anatomy is taught or more exactly the way anatomical knowledge is transmitted to the students using interactive 3D scenes.
Museography is another related application domain for our research, with a high demand for novel tools allowing to populate and animate virtual reconstructions of art works into stories that make sense to museum audiences of all ages. Our research is also applicable to scientific museography, where animated story worlds can be used to illustrate and explain complex scientific concepts and theories.
Industrial design
Our research in designing shapes and motions from storyboards is also relevant to industrial design, with applications in the fashion industry, the automotive industry and in architecture. Those three industries are also in high demand for tools exploiting spatial interaction in virtual reality. Our new research program in physical modeling is also applicable to those industries. We have established strong partnerships in the past with PSA and Vuitton, and we will seek to extend them to architectural design as well in the future,
5 Social and environmental responsibility
5.1 Footprint of research activities
ANIMA is a small team of four permanent researchers and six students, so our footprint is limited. We estimate that we run approximately twenty computers, including laptops, desktops and shared servers, at any given time.
Research in computer graphics is not (yet) data intensive. We mostly devise procedural algorithms, which require limited amounts of data. One notable exception is our work on computational editing of live performances, which produces large amounts of ultra high definition video files. We have opted for a centralized video server architecture (KinoAI) so that at least the videos are never duplicated and always reside on a single server.
On the other hand, our research requires prowerful graphics processing units (GPU) which significantly increase the power consumption of our most powerful desktops.
The COVID19 crisis has changed our working habits heavily. We have learned to work from home most of the week, and to abandon international travel entirely. This holds the promise of a vastly reduced footprint. But it is too early to say whether this is sustainable. While the team as a whole has been able to function in adverse conditions, two of our PHD students have abandoned their theses. Probably, a balance needs to be found between working on site to maintain social contact, motivation and creativity; and working from home to save time and reduce our footprint.
5.2 Impact of research results
Our research does not directly address social and environmental issues. Our work on 3D printing may have positive effects by allowing the more efficient use of materials in the production of prototypes. Our work on virtual medical simulation and training may provide an alternative in some cases to animal experiments and costly robotic simulations.
Our most important application domain is arts and culture, including computer animation and computer games. Globally, those sectors are creating jobs, rather than destroying them, in France and in Europe. The impact of our discipline is therefore positive at least in this respect.
We are more concerned with the impact of the software industry as a whole, i.e. private companies who implement our research papers and include them in their products. We note a tendency to increase the memory requirements of software. While much effort in software engineering is devoted to improving the execution speed of graphics programs, there is not enough effort in optimizing their footprint. This is an area that may be worth investigating in our future work.
6 Highlights of the year
6.1 PhD defenses
Youna Le Vaou defended her PhD thesis in June 2020. The thesis was part of a fruitful collaboration between PSA and the ANIMA team since 2016. The context of the thesis was the following: When creating an object, designers have a mental representation of desirable shapes that can be referred to as Mental Shape Space (MSS). At an early design stage, designers explore and modify this MSS, notably through design reviews. In the automotive industry, design reviews are commonly performed using a 3D digital model inside a virtual reality system, where it is visualized at real-size with depth perception and realistic rendering. However, such a digital model cannot be modified in the immersive environment for lack of appropriate methods. Presently, the design review supervisor only gives feedback through verbal communication, which may lead to ambiguities. Our goal was to improve communication by providing the user with a fast and efficient tool that let him/her explore new 3D shapes during the immersive design review. We contributed a new surface deformation method by introducing virtual aisotropic materials of shape 2 and an aesthetic surface modification method guided by a deforming feature line (under review).
6.2 Awards
Mélina Skouras received the title of Young Researcher Fellow from the French Chapter of Eurographics.
6.3 Apppointmants
Olivier Palombi was appointed as member of the Comité d’Expertise pour les Recherches les Etudes et les Evaluations dans le domaine de la Santé (CEREES)
7 New software and platforms
Due to the diversity of topics in the ANIMA team, there is not a single software platform where our software developments can be deployed. The ANIMA project will focus its software development effort on targeted actions such as My Corporis Fabrica, which we will continue to maintain; Kino Ai, which will be made available as an open source project; and new software packages targeting the interactive storytelling and virtual reality communities. In the last category, we are already contributing software to the 3D drawing tool VairDraw 1 for use in artistic projects such as L'Ebauchoir 2.
The ANIMA team is also committed to the Graphics Replicability Stamp Initiative 3, which is an independent group of volunteers who help the community by enabling sharing of code and data as a community resource for non-commercial use.
7.1 New software
7.1.1 Kino AI
- Name: Artificial intelligence for cinematography
- Keywords: Video analysis, Post-production
- Functional Description: Kino AI is an implementation of the method described in our patent "automatic generation of cinematographic rushes using video processing". Starting from a single ultra high definitiion (UltraHD) recording of a live performance, we track and recognize all actors present on stage and generate one or more rushes suitable for cinematographic editing of a movie.
-
URL:
https://
kinoai. inria. fr/ - Publications: hal-01067093, hal-01482165
- Contact: Rémi Ronfard
- Partner: IIIT Hyderabad
8 New results
8.1 KinoAI and the audiovisual notebook : a plural solution for rehearsal studies
In this communication, we report on the work of an interdisciplinary team of computer scientists, performing art researchers and stage and video artists, towards the common goal of transforming raw video recordings of theatre rehearsals into meaningful movie narratives 3.
Our prototype system, KinoAI, combines sophisticated artificial intelligence techniques with a careful dramaturgic analysis of the rehearsals. We present extensive field experiments and propose new research directions.
8.2 Recognition of Laban Effort Qualities from Hand Motion
We present a method for clasifying hand gestures into Laban effort categories : light and strong, direct and indirect, sudden and sustained 4.
8.3 Printing-on-Fabric: Meta-Material for Self-Shaping Architectural Models
In this work, we describe a new meta-material for fabricating lightweight architectural models,consisting of a tiled plastic star pattern layered over pre-stretched fabric, and an interactive system for computer-aided design of doubly-curved forms using this meta-material. 3D-printing plastic rods over pre-stretched fabric recently gained popularity as a low-cost fabrication technique for complex free-form shapes that automatically lift in space. Our key insight is to focus on rods arranged into repeating star patterns, with the dimensions (and hence physical properties) of the individual pattern elements varying over space. Our star-based meta-material on the one hand allows effective form-finding due to its low-dimensional designspace, while on the other hand it is flexible and powerful enough to express large-scale curvature variations. Users of our system design free-form shapes by adjusting the star pattern; our system then automatically simulates the complex physical coupling between the fabric and stars to translate the design edits into shape variations. We experimentally validate our system and demonstrate strong agreement between the simulated results and the final fabricated prototypes 5.
8.4 As-Stiff-As-Needed Surface Deformation Combining ARAP Energy with an Anisotropic Material
The creation of man-made shapes can be seen as the exploration of designers’ ‘Mental Shape Space’, often supported by design reviews. To improve communication among the designers during these reviews, we introduced a new physically-based method to intuitively deform man-made shapes. This method is based on as-rigid-as possible (ARAP) shape deformation methods, known to offer a direct surface manipulation and to generate visually pleasant shapes by minimizing local deviations from rigidity. However, the organic character of ARAP shape deformations leads to undesired effects, such as surface collapsing or bulging because of an inappropriate stiffness model over the object. Our insight is that controlling the material stiffness could prevent the undesirable organic effects. Yet, we shed light on the fact that none of the ARAP-based methods offers an appropriate stiffness distribution over the object from a mechanical standpoint. We do so by introducing virtual anisotropic materials for the shape in order to improve the stiffness distribution over the surface and its deformation behavior for man-made shapes. This material is associated with a membrane-like structural behavior to further improve the stiffness distribution. Thanks to these settings, we derive a robust and intuitive deformation process that produces an anisotropic mesh deformation based on new edge weights in the ARAP formulation. The benefits of our new method are finally illustrated by typical design examples from the automotive industry and other man-made shapes. 2.
9 Bilateral contracts and grants with industry
9.1 CIFRE PhDs
9.1.1 PSA
Participants: Stefanie Hahmann, Youna Le Vaou.
We had a very successful CIFRE PhD contract with PSA on the topic of aesthetic shape modeling in immersive virtual reality environments, which was funding the PhD of Youna Le Vaou. The thesis was defended in June 2020.
10 Partnerships and cooperations
10.1 European initiatives
10.1.1 ADAM2
Participants: Stefanie Hahmann, Mélina Skouras, Emmanuel Rodriguez.
ANIMA is an active member of the H2020 FET OPEN RIA ADAM2 consortium (Analysis, Design, And Manufacturing using Microstructures), with partners BCAM Bilbao, Technion, EPFL, TU Wien, Univ. Del Pais Vasco, Stratasys LTD, Trimtek SA, Hutchinson SA, Seoul National University.
Research within the ADAM2 project combines user-guided shape modelling using microstructures, followed by validation and structural optimization using physical process simulation, and finalized by physical realizations via additive and hybrid manufacturing.
The contract is funding the PhD thesis of Emmanuel Rodriguez.
10.2 National initiatives
10.2.1 ANR E-ROMA
Participants: Stefanie Hahmann, Rémi Ronfard, Nachwa Aboubakr, Mélina Skouras.
E-ROMA is a collaborative project funded by ANR with LIRIS (Raphaëlle Chaine, Julie Digne): Musée Gallo-Romain (Hugues Savay-Guerraz, Maria-Pia Darblade-Audoin) and Université Paris-Sorbonne (Emmanuelle Rosso).
The project was funding the PhD thesis of Pierre Casati, which was interrupted in June 2020.
Nachwa Aboubakr (PhD in June 2020) started a one-year postdoc contract in September 2020 and we are investigating new research directions with her, including end-to-end reconstruction of 3D scenes from bas reliefs and animation of the reconstructed 3D scenes using deep learning methods.
10.2.2 ANR ANATOMY2020
Participants: Olivier Palombi, Rémi Ronfard, Ameya Murukutla.
ANATOMY2020 is a collaborative projet funded by ANR with TIMC, Anatoscope and EHCI/LIG.
The project is funding the PhD thesis of Ameya Murukutla (May 2021) : Text-to-movie authoring of anatomy lessons.
During this PhD we have extend our « prose storyboard language » with Petri net semantics and used it to generate short animated "movie lessons" useful for teaching anatomy to medical students. Future work will investigate the applicability of the method for authoring interactive games (exercises, quizzes, tests, exams) and virtual reality « experiences » in AR / VR.
10.2.3 ANR FOLD-DYN (November 2017 - October 2020)
Participants: Thomas Buffet.
The FOLDDyn project (Field-Oriented Layered Dynamics animating 3D characters) proposed the study of new theoretical approaches for the effective generation of virtual characters deformations, when they are animated.
This 3-year contract was a joint project with the University of Toulouse. The contract was funding the PhD thesis of Thomas Buffet, which was defended in January 2021.
10.2.4 MONTAGE SPATIAL
Participants: Rémi Ronfard.
Montage Spatial is a contract between ANIMA and ENSADLAB funded by the Ministery of Culture and Communication for reviewing the state of the art in computational directing tools for virtual reality.
Thèse de Rémi Sagot, école doctorale SACRE, PSL
10.2.5 LIVING ARCHIVE OF ISADORA
Participants: Manon Vialle, Rémi Ronfard, Mélina Skouras.
The Living Archive of Isdaora was a collaboration between artist Elisabeth Rémy-Schwartz, Inria project team EXSITU in Saclay and ANIMA. The project was funded by Centre National de la Danse in 2020 and allowed us to acquire a unique motion capture dataset of Isodara Duncan choreographies. This was used during Manon Vialle's Masters internship, and will continue to be used during Manon Vialle's Phd thesis.
10.3 Regional initiatives
10.3.1 Kino Ai
Participants: Rémi Colin de Verdière, Rémi Ronfard, Qianqian Fu.
Kino Ai was an INRIA ADT which funded the research engineer position of Rémi Colin de Verdière from October 2018 to September 2020.
Following our previous work in “multiclip video editing” and “split screen video generation”, Kino Ai was dedicated to to provide a user-friendly environment for editing and watching ultra-high definition movies online, with an emphasis on recordings of live performances.
The code from Vineet Gandhi's PhD thesis was entirely re-designed for supporting ultra high definition video.
The goal of the Kino AI ADT was to allow the Kino Ai python code to run in a web server, and to provide a redesigned user interface (in javascript) running on a web client. The user interface will be designed, tested and evaluated with the Litt&Arts team at Univ. Grenoble Alpes, as part of CDP project Performance Lab.
The software was extensively tested on a large dataset of 4K video recordings of theatre rehearsals, in collaboration with the Litt&Arts team at Univ. Grenoble Alpes, theatre director Jean-Francois Peyret in Paris, Theatre de l'Hexagone in Meylan and Theatre de Vidy in Lausanne. In February 2020, new experiments were made with 5 student actors from Conservatoire National d’Art Dramatique under the direction of Jean-Francois Peyret (Théâtre Numérique Populaire / Petit Bréviaire), using stereoscopic cameras and multi-track audio recordings.
The Kino Ai server was made available publicly at https://
The code for KinoAi is open source and can be found at https://
10.3.2 Performance Lab
Participants: Rémi Ronfard, Manon Vialle, Qianqian Fu, Rémi Colin de Verdière.
The Performance Lab is a Cross Disciplinary Project (CDP) funded by IDEX UGA with the goal of fostering inter-disciplinary research in the performing arts (dance, theater), social sciences (geography) and computer science.
ANIMA is part of the « Digital dramaturgy » work package (Rémi Ronfard, Julie Valéro).
We are taking an active part in several artistic residencies (Jean-Francois Peyret, Agnes De Cayeux). We organized workshops on augmented reality theater, rehearsal studies, dramatic scores.
The Performance Lab was funding the PhD thesis of Qianqian Fu : Computational video editing of live performances, supervised by Remi Ronfard and with Benjamin Lecouteux, speech processing team at LIG, which was unfortunately left unfinished.
10.3.3 Theatre Numérique Populaire
Participants: Rémi Ronfard, Qianqian Fu, Rémi Colin de Verdière.
TNP was a research project funded by IDEX UGA on the occasion of the artistic residency of Jean-François Peyret in February 2020. During this short project, we extended the Kino Ai system with multi-track audio and stereoscopic video recording and performed extensive testing and validation during two weeks of rehearsals. The project is expected to lead to a public presentation at Studio Theatre de Vitry in October 2021.
10.3.4 L'ébauchoir
Participants: Stefanie Hahmann.
This project is an informal collaboration with artist Pauline de Chalendar and Samuel Degrande et Paul-Elian Tabarant at PIRVI du laboratoire CRIStAL (UMR 9189), Université de Lille, CNRS - Lille
The project is an artistic application of the surface scuplting software developed by Paul-Elian Tabarant during his M2 internship with Stefanie Hahmann in 2019, and was presented publicly by Pauline de Chalendar at the Bennale Arts et Science in Grenoble in February 2020.
11 Dissemination
11.1 Promoting scientific activities
11.1.1 Scientific events: organisation
Rémi Ronfard is a member of the steerng committee for the Eurographics workshop on intelligent cinematography and editing (WICED). The ninth workshop in the series was held online in
General chair, scientific chair
Remi Ronfard was the general chair (with Julie Valéro) for the first Workshop on Computer Theater (Journées d'nformatique Théâtrale), February 12-14, 2020. The workshop was co-organized by Litt & Arts team at UGA and Inria, in association with Biennale Art et Science EXPERIMENTA, Théâtre de l’Hexagone Meylan.
The workshop was quite successful. 20 papers were selected by a scientific committee of computer scientists and theater studies. The workshop was attended by 20 participants from France, Switzerland, Canada, Germany, Greece, Italy and New Zealand and an audience of 100 people during three days. All presentations can be watched on the workshop Youtube channel 4, including an introduction by Rémi Ronfard and Julie Valéro and two short papers presented the ANIMA team. The workshop proceedings are being revised for online publication in 2021. We are planning a second edition in October 2022.
Chair of conference program committees
Mélina Skouras was a technical papers chair for SMI'2020 Fabrication and Sculpting Event (FASE), and the student research competition chair for Siggraph 2020.
Member of the conference program committees
Mélina Skouras was a member of the program committees for Siggraph 2020, Shape Modeling International 2020, ACM/Eurographics Symposium on Computer Animation 2020, Advances in Architectural Geometry 2020, and Pacific Graphics 2020.
Stefanie Hahmann was a member of the program committees for the Symposium on Geometry Processing (SGP) and the Symposium on Solid and Physical Modeling (SPM).
Reviewer
Rémi Ronfard was an external reviewer for the Siggraph 2020 and Siggraph Asia 2020 conferences.
11.1.2 Journal
Member of the editorial boards
Rémi Ronfard is an associate editor of the journal Computer Animation and Virtual Worlds.
Stefanie Hahmann is an Associate Editor of the journals Compters & Graphics (Elsevier) and Computer Aided Design (Elsevier).
Reviewer - reviewing activities
Mélina Skouras was an external reviewer for Computer-Aided Design, PNAS, ACM UIST, SN Applied Sciences, Siggraph Asia, ACM Transactions on Graphics.
Rémi Ronfard was an external reviewer for ACM transactions on Graphics.
Stefanie Hahmann was a reviewer for the journals Computer Graphics Forum and GMOD.
11.1.3 Invited talks
Mélina Skouras gave invited talks at the Bellairs Workshop on Computer Animation 2020 and at the CECAM-Lorentz workshop on Computing Complex Mechanical Systems in January 2020.
Stefanie Hahmann was invited to the Dagstuhl Seminar on Geometric Modeling in June 2020 at the Leibniz-Zentrum für Informatik. The seminar was postponed because of Coronavirus pandemic.
11.1.4 Research administration
Stefanie Hahmann is an elected member of the European Association for Computer Graphics – chapitre français (EGFR) and serves as secretary in the streering committee. Stefanie Hahmann was elected member of Association Française d’Informatique Graphique (AFIG).
Stefanie Hahmann is a member of the Comité d’Etudes Doctorales (CED) at Inria Grenoble.
Stefanie Hahmann is an elected memer of the Conseil Scientifique of Grenoble INP.
11.2 Teaching - Supervision - Juries
Teaching
- Bachelor : Stefanie Hahmann, Numerical Methods, 45 HETD, 240 students, L3, Ensimag-Grenoble INP.
- Master : Stefanie Hahmann, Geometric Modeling, 58 HETD, 70 students, M1, Ensimag-Grenoble INP.
- Master : Stefanie Hahmann, Surface Modeling, 52 HETD, 45 students, M2, Ensimag-Grenoble INP.
- Master: Mélina Skouras, Surface modeling, 13.5 HETD, M2, Ensimag, Grenoble, France
- Master: Mélina Skouras, Numerical Mechanics, 12 HETD, M2, ENS de Lyon, France
- Master: Rémi Ronfard teaches computer animation to MOSIG M2 students, 18 HETD, Grenoble INP, Univ. Grenoble Alpes.
- PhD: Rémi Ronfard is an associate researcher in the Spatial Media team at ENSADLAB, where he teaches computer graphics to doctoral art students in the SACRE doctoral school, 60 HETD, Univ. Paris Sciences et Lettres (PSL).
Stefanie Hahmann is co-responsible of the department MMIS (Images and Applied Maths) at Grenoble INP with 120 students.
Supervision
- PhD: Youna Le Vaou, Virtual Sculpture: shape creation and modification through immersive CAVE-like systems, supervised by Jean-Claude Léon and Stefanie Hahmann, was defended on June 10, 2020.
- PhD : Nachwa Aboubakr, Observation and modeling of human activities, supervised by James Crowley and Rémi Ronfard, was defended on June 12, 2020.
- PhD : Thomas Buffet, Efficient multi-layered cloth animation using implicit surfaces, supervised by Marie-Paule Cani and Damien Rohmer, was defended on January 13, 2021.
- PhD in progress : Ameya Murukutla, Storyboarding augmented reality anatomy lessons, since Octobre 2017, supervised by Rémi Ronfard and Olivier Palombi
- PhD in progress: David Jourdan, Design of free-from surfaces using self-actuated materials, since October 2018, supervised by Adrien Bousseau and Mélina Skouras.
- PhD in progress: Mickael Ly, Inverse elastic shell design with contact and friction with applications to garment design, since October 2017, supervised by Florence Descoubes and Mélina Skouras.
- PhD in progress: Emmanuel Rodriguez, Direct and inverse modeling of laser-cut meta-materials, since October 2020, supervised by Georges-Pierre Bonneau, Mélina Skouras and Stefanie Hahmann.
- PhD in progress: Manon Vialle, Choreographic Style Transfer, since October 2020, supervised by Rémi Ronfard and Mélina Skouras.
- PhD in progress: Nicolas Parent, Numerical modeling of knitted fabric, since October 2020, supervised by Florence Bertails-Descoubes, Thibaut Métivet and Mélina Skouras.
- PhD in progress: Sandrine Barbois, Formalization od surgical procedures, since March 2020, supervised by Olivier Palombi and Rémi Ronfard.
- PhD in progress: Rémi Sagot-Duvauroux, Spatial and temporal structures of virtual realities, supervised by Rémi Ronfard and Guillaume Soulez, Univ. Paris Sciences et Lettres, SACRE program, ED 540, since October 2020.
- Master’s thesis: Johana Marku, Geometric construction of auxetic metamaterials, supervised by Georges-Pierre Bonneau, Stefanie Hahmann, was defended in June 2020.
- Master’s thesis: Manon Vialle, Choreographic style transfer, June 2020, supervised by Mélina Skouras and Rémi Ronfard, was dfended in June 2020.
The PhD thesis of Pierre Casati, Expressive restoration of Gallo-Roman statues by virtual sculpture and animation, supervised by Rémi Ronfard and Stefanie Hahmann, was left unfinished in June 2020, partly due to the COVID crisis.
The PhD thesis of Qianqian Fu, Computational video editing of live performances, supervised by Rémi Ronfard and Benjamin Lecouteux (GETALP, LIG) since November 2018, was left unfinished in March 2021, mostly due to the COVID crisis.
Juries
Stefanie Hahmann was a reviewer of the PhD theses of Thibault Blanc-Beyne (Université de Toulouse) and Ahmed Blidia (Université Côte d'Azur, Nice).
Mélina Skouras was an examiner for the PhD thesis of Thibault Lescoat (Institut Polytechnique de Paris) and Tristan Djourachkovitch (INSA Lyon).
12 Scientific production
12.1 Publications of the year
International journals
International peer-reviewed conferences
Conferences without proceedings
12.2 Cited publications
- 7 articleA Generative Audio-Visual Prosodic Model for Virtual ActorsIEEE Computer Graphics and Applications376November 2017, 40-51
- 8 articleDepicting as a method of communicationPsychological Review1232016, 324–347
- 9 articleAttention and the Evolution of Hollywood FilmPsychological Science2132010, 432-439
- 10 inproceedings dAIrector: Automatic Story Beat Generation through Knowledge Synthesis Proceedings of the Joint Workshop on Intelligent Narrative Technologies and Workshop on Intelligent Cinematography and Editing 2018
- 11 inproceedingsCamera-on-rails: Automated Computation of Constrained Camera PathsACM SIGGRAPH Conference on Motion in GamesParis, FranceACMNovember 2015, 151-157
- 12 inproceedingsContinuity Editing for 3D AnimationAAAI Conference on Artificial IntelligenceAustin, Texas, United StatesAAAI PressJanuary 2015, 753-761
- 13 inproceedings Multi-Clip Video Editing from a Single Viewpoint European Conference on Visual Media Production London, United Kingdom ACM November 2014
- 14 inproceedings Spatial Motion Doodles: Sketching Animation in VR Using Hand Gestures and Laban Motion Analysis Motion, Interaction and Games October 2019
- 15 inproceedings Holojam in Wonderland: Immersive Mixed Reality Theater ACM SIGGRAPH 2018 Art Gallery SIGGRAPH '18 2018
- 16 inproceedings Improvisational puppets, actors, and avatars Proc Comp Game Dev Conf 1996
- 17 inproceedings Cave: Making Collective Virtual Narrative ACM SIGGRAPH 2019 Art Gallery SIGGRAPH '19 2019
- 18 article Functionality Preserving Shape Style Transfer ACM Transactions on Graphics 35 6 2016
- 19 article An empirical study of cognition and theatrical improvisation Creativity and Cognition 2009
- 20 book Hamlet on the Holodeck, The Future of Narrative in Cyberspace, Updated Edition MIT Press 2017
- 21 inproceedingsMy Corporis Fabrica: A Unified Ontological, Geometrical and Mechanical View of Human AnatomyProceedings of the 2009 International Conference on Modelling the Physiological Human3DPH'092009, 209--219
- 22 article The Prose Storyboard Language: A Tool for Annotating and Directing Movies CoRR arXiv:1508.07593v3 2019
- 23 inproceedingsSeeing Behind the Camera: Identifying the Authorship of a Photograph2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)2016, 3494-3502
- 24 articleCombining 3D Models and Functions through Ontologies to Describe Man-made Products and Virtual Humans: Toward a Common FrameworkComputer-Aided Design and Applications1222015, 166-180
- 25 articleThespian : An architecture for interactive pedagogical dramaArtificial Intelligence in Education1252005, 595–602