2.1 Understanding stories

Stories can come in many forms. An anatomy lesson is a story. A cooking recipe is a story. A geological sketch is a story. Many paintings and sculptures are stories. Stories can be told with words, but also with drawings and gestures. For the purpose of creating animated story worlds, we are particularly interested in communicating the story with words in the form of a screenplay or with pictures in the form of a storyboard. We also foresee the possibility of communicating the story in space using spatial gestures. The first scientific challenge for the ANIMA team is to propose new computational models and representations for screenplays and storyboards, and practical methods for parsing and interpreting screenplays and storyboards from multimodal user input. To do this, we reverse engineer existing screenplays and storyboards, which are well suited for generating animation in traditional formats. We also explore new representations for communicating stories with a combination of speech commands, 3D sketches and 3D gestures, which promise to be more suited for communicating stories in new media including virtual reality, augmented reality and mixed reality.

2.2 Authoring story worlds

Telling stories visually creates additional challenges not found in traditional, text-based storytelling. Even the simplest story requires a large vocabulary of shapes and animations to be told visually. This is a major bottleneck for all narrative animation synthesis systems. The second scientific challenge for the ANIMA team is to propose methods for quickly authoring shapes and animations that can be used to tell stories visually. We devise new methods for generating shapes and shape families, understanding their functions, styles, material properties and affordances, authoring animations for a large repertoire of actions, and printing and fabricating articulated and deformable shapes suitable for creating physical story worlds with tangible interaction.

2.3 Directing story worlds

Lastly, we develop methods for controlling virtual actors and cameras in virtual worlds and editing them into movies in a variety of situations ranging from 2D and 3D professional animation, to virtual reality movies and real-time video games. Starting from the well-established tradition of the storyboard, we create new tools for directing movies in 3D animation, where the user is really the director, and the computer is in charge of its technical execution using a library of film idioms. We also explore new areas, including the automatic generation of storyboards from movie scripts for use by domain experts, rather than graphic artists.

3.1 Geometric modeling of story worlds

Scientist in charge: Stefanie Hahmann

Other participants: Rémi Ronfard, Mélina Skouras

We aim to create intuitive tools for designing 3D shapes and animations which can be used to populate interactive, animated story worlds, rather than inert and static virtual worlds. In many different application scenarios such as preparing a product design review, teaching human anatomy with a MOOC, composing a theatre play, directing a movie, showing a sports event, 3D shapes must be modeled for the specific requirements of the animation and interaction scenarios (stories) of the application.

We will need to invent novel shape modelling methods to support the necessary affordances for interaction and maintain consistency and plausibility of the shape appearances and behaviors during animation and interaction. Compared to our previous work, we will therefore focus increasingly on designing shapes and motions simultaneously, rather than separately, based on the requirements of the stories to be told.

Previous work in the IMAGINE team has emphasized the usefulness of space-time constructions for sketching and sculpting animation both in 2D and 3D. Future work in the ANIMA team will further develop this line of research, with the long-term goal of choreographing complex multi-character animation and providing full authorial and directorial control to the user.

3.2 Physical modeling of story worlds

Scientist in charge: Mélina Skouras

Other participants: Stefanie Hahmann, Rémi Ronfard

When authoring and directing story worlds, physics is important to obtain believable and realistic behaviors, e.g. to determine how a garment should deform when a character moves, or how the branches of a tree bend when the wind starts to blow. In practice, while deformation rules could be defined a priori (e.g. procedurally), relying on physics-based simulation is more efficient in many cases as this means that we do not need to think in advance about all possible scenarii. In ANIMA, we want to go a step further. Not only do we want to be able to predict how the shape of deformable objects will change, but we also want to be able to control their deformation. In short, we are interested in solving inverse problems where we adjust some parameters of the simulation, yet to be defined so that the output of the simulation matches what the user wants.

By optimizing design parameters, we can get realistic results on input scenarii, but we can also extrapolate to new settings. For example, solving inverse problems corresponding to static cases can be useful to obtain realistic behaviors when looking at dynamics. E.g. if we can optimize the cloth material and the shape of the patterns of a dress such that it matches what an artist designed for the first frame of an animation, then we can use the same parameters for the rest of the animation. Of course, matching dynamics is also one of our goals.

Compared to more traditional approaches, this raises several challenges. It is not clear what the best way is for the user to specify constraints, i.e. how to define what she wants (we do not necessarily want to specify the positions of all nodes of the meshes for all frames, for example). We want the shape to deform according to physical laws, but also according to what the user specified, which means that the objectives may conflict and that the problem can be over-constrained or under-constrained.

Physics may not be satisfied exactly in all story worlds i.e. input may be cartoonish, for example. In such cases, we may need to adapt the laws of physics or even to invent new ones. In computational fabrication, the designer may want to design an object that cannot be fabricated using traditional materials for example. But in this case, we cannot cheat with the physics. One idea is to extend the range of things that we can do by creating new materials (meta-materials), creating 3D shapes from flat patterns, increasing the extensibility of materials, etc.

To achieve these goals, we will need to find effective metrics (how to define objective functions that we can minimize); develop efficient models (that can be inverted); find suitable parameterizations; and develop efficient numerical optimization schemes (that can account for our specific constraints).

3.3 Semantic modeling of story worlds

Scientist in charge: Olivier Palombi

Other participants: Rémi Ronfard, Nicolas Szilas

Beyond geometry and physics, we aim at representing the semantics of story worlds. We use ontologies to organize story worlds into entities described by well defined concepts and relations between them. Especially important to us is the ability to "depict" story world objects and their properties during the design process 12 while their geometric and material properties are not yet defined. Another important aspect of this research direction is to make it possible to quickly create interactive 3D scenes and movies by assembling existing geometric objects and animations. This requires a conceptual model for semantic annotations, and high level query languages where the result of a semantic query can be a 3D scene or 3D movie.

One important application area for this research direction is the teaching of human anatomy. The PhD thesis of Ameya Murukutla focuses on automatic generation of augmented reality lessons and exercises for teaching anatomy to medical students and sports students using the prose storyboard language which we introduced during Vineet Gandhi's PhD thesis 26. By specializing to this particular area, we are hoping to obtain a formal validation of the proposed methods before we attempt to generalize them to other domains such as interactive storytelling and computer games.

3.4 Aesthetic modeling of story worlds

Scientist in charge:Rémi Ronfard

Other participants: Stefanie Hahmann, Mélina Skouras, François Garnier

Data-driven methods for shape modeling and animation are becoming increasingly popular in computer graphics, due to the recent success of deep learning methods. In the context of the ANIMA team, we are particularly interested in methods that can help us capture artistic styles from examples and transfer them to new content. This has important implications in authoring and directing story worlds because it is important to offer artistic control to the author or director, and to maintain a stylistic coherence while generating new content. Ideally, we would like to learn models of our user's authoring and directing styles, and create contents that matches those styles.

Arts and entertainement

Animated story worlds are central to the industries of 3D animation and video games, which are very strong in France. Designing 3D shapes and motions from storyboards is a worthwhile research goal for those industries, where it is expected to reduce production costs while at the same time increasing artistic control, which are two critical issues in those domains. Furthermore, story is becoming increasingly important in video games and new authoring and directing tools are needed for creating credible interactive story worlds, which is a challenge to many video game companies. Traditional live action cinematography is another application domain where the ANIMA team is hoping to have an impact with its research in storyboarding, virtual cinematography and film editing.

Performance art, including dance and theater, is an emergent application domain with a strong need for dedicated authoring and directing tools allowing to incorporate advanced computer graphics in live performances. This is a challenging application domain, where computer-generated scenography and animation need to interact with human actors in real-time. As a result, we are hoping that the theater stage becomes an experimental playground for our most exploratory research themes. To promote this new application domain, we are organizing the first international workshop on computer theater in Grenoble in February 2020, under the name Journées d’Informatique Théâtrale (JIT). The workshop will assemble theater researchers, artists and computer engineers whose practice incorporates computer graphics as a means of expression and/or a creative tool. With this workshop, our goal is to create a new research discipline that could be termed “computer theater”, following the model of computer music, which is now a well established discipline.

Education

Teaching of Anatomy is a suitable domain for research. As professor of Anatomy, Olivier Palombi gives us the opportunity to experiment in the field. The formalization of anatomical knowledge in our ontology called My Corporis Fabrica (MyCF) is already operational. One challenge for us is to formalize the way anatomy is taught or more exactly the way anatomical knowledge is transmitted to the students using interactive 3D scenes.

Museography is another related application domain for our research, with a high demand for novel tools allowing to populate and animate virtual reconstructions of art works into stories that make sense to museum audiences of all ages. Our research is also applicable to scientific museography, where animated story worlds can be used to illustrate and explain complex scientific concepts and theories.

Industrial design

Our research in designing shapes and motions from storyboards is also relevant to industrial design, with applications in the fashion industry, the automotive industry and in architecture. Those three industries are also in high demand for tools exploiting spatial interaction in virtual reality. Our new research program in physical modeling is also applicable to those industries. We have established strong partnerships in the past with PSA and Vuitton, and we will seek to extend them to architectural design as well in the future.

5.1 Footprint of research activities

ANIMA is a small team of four permanent researchers and three PhD students, so our footprint is limited. We estimate that we run approximately twelve computers, including laptops, desktops and shared servers, at any given time.

Research in computer graphics is not (yet) data intensive. We mostly devise procedural algorithms, which require limited amounts of data. One notable exception is our work on computational editing of live performances, which produces large amounts of ultra high definition video files. We have opted for a centralized video server architecture (KinoAI) so that at least the videos are never duplicated and always reside on a single server.

On the other hand, our research requires powerful graphics processing units (GPU) which significantly increase the power consumption of our most powerful desktops.

The COVID19 crisis has changed our working habits heavily. We have learned to work from home most of the week, and to abandon international travel entirely. This holds the promise of a vastly reduced footprint. But it is too early to say whether this is sustainable. While the team as a whole has been able to function in adverse conditions, it has become increasingly difficult to welcome interns and Masters students.

5.2 Impact of research results

Our research does not directly address social and environmental issues. Our work on 3D printing may have positive effects by allowing the more efficient use of materials in the production of prototypes. Our work on virtual medical simulation and training may provide an alternative in some cases to animal experiments and costly robotic simulations.

Our most important application domain is arts and culture, including computer animation and computer games. Globally, those sectors are creating jobs, rather than destroying them, in France and in Europe. The impact of our discipline is therefore positive at least in this respect.

We are more concerned with the impact of the software industry as a whole, i.e. private companies who implement our research papers and include them in their products. We note a tendency to increase the memory requirements of software. While much effort in software engineering is devoted to improving the execution speed of graphics programs, there is not enough effort in optimizing their footprint. This is an area that may be worth investigating in our future work.

6.1 Computer theater conference / Journées d'informatique théâtrale

The third edition of the international conference on computer theater was co-organized by Rémi Ronfard and Cyrielle Garson, at the University of Avignon, October 9, 10 and 11, 2024.

The scientific committee selected 30 papers out of 40 submissions, both in English and French. Accepted papers will be published in the conference proceedings, due in June 2025.

The three-day conference was attended by 80 researchers, practitioners, and independent artists from France, Italy, Switzerland, UK, USA and Canada.

Many different academic disciplines were represented, including computer science, theatre studies, artificial intelligence, virtual reality, literature, media studies and applied arts.

The three-day conference featured keynotes by Brendan Bradley (NYU Tisch School of the Arts), Lisa Bretzner (Compagnie Atropos), Laurence Devillers (Sorbonne Université) and Serge Abiteboul (Inria).

6.2 Awards

Stefanie Hahmann was awarded with the John Gregory Memorial Award by the international Geometric Modeling community. The award ceremony took place at the Leibniz Center for Informatics in Schloss Dagstuhl, June 2024. View award site.

Stefanie Hahmann was awarded the T. Kunii Distinguished Researcher Award by the International Shape Modeling Committee during the Shape Modeling conference (SMI) in Detroit in July 2024. View award site

7.1 3D sketching in immersive environments: Shape from disordered ribbon strokes.

Participants: Stefanie Hahmann, Anandhu Sureshkumar.

Show image description

This figure describes six steps of the creation of a surface in VR. The user draws ribbons in space using a handheld controller. Our algorithm simultaneously calculates a 3D shape that evolves with the ribbon drawings.

This work is done with PhD student Anandhu Sureshkumar in collaboration with Georges-Pierre Bonneau (Maverick team), Amal Dev Parakkat (Télécom Paris) and Marie-Paule Cani (LIX/Ecole Polytechnique). We developed a new 3D sketching system in Virtual Reality. Indeed, immersive environments with head mounted displays (HMD) and hand-held controllers offer new possibilities for the creation of artistic 3D content. Some of them are exploited by mid-air drawing applications: the user's hand trajectory generates a set of stylized curves or ribbons in space, giving the impression of painting or drawing in 3D. We propose a method to extend this approach to the sketching of surfaces with a VR controller. The idea is to favor shape exploration, offering a tool, where the user creates a surface just by painting ribbons. These ribbons are not constrained to form patch boundaries for example or to completely cover the shape. They can be very sparse, disordered, overlap or not, intersect or not. The shape is computed simultaneously, starting with the first piece of ribbon drawn by the user and continuing to evolve in real-time as long as the user continues sketching. An example is shown in Figure 1. Our method involves minimizing an energy function based on the projections of the ribbon strokes on a proxy surface by taking the controller's orientations into account.

During this work we collaborated also with a visual artist, Pauline de Chalandar. In the corresponding publication, we show images of one of her artistic creations that combines stylized curve drawings in VR with our surface sketching tool.

7.2 Modern Dance Retargeting using Ribbons as Lines of Action.

Participants: Mélina Skouras, Rémi Ronfard.

Show image description

Retargeting results. From top to bottom: input ribbon, retargeted humanoid skeleton with the same morphology as the input data, retargeted skeleton with added bones and different T-Pose shape.

This work, done in collaboration with former PhD student Manon Vialle, presents a method for retargetting dancing characters represented as articulated skeletons with possibly different morphologies and topologies. Our approach relies on the use of flexible ribbons that can bend and twist as an intermediate representation, and that can be seen as animated lines of action. These ribbons allow us to abstract away the specific morphology of the bodies and to well transmit the fluidity of modern dance movement from one character to another.

7.3 Designing Bending-Active Freeform Surfaces.

Participants: Mélina Skouras, Stefanie Hahmann.

Show image description

Given a freeform input surface segmented into strips by the user (top left), we automatically generate the layout of flat ribbons (i.e their external boundaries and their internal structures) such that their deformed shapes, when their are clamped at both ends, match the ruled surfaces at equilibrium (top right). Once laser-cut, the fabricated ribbons can be actively bent to approximate the given input surface (bottom).

This work, done in collaboration with Georges-Pierre Bonneau and former PhD student Emmanuel Rodriguez, presents a method for designing bending-active structures, i.e. curved structures created using initially planar components that have been elastically bent so that they take on a three-dimensional shape. The 3D shape that the structure assumes depends not only on the applied external forces, but also on the elastic properties of the physical material used. By altering the fine-scale structure of this material, we are able to locally control its macroscopic elastic properties.

We introduce a computational framework for the inverse design of custom 3D surfaces using bending-active structures. We optimize the internal structure of laser-cut planar surface strips, so that they approximate a desired 3D freeform surface when being clamped at both extremities. The specific orthotropic design of the structure of the strips allows us to locally control their bending behavior. In contrast to past work the planar surface strips can be of arbitrary shape, not necessarily rectangular, and when deformed, they form a piecewise continuous surface.

7.4 MatAIRials: Inflatable Metamaterials for Freefrom Surface Design.

Participants: Mélina Skouras, Siyuan He.

Show image description

We propose isotropic inflatable metamaterials for creating inflatable structures of desired 3D shapes. Our approach starts by precomputing the mapping between the parameters of the sealing patterns and the contraction ratios of the inflated metamaterials (a). Then, given an input 3D triangular mesh (b), we flatten it to the plane using conformal parametrization (c) and use the local scaling factor (color plot in Figure (c)) of the surface to trace the layout of the sealing patterns of the triangles of a superimposed regular grid (d). The resulting graded patterns allows us to generate a curved inflated structure that approximates the input mesh through metric frustration (e).

This work, done in collaboration with Arthur Lebée, investigates the modeling of inflatable pads, made of two planar membranes sealed according to periodic patterns, typically parallel lines or dots, as metamaterials.

By considering novel sealing patterns with 6-fold symmetry, we are able to generate a family of inflatable materials whose macroscale contraction is isotropic and can be modulated by controlling the parameters of the seals. We leverage this property of our inflatable materials family to propose a simple and effective algorithm based on conformal mapping that allows us to design the layout of inflatable structures that can be fabricated flat and whose inflated shapes approximate those of given target freeform surfaces.

8.1 International research visitors

8.2 National initiatives

9.1 Promoting scientific activities

9.2 Teaching - Supervision - Juries

10.1 Major publications

• 1 articleZ.Zhen Chen, H.-Y.Hsiao-Yu Chen, D. M.Danny M Kaufman, M.Mélina Skouras and E.Etienne Vouga. Fine Wrinkling on Coarsely Meshed Thin Shells.ACM Transactions on Graphics405August 2021, 1-32HALDOI
• 2 articleA.Amélie Fondevilla, D.Damien Rohmer, S.Stefanie Hahmann, A.Adrien Bousseau and M.-P.Marie-Paule Cani. Fashion Transfer: Dressing 3D Characters from Stylized Fashion Sketches.Computer Graphics Forum4062021, 466-483HALDOI
• 3 articleR.Rémi Ronfard. Film Directing for Computer Games and Animation.Computer Graphics Forum402May 2021, 713-730HALDOI
• 4 articleR.Rémi Ronfard and J.Julie Valéro. KinoAI and the audiovisual notebook : a plural solution for rehearsal studies.European Journal of Theatre and Performance2June 2020, 334-375HAL
• 5 articleX.Xiaoting Zhang, G.Guoxin Fang, M.Mélina Skouras, G.Gwenda Gieseler, C. C.Charlie C.L. Wang and E.Emily Whiting. Computational Design of Fabric Formwork.ACM Transactions on Graphics384July 2019, 1-13HALDOI

10.2 Publications of the year

10.3 Cited publications

• 11 articleA.Adela Barbulescu, R.Rémi Ronfard and G.Gérard Bailly. A Generative Audio-Visual Prosodic Model for Virtual Actors.IEEE Computer Graphics and Applications376November 2017, 40-51HALDOIback to text
• 12 articleH.Herbert Clarck. Depicting as a method of communication.Psychological Review1232016, 324–347back to text
• 13 articleJ. E.James E. Cutting, J. E.Jordan E. DeLong and C. E.Christine E. Nothelfer. Attention and the Evolution of Hollywood Film.Psychological Science2132010, 432-439back to text
• 14 inproceedingsM.Markus Eger and K. W.Kory W Mathewson. dAIrector: Automatic Story Beat Generation through Knowledge Synthesis.Proceedings of the Joint Workshop on Intelligent Narrative Technologies and Workshop on Intelligent Cinematography and Editing2018back to text
• 15 inproceedingsQ.Quentin Galvane, M.Marc Christie, C.Christophe Lino and R.Rémi Ronfard. Camera-on-rails: Automated Computation of Constrained Camera Paths.ACM SIGGRAPH Conference on Motion in GamesParis, FranceACMNovember 2015, 151-157HALDOIback to text
• 16 inproceedingsQ.Quentin Galvane, R.Rémi Ronfard, C.Christophe Lino and M.Marc Christie. Continuity Editing for 3D Animation.AAAI Conference on Artificial IntelligenceAustin, Texas, United StatesAAAI PressJanuary 2015, 753-761HALback to text
• 17 inproceedingsV.Vineet Gandhi, R.Rémi Ronfard and M.Michael Gleicher. Multi-Clip Video Editing from a Single Viewpoint.European Conference on Visual Media ProductionLondon, United KingdomACMNovember 2014back to text
• 18 inproceedingsM.Maxime Garcia, R.Rémi Ronfard and M.-P.Marie-Paule Cani. Spatial Motion Doodles: Sketching Animation in VR Using Hand Gestures and Laban Motion Analysis.Motion, Interaction and GamesOctober 2019HALDOIback to text
• 19 inproceedingsD.David Gochfeld, C.Corinne Brenner, K.Kris Layng, S.Sebastian Herscher, C.Connor DeFanti, M.Marta Olko, D.David Shinn, S.Stephanie Riggs, C.Clara Fernández-Vara and K.Ken Perlin. Holojam in Wonderland: Immersive Mixed Reality Theater.ACM SIGGRAPH 2018 Art GallerySIGGRAPH '182018back to text
• 20 inproceedingsB.Barbara Hayes-Roth and R. V.Robert Van Gent. Improvisational puppets, actors, and avatars.Proc Comp Game Dev Conf1996back to text
• 21 inproceedingsK.Kris Layng, K.Ken Perlin, S.Sebastian Herscher, C.Corinne Brenner and T.Thomas Meduri. Cave: Making Collective Virtual Narrative.ACM SIGGRAPH 2019 Art GallerySIGGRAPH '192019back to text
• 22 articleZ.Zhaoliang Lun, E.Evangelos Kalogerakis, R.Rui Wang and A.Alla Sheffer. Functionality Preserving Shape Style Transfer.ACM Transactions on Graphics3562016back to text
• 23 articleB.Brian Magerko, W.Waleed Manzoul, M.Mark Riedl, A.Allan Baumer, D.Daniel Fuller, K.Kurt Luther and C.Celia Pearce. An empirical study of cognition and theatrical improvisation.Creativity and Cognition2009back to text
• 24 bookJ. H.Janet H. Murray. Hamlet on the Holodeck, The Future of Narrative in Cyberspace, Updated Edition.MIT Press2017back to text
• 25 inproceedingsO.Olivier Palombi, G.Guillaume Bousquet, D.David Jospin, S.Sahar Hassan, L.Lionel Revéret and F.François Faure. My Corporis Fabrica: A Unified Ontological, Geometrical and Mechanical View of Human Anatomy.Proceedings of the 2009 International Conference on Modelling the Physiological Human3DPH'092009, 209--219back to text
• 26 inproceedingsR.Rémi Ronfard, V.Vineet Gandhi, L.Laurent Boiron and V. A.Vaishnavi Ameya Murukutla. The Prose Storyboard Language: A Tool for Annotating and Directing Movies.WICED 2022 - 10th Workshop on Intelligent Cinematography and EditingReims, FranceApril 2022, 1-16HALback to text
• 27 inproceedingsC.C. Thomas and A.A. Kovashka. Seeing Behind the Camera: Identifying the Authorship of a Photograph.2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)2016, 3494-3502back to text
• 28 articleF.Federico Ulliana, J.-C.Jean-Claude Léon, O.Olivier Palombi, M.-C.Marie-Christine Rousset and F.François Faure. Combining 3D Models and Functions through Ontologies to Describe Man-made Products and Virtual Humans: Toward a Common Framework.Computer-Aided Design and Applications1222015, 166-180HALDOIback to text
• 29 articleM. S.Mei Si et al.. Thespian : An architecture for interactive pedagogical drama.Artificial Intelligence in Education1252005, 595–602back to text