Images, often accompanied by sound effects, have become increasingly present in our everyday lives; this has resulted in greater needs for content creation. Despite the fact that many traditional means exist, such as photography, artistic graphic design, audio mixing, they typically still remain the reserve of the expert, and require significant investment in time and expertise.
Our main interest is computer image and sound synthesis, with an emphasis on automated methods. Our main goals include the simplification of the tasks required for the production of sound and images, as well as the development of new techniques for their generation.
The application domain is vast. It ranges from audiovisual production, which typically requires long, offline computation to obtain high quality results, all the way to real-time applications such as computer games or virtual reality, for which the main consideration is to guarantee 60 frames per second frame rates, or, in general the reduction of latency to user reaction.
The process of generation of images and sound, generally called rendering
is our primary interest; our second main interest are virtual
environments (VE's) as well as augmented (AE's) or mixed environments (ME's),
that is scenes containing both real objects (often digitized) as well as purely
synthetic objects. We are interested in both the generation and the interaction
with these environments. We use the term virtual environments for scenes
with a certain degree of interactivity, potentially in a semi-immersif
(stereo and tracking, workbench) or immersive (CAVE, RealityCenter) context.
We consider plausible rendering to be a first promising research direction, both for images
and for sound.
Recent developments, such as point rendering, image-based modeling and rendering, and
work on the simulation of aging indicate high potential for the development of techniques
which render plausible rather than extremely accurate images.
In particular, such approaches can result in more efficient renderings of very
complex scenes (such as outoors environments). This is true both for visual (image) and sound rendering.
In the case of images, such techniques are naturally related to image- or point-based methods.
It is important to note is that these models are becoming more and more important in the
context of network or heterogeneous rendering, where the traditional polygon-based approach is
rapidly reaching its limits.
Another research direction of interest is realistic rendering using simulation methods, both for images and sound. In some cases, research in these domains has reached a certain level of maturity, for example in the case of lighting and global illumination. For some of these domains, we investigate the possibility of technology transfer with appropriate partners. Nonetheless, certain aspects of these research domains, such as visibility or high-quality sound still have numerous and interesting remaining research challenges.
The key elements required to obtain visually rich simulations, are sufficient geometric detail, textures and lighting effects. A variety of algorithms exist to achieve these goals, for example displacement mapping, that is the displacement of a surface by a fucntion or a series of functions, which are often generating stochastically. With such methods, it is possible to generate convincing representations of terrains or mountains, or of non-smooth objects such as rocks. Traditional approaches used to represent such objects require a very large number of polygons, resulting in slow rendering rates. Much more efficient rendering can be achieved by using point or image based rendering, where the number of elements used for display is view- or image resolution-dependent, resulting in a significant decrease in geometric complexity.
Such approaches have very high potential. For example, if all object can be rendered by points, it could be possible to achieve much higher quality local illumination or shading, using more sophisticated and expensive algorithms, since geometric complexity will be reduced. Such novel techniques could lead to a complete replacement of polygon-based rendering for complex scenes. A number of significant technical challenges remain to achieve such a goal, including sampling techniques which adapt well to shading and shadowing algorithms, the development of algorithms and data structures which are both fast and compact, and which can allow interactive or real-time rendering. The type of rendering platforms used, varying from the high-performance graphics workstation all the way to the PDA or mobile phone, is an additional consideration in the development of these structures and algorithms.
Such approaches are clearly a suitable choice for network rendering, for games or the modelling of certain
natural object or phenomena (such as vegetation, or clouds). Other representations merit
further research, such
as image or video based rendering algorithms, or structures/algorithms such as the "render cache"
Point- or image-based representations can also lead to novel solutions for capturing and representing real objects. By combining real images, sampling techniques and borrowing techniques from other domains (e.g., computer vision, volumetric imaging, tomography etc.) we hope to develop representations of complex natural objects which will allow rapid rendering. Such approaches are closely related to texture synthesis and image-based modeling. We believe that such methods will not replace 3D (laser or range-finger) scans, but could be complementary, and represent a simpler and lower cost alternative for certain applications (architecture, acheology etc.).
We are also investigating methods for adding "natural appearance" to synthetic
objects. Such approaches include weathering or aging techniques,
based on physical simulations
Similar to image rendering, plausible approaches can be designed for audio rendering. For instance, the complexity of rendering high order reflections of sound waves makes current geometrical approaches inappropriate. However, such high order reflections drive our auditory perception of "reverberation" in a virtual environment and are thus a key aspect of a plausible audio rendering approach. In complex environments, such as cities, with a high geometrical complexity, hundreds or thousands of pedestrians and vehicles, the acoustic field is extremely rich. Here again, current geometrical approaches cannot be used due to the overwhelming number of sound sources to process. We study approaches for statistical modeling of sound scenes to efficiently deal with such complex environments. We also study perceptual approaches to audio rendering which can result in high efficiency rendering algorithms while preserving visual and auditory consistency if required.
A large body of global illumination research has concentrated on finite element methods for the simulation of the diffuse component and stochastic methods for non-diffuse component. Mesh-based finite element approaches have a number of limitations, in terms of finding appropriate meshing strategies and form-factor calculations. Error analysis methodologies for finite element and stochastic methods have been very different in the past, and a unified approach would clearly be interesting. Efficient rendering, which is a major advantage of finite element approaches, remains an overall goal for all general global illumination research.
For certain cases, stochastic methods can be efficient for all type of light transfers,
in particular if we require a view-dependent solution. We are also interested both in
pure stochastic methods, which do not use finite element techniques.
Interesting future directions include filtering for improvement of final image quality as well
as beam tracing type approaches
Visibility calculations are central to all global illumination simulations,
as well as for all rendering algorithms of images and sound. We have investigated
various global visibility structures, and developed robust solutions for scenes
typically used in computer graphics. Such analytical data structures
Our solutions to date are based on general and flexible formalisms which
describe all visibility event in terms of generators (vertices and edges);
this approach has been published in the past
For purely diffuse scenes, the radiosity algorithm remains one of the most well-adapted solutions. This area has reached a certain level of maturity, and many of the remaining problems are more technology-transfer oriented (see also the paragraph on applications). We are interested in interactive or real-time renderings of global illumination simulations for very complex scenes, the "cleanup" of input data, the use of application-dependent semantic information and mixed representations and their management.
Hierarchical radiosity can also be applied to sound, and the ideas used in clustering methods for lighting can be applied to sound.
Our research on high quality audio rendering is focused on developing efficient algorithms for simulations
of geometrical acoustics.
It is necessary to develop techniques that can deal with complex scenes, introducing
efficient algorithms and data structures (for instance, beam-trees
Validation of the algorithms is also a key aspect that is necessary in order to determine important acoustical phenomena,
mandatory in order to obtain a high-quality result. Recent work by Nicolas Tsingos at Bell Labs
Finally, several signal processing issues remain in order to properly and
efficiently restitute a 3D soundfield to the ears of the listener over
a variety of systems (headphones, speakers). We would like to develop an
open and general-purpose API for audio rendering applications.
We already completed a preliminary version of a software library: AURELI
The second major research direction of our group is on virtual, augmented or mixed environments,
which include both visual and sound representations. We are mainly interested in interactive
environments, permitting the user to create and maniuplate scenes consisting
of both real and synthetic objects. As a first step, we consider real objects
to be digitised representations of reality, rather than the real world.
Our first goal is to apply and adapt our rendering expertise, presented in the previous paragraphs to virtual and augmented realy. There are three areas in which we concentrate our efforts: consistent lighting between real and synthetic illumination, for shadows and reflections, enriching virtual and augmented environments with sound, in a consistent manner and finally appropriate interaction and visual paradigms for virtual and augmented environments.
We wish to develop relighting and consistent real/virtual lighting methods which have simple input requirements: i.e., a small number of input images, and the smallest number of restrictions on the lighting conditions. The goal is to get high quality results for both interior and outdoors environments. To achieve these goals, we investigate ways to extract approximate reflectances in real scenes, potentially using scene or image statistics, and by including some level of user interaction in the process. For efficient display, texture capacities of modern graphics hardware will definitely be advantageous.
Our previous work on interior relighting has given satisfactory solutions,
allowing us to add virtual object with consistent lighting,
but implies severe restrictions on the lighting conditions of input images
The long-term goal is to turn on a video camera in a scene (potentially with partially pre-reconstructed geometry), and be able to add virtual objects or light sources interactively in a consistent manner into the video stream. Relighting could also be achieved in this manner, or using semi-transparent glasses or headsets. Applications of such an approach are numerous, for archeology, architecture and urban planning, special effects, manufacturing, design, training, computer games etc.
This long term vision will require a way to smoothly vary from low-quality methods
Consistent rendering of real and synthetic sounds is a key aspect for virtual reality applications. Solving the problem would make it possible to mix natural sounds with synthesized spatial audio for augmented reality applications. This can be used to enrich the natural soundscape with additional auditory information through wearable devices (e.g., virtual museums, etc.). Another application would be to provide auditory feedback to visually-impaired people while preserving their natural auditory perception.
Another future direction of research is active control of rooms and listening spaces. Such control can be achieved by coupling microphones and speaker arrays and allow for modifying the natural acoustical properties of the space (e.g., reverberation time) in real-time. Such technologies have already been used to improve acoustics in concert halls that, for a variety of reasons, do not sound as good as designed for. They appear to be promising for VR/AR applications. However, existing techniques yet have to be improved to be applied in this context.
The use of immersive or semi-immersive systems opens a large number of new types of interaction with virtual or augmented environments. There is a vast body of research on interfaces for 3D environments, and in particular for immersive systems. Our focus will be on specific interfaces, interaction or visual paradigm problems which inevitably appear in the course of our research. When necessary, we will work with completent partners in Computer-Human Interaction to find solutions to these problems.
One question we consider important is finding appropriate interface paradigms which replace 2D (menu or button-based) interfaces both in the context of the actual rendering research process and for the applications we investigate. Despite significant previous work in the domain, there is yet to be a standard which has been widely adopted. It is entirely possible that the lack of usable interfaces is part of the reason that immersive systems are not being adopted as widely nor as rapidly as their inventors would have hoped.
In terms of visual representation, non-photorealistic (NPR) or expressive, renderings are an interesting avenue of investigation. In particular, NPR can allow abstraction of unimportant details and more efficient communication of certain concepts. Since a number of the algorithms developed are based on inherently 2D drawing, their transposition to immersive, stereo-display environments poses a number of very interesting and challenging questions. There are also some applications domains, for example archeology or architecture, where drawing-style renderings are part of the current workflow, and which will naturally fit into a EVs adapted to these domains. Virtual storytelling is another domain in which NPR has a natural application.
Immersive, stereo-based systems seem a well-adapted platform for more intuitive interactive modelling in 3D. The development of efficient and flexible structures such as procedural point-based representations, or rapid aging techniques in a true 3D context could result in systems which are much more efficient that 2D displays, in which the sensation of 3D depth and immersion is missing.
Finally, the inclusion of spatialised sound for 3D interfaces is clearly a promising research direction. The benefit of consistent 3D sound is evident, since it results in better spatial perception for the user, can help for example in determining spatial or visibility relationships, resulting in improved usability. The actual inclusion of sound effects or sound metaphors in interface design is clearly an interesting challenge.
Virtual heritage is a recent area which has seen spectacular growth over the past few years. Archeology and heritage exhibits are natural application areas for virtual environments and computer graphics and sound, since our research can reconstruct both in images and in sound, artefacts, monuments building of lost civilisations.
We are interested both in "general public" presentations of cultural heritage artefacts and sites, and in the actual archeological research of the experts.
Our cultural heritage work has concentrated around three main collaborations.
The first is with the Foundation of the Hellenic World (FHW)
Our collaboration with the FHW is in the context of the EU IST project CREATE (see also Section ). The combination of educational theories and field expertise allows a novel approach to the usage of Virtual Environments for both experts and novices. The first application of the project is in cultural heritage, and REVES contributes significantly to the development of higher quality virtual environments based on real world sources as well as virtual environment enhancements (view-dependent texturing, efficient vegetation rendering, lighting, shadows and spatialised sound). Work takes place both with archeologists who are interested in virtual reconstruction of monuments and with children in the context of the educational mission of the FHW.
In the context of ARCHEOS, we have worked on the Agora site of ancient Argos, and in particular on the Tholos monument. We have contacted and worked with two archeologists who have conflicting hypotheses on the utility of the monument, and reconstructed the two different possibilities in a virtual world. In addition, we have investigated the relative effectiveness or utility of realistic vs. non-photo realistic (or expressive) renderings for archeological reconstruction. Although this is very experimental work, but we have had very positive feedback from the archeologists.
Finally, we are participating in a cultural heritage project with the VR department of EDF (see also ). The goal here is the reconstruction of a complex monument at Delphes (the Danser column and the Omphalos), and to test a number of hypotheses (structural, aesthetic etc.). Our contribution is the development of efficient and high-quality reconstruction and rendering techniques, in the context of the thesis of F. Duguet, using a point-based approach.
We are also interested in the use of sound in a cultural heritage context. The acoustic properties of many sites could be particularly interesting to study, for example the famous acoustics of ancient Hellenic theatres.
Urban planning and architecture are very appropriate application domains for computer graphics and virtual environments, since they often deal with future projects requiring visual or auditive representations. In addition, evaluation, design review and training can also greatly benefit from digital or virtual mock-ups. Our expertise in rendering and its application to VE's can greatly benefit the process.
Our work in this domain has been with two main partners, the CSTB (the French
National Scientific and Technical Center for Construction
Training and evaluation are domains in which VE's can be applied naturally. The use of coherent sound and image renderings in VE's can be particularly important for training in risk-critical environments (for example nuclear power plants). We have several contacts with companies working in these domains (notably EDF).
Computer Games have been the driving force in the development of rendering and computer graphics research, especially in terms of low-cost hardware over the past few years. Interactive rendering for ever-more complex scenes, both in terms of geometry and lighting effects is clearly of great interest for games companies. Despite initial encouraging contacts, the current economic downturn in the industry has limited our collaboration with industrial partners in this domain.
Integration of sound spatialisation, either geometric or statistical is clearly promising, and is currently almost entirely missing in exisiting games. We hope to be able to convince industrial partners of the importance of these techniques and of their maturity for technology transfer.
A more long-term goal for this application domain is the use of virtual environments in low-cost immersive or semi-immersive contexts. Their augmentation with both synthetic images and sound should have great potential. As a first step, such applications would be limited to more "theme-park" style environments. However, the emergence of low-cost "wall-projection" stereo and tracking systems could result in the development of installations that have cost equivalent to that of a home-cinema setup today, making them potentially viable in a mass-market context.
Although our emphasis is on interactive applications, our high-quality rendering research, both in sound and images could be of interest in post-production or the film industry. We are also interested in combined interactive/offline approaches, such as the previsualisation tools in certain modelling/animation packages.
Integrating vision-based match-moving techniques for placement of sound tracks could also be useful, but care must be taken to preserve artistic control for sound engineers and technical directors.
We are developing a global illumination system, ECLAIRES, based on the work carried out by Xavier Granier during his Ph.D. thesis. ECLAIRES is a radiosity system based on hierarchical clustering methods. ECLAIRES is a multi-platform system running under Linux, Windows and Irix and is based on the REVESAPI common library (see ).
The aim of this developement, supported by a national Inria funding for a software engineer (G. Braconier), is to provide a robust library for computing global illumination for realistic environments. The two applications concerned are architecture and archeology. For urban planning and architectural applications, we collaborate closely with the CSTB, which provides us models of real buildings allowing us to validate our algorithms.
We are investigating the use of semantic information for architectural scenes, based on the industry standard IFC (Industry Foundation Classes), and the unified approach which combined particle tracing and radiosity, for non-diffuse and diffuse transfer respectively.
REVES is developing an API, AURELI (AUdio REndering LIbrary), as a tool supporting our research in acoustic modeling and audio rendering. Several prototype algorithms for sound spatialization, geometrical and statistical reverberation modeling, sound source clustering and audio rendering server have been implemented using AURELI's core functionnalities or as an extension to the API itself. Core fonctionalities include audio i/o plug-ins, audio buffer handling, basic signal processing. Higher level functions perform geometrical processing and audio rendering on a variety of restitution systems. AURELI is a cross-platform, object oriented, C++ API. It runs on LINUX/Windows/IRIX and also features primitives for parallel signal processing on multi-processor systems and network communication (used for instance to design audio rendering servers).
We are investigating possibilities for public release of the API as a tool for researchers in acoustics and virtual/augmented reality.
The work developed in 2001-2002 by Marc Stamminger at REVES has been integrated into the common software platform of the CREATE project. As a precursor to this step, two separate libraries have been developed, one for shadows and another for point-based rendering, which have been tested on real-world data for evaluation purposes with local companies with visualization needs.
We started last year the development of a common platform for research
algorithm implementation. The platform is now operational and a significant
amount of code has been developed in common by interns, PhD students, and
researchers.
This development platform exposes an API for applications which involve 3D
graphics. This API is implemented on multiple platforms and can be compiled using
different compilers.
The platform is independent of the widget toolkit, and
two different toolkits are currently available: Qt and the native Win32 API.
The system runs on Windows using Visual C++ compiler, or the MinGW compiler
(
Tools for 3D graphics have been developed and are shared amongst the team members including a scene graph structure, an OpenGL renderer for this scene graph, and a (partial) VRML 2.0 parser. Application templates have been provided as well; using these templates a new user can have a customizable VRML viewer running with 4 lines of C++ code. The main motivation of this work was to gather and share as much code as possible so that current as well as new users can reuse code from another contributor with a minimal integration work. For example, a new user can write a specific renderer without having to care about the scene graph construction, the parsing of the geometry file (VRML 2.0 is partially supported), and the specificities of the platform the user works on; the only code (s)he has to write is the pure rendering code (see image for dynamic renderer selection).
This approach has recently been tested in practice, with complex data structures developed for one group member being used by another, thus greatly accelerated development time. Interface modules are also shared amongst group members.
We have developed a flexible point-based rendering algorithm that works on PDAs. A 3D object is first encoded into a hierarchical recursive grid data structure, and then efficiently rendered using a dedicated point-based rendering algorithm. We presented a study of the type of hierarchical structure which should be used. We chose the 3x3x3 recursive grid (which we call the tri-grid), for its good compromise on rendering speed and storage compactness. We extended a rendering algorithm presented previously and made it flexible to allow rendering of the structure at different levels of the hierarchy in the same rendering pass. With this algorithm, we can achieve efficient view-frustum culling, and automatic, view-dependent level of detail. We also presented a one-pass shadow-map algorithm, avoiding expensive matrix transformations.
These algorithms have been implemented on an iPAQ HP3850, with an ARM processor at 200MHz, running PocketPC 2002. Objects sampled at 1.3 Million points can be rendered with shadows at interactive framerates, i.e, 2.11 fps.
This work is described in the research report RR-4833
Collaboration with Marc Stamminger at the University of Erlangen.
Our work on perspective shadows maps, presented last year
at SIGGRAPH 2002 was extended and adapted to
a Computer Games context. This work was presented in a more "wider public"
context as a chapter part Computer Games Gems IV
In collaboration with Pierre Poulin at the University of Montréal, and Marc Stamminger at the University of Erlangen.
We have developed a novel approach which permits point-based modeling from images. As described previously (Sec. ), complex objects can be displayed rapidly when represented by points rather than polygons. Based on this idea, we have built an interactive system which permits automatic reconstruction of complex geometric objects such as plants, organic matter and generally irregular geometry. The main emphasis of our solution is the fact that the user can interactively control the process at every step of the reconstruction, thus significantly improving the quality of the geometry generated and rendered.
The system starts with a small number of images, without any restrictions on camera pose. After the calibration phase of the camera, and the interactive definition of an approximate volume containing the object to be reconstructed, the system generates a large number of points inside this volume. A set of known or improved techniques are them combined to test the quality of the points generated, for example colour or depth comparisons. As mentioned above, interactivity and user intervetion are central to our approach; the user can designate regions in which the system should add more points, regions where a mask can be applied to guide point generation, or regions where a curve filling approach (for example tree branches) is more appropriate. At all intermediate steps, and of course once the model is complete, the system can generate images from any viewpoint using the point-based representation and a view-dependent texturing approach. The results of this research (see Fig. ), are very satisfactory for objects which are hard to reconstruct using other methods because of small, irregular details.
This work was presented at the international conference Graphics Interface 2003
This project aims at helping artists build depth maps typically used to add details to their scenes. Most of the time, artists have to build their displacement or depth maps by hand; this can be tedious when the textures are large. We have developed a method to automate this process. To add detail to a texture, the artist only has to supply a few detail extracts (typically 2 or 3), along with the associated hand-drawn depth. The system will use these patches to produce a complete depth map for the entire input texture according to the information in the patches. The details on these objects are then generated according to the depth maps and junctions between two displaced textures are handled correctly to produce a seamless blend (see Fig. ).
In the context of the european IST project CREATE, we developed several techniques aimed at accelerating the audio rendering of complex scenes, comprising a large number of point sources. The first techniques groups sound sources based on geometrical and psycho-acoustical information updated in real-time. This technique dynamically best-fits a specified number of clusters to the sound scene and, as such, can be used for dynamic allocation of computational resources. We also explored how programmable graphics hardware can be used in this context to speed up audio calculations. A second technique uses psycho-acoustics to evaluate auditory masking occuring in the sound scene in real-time. This information is then used to cull (i.e., not render) inaudible sound sources. This work was published as an INRIA technical report and also appeared as a technical paper in the 2003 Online Audio Resource Guide of Gamasutra.com, a web site specializing in computer gaming. We further improved the techniques and began a series of perceptive tests to validate our approaches.
In the context of Charles-Felix Chabert's master work
Finally, a state-of-the-art paper on audio rendering techniques for virtual reality applications, co-authored by Tom Funkhouser at Princeton and Jean-Marc Jot at Creative Labs, has also been accepted for publication in Presence, the reference journal of the virtual reality community
The goal of this research is to efficiently render virtual scenes taking advantage of the human perception
of the environment
We started work on this research direction by initially concentrating on the study of attention only for the visual aspect, which represents a fairly large part of the problem. The aim was to develop an algorithm capable of detecting the most important zones in a 3D scene, depending on the point of view without using a tracking device. The idea we exploited was to use two different kinds of analysis and make them converge into a single, plausible, saliency map.
The first analysis only takes the scenegraph data into account whereas the second makes use only of the information given in the pixels. Two sorts of applications could use this algorithm. On the one hand, the rendering process could be accelerated using an attention guided level of details algorithm. On the other hand, we could refine the important parts of a scene if they do not appear with the required importance, effecting attention control. Prototype scenegraph and pixel based analysis tools have already been developed. We are now working on level of detail control issues and merging of our two analysis tools.
Collaboration with Bell Laboratories and Princeton University
A paper compiling all the research work, started at Bell Laboratories, concerning efficient high-quality simulation of sound propagation has been published in JASA (Journal of the Acoustical Society of America), the reference journal in the field. Our techniques, based on beam tracing, allow for efficient and aliasing free construction of reflection and diffraction paths in 3D architectural environments.
Capturing and rendering of real scenes in an immersive virtual environment is still a challenging task. We have developed a new technique to generate view dependent textures using image layers that produces high quality images. In addition the new approach reduces texture memory consumption. The algorithm has two steps: the first part builds the layered textures and the second part uses these textures to render the scene.
To generate the layered textures the algorithm takes high quality images as input, the corresponding calibrated cameras and the 3D model of the objects in the photographs. Then we use the camera information to sort the geometry by visibility, grouping the objects with the same level of visibility into visibility layers. These layers are projected into the image space to generate image Layers. The image layers are then split into smallest layers using a criteria based on OpenGL, to optimise the texture memory used. At the end of the process all the images have been segmented into a sequence of image layers, each one of them corresponding to a level of visibility.
For rendering, we extended the traditional projective texture approach to be able to manage layered textures. The main difference with standard projective textures is that the matrix projection for a layer is an asymmetric projective matrix and we need to keep the relation between each object and the associated layer into the image. We used this new projective texture technique in a view dependent projective texture algorithm. Compared with precedent algorithms, our new technique does not require the subdivision of geometry (imposed by the visibility problems during the texture projection in previous techniques) and allows the artist to edit the texture using standard tools such as clone brushing to fill holes due to occlusion. This step was impossible using previous approaches; our method gives complete control of the final result to the artists.
A short version of this research was published in the Proceedings of the Pacific Graphics Conference 2003
This work is part of the EU IST project CREATE.
As part of the EU IST project CREATE, we have developed an series of virtual environment enhancements in an integrated software platform, incorporating view-dependent texture mapping, point-based rendering for vegetation, perspective shadow maps and 3D spatialised sound. All of these components have been integrated into the common CREATE platform based on Performer, CAVElib and the high-level scripting language XP developed at EVL and FHW allowing easy integration of novel functionalities.
This system has been used in many demonstrations, and we have started an official collaboration
with the architects of the Tramway project of the city of Nice using our tools.
An initial presentation of these enhancements in the context of the CREATE
project was presented as a poster at the international symposium on Mixed
and Augmented Reality (ISMAR) 2003
Research in collaboration with ARTIS and MIT, in the context of ARC ARCHEOS.
In the context of the ARCHEOS national research action (see ), we have
investigated questions related to expressive (better known as non-photorealistic) renderings
in the context of archeological applications.
The research developed this year concentrated on two main axes, that of
rendering styles for walkthroughs and the use of expressive rendering in semi-immersive
or immersive systems.
Matthieu Cunzi, in collaboration with his DEA stage supervisor J. Thollot and other
colleagues at ARTIS and MIT, developed a novel approach for rendering the effect
of canvas granularity for expressive rendering.
The solution consisted of developing a solution for infinite zoom based on
repetitive textures, and appropriate texutre warping for rotation.
This work was published at the international conference Graphics Interface 2003
in Canada
In the context of his 6-month internship Roman Bayon developed a number
of solutions for expressive rendering on our workbench platform. He investigated
a number of software solutions, including the use of our common API.
The final solution adopted involved the use of standard tools i.e., Performer
and CAVElib. In his internship, the idea of transposing the canvas solution
to stereo was also partially investigated; this issue proved to be more
complex than initially expected, and further research is required.
An internship report describing this work is available
In collaboration with M. Roussou, and in the context of
both the EU CREATE project and ARC ARCHEOS we presented a position paper
on the relative advantages and disadvantages of realistic vs. non-photorealistic
renderings for cultural heritage applications. This paper was presented
at the first EG workshop on Graphics and Heritage, which was common
event with VAST 2003
The thesis fellowship of Alex Reche is a CIFRE contract which is an industry-research collaboration. In this context, Alex Reche spends a significant amount of time at the CSTB and strives to apply his work directly on urban planning and architectural applications.
We have an ongoing research contract with REALVIZ which is related to the development of novel algorithms and procedures for reconstruction and from images and view-dependent rendering, in the context of the CREATE project.
We are part of the Alias|Wavefront software donation program. Our artist Alexandre Olivier uses Maya extensively to model various objects used in our animations and results, and especially in the context of the ARCHEOS project.
In the context of cultural heritage preservation and studies, EDF Foundation has funded the 3D scanning of the Omphalos column in Delphi, Greece. This column, also known as the "Dancers column" is stored in several pieces in creates in the Delphi museum. The goal of the project is to scan the different parts of the column, to test archeological hypotheses using digital data acquired with the computer, including positioning of the stones, comparison of statues etc. The project includes partners from the Louvre Musuem (J-L. Martinez, the curator who has studied this monument extensively, INSIGHT, a not-for-profit company specialising in virtual heritage and scanning and archeologists from the Ecole Normale Superieure, Paris.
The data set will include up to 300 million points, which is a real challenge to display. Most recent graphics cards can display 60 million points per second, but the memory required for such a point set is 3.6 Giga bytes. Our work will concentrate on post-processing of the data, for better reconstruction and for efficient interactive rendering, as part of the F. Duguets thesis.
We are in the process of filing a French and US patent on clustering and masking algorithms for 3D sound "Dispositif et méthode perfectionnés de spatialisation du son" (Advanced System and Method for Sound Spatialisation).
We collaborate with CSTB in the context of the European IST project CREATE. We are also conducting tests for global illumination simulation for architectural environments in the context of the ECLAIRES project, developed by G. Braconier. Such tests allow us to validate our lighting algorithms.
The regional Provence-Alpes-Cote d'Azur government has co-funded (with INRIA) the acquisition of semi-immersive platform for virtual environments, also known as "workbench". David Geldreich setup, integrated and continues support for the system.
The platform is composed of a Barco Baron screen (1.5 m diagonal) which can be tilted from near horizontal (table) to near vertical position. The screen is equiped with a BarcoReality 908 CRT projector driven by an off-the-shelf PC (2 PIII 1GHz + GeForce 4 Ti 4600 graphics) running under Linux. Stereo display is achieved through a frequency-doubler StereoGraphics EPC-2 and active LCD shutter-glasses (StereoGraphics CrystalEyes and NuVision/60GX). Finally, we also use a 6-DOF Polhemus Fastrak 3D tracking system interfaced with a stylus for interaction and an additional captor for view-point tracking and view-dependent rendering.
D. Geldreich installed the system and developed a suite of APIs and tools allowing for easy integration of the stereo display and tracking system in any Performer/VTK- based application. D. Geldreich also installed the commercial library CAVElib, used in the context of european IST project CREATE. Several members of the group adapted their applications so they can run on the platform.
D. Geldreich and N. Tsingos also worked on the final set-up of the workbench in an acoustically-treated room where the immersive display will be combined with an immersive binaural or 8.1-surround sound restitution system. The final system should be operational by early 2004.
REVES is the coordinator of the national research action ARCHEOS, whose initial goal was to investigate the use of expressive (non-photorealistic) rendering for archeology. The partners of the action are ARTIS in Grenoble, the Homeric Studies group ERGA in Grenoble, the architecture School of Lyon, the Ecole Normale Supérieure in Paris and the Foundation of the Hellenic World in Greece. One of the main goals of this project is to initiate a dialogue between the computer graphics and VE researchers on the one hand, and the historians and archeologists on the other.
We have concentrated our work on a specific archeological example, which is the Agora of Argos, an ancient city in Greece. One monument of particular interest is the Tholos (see Fig. and ), where there is some doubt of its exact utility. Different archeological hypotheses have been proposed concerning its form and its utility.
In its second year, the ARC ARCHEOS funded the internship of Roman Bayon (see also Section ) as well as the summer work of Pascal Barla at ARTIS in Grenoble after his DEA.
Two main meetings took place this year. The first meeting, in Grenoble in May, was particularly interesting, since two of the main archeologists who have excavated the Argos site were present (A. Pariente and M. Pierart). The interest of the archeologists was evident, both for the general aspect of VE's and for expressive renderings. The second meeting was a research brainstorming on Computer Graphics aspect of the work in July in Sophia-Antipolis.
The work of ARCHEOS was also presented at two archeological research meetings, one in Boston as a poster and one in Athens at the Ecole Française d'Athènes, which is responsible for the Argos site. In both cases, the archeologists present expressed great interest in the work, and were particularly interested to see its continuation.
The closing event of the ARC ARCHEOS is planned for early December; the morning will include a presentation of the research results and demos (see Fig. ), and the afternoon will include a round-table of archeologists, computer graphics and virtual environment researchers.
A national project, coordinated by ARTIS in Grenoble, started this autumn. Our paticipation in this project will be on point-based or alternative rendering of very large data sets, such as scans of statues or alternative representations of trees. The other participants of this project are IPARLA in Bordeaux and ISA in Nancy.
REVES is also coordinator of a new RNTL project: OPERA, which aims at further studying how audio rendering can be optimized using perceptual knowledge with two applications in mind: telecommunication ("chat rooms", MMOGs) and virtual reality (e.g,. urban planning). In this context, REVES collaborates with IRCAM, France Telecom R&D, LIMSI, CSTB and the company VIRTOOLS. The project will start early 2004.
This year the following researchers visited the group Ken Perlin (New York University), Mat Kaplan (University of Utah), Alexandre Meyer (University of London) and Xavier Decoret (ARTIS, Grenoble). Several other researchers visited the group in the context of ARCHEOS and CREATE.
As metioned previously in various sections, a significant part of our activities this year
has concentrated around the EU IST CREATE project``Constructivist Mixed Reality for
Design, Education, and Cultural Heritage''
The capture and display pipeline starts using improved versions of the REALVIZ Stitcher and ImageModeler tools. We developed an improved capture pipeline which starts with the creation of panoramic images using Stitcher at various locations in the environment. These are then passed to ImageModeler for calibration, and then are the basis for Image-Based modeling. We then use the VE enhancement tools to display the environments created in an immersive or semi-immersive context.
Two application demonstrators are under development as part of the contract. The first concerns the site of ancient Messene, which is being built by the Foundation of the Hellenic World. Alex Reche participated in the original capture of the Messene site in Greece this May, which involved 3 days on site and another 2 days at the FHW to calibrate the model. The goal is to build a demonstrator which will be used by children to improve learning in the architectural and archeological domain.
The second demonstrator is the Tramway project of the city of Nice, for which public works have started this autumn. After a series of demonstrations to officials and engineers of the real project (see ), we have established an official convention of collaboration with the "Mission Tramway" which is the organisation in charge of the implementation of the project.
Initially we created a pilot captured model of Place Massena, which was enhanced with point-based vegetation, view-dependent texturing, perspective shadows and Initially a series of photographs were taken from various viewpoints. Then 3 photographs were selected to represent the point of view and we extracted textures from them. Virtual elements, such as the Tramway or the trees, were also integrated into the scene (see Fig. and (a)). The system has been successfully ported to the CSTB RealityCenter (see Fig. (b)).
After establishing contact with the real Tramway project and discussing choices in detail with the engineers and the architects, we decided to develop Place Garibaldi as the final demostrator of the urban planning CREATE scenario. A series of 5 panoramas were taken to capture and represent different points of view and to build a complete 3D model. This project is one of the most detailed and most complex to be developed using the REALVIZ tools, which have been successively adapted and extended to meet the needs of this project. The initial capture and modelling of this site is complete (see Fig. ).
Current discussions with the architects are centered on the different choices for the development of the square, their presentation and the alternatives which are possible. We expect this activity to result in interesting applied VE research for a real application.
Our collaboration with the University of Montréal and Pierre Poulin continued
successfully this year with the presentation of our common work at Graphics Interface 2003
Our work in common with the FHW has continued in the context of the CREATE project,
and with the presentation of two publications (ISMAR
Our work with M. Stamminger at the university of Erlangen has continued
with the presentation of the "wider public" presentation of the
perspective shadow maps
in Game Programming Gems IV
We have re-established contact with the research group of Girona, with the visit of Ignacio Martin from this group for a month and a half. We intend to continue this collaboration next year.
During the summer, Florent Duguet visited Fredo Durand in the Graphics lab at MIT funded by the MIT-France project for two months. A new research project has been started during this visit putting together several techniques from both parties. The focus of the project is mesh smoothing and surface reconstruction. The collaboration of MIT and the REVES team is continuing with a publication project.
Nicolas Tsingos is actively collaborating with Bell Laboratories, Princeton University and Creative Labs.
We currently have active research contacts with Columbia (NY) and Cornell (NY).
G. Drettakis was a member of the programme committees for the Eurographics Symposium on Rendering 2003, and VAST2003/1st Eurographics workshop on Graphics and Heritage. G. Drettakis and N. Tsingos reviewed papers for the following conferences: SIGGRAPH, Eurographics, ACM SIGGRAPH Symposium on Interactive 3D Graphics, as well as several journals (JASA, IEEE CG & A, IEEE TVCG etc.).
G. Drettakis was an external examiner for the thesis of P. Reuter at the University of Bordeaux.
The web server was completely redesigned and put online in Spring this year.
Most of the publications of REVES can be found online, and often
include short movies demonstrating our research results. See
George Drettakis was responsible for the Computer Graphics course at ISIA (École des mines) in January 2003 (15 hours). He also taught three hours of the Medical Robotics course at the Graphics DEA at the UNSA and one hour in Computer Graphics for second-year engineering students of the ESSI. Alex Reche taught 12 hours of the Image-Based reconstruction and OpenGL course and laboratories for the second year students of the LPMI at the Nice IUT. Florent Duguet taught three hours of the Computer Graphics course and four hours of associated laboratories for third-year engineering students of the ESSI. He is also a regular teaching assistant (as part of his PhD AMX fellowship) at the University of Nice, supervising laboratories of Java beginner programming courses and general computer science laboratories. Marie-Claude Frasson gave two hours of a Ray-Tracing course to second-year engineering students at the ESSI, plus eight hours of associated laboratories. She also supervised the 4 hours OpenGL laboratories of the second-year LPMI students of the IUT of Nice. Nicolas Tsingos was responsible for the ``advanced 3D rendering" module of second-year LPMI of the IUT of Nice (20 hours).
Marie-Claude Frasson and Alex Reche registered at the University of Nice for their third PhD year, while Florent Duguet is beginning is second year, also at the University of Nice Sophia-Antipolis (UNSA).
Alex Reche presented his short paper as a poster at the Pacific Graphics 2003 conference in Canmore, Canada and its french version at the Journées de l'AFIG 2003 in Paris. Matthieu Cunzi presented his paper at Graphics Interface, and Pierre Poulin presented the paper on point reconstruction. The ISMAR 2003 poster was presented by Celine Loscos of UCL and the VAST 2003 paper was presented by Maria Roussou of FHW/UCL. Almost all members of the team held a booth and participated at the Imagina 2003 conference in Monaco.
In addition to the above-mentioned conference presentations, the researchers, Ph.D. students and engineers went to the 2003 Eurographics Symposium on Rendering. G. Drettakis, F. Duguet and N. Tsingos were at the San Diego SIGGRAPH 2003 conference. M-C. Frasson and Alex Reche went to the Eurographics 2003 conference in Granada.
Since the workbench has become operational, our group is often solicited for demonstrations.
In the context of CREATE, we established contact with the real Tramway project, by first presented the prototype CREATE demonstrator (Massena) to M. Bonis chief engineer of the Tram Project and two public relations officers in April. We then presented the work to M. Sanz municipal councillor in charge of Transportation of Nice and M. Cherifi PR delegate for the CANCA in May, which was a necessary step in establishing contact.
More recently, we have demonstrated the work to the the architect company MOER (June and August), which is in charge of implementing the Tramway project.
We also presented research demonstrator on the workbench to the companies OKTAL SA and VIRNET (April/May), INRIA directors of national units (May), Toyota (July), the Mayor of Antibes and municipal delegates (August 2003) and the vice-president of the Regional local government, as well as other local officials (December 2003). We also presented the workbench during "La fete de la Science" in October (150 people over two days).
We also contributed to the INRIA press release on virtual reality,