Iparla is a joint project of INRIA and LaBRI, situated in Bordeaux. LaBRI is a joint laboratory of CNRS (UMR 5800), University Bordeaux 1, and ENSEIRB.
The Laboratory of Cognitive Sciences (University Bordeaux 2 and Institut de cognitique) is associated with the project.
The project has officially been created the 13/11/2003.
Mobile and connected devices (MCD), such as cell phones and PDAs, allow benefiting from computing resources in mobile settings. On the one hand, embedded CPUs enable local computation, and on the other hand, the wireless communication makes it possible to use client/server infrastructures. The goal of the Iparla project is to benefit from these mobile and remote computing resources for the visualization and manipulation of complex datasets on MCD. We understand complex datasets as large amounts of multi-dimensional data. The complex datasets can have their origin either from acquired real-world datasets, such as scanned objects, automatically generated cities, and terrains, or from abstract information, such as financial or statistical data.
Compared to classical computers, MCD have several limitations that imply various challenges for the development of interactive applications. First, the computing resources (eg. CPU and memory) are limited, in particular because of the restricted power consumption. Second, the small size of the devices implies the use of small screens, which decreases the provided area for visualizing complex datasets. Finally, the input devices for the interaction with MCD are limited. Consequently, we state that classic 3D pipelines are not always the most appropriate solution for the development of interactive 3D applications on mobile devices. In the Iparla project, we develop new techniques for the visualization and manipulation of complex datasets. In particular, our team focuses on evolutive modeling, rendering, and interaction techniques for MCD.
Let us first consider the complete pipeline that is involved in interactive 3D graphics applications. First of all, 3D environments are created, either by using modeling tools, or by directly acquiring real-world data. The 3D environments, which are stored in the memory of the computers, can then be visualized by using rendering techniques. Finally, the interaction with the 3D environments is possible through adapted user interfaces.
There is a variety of previous work that adresses the acquisition, visualization, and interaction with 3D environments. This work have contributed to the evolution of 3D graphics since the early 80's, mainly for classical workstations. Today, we are convinced that the classical pipeline must be adapted to requirements of mobile settings. The Iparla project aims at enhancing the entire pipeline by conjointly developing new techniques in the scope of modeling, rendering and interaction for the visualization and the manipulation of complex data on mobile devices.
For a better understanding of our technological environment, we first describe the technical characteristics of mobile and connected devices (MCD).
MCD have primarily been devoted to voice transmission using the GSM standard (9.6kbps). The same standard has been used for the transmission of short-text messages (SMS protocol). Today, new standards with higher bandwidths enable the transmission of multimedia data such as images and movies. The GPRS standard appeared in 2002 with a bandwidth of 20 to 60 kbps, and today UMTS is around with bandwidths from 400 kbps to 20 Mbps. To complete the development of these international networks, some standard protocols for local wireless communications have been introduced, like BlueTooth (1 Mbps, 100 m) or IEEE 802.11 (10 Mbps, 500m, recently renamed WIFI). There are more and more mobile devices ranging from cell phones to PDA that have now integrated these standards.
The hardware capacities of mobile devices have spectacularly increased recently. The today capacities of the best mobile devices are comparable to about five-year-old graphics workstations. However, the gap between mobile devices and standard workstations will never be filled due to the inherent limitations of the handhelds. In particular, low power consumption, low weight and low cost are limiting issues in the heart of the mobile devices market. Concerning graphics, the first 3D architectures were presented (eg. the Nvidia GoForce 3D wireless processors and the ARM-based Nexperia pnx 4008). We assume that many products will follow these first examples. Furthermore, software standards are defined as well. In particular, OpenGL|ES 2.0 is the mobile version of the well-known graphics API OpenGL. There are various implementations available for the different mobile platforms. Similarly, OpenVG 1.0 is a standard for vector graphics acceleration. We believe that other standards will be dedicated to the mobile architectures soon.
In computer graphics, we are primarily concerned with the surfaces of geometric objects since the surfaces are what we actually see, animate, and physically process. In recent years, multiresolution modeling has proved to be valuable in 3D geometric surface modeling and computer graphics. It deals with the generation, representation, visualization, and manipulation of surfaces at various levels of detail or accuracy in a single model. Applications include fast rendering, level of detail editing, collision detection, scientific visualization, as well as compression and progressive transmission. In the context of MCD, the requirement of multiresolution is even indispensable due the enormous differences of hardware capacities.
A widespread example for multiresolution surfaces is the subdivision surface. Starting from very simple primitives, such as cubes or spheres, the user can progressively deform and enrich the surface with tools like, for instance, extrusion, shear or twist. As soon as a coarse level is correctly modeled, the user can refine the model by applying a subdivision pass, and add finer details, tessellate the areas where more features are required, and thus model the shape more and more accurate by using the different levels of refinement of the subdivision surface. Indeed, a major advantage of the subdivision surface is flexibility: the user can modify the shape at any resolution - the deformations at coarser levels are automatically propagated to the finer levels.
Multiresolution is also the link between geometric modeling and rendering, providing for instance an appropriate level of detail for a given viewpoint in order to ensure real-time rendering. A simple example, introduced in the 90s, is the progressive polygonal mesh. Starting from a detailed single model, multiple coarser mesh resolutions can be generated by the successive application of an edge collapse operator. Of course, coarser mesh resolutions can be rendered more efficiently, at the expense of lost fine details. All these multiresolution techniques must now be adapted to acquired surface data, since due to the recent advances in 3D acquisition devices, the surfaces are more and more scanned from the real world rather than modeled. A challenging task is to handle the modeling and rendering of the large amount of data usually provided by 3D scanners in real-time. For example, the significant overhead of dealing with the connectivity of polygonal meshes has motivated various researchers to seek for alternative multiresolution surface representations, as for example point-based surfaces. In our Iparla project, multiresolution surface modeling is present at different scales, from surface reconstruction to visualization. We use and develop surface definitions that have gained much attention recently as suitable multiresolution representations, such as subdivision surfaces, implicit surfaces, and point-based surfaces. Our special interest focuses on how these surface representations are adapted for multiresolution modeling and rendering in the above-mentioned applications.
One of the main goals of the Iparla project is the interactive visualization of complex data on MCD. For example, a very rich and realist visualization stream including shadows and a complete set of light effects is required when a user has to feel parts of a virtual world. Realistic rendering is also required when it comes to augmented reality applications. Keeping the coherence between the virtual world and some virtual objects as well as between real objects and the virtual world is a challenging research domain. For the MCD, these technologies can be used for example for virtual visits, virtual presentations or, more generally, when the MCD is used as an interface to the real world.
On the other hand, in order to easily focus on what is really important to visualize, a more legible rendering is more appropriate. As a consequence, non-photorealistic rendering (NPR) techniques have recently become popular in the computer graphics community. We believe that these techniques are helpful for depiction. Based on a cognitive study, we choose the appropriate rendering style in order to provide the user with the desired information for a faster communication.
Despite the progress of MCDs, these client devices which are designed for mobility will always have less computing and storage capacity compared to the server. Consequently, we have to think about distributed approaches by re-investigating the entire pipeline, from storage, over transmission, to visualization. We have to define the correct representation for the data, for transmission and streaming. Moreover, we have to define how to visualize the data when received, both for realistic rendering and NPR rendering. We think that NPR rendering reduces the amount of information to transmit by focusing on what is really important.
MCDs represent a high range of devices, starting from small cell phones or old PDAs, to high-end tablet PCs. In order to achieve an interactive visualization for every device, we rely on the use of multiresolution representations as a unified representation for the data for adaptive rendering according to the specific resources. For devices with embedded graphics pipelines, we exploit the mutliresolution representation in order to provide the best possible performance. Since most of the MCDs do not provide this high-end power capacity, we need to investigate alternative rendering techniques, like image-based or point-based rendering techniques, or to simply use simple 2D approaches.
The Iparla project aims at improving the development of 3D interactive applications for mobile and connected devices. Consequently, as we have seen above, an essential part of this project consists in adapting the classical 3D graphics pipeline to the characteristics of the mobile devices. However, we think that the development of modeling and rendering techniques cannot go without the development of adapted user interfaces. Indeed, the interest of mobile applications where complex data can be visualized in real-time is limited when the interaction with the data is difficult.
We believe that human factors have to be taken into account in the early stage of development. Indeed, the choice of the user interface can influence the modeling and rendering techniques to use. For example, an object-oriented construction of the scene has to be preferred when the main user task of a given application consists in selecting individual objects. In the Iparla project, we want to control the entire pipeline, from the creation of the 3D environments to the interaction with these environments. Each of the components of this pipeline are often strongly linked and should not be considered independently.
When dealing with mobile devices, the classical user interfaces that have been developed for desktop workstations are not the most appropriate. For example, the lack of keyboards has led to the development of intuitive writing interfaces. The classical side-menus cannot be used for the control of the application without occluding a large amount of the screen and, consequently, without occluding a large part of the data to be visualized. Last but not least, the lack of pointing devices with cell-phones makes the manipulation of the data very difficult. In the Iparla project, we develop interaction techniques that are adapted to the user, to the task, and to the characteristics of mobile devices, for efficient interaction with 3D datasets.
Based on our experience in Virtual Reality (VR) settings, one of our principal goals is to improve the immersion of the users in order to increase their performance. With mobile devices, where the technological immersion is very limited, the efficiency of the interaction techniques is primordial for the success of the interactive applications. From previous work in the scope of VR and general Human Computer Interfaces (HCI), we investigate mobile HCI techniques. In particular, our work is based on the following foundations:
It has been shown that the use of both hands can be more efficient than the use of one single hand.
It is necessary to adapt the structure of the interface to the structure of the task. Consequently, interaction with 3D data generally requires more than 2-dof interfaces.
Non-intrusive and easy-to-learn interaction can be obtained from natural gesture recognition.
Modern mobile devices are equpped with embedded cameras. The video stream analysis can be used as input for the development of interaction techniques.
The interaction techniques are developed in concordance with the user and the task. They are evaluated via experiments. Hence, the user performance can be qualitatively and quantitatively measured, which indicates whether a new technique is more or less efficient than another one.
We think that is out of the scope of this report to establish an exhaustive list of application domains that could benefit from the interactive visualization of complex data on mobile devices. Consequently, we only present some key applications here.
Mobile and connected devices equipped with GPS are currently used as digital assistants for navigation. Such systems can help car drivers for route planning. They also can assist pedestrians or bike users when exploring cities, or when hiking in countryside. Existing solutions are mainly based on 2D or 2.5D visualization of data, which are generally stored on CD-ROMs or memory-sticks. Our project aims to provide 3D navigation tools where the data can be accessed from an up-to-date database stored on distant servers. Hence, for example, a hiker visualizes on its mobile device a 3D representation of the surrounding landscape that embeds information such as the way to follow, or the direction to the next mountain hut.
The majority of today's mobile devices is equipped with embedded cameras. Consequently, the use of these setups for augmented reality allows to imagine a wide variety of useful applications in our everyday life. For example, in the domain of cultural heritage, some extra information coming from distant servers can enhance the images coming from the cameras of the mobile devices. More precisely, for example the interest of merging synthetic reconstructions of partially destroyed buildings with the images of the real buildings can easily be understood. The same approach can be useful for many domains such as tourism, maintenance, emergency management (eg. firemen), and so on.
Mobile devices ensure mobility while keeping connected to networks. Consequently, these devices can be useful for visualization of distant information and control of remote applications. For example, a scientist could supervise a several day long experiment, without the need of physically being where the experiment takes place. He or she would be able to visualize the state of numerous sensors (e.g. temperature and pressure). The scientist would be able to modify the experiment's conditions by means of adapted user interfaces. This remote supervision can be helpful in many situations, such as for example the control of production machines in factories, the supervision of dangerous areas, or the real-time understanding of financial data.
Entertainment and especially video games are key applications directly related with our project as well. Some mobile devices have been designed for entertainment, and video games have been specifically developed for such setups. The results of our research in the scope of rendering or interaction can directly contribute to the development of the entertainment industry. Moreover, we are investigating new approaches for entertainment, in particular concerning the continuum between different platforms. For example, we can imagine a user to start a game at home with a PC/console, and to continue later the same game with MCD in public transportation.
GLUT|ESis a C library that allows the easy development of 3D portable applications on mobile devices (cf. ) without programming the operating system on a lower level. GLUT|ES is a port of the OpenSource freeglutimplementation of the world-wide used glutlibrary for WindowsMobile and Win32 systems based on OpenGL|ES. Most of GLUT API v.3 functionalities are present: window creation, callbacks, menus, timers etc. Only functions that cannot exist with OpenGL|ES or are not pertinent have not been ported (like overlays, joysticks on PocketPC, ...). GLUT|ES is freely distributed for non-commercial applications, and it is more and more used by the OpenGL ES developers community.
AutoDEMis a GIS software to create Digital Elevation Models (DEM) from color-scanned topographic maps. The purpose is to reconstruct as automatically as possible 3D landscapes and cities from paper maps. The software provides a specialized graphical user interface that lets the user manipulate different layers of data (map, contours or elevations) and offers a lot of tools and filters to work with them. Many file formats are supported, and furthermore the software can be extended easily through a plug-in mechanism.
The main purpose of the Magellan framework is to allow easy and fast development of new distributed solutions for the remote visualization of 3D models. The framework is coded using the C++ language and makes an intensive use of object-oriented development. Magellan is fully compatible with Linux, Windows, and WindowsMobile based systems. Moreover, it allows to implement adaptive algorithms to get the best performance on heterogeneous platforms.
Some members of Iparla have contributed to others softwares:
OpenSGis a portable scenegraph system to create realtime graphics programs, e.g. for virtual reality applications.
HyperFunis a free software development for modeling, visualization and animation of 3D shapes based on a so-called "Function Representation" (F-Rep).
Following a research result form the previous year (see below), a plug-in available for download at http://www.labri.fr/ boubek/cotex/of this new interactive constructive solid texturing approach has been developed for the PointShop3D system. http://graphics.ethz.ch/pointshop3d
Osiris provides various surface processing tools and visualization methods for discrete surfaces (point sets and polygons). It focuses on efficient local methods, suitable for large data sets. http://www.labri.fr/ boubek/osiris/
GenRef is a GPU framework for Generic Mesh Refinement methods in real-time applications . GenRef is a small module, easy to integrate in existing applications, and efficiently replaces usual on-CPU refinements required in displacement surfaces, Bezier patches, or subdivision surfaces.
The ST Mesh Tool is a simple 3D modeler for creating and deforming Scalar Tagged Meshes . It takes as input existing meshes, and allows to configure the various scalar tags for controlling the global shape at rendering time.
Recent 3D acquisition technologies provide a huge number of unorganized 3D points, and our goal is to develop new methods to reconstruct surfaces from such large unorganized point sets . One of the most commonly used surface representations are Point Set Surfaces (PSS) defined as the set of stationary points of a Moving Least Squares (MLS) projection operator. One interesting property of the MLS projection is to automatically filter out high frequency noise, that is usually present in raw data due to scanning errors. Unfortunately, the MLS projection also smoothes out any high frequency feature, such as creases or corners, that may be present in the scanned geometry, and does not offer any possibility to distinguish between such feature and noise.
In order to tackle this problem, we have developed an alternative projection operator for Point Set Surfaces based on the Enriched Reproducing Kernel Particle Approximation (ERKPA) , that allows to account for high frequency features, by letting the user explicitly tag the corresponding areas of the scanned geometry , . Furthermore, we have shown that the ERKPA can also be used to reconstruct implicit surfaces with sharp features. See for an example of the reconstruction of a screwdriver using ERKPA, where the sharp edge was tagged explicitly.
In this work , we propose a solution to adapt the differential point renderingtechnique to implicit surfaces. Differential point rendering was initially designed for parametric surfaces as a two-stage sampling process that strongly relies on an adjacency relationship for the samples which does not naturally exist for implicit surfaces. This fact made it particularly challenging to adapt the technique to implicit surfaces. To overcome this difficulty, we extended the particle samplingtechnique developed by Witkin and Heckbert in order to locally account for the principal directions of curvatures of the implicit surface. The final result of our process is a curvature driven anisotropic samplingwhere each sample "rules" a rectangular or elliptical surrounding domain and is locally oriented according to the directions of maximal and minimal curvatures. Like in the differential point rendering technique, these samples can then be efficiently rendered using a specific shader on a programmable GPU.
This sampling technique in described in the modeling section rather than the rendering one, because our final goal with anisotropic particles is not specifically to render the corresponding implicit surfaces, but to consider these particles as a set of control parameters that can be used to manipulate the surfaces, for instance, in a virtual sculpting environment.
We are currently investigating techniques to accelerate the generation of particles on an implicit surface. Particle systems are well suited to sample implicit surfaces but, due to the computational cost of the relaxation process, sampling a surface with more than 5,000 particles is time consuming. We propose a technique that allows to rapidly generate a set of particles (half a second to generate 5,000 particles) with near-optimal positions (for a particle system). Because of these characteristics, we only need around ten relaxations of the particle system to get a high-quality sampling. Our technique works both for uniform and non-uniform (anisotropic) sampling.
When it comes to the simplification of 3D geometry, either for streaming or for device-dependent quality rendering, multiresolution is a natural approach and has been widely used. Unfortunately, most of these solutions are geometry-specific and cannot be directly applied to the appearance since it spans in a 4D directional space.
We are currently developing new representations for the light interaction with a 3D model that allow a better preservation of the object's appearance via a multiresolution simplification. With this model, we are able to increase the richness of the visualization, even for displays with a limited resolution. Note that the lighting variation provides information about the underlying simplified geometry.
Paper maps (topographic maps, city maps, etc.) are the classical supports for geographic information. Nowadays maps are created by geographical information systems, but in many cases the original information that was used to create the map are not - or no longer - available. This is especially the case for maps designed before the emergence of our information society. However, for various purposes and historical studies in general, a numerical exploitation of information available on these maps would be very useful.
The classical method to extract features from paper maps is the manual acquisition using a digitizing tablet. This represents an accurate but tedious and time-consuming work. There is a large amount of research work that aims to automate this task.
In previous work, we have created a workflow and techniques to extract topographic information in order to reconstruct a digital elevation model. More recently, we have proposed an algorithm to extract and recognize textual information in maps (toponyms, elevation values, etc.). Our approach is based on potential characters extraction based on different criteria. Near characters are then grouped in potential strings before being automatically recognized by an optical character recognition (OCR) system. Different filterings are operated at each step of the process in order to remove outliers. Using our technique, we reach more than 90% of well recognized items. We then exploit extracted strings as floating 3D labels (eg. city names) in virtual environments.
In the same research topic, we also developed a technique to extract building footprints in city maps in order to reconstruct 3D cities in an automatic way.
Note that all the work lead in this area is implemented using our GIS framework software called AutoDEM.
Accurately capturing the near field emission of complex luminaires is still very difficult. We have developed a new acquisition pipeline of such luminaires that performs in a two-step procedure an orthogonal projection onto a given basis. In a first step, we use an optical low-pass filter that corresponds to the reconstruction filter and guarantees high precision measurements. The second step is a numerical process on the acquired data that finalizes the projection. Based on this concept, we have developed new experimental setups for automatic acquisition and performed a detailed error analysis of the acquisition process. This work has been submitted to the Journal of Applied Optics, and is in the second pass of the reviewing process.
Point-based surfaces (i.e. surfaces represented by discrete point sets which are either directly obtained by current 3D acquisition devices or converted from other surface representations) are well designed for multiresolution, storage and transmission of complex objects. This surface definition is enough generic to be used as a core representation of 3D objects, that can be used for different hardware setups, ranging from mobile devices (PDA, cell phone, Sony PSP) to high-end workstations. Unfortunately, the visualization of point-based surfaces requires to develop specific rendering techniques (e.g. surface splatting) since the point sets are not well adapted to existing graphics hardware (that is rather optimized for polygonal meshes). In this project, we propose a fast local reconstruction technique to provide an efficient visualization of point-based surfaces (see Figure ) that takes full benefit from the whole optimized pipeline implemented in graphics hardware.
This project allowed us to develop a new adaptive multiresolution data structure for point-based surfaces . Instead of using surface reconstruction algortihms and thus loosing the advantage of point-based modeling, we propose a mixed point-strip rendering that can be seen as a "layer" between point-based surfaces and polygonal renderers. The strips are efficiently generated using an improved adjacency data structure . This structure can be efficiently maintained during point-based deformation since updates are completely localized. The rendering takes full benefit of existing polygonal techniques such as shadows, reflections, NPR, etc., and provides higher frame rates and screen resolutions than classical point-based rendering techniques.
The next challenge in this area is clearly the growing size of the models to deal with. On one hand, modern applications can deal with hundred millions of samples. On the other hand, real-time rendering strongly relies on the optimized graphics pipeline, where techniques such as normal mappingare more amenable to high frame rates than pure geometry rendering. We propose an efficient out-of-core conversion of point clouds to low-resolution meshes and high resolution normal maps, that avoids any kind of surface reconstruction or parameterization at the full resolution of the large input point cloud . The resulting appearance preservingsimplification (see Figure ) provides an efficient realtime rendering on dedicated graphics hardware, such as GPUs present in modern mobile devices (PDA, cell phones, Sony PSP) or workstations.
A 3D scene can be expressed with objects at low resolution at CPU level, where only a coarse 3D representation is necessary for animation, collision, deformation, visibility, etc. But today's state-of-the-art rendering imposes 3D surfaces at high resolution, even for real-time rendering.
We propose a low-cost surface definition, the scalar tagged meshes, that are designed to efficiently perform an on-the-fly surface refinement . This allows to produce high quality visually smoothsurfaces in real-time.
In order to propose any kind of local refinement for real-time applications, such as procedural displacements, Bezier patches or subdivision surfaces, we have developed a Generic GPU refinement method, which reduces the CPU-GPU bottleneck . This approach is also suitable for network optimization of object transmission : the transmission of 3D objects toward handheld devices is reduced to the transmission of coarse ST meshes, with a realtime refinement performed on the mobile device.
The combination of scalar tagged meshes with the GPU refinement pipeline makes possible the real-time rendering of refined meshes without neither storing nor transmitting meshed at full resolution (see Figure ).
Terrain rendering is an important factor in the rendering of virtual scenes. If they are large and detailed enough, the geometry and textures of digital terrains can represent a huge amount of data and thus a huge amount of graphical primitives to render in real-time. Common approaches are based on runtime creation and modification of mesh structures which can be expensive, especially on handheld devices. In , we proposed an efficient technique for the streaming and rendering of pseudo-infinite terrains in an adaptive way. The terrain height field is divided into regular tiles which are managed adaptively, and each tile is also rendered in an adaptive way according to an importance metric. Our approach has been thought as a general technique to render terrains on any kind of devices - from slow handheld device (see Figure ) to recent desktop PCs by exploiting the device capacity to draw as much triangles as possible.
We are particularly interested in the navigation through large virtual urban environments. Usually, 3D urban models contain a set of buildings that are simple blocks on which some realistic textures are mapped. When rendering these environments on a PDA, the cost for texture storing and transmission can be prohibitive. We propose to use a Non-Photorealistic-Rendering (NPR) style based on feature lines for the modeling and rendering of urban environments. According to the literature on feature line rendering, the lines are extracted on a 3D model guided by the geometric properties (the principal directions of curvature as well as the silhouette) or on a final image (using the gradient and segmentation). We have chosen to work in image space because of the richness of information we can extract. We have implemented and integrated a pipeline of image processing by using the Gimp software. The input is the image of a building facade where feature lines are created thanks to the vectorization of contour lines. The initial line set is dense and noisy. Considering some knowledge on the facade structure (majority of horizontal / vertical lines), we post-process this set in order to obtain a minimal set of lines that can be transmitted more rapidly than textures via a network. We are working on a hierarchical classification of lines in order to transmit them progressively on the network . For the final rendering, our future work is to adapt stylization techniques (pen and ink) for a more aesthetic (but still interactive) rendering on the PDA. We will also explore mixed rendering (water-color) for a legible rendering.
Line drawings, where strokes outline the objects' features, allow an efficient and intuitive depiction of objects. Therefore, recent NPR research tools make it possible to extract feature lines from 3D objects (silhouettes, boundaries, ridges, valleys, suggestive contours, etc.). Since the extraction processes rely on geometric computations, the resulting lines may not be coherent with what we actually perceive from objects' shapes. We have presented a perception-based evaluation criterion that allows an estimation of the relevance of feature lines . We have shown the use of such a measure to increase the quality of line-based renderings by providing a filtering scheme that is intuitive and does not depend on the type of the computed lines. Furthermore, we propose to use perception-based image-processing tools in order to extract lines of optimal perceptual relevance. To this end, we are currently investigating biologically-inspired computer vision techniques and applications. Once we are able to extract lines from static images, we are going to study how we can animate the resulting lines to produce non-photorealistic line-based animations.
A classical rendering pipeline is not always well suited to mobile terminals due to their limited performance. Sometimes, the better alternative is to use an image-based rendering (IBR) approach. In order to render highly complex scenes on MCD, we have developed an IBR algorithm in the framework of a client/server architecture. The server computes some reference images and transmits them to the client that can be a PDA or a mobile phone, for example. On the client side, the navigation through a scene is performed by warping a reduced set of reference images. In this way, the rendering complexity is only dependent on the image resolution and not on the geometric complexity. A first implementation proved the feasibility of the IBR approach on MCDs. This year, we address the problem of camera placement that allows an efficient warping by avoiding artifacts, such as holes, due to occlusions and exposures. Providing a general solution to this problem is a hard task; and in we present results in the case of urban scenes.
While many solutions for realistic illumination have been developed during the last twenty years, few of them are easy to port on heterogeneous platforms. Furthermore, few of them allow an intuitive control of the result. To improve these solutions, we are looking for new representations of the different phenomena involved in global illumination. Our new representations should be compact, compatible with the existing rendering architecture, and easy to handle for the user.
We are currently studying the following issues :
Representation of the illumination values. This representation should be compact, multiresolution and easy to edit. With such a representation, the illumination computation could be done on a server, and the result sent to the clients (e.g PDAs). Moreover, it allows the user to improve qualitatively the solution. A first representation has been proposed in . It is compact, robust to BRDF and geometry variations (see Figure ), and we are currently improving it to become multiresolution.
Shadow representation. The goal of a new representation for the triplet (light, occluder, receiver) is to facilitate the introduction of new objectsin a scene that has been already shaded (as done in augmented reality applications). This representation should handle how the light stream is modified when a new object is introduced. We emphasize on the possibility for the user to create arbitrary shadows.
Caustic representation. In the continuity of the preceding work we would like to adapt the representation of shadows to handle the caustic phenomenona.
Finally we will study the reflectance model (BRDF) representations. We will favor the availability for the user to define the BRDF behaviour. This representation will follow the user constraints, and therefore it can also be used to render physical phenomena by choosing the constrains appropriately, i.e. by taking the constraints of the reality.
The stylus is the principal input device for interaction with PDAs. Consequently, a stylus-based sketching interface for modeling seems promising. Since it is difficult to perform precise 3D rotations with a stylus, we developed a shape from shading approach for 3D modeling . This approach is based on the ability of a user to coarsely draw a shading, under different lighting directions. With this intuitive process, users can create or edit a height-field (locally or globally), that will correspond to the drawn shading values.
Moreover, we propose a new sketching tool that allow to reconstruct a model with respect to defined profile curves. Starting from only two different sketches (one for the silhouette of the model and one for the profile curve), users can design a large variety of shapes. There is no limitation concerning the form of the profile curve since it doesn't self-intersect. As the reconstruction of the model is driven by this profile curve, models are not necessarily topologically equivalent to a sphere, but we can instead obtain models with holes or straight lines (see Figure ).
Recent advances in mobile computing allow the users to deal with 3D interactive graphics on handheld computers. Although the computing resources and screen resolutions grow steadily, user interfaces for handheld computers do not change significantly. Consequently, we designed a new 3 degrees of freedom interface adapted to the characteristics of handheld computers . This interface, illustrated in Figure , tracks the movement of a target that the user holds behind the screen by analyzing the video stream of the handheld computer camera. Consequently, the users benefit from bi-manual interaction. The position of the target is directly inferred from the color-codes that are printed on it by using an efficient algorithm. The users can easily interact in real-time in a mobile setting. This technique is appropriate for visualization on handhelds since the target does not occlude the screen, and the interaction techniques are not dependent on the orientation of the handheld computer. We used the interface in several test applications for the visualization of large images such as the navigation through planar maps, the manipulation of 3D models, and the navigation in 3D scenes. This new interface favors the development of 2D and 3D interactive applications on handheld computers. An experiment allowed us to demonstrate that our interface was faster than a classical stylus interface for a 2D searching task. Moreover, the user preferences were in favor of our new bi-manual interface .
Martin Hachet spent one month in the Human Interface Engineering Lab(University of Osaka), where he has been invited by Professor Kitamura. During his stay, they developed 3D interfaces for collaborative and immersive interaction. The first interface aims at enhancing the interaction with the Illusionhole, a collaborative visualization interface. By rotating a circular interface (see Figure ), the users are able to easily manipulate virtual objects, as if they were rotating real ones. The second interface consists in using mobile devices as remote controls for interaction with 3D environments displayed on large screens . In such settings, numerous users can interact from their own device. We use the new camera-based bi-manual interface described above for the interaction from the mobile devices.
We extract the hand or body posture of an unequipped user from a live video flow in order to perform interaction in a large display setting. This induced a real-time analysis of monochrome input images. According to a predefined interaction technique, this data is interpreted to achieve 3D selection, manipulation or navigation. While most of such systems rely on appearance-based approaches, we have chosen to investigate how far a model-based one could be efficient. The real-time algorithm we are developing modifies a 3D hand or body model pose so that its projection matches the input silhouette. Great tracking speeds were reached, while using several cameras helps to disambiguate the model pose extraction . Further improvements include the integration of a tight collaboration between image-based and model-based approaches.
The main objective of this work is to understand the differences of human performance during travel and manipulation tasks in Virtual Reality setups. The better understanding of how subjects use virtual reality interfaces can help us to design interfaces that are more adapted to the cognitive processes of the user. We set up a double experimentation that we have performed on one hundred subjects. For each subject, we measured a set of cognitive factors such as attention and motor skills as well as the subject's performances in two virtual reality tasks (manipulation and travel) by using two different screens (large and small) and the CAT. After the analysis of the statistical data, the major result concerns the screen size influence . Subjects with lower spatial abilities improve their performance with larger screens whereas the others do not show significant changes depending on the distinct screens. With some small adaptations, the experiment can be performed for an analysis of interactions on small devices.
Title:The CAT.
Dates:From 2004
Overview:
The CAT is a 6 degrees of freedom input device that has been developed to favor interaction with virtual environments displayed on large screens (see the 2004 activity report). The success of this new interface lead us to work with a company, Immersion, for the commercialization of the product. The CAT has jointly been presented by INRIA and Immersion at SIGGRAPH 2005. This collaboration comes from a support by EITICA, a technology transfer organism of the Aquitaine Region.
Title:Modeling and rendering of non-photorealistic urban environments and digital territories.
Dates:2004-2005, 2005-2006
Overview:This project concerns the modeling and rendering of non-photorealistic urban environments and digital territories. The target platforms for visualization are small devices. We propose an entire graphics pipeline (data modeling, transmission and rendering) that is adapted to the constraints imposed by the small devices. We will implement a client-server application where the server has a huge 3D database which is rendered on the client (a small device). As an alternative to photorealistic rendering and texture based rendering which generate large models, we will explore non-photorealistic techniques, such as line-based rendering. Point-based rendering will also be considered. Some multiresolution models of cities or territories (efficiently transmitted and rendered) will be constructed with those techniques.
Title:Multi-scale techniques for scalable coding of 3D representations.
Dates:2005-2008
Overview:The goal of this project is to develop new techniques for progressive representations of point-based modeling and rendering.
"Lab for Interactive Graphics on Handheld and Tabletop displays"
Grant:INRIA-DREI
Dates:2004-2007
Partners:IMAGER Lab- University of British Columbia - Vancouver - Canada
Grant:INRIA ARC
Dates:2005-2006
Partners:INRIA (IPARLA, ARTIS) + ISTI-Pise, AUSONIA-Bordeaux, LSC-Bordeaux, DIGISENS-Annecy
Overview:MIRO stands for "Methods for the legible and Interactive Rendering Of Complex data". In this project, we are interested in using NPR for the interactive and legible rendering of complex 3D scenes. Many examples of scientific visualization prove that photorealism does not always offer meaningful images. To address the legibility question we will rely (i) on the skills and experience of the IPARLA and ARTIS teams; (ii) on the knowledge of specific users: archeologists, museum curators, industrial users at Digisens (the startup specialized in CTscan reconstruction and dental chirurgy); (iii) on the skills of the Bordeaux lab of cognitive psychology that will help us how to validate our new methods.
Grant:French Ministery for Foreign Affairs, CNRS, INRIA
Dates:2004-2005
Partners:France, China, Corea, Japan, Singapore, Taiwan
Grant:ACI "Masse de données" (French Ministry of Research)
Dates:12/2005 - 12/2008
Partners:IRIT (Toulouse 2), EVASION (Inria Rhones-Alpes), AMAP (UMR Montpellier), LIAMA (Beijing)
Overview:This project deals with natural simulations (vegetal, watercourses, clouds). It aims to adapt this huge amount of heterogenous data in terms of data structures, techniques and algorithms, in a unified framework to both to the content and navigation context (from mobile phones to display walls).
Grant:ACI "Masse de données" (French Ministry of Research)
Dates:2003-2005
Partners:Grenoble, Nancy, Sophia, Iparla
Overview:The goal of this collaboration is to develop a software architecture for the 3D visualization of very large datasets (more than hundred millions of polygons or points). Iparla has in charge the client-server architecture for the data streaming, and also the point-based rendering.
Grant:GRID 5000 (French Ministry of Research and Coneil Régional d'Aquitaine
Dates:2004-2006
Partners:RunTime (INRIA Futurs), Scalapplix (INRIA Futurs), SOD (LaBRI), Iparla
Overview:This project aims at developing a large PC grid on French territory. We are focusing on the development of a specialized PC cluster for the visualization of large and complex data.
Grant:Conseil Régional d'Aquitaine)
Dates:2005-2007
Partners:Sound analysis and synthesis group (LaBRI)
Overview:We collaborate with the sound analysis and synthesis group of our lab for the development of interfaces aiming at enhancing interaction with sounds in virtual reality setups.
In 2005, some members of Iparla have been involved in conference programme committees and journal editorial boards:
Shape Modeling International 2005: programme committee
Graphics Interface 2005: programme committee
International Conference on Entertainment Computing 2005: programme committee
Virtual concept 2005: programme committee
Journal Computer Graphics Forum (Eurographics): editorial board
"Le traité de la réalité virtuelle" (3rd edition): editorial board
They have also participated to the reviewing process for conferences (SIGGRAPH 2005, Eurographics 2005, Graphics Interface 2005, UIST 2005, I3D 2005, IEEE VR 2006, IEEE Visualization 2005, IHM 2005) and journals (The Visual Computer, ACM Transaction on Graphics, Computer Graphics Forum 2005, Parallel Computing 2005).
AFIG 2006 will be held in Bordeaux and organized by the Iparla project.
In 2005, the members of Iparla have been involved in the following committees:
scientific committee of INRIA Futurs
correspondent for the "formation par la recherche" at the INRIA Futurs.
scientific committee of Visitor project (Grenoble, Marie Curie action)
scientific committee for GDR Algorithmique, Langages et Programmation (CNRS).
administrating committee of the AFIG.
The expertise of some members has been required for
Evaluation committee of LIRIS (Lyon)
Comité d'évaluation de la Fédération de Recherche "Physique et image de la ville (Nantes, CNRS, Ecole Centrale, Ecole des Mines, Ecole d'Architecture, LCPC, CSTB)
ACI "Masses de données"
The members of our team are implied in teaching computer science in the scope of general computer science, image synthesis and virtual reality. The members of Iparla teach at University Bordeaux 1, University Bordeaux 2, University Bordeaux 3, and ENSEIRB.
The project members have participated to a number of international workshop and conferences (cf bibliography).
They also have been invited to several seminars and conferences:
Human interfaces engineering lab and Cybermedia center, Osaka (Japan) [Martin Hachet]
"La simulation au service de l'innovation", Biarritz (France) [Martin Hachet]
Multiresolution in Geometric Modeling, Virtual Concept 2005 (Biarritz) [Patrick Reuter and Tamy Boubekeur]