Keywords
Computer Science and Digital Science
- A5.3. Image processing and analysis
- A5.3.2. Sparse modeling and image representation
- A5.3.3. Pattern recognition
- A5.5.1. Geometrical modeling
- A8.3. Geometry, Topology
- A8.12. Optimal transport
- A9.2. Machine learning
Other Research Topics and Application Domains
- B2.5. Handicap and personal assistances
- B3.3. Geosciences
- B5.1. Factory of the future
- B5.6. Robotic systems
- B5.7. 3D printing
- B8.3. Urbanism and urban planning
1 Team members, visitors, external collaborators
Research Scientists
- Pierre Alliez [Team leader, Inria, Senior Researcher, HDR]
- Florent Lafarge [Inria, Researcher, HDR]
Post-Doctoral Fellows
- Shenlu Jiang [Inria, from Oct 2020]
- Johann Lussange [Inria, from Sep 2020]
- Xiao Xiao [Inria, from Nov 2020]
PhD Students
- Gaetan Bahl [IRT Saint Exupéry]
- Nicolas Girard [Inria, until Nov 2020]
- Muxingzi Li [Inria]
- Lionel Matteo [UMR Géoazur, until Aug 2020]
- Onur Tasar [Inria, until Sep 2020]
- Vincent Vadez [Dorea Technology]
- Julien Vuillamy [Dassault Systèmes]
- Mulin Yu [Inria]
- Tong Zhao [Inria]
Technical Staff
- Fabrizio Corda [Inria, Engineer, from Apr 2020 until Jul 2020]
- Lucas Dubouchet [Inria, Engineer]
- Fernando Ireta Munoz [Inria, Engineer]
- Matthieu Lecce [Inria, Engineer, from Apr 2020 until Jul 2020]
- Cédric Portaneri [Inria, Engineer]
Interns and Apprentices
- Ilona Charipov [Inria, from Jun 2020 until Aug 2020]
- Paul Vinh Le [Inria, from Sep 2020]
Administrative Assistant
- Florence Barbara [Inria]
Visiting Scientist
- Benoit Morisset [Ai Verse, from Apr 2020]
External Collaborator
- Yuliya Tarabalka [Luxcarta, HDR]
2 Overall objectives
2.1 General Presentation
Our overall objective is the computerized geometric modeling of complex scenes from physical measurements. On the geometric modeling and processing pipeline, this objective corresponds to steps required for conversion from physical to effective digital representations: analysis, reconstruction and approximation. Another longer term objective is the synthesis of complex scenes. This objective is related to analysis as we assume that the main sources of data are measurements, and synthesis is assumed to be carried out from samples.
The related scientific challenges include i) being resilient to defect-laden data due to the uncertainty in the measurement processes and imperfect algorithms along the pipeline, ii) being resilient to heterogeneous data, both in type and in scale, iii) dealing with massive data, and iv) recovering or preserving the structure of complex scenes. We define the quality of a computerized representation by its i) geometric accuracy, or faithfulness to the physical scene, ii) complexity, iii) structure accuracy and control, and iv) amenability to effective processing and high level scene understanding.
3 Research program
3.1 Context
Geometric modeling and processing revolve around three main end goals: a computerized shape representation that can be visualized (creating a realistic or artistic depiction), simulated (anticipating the real) or realized (manufacturing a conceptual or engineering design). Aside from the mere editing of geometry, central research themes in geometric modeling involve conversions between physical (real), discrete (digital), and mathematical (abstract) representations. Going from physical to digital is referred to as shape acquisition and reconstruction; going from mathematical to discrete is referred to as shape approximation and mesh generation; going from discrete to physical is referred to as shape rationalization.
Geometric modeling has become an indispensable component for computational and reverse engineering. Simulations are now routinely performed on complex shapes issued not only from computer-aided design but also from an increasing amount of available measurements. The scale of acquired data is quickly growing: we no longer deal exclusively with individual shapes, but with entire scenes, possibly at the scale of entire cities, with many objects defined as structured shapes. We are witnessing a rapid evolution of the acquisition paradigms with an increasing variety of sensors and the development of community data, as well as disseminated data.
In recent years, the evolution of acquisition technologies and methods has translated in an increasing overlap of algorithms and data in the computer vision, image processing, and computer graphics communities. Beyond the rapid increase of resolution through technological advances of sensors and methods for mosaicing images, the line between laser scan data and photos is getting thinner. Combining, e.g., laser scanners with panoramic cameras leads to massive 3D point sets with color attributes. In addition, it is now possible to generate dense point sets not just from laser scanners but also from photogrammetry techniques when using a well-designed acquisition protocol. Depth cameras are getting increasingly common, and beyond retrieving depth information we can enrich the main acquisition systems with additional hardware to measure geometric information about the sensor and improve data registration: e.g., accelerometers or gps for geographic location, and compasses or gyrometers for orientation. Finally, complex scenes can be observed at different scales ranging from satellite to pedestrian through aerial levels.
These evolutions allow practitioners to measure urban scenes at resolutions that were until now possible only at the scale of individual shapes. The related scientific challenge is however more than just dealing with massive data sets coming from increase of resolution, as complex scenes are composed of multiple objects with structural relationships. The latter relate i) to the way the individual shapes are grouped to form objects, object classes or hierarchies, ii) to geometry when dealing with similarity, regularity, parallelism or symmetry, and iii) to domain-specific semantic considerations. Beyond reconstruction and approximation, consolidation and synthesis of complex scenes require rich structural relationships.
The problems arising from these evolutions suggest that the strengths of geometry and images may be combined in the form of new methodological solutions such as photo-consistent reconstruction. In addition, the process of measuring the geometry of sensors (through gyrometers and accelerometers) often requires both geometry process and image analysis for improved accuracy and robustness. Modeling urban scenes from measurements illustrates this growing synergy, and it has become a central concern for a variety of applications ranging from urban planning to simulation through rendering and special effects.
3.2 Analysis
Complex scenes are usually composed of a large number of objects which may significantly differ in terms of complexity, diversity, and density. These objects must be identified and their structural relationships must be recovered in order to model the scenes with improved robustness, low complexity, variable levels of details and ultimately, semantization (automated process of increasing degree of semantic content).
Object classification is an ill-posed task in which the objects composing a scene are detected and recognized with respect to predefined classes, the objective going beyond scene segmentation. The high variability in each class may explain the success of the stochastic approach which is able to model widely variable classes. As it requires a priori knowledge this process is often domain-specific such as for urban scenes where we wish to distinguish between instances as ground, vegetation and buildings. Additional challenges arise when each class must be refined, such as roof super-structures for urban reconstruction.
Structure extraction consists in recovering structural relationships between objects or parts of object. The structure may be related to adjacencies between objects, hierarchical decomposition, singularities or canonical geometric relationships. It is crucial for effective geometric modeling through levels of details or hierarchical multiresolution modeling. Ideally we wish to learn the structural rules that govern the physical scene manufacturing. Understanding the main canonical geometric relationships between object parts involves detecting regular structures and equivalences under certain transformations such as parallelism, orthogonality and symmetry. Identifying structural and geometric repetitions or symmetries is relevant for dealing with missing data during data consolidation.
Data consolidation is a problem of growing interest for practitioners, with the increase of heterogeneous and defect-laden data. To be exploitable, such defect-laden data must be consolidated by improving the data sampling quality and by reinforcing the geometrical and structural relations sub-tending the observed scenes. Enforcing canonical geometric relationships such as local coplanarity or orthogonality is relevant for registration of heterogeneous or redundant data, as well as for improving the robustness of the reconstruction process.
3.3 Approximation
Our objective is to explore the approximation of complex shapes and scenes with surface and volume meshes, as well as on surface and domain tiling. A general way to state the shape approximation problem is to say that we search for the shape discretization (possibly with several levels of detail) that realizes the best complexity / distortion trade-off. Such a problem statement requires defining a discretization model, an error metric to measure distortion as well as a way to measure complexity. The latter is most commonly expressed in number of polygon primitives, but other measures closer to information theory lead to measurements such as number of bits or minimum description length.
For surface meshes we intend to conceive methods which provide control and guarantees both over the global approximation error and over the validity of the embedding. In addition, we seek for resilience to heterogeneous data, and robustness to noise and outliers. This would allow repairing and simplifying triangle soups with cracks, self-intersections and gaps. Another exploratory objective is to deal generically with different error metrics such as the symmetric Hausdorff distance, or a Sobolev norm which mixes errors in geometry and normals.
For surface and domain tiling the term meshing is substituted for tiling to stress the fact that tiles may be not just simple elements, but can model complex smooth shapes such as bilinear quadrangles. Quadrangle surface tiling is central for the so-called resurfacing problem in reverse engineering: the goal is to tile an input raw surface geometry such that the union of the tiles approximates the input well and such that each tile matches certain properties related to its shape or its size. In addition, we may require parameterization domains with a simple structure. Our goal is to devise surface tiling algorithms that are both reliable and resilient to defect-laden inputs, effective from the shape approximation point of view, and with flexible control upon the structure of the tiling.
3.4 Reconstruction
Assuming a geometric dataset made out of points or slices, the process of shape reconstruction amounts to recovering a surface or a solid that matches these samples. This problem is inherently ill-posed as infinitely-many shapes may fit the data. One must thus regularize the problem and add priors such as simplicity or smoothness of the inferred shape.
The concept of geometric simplicity has led to a number of interpolating techniques commonly based upon the Delaunay triangulation. The concept of smoothness has led to a number of approximating techniques that commonly compute an implicit function such that one of its isosurfaces approximates the inferred surface. Reconstruction algorithms can also use an explicit set of prior shapes for inference by assuming that the observed data can be described by these predefined prior shapes. One key lesson learned in the shape problem is that there is probably not a single solution which can solve all cases, each of them coming with its own distinctive features. In addition, some data sets such as point sets acquired on urban scenes are very domain-specific and require a dedicated line of research.
In recent years the smooth, closed case (i.e., shapes without sharp features nor boundaries) has received considerable attention. However, the state-of-the-art methods have several shortcomings: in addition to being in general not robust to outliers and not sufficiently robust to noise, they often require additional attributes as input, such as lines of sight or oriented normals. We wish to devise shape reconstruction methods which are both geometrically and topologically accurate without requiring additional attributes, while exhibiting resilience to defect-laden inputs. Resilience formally translates into stability with respect to noise and outliers. Correctness of the reconstruction translates into convergence in geometry and (stable parts of) topology of the reconstruction with respect to the inferred shape known through measurements.
Moving from the smooth, closed case to the piecewise smooth case (possibly with boundaries) is considerably harder as the ill-posedness of the problem applies to each sub-feature of the inferred shape. Further, very few approaches tackle the combined issue of robustness (to sampling defects, noise and outliers) and feature reconstruction.
4 Application domains
In addition to tackling enduring scientific challenges, our research on geometric modeling and processingis motivated by applications to computational engineering, reverse engineering, digital mapping and urbanplanning. The main deliverable of our research will be algorithms with theoretical foundations. Ultimately, we wish to contribute making geometry modeling and processing routine for practitioners who deal with real-world data. Our contributions may also be used as a sound basis for future software and technology developments.
Our first ambition for technology transfer is to consolidate the components of our research experiments in the form of new software components for the CGAL (Computational Geometry Algorithms Library). Consolidation being best achieved with the help of an engineer, we will search for additional funding. Through CGAL, we wish to contribute to the “standard geometric toolbox”, so as to provide a generic answer to application needs instead of fragmenting our contributions. We already cooperate with the Inria spin-offcompany Geometry Factory, which commercializes CGAL, maintains it and provide technical support.
Our second ambition is to increase the research momentum of companies through advising Cifre Ph.D. theses and postdoctoral fellows on topics that match our research program.
5 Social and environmental responsibility
5.1 Impact of research results
We are collaborating with Ekinnox, an Inria spin-off which develops human movement analysis solutions for healthcare institutions. We provide scientific advise on automated analysis of human walking, in collaboration with Laurent Busé from the Aromath project-team.
6 Highlights of the year
Since April 2020 Titane is hosting the AI Verse startup project led by Benoit Morisset, focused on the generation of 3D training data for deep learning.
6.1 Awards
- Oussama Ennafii received the EuroSDR PhD Award 2020.
- Onur Tasar and Pierre Alliez received a best paper award at the EarthVision Workshop organized during the CVPR conference (Conference on Computer Vision and Pattern Recognition).
7 New software and platforms
7.1 New software
7.1.1 Module CGAL : Point Set Processing
- Keyword: Geometry Processing
- Functional Description: This CGAL component implements methods to analyze and process unorganized point sets. The input is an unorganized point set, possibly with normal attributes (unoriented or oriented). The point set can be analyzed to measure its average spacing, and processed through functions devoted to the simplification, outlier removal, smoothing, normal estimation, normal orientation and feature edges estimation.
-
URL:
http://
doc. cgal. org/ latest/ Point_set_processing_3/ index. html#Chapter_Point_Set_Processing - Contact: Pierre Alliez
- Participants: Clément Jamin, Laurent Saboret, Pierre Alliez
7.1.2 DPP
- Name: Delaunay Point Process for image analysis
- Keywords: Computer vision, Shape recognition, Delaunay triangulation, Stochastic process
- Functional Description: The software extract 2D geometric structures (planar graphs, polygons...) from images
- Publication: hal-01950791
- Contact: Florent Lafarge
- Participants: Jean-Dominique Favreau, Florent Lafarge, Adrien Bousseau
7.1.3 KIPPI
- Name: KInetic Polygonal Partitioning of Images
- Keywords: Computer vision, Computational geometry, Image segmentation
- Scientific Description: The scientific description of the algorithm is detailed in [Bauchet and Lafarge, KIPPI: KInetic Polygonal Partitioning of Images, CVPR 2018]
- Functional Description: KIPPI decompose an image, or a bounded 2D space, into convex polygons. The method exploits a kinetic framework for propagating and colliding line-segments until forming convex polygons.
- Contact: Florent Lafarge
- Participants: Jean-Philippe Bauchet, Florent Lafarge
7.1.4 Module CGAL: 3D Point-Set Shape Detection
- Keyword: CGAL
- Functional Description: This package implements the efficient RANSAC method for shape detection, contributed by Schnabel et al. From an unstructured point set with unoriented normals, the algorithm detects a set of shapes. Five types of primitive shapes are provided by this package: plane, sphere, cylinder, cone and torus. Detecting other types of shapes is possible by implementing a class derived from a base shape.
- Contact: Pierre Alliez
- Participants: Clément Jamin, Pierre Alliez, Sven Oesau
7.1.5 CGAL module: Classification
- Keywords: Classification, Point cloud, Mesh
- Functional Description: This CGAL module aims at classifying 3D data, typically point clouds, into arbitrary classes of interest. The module offers the user the possibility to segment data i) locally or globally, and ii) in an supervised or unsupervised way.
- Authors: Florent Lafarge, Simon Giraudot
- Contact: Florent Lafarge
7.1.6 Stochastic Vectorization
- Keywords: Vector graphics, Stochastic models
- Functional Description: The software converts a line-drawing image into Bezier curves.
- Publication: hal-01309271
- Contact: Florent Lafarge
- Participants: Jean-Dominique Favreau, Florent Lafarge, Adrien Bousseau
7.1.7 Geometry Sanitizer
- Name: Watertight reconstruction from raw triangle soups
- Keyword: Geometry Processing
-
Scientific Description:
In the context of motion planning and collision detection, the objects in the scene as well as the robotic arms must initially have a consistent boundary representation as a solid mesh before even thinking of any further collision detection algorithms. Without it, any standard geometry processing tasks would be either a gamble or are significantly slowed down.
Indeed, today's digitized meshes are often very detailed and imprecise. Precisely, we see triangular meshes that even model the laser imprint of the manufacturer, while falling short on providing a valid boundary representation of the physical real world objects they are supposed to model. The most common flaws are open boundary edges of triangular patches that do not match their corresponding counterparts, which results in gaps or self intersections.
Therefore, we want to “solidify” a triangle soup to get a consistent boundary representation. That is, we want a conservative outer approximation of the input which is a tight and solid mesh for later geometry processing tasks.
-
Functional Description:
Watertight reconstruction from raw triangle soups
This codes produces an outer solid representation from a triangle soup by computing its alpha hull. First apply a dense sampling on the input and insert sampled points in a delaunay triangulation. Then performs a flood-fill on the delaunay triangulation with an alpha criteria to tag cells as interior/exterior. Cases where input holes are larger than alpha are considered alpha-invalid and discarded.
- Contact: Pierre Alliez
- Participant: Pierre Alliez
- Partner: Google
8 New results
8.1 Analysis
8.1.1 ColorMapGAN: Unsupervised Domain Adaptation for Semantic Segmentation Using Color Mapping Generative Adversarial Networks
Participants: Onur Tasar, Pierre Alliez.
In collaboration with Yuliya Tarabalka (Luxcarta) and S L Happy (Stars Inria project-team).
Due to various reasons such as atmospheric effects and differences in acquisition, it is often the case that there exists a large difference between spectral bands of satellite images collected from different geographic locations. The large shift between spectral distributions of training and test data causes the current state of the art supervised learning approaches to output unsatisfactory maps. We present a novel semantic segmentation framework that is robust to such shift. The key component of the proposed framework is Color Mapping Generative Adversarial Networks (ColorMapGAN), which can generate fake training images that are semantically exactly the same as training images, but whose spectral distribution is similar to the distribution of the test images (See Figure 1). We then use the fake images and Ground Truth for the training images to fine-tune the already trained classifier. Contrary to the existing Generative Adversarial Networks (GANs), the generator in ColorMapGAN does not have any convolutional or pooling layers. It learns to transform the colors of the training data to the colors of the test data by performing only one element-wise matrix multiplication and one matrix addition operations. Thanks to the architecturally simple but powerful design of ColorMapGAN, the proposed framework outperforms the existing approaches with a large margin in terms of both accuracy and computational complexity. This work was published in IEEE Transactions on Geoscience and Remote Sensing 14.
8.1.2 Extracting geometric structures in images with Delaunay point processes
Participants: Florent Lafarge.
In collaboration with Jean-Dominique Favreau (Ekinnox), Adrien Bousseau (GraphDeco Inria team) and Alex Auvolat (Wide Inria team).
We introduce Delaunay Point Processes, a framework for the extraction of geometric structures from images. Our approach simultaneously locates and groups geometric primitives (line segments, triangles) to form extended structures (line networks, polygons) for a variety of image analysis tasks. Similarly to traditional point processes, our approach uses Markov Chain Monte Carlo to minimize an energy that balances fidelity to the input image data with geometric priors on the output structures. However, while existing point processes struggle to model structures composed of interconnected components, we propose to embed the point process into a Delaunay triangulation, which provides high-quality connectivity by construction. We further leverage key properties of the Delaunay triangulation to devise a fast Markov Chain Monte Carlo sampler. We demonstrate the flexibility of our approach on a variety of applications, including line network extraction, object contouring, and mesh-based image compression (See Figure 2). This work was published in IEEE Transactions on Pattern Analysis and Machine Intelligence 12.
8.1.3 DAugNet: Unsupervised, Multi-source, Multi-target, and Life-long Domain Adaptation for Semantic Segmentation of Satellite Images
Participants: Onur Tasar, Pierre Alliez.
In collaboration with Yuliya Tarabalka (Luxcarta), Alain Giros (CNES) and Sébastien Clerc (ACRI-ST).
The domain adaptation of satellite images has recently gained an increasing attention to overcome the limited generalization abilities of machine learning models when segmenting large-scale satellite images. Most of the existing approaches seek for adapting the model from one domain to another. However, such single-source and single-target setting prevents the methods from being scalable solutions, since nowadays multiple source and target domains having different data distributions are usually available. Besides, the continuous proliferation of satellite images necessitates the classifiers to adapt to continuously increasing data. We propose a novel approach, coined DAugNet, for unsupervised, multi-source, multi-target, and life-long domain adaptation of satellite images. It consists of a classifier and a data augmentor (See Figure 3). The data augmentor, which is a shallow network, is able to perform style transfer between multiple satellite images in an unsupervised manner, even when new data are added over the time. In each training iteration, it provides the classifier with diversified data, which makes the classifier robust to large data distribution difference between the domains. Our extensive experiments prove that DAugNet significantly better generalizes to new geographic locations than the existing approaches. This work was published in IEEE Transactions on Geoscience and Remote Sensing 13.
8.2 Reconstruction
8.2.1 Kinetic Shape Reconstruction
Participants: Jean-Philippe Bauchet, Florent Lafarge.
Converting point clouds into concise polygonal meshes in an automated manner is an enduring problem in Computer Graphics. Prior work, which typically operate by assembling planar shapes detected from input points, largely overlooked the scalability issue of processing a large number of shapes. As a result, they tend to produce overly simplified meshes with assembling approaches that can hardly digest more than one hundred shapes in practice. We propose a shape assembling mechanism which is at least one order magnitude more efficient, both in time and in number of processed shapes. Our key idea relies upon the design of a kinetic data structure for partitioning the space into convex polyhedra. Instead of slicing all the planar shapes exhaustively as prior methods, we create a partition where shapes grow at constant speed until colliding and forming polyhedra. This simple idea produces a lighter yet meaningful partition with a lower algorithmic complexity than an exhaustive partition. A watertight polygonal mesh is then extracted from the partition with a min-cut formulation. We demonstrate the robustness and efficacy of our algorithm on a variety of objects and scenes in terms of complexity, size and acquisition characteristics. In particular, we show the method can both faithfully represent piecewise planar structures and approximating freeform objects while offering high resilience to occlusions and missing data (See Figure 4). This work was published in ACM Transactions on Graphics 11 and presented at the ACM SIGGRAPH conference.
8.2.2 Real-Time Multi-SLAM System for Agent Localization and 3D Mapping in Dynamic Scenarios
Participants: Fernando Ireta Munoz, Pierre Alliez.
In collaboration with Fabien Bonardi, Samia Bouchafa, Jean-Yves Didier, Hicham Hadj-Abdelkader, Viachaslau Kachurka, David Roussel (IBISC), Bastien Rault and Maxime Robin (Innondura TB).
This paper introduces a Wearable SLAM system that performs indoor and outdoor SLAM in real time. The related project is part of the MALIN challenge which aims at creating a system to track emergency response agents in complex scenarios (such as dark environments, smoked rooms, repetitive patterns, building floor transitions and doorway crossing problems), where GPS technology is insufficient or inoperative. The proposed system fuses different SLAM technologies to compensate the lack of robustness of each, while estimating the pose individually (See Figure 5). LiDAR and visual SLAM are fused with an inertial sensor in such a way that the system is able to maintain GPS coordinates that are sent via radio to a ground station, for real-time tracking. More specifically, LiDAR and monocular vision technologies are tested in dynamic scenarios where the main advantages of each have been evaluated and compared. Finally, 3D reconstruction up to three levels of details is performed. This work was presented at the IROS robotics conference 16.
8.2.3 Lexicographic optimal homologous chains and applications to point cloud triangulations
Participants: Julien Vuillamy.
In collaboration with David Cohen-Steiner (Datashape Inria team) and André Lieutier (Dassault Systèmes).
This work considers a particular case of the Optimal Homologous Chain Problem (OHCP), where optimality is meant as a minimal lexicographic order on chains induced by a total order on simplices. The matrix reduction algorithm used for persistent homology is used to derive polynomial algorithms solving this problem instance, whereas OHCP is NP-hard in the general case. The complexity is further improved to a quasilinear algorithm by leveraging a dual graph minimum cut formulation when the simplicial complex is a strongly connected pseudomanifold. We then show how this particular instance of the problem is relevant, by providing an application in the context of point cloud triangulation (See Figure 6). This work was presented at the International Symposium on Computational Geometry (SoCG) conference 17.
8.2.4 Connect-and-Slice: an hybrid approach for reconstructing 3D objects
Participants: Florent Lafarge.
In collaboration with Hao Fang (BeiKe).
Reconstructing in 3D complex scenes captured by Laser scanning, multiview stereo imagery or depth cameras into compact polygon meshes is a challenging problem in Vision. Existing methods are either robust to imperfect data or scalable, but rarely both. In this work, we address this issue with an hybrid method that successively connects and slices planes detected from 3D data. The core idea consists in constructing an efficient and compact partitioning data structure. The later is i) spatially-adaptive in the sense that a plane slices a restricted number of relevant planes only, and ii) composed of components with different structural meaning resulting from a preliminary analysis of the plane connectivity. Our experiments on a variety of objects and sensors show the versatility of our approach as well as its competitiveness with respect to existing methods. Figure 7 shows the algorithm can also be used to approximate the geometry of a free-form object at different levels of details. This work was presented at the CVPR conference 18.
8.3 Approximation
8.3.1 Approximating shapes in images with low-complexity polygons
Participants: Muxingzi Li, Florent Lafarge.
In collaboration with Renaud Marlet (Valeo.ai).
We present an algorithm for extracting and vectorizing objects in images with polygons. Departing from a polygonal partition that oversegments an image into convex cells, the algorithm refines the geometry of the partition while labeling its cells by a semantic class. The result is a set of polygons, each capturing an object in the image (See Figure 8). The quality of a configuration is measured by an energy that accounts for both the fidelity to input data and the complexity of the output polygons. To efficiently explore the configuration space, we perform splitting and merging operations in tandem on the cells of the polygonal partition. The exploration mechanism is controlled by a priority queue that sorts the operations most likely to decrease the energy. We show the potential of our algorithm on different types of scenes, from organic shapes to man-made objects through floor maps, and demonstrate its efficiency compared to existing vectorization methods. This work was presented at the CVPR conference 19.
8.3.2 Progressive Geometric View Factors for Radiative Thermal Simulation
Participants: Vincent Vadez, Pierre Alliez.
In collaboration with François Brunetti (Dorea technology).
Radiative heat transfer or light transport are primarily governed by geometric view factors between surface elements. For general surfaces, calculating accurate geometric view factors requires solving integrals via quadrature methods. For complex scenes with free-form objects (Figure 9) and obstacles such calculations are compute-intensive, preventing real-time simulations. The progressive approach detailed here takes as input objects represented by surface triangle meshes and generates as output a dense square matrix of geometric view factors whose accuracy improves over time. The technical parameters of the approaches explained in this paper (hybrid triangle-based and point-based quadratures, tree-based data structure, segment-based probing and prediction) are selected to best trade accuracy for time. This work was presented at the International Conference on Environmental Systems (ICES) conference 22.
9 Bilateral contracts and grants with industry
9.1 Bilateral contracts with industry
X Development LLC (formerly Google X)
Participants: Cédric Portaneri, Pierre Alliez.
We explore an adaptive framework to repair and generate levels of detail from defect-laden 3D measurement data. Repair herein implies to convert raw 3D point sets or triangle soups into watertight solids, which are suitable for computing collision detection and situational awareness of robots. In this context, adaptive stands for the capability to select not only different levels of detail, but also the type of representation best suited to the environment of the scene and the task of the robot. This project is tightly focused on geometry processing for collision detection of robots with mechanical parts and other robots in motion.
Dorea technology
Participants: Vincent Vadez, Pierre Alliez.
In collaboration with SME Dorea Technology, our objective is to advance the knowledge on the radiative thermal simulation of satellites, via geometric model reduction. The survival of a satellite is related to the temperature of its components, the variation of which must be controlled within safety intervals. In this context, the thermal simulation of the satellite for its design is crucial to anticipate the reality of its operation. This CIFRE project started in August 2018, for a total duration of 3 years.
Luxcarta
Participants: Johann Lussange, Florent Lafarge.
The goal of this collaboration is to develop efficient, robust and scalable methods for extracting geometric shapes from the last generation of satellite images (at 0.3m resolution). Besides satellite images, input data will also include classification maps that identify the location of buildings and Digital Surface Models that bring rough pixel-based estimation of the urban object elevation. The geometric shapes will be first restricted to planes that typically describes well the piecewise planar geometry of buildings. The goal will be then to detect and identify each roof section of facade component of a building by a plane in the 3D space. This DGA Rapid project started in September 2020, for a total duration of 2 years.
CNES and Acri-ST
Participants: Onur Tasar, Yuliya Tarabalka, Pierre Alliez.
The aim is to devise efficient representations for satellite images. The project started in October 2017, for a total duration of 3 years.
CSTB
Participants: Mulin Yu, Florent Lafarge.
This collaboration takes the form of an independent contract. The project investigates the design of as-automatic-as-possible algorithms for repairing and converting Building Information Modeling (BIM) models of buildings in different urban-specific CAD formats using combinatorial maps. This project started November 2019, for a total duration of 3 years.
IRT Saint-Exupéry
Participants: Gaetan Bahl, Florent Lafarge.
This project investigates low-power deep learning architectures for detecting, localizing and characterizing changes in temporal satellite images. These architectures are designed to be exploited on-board satellites with low computational resources. The project started in March 2019, for a total duration of 3 years.
Dassault Systèmes
Participants: Julien Vuillamy, Pierre Alliez, Florent Lafarge.
This project investigates algorithms for reconstructing city models from multi-sourced data. 3D objects are reconstructed by filtering, parsing and assembling planar shapes. The project started in April 2018, for a total duration of 3 years.
10 Partnerships and cooperations
10.1 European initiatives
10.1.1 FP7 & H2020 Projects
BIM2TWIN
- Title: BIM2TWIN: Optimal Construction Management & Production Control
- Duration: 42 months
- Coordinator: CSTB, France
-
Partners:
- AARHUS UNIVERSITET (Denmark)
- ACCIONA CONSTRUCCION SA (Spain)
- CENTRE SCIENTIFIQUE ET TECHNIQUE DU BATIMENT (France)
- FUNDACION TECNALIA RESEARCH & INNOVATION (Spain)
- IDP INGENIERIA Y ARQUITECTURA IBERIA SL (Spain)
- INTSITE LTD (Israel)
- ORANGE SA (France)
- RUHR-UNIVERSITAET BOCHUM (Germany)
- SPADA CONSTRUCTION (France)
- TECHNISCHE UNIVERSITAET MUENCHEN (Germany)
- UNISMART - FONDAZIONE UNIVERSITA DEGLI STUDI DI PADOVA (Italy)
- UNIVERSITA DEGLI STUDI DI PADOVA (Italy)
- UNIVERSITA POLITECNICA DELLE MARCHE (Italy)
- Inria contact: Pierre Alliez
- Summary: The BIM2TWIN research and innovation action is sponsored by the H2020 “Digital Twins” programme. Our overall objective is to build a virtual replica platform, referred to as a digital building twin, to improve construction management, i.e. enhance quality and safety, reduce operational waste, carbon footprints and costs, and shorten schedules. The platform should provide full situational awareness via various sensing modalities in order to monitor human activities during construction and assess conformity to the digital plan. My team will contribute to the conformity assessment part of the platform by providing novel geometry processing pipelines and reliable software components for 3D urban reconstruction.
GRAPES
- Title: learninG, Representing, And oPtimizing shapES.
- Duration: 2019 - 2023
- Coordinator: ATHENA
-
Partners:
- ATHENA, ATHINA-EREVNITIKO KENTRO KAINOTOMIAS STIS TECHNOLOGIES TIS PLIROFORIAS, TON EPIKOINONION KAI TIS GNOSIS (Greece)
- AROMATH, INRIA SOPHIA-ANTIPOLIS (FRANCE)
- VISUAL COMPUTING, RHEINISCH-WESTFAELISCHE TECHNISCHE HOCHSCHULE AACHEN (Germany)
- OSLO, SINTEF AS (Norway)
- MATH DEPARTMENT, UNIVERSITA DEGLI STUDI DI ROMA TOR VERGATA (Italy)
- INFORMATICS DEPARTMENT), UNIVERSITA DELLA SVIZZERA ITALIANA (Switzerland)
- MATH DEPARTMENT, UNIVERSITAT DE BARCELONA (Spain)
- MATH Dept, UNIVERSITAT LINZ (Austria)
- Naval engineering, University of Strathclyde (UK)
- Math DEPARTMENT, VILNIAUS UNIVERSITETAS (Lithuania)
- GeometryFactory SARL (France)
- Inria contact: Laurent Busé
- Summary: GRAPES aims at considerably advancing the state of the art in Mathematics, CAD, and Machine Learning in order to promote game changing approaches for generating, optimising, and learning 3D shapes, along with a multisectoral training for young researchers. The scientific goals of GRAPES rely on a multidisciplinary consortium composed of leaders in their respective fields. Top-notch research is also instrumental in forming the new generation of European scientists and engineers. Their disciplines span the spectrum from Computational Mathematics, Numerical Analysis, and Algorithm Design, up to Geometric Modelling, Shape Optimisation, and Deep Learning. This allows the 15 PhD candidates to follow either a theoretical or an applied track and to gain knowledge from both research and innovation through a nexus of intersectoral secondments and Network-wide workshops. Horizontally, our results lead to open-source, prototype implementations, software integrated into commercial libraries as well as open benchmark datasets. These are indispensable for dissemination and training but also to promote innovation and technology transfer. Innovation relies on the active participation of SMEs, either as a beneficiary hosting an ESR or as associate partners hosting secondments. Concrete applications include simulation and fabrication, hydrodynamics and marine design, manufacturing and 3D printing, retrieval and mining, reconstruction and urban planning. In this context, Pierre Alliez advises a PhD student in collaboration with the beneficiary partner Geometry Factory, an Inria spin-off company commercializing components from the CGAL library. Our objective is to advance the state of the art in applied mathematics, computer-aided design (CAD) and machine learning in order to promote novel approaches for generating, optimizing, and learning 3D shapes. We will focus on dense semantic segmentation of 3D point clouds and semantic-aware reconstruction of 3D scenes. Our goal is to enrich the final reconstructed 3D models with labels that reflect the main semantic class of outdoor scenes such as ground, buildings, roads and vegetation. The Titane project-team will also contribute to the training of young researchers via tutorials on the CGAL library and practical exercises on real-world use cases.
10.2 National initiatives
10.2.1 ANR
PISCO: Perceptual Levels of Detail for Interactive and Immersive Remote Visualization of Complex 3D Scenes
Participants: Pierre Alliez, Lucas Dubouchet, Florent Lafarge.
The way of consuming and visualizing this 3D content is evolving from standard screens to Virtual and Mixed Reality (VR/MR). Our objective is to devise novel algorithms and tools allowing interactive visualization, in these constrained contexts (Virtual and Mixed reality, with local/remote 3D content), with a high quality of user experience. Partners: Inria, LIRIS INSA Lyon Institut National des Sciences Appiquées (coordinator), Laboratoire d'Informatique en Images et Systèmes d'Information LS2N Nantes University. Total budget 550 KE, 121 KE for TITANE. The PhD thesis of Flora Quilichini is funded by this project which started in January 2018, for a total duration of 4 years.
LOCA-3D: Localization Orientation and 3D CArtography
Participants: Fernando Ireta Munoz, Florent Lafarge, Pierre Alliez.
This project is part of the ANR Challenge MALIN LOCA-3D (Localization, orientation and 3D cartography). The challenge is to develop and experiment accurate location solutions for emergency intervention officers and security forces. These solutions must be efficient inside buildings and in conditions where satellite positioning systems do not work satisfactorily. Our solution is based on an advanced inertial system, where part of the inertial sensor drift is compensated by a vision system. Partners: SME INNODURA TB (coordinator), IBISC laboratory (Evry university) and Inria. Total budget: 700 KE, 157 KE for TITANE. The engineer position of Fernando Ireta Munoz is funded by this project which started in January 2018, for a total duration of 4 years.
EPITOME: efficient representation to structure large-scale satellite images
Participants: Nicolas Girard, Yuliya Tarabalka.
The goal of this young researcher project is to devise an efficient multi-scale vectorial representation, which would structure the content of large-scale satellite images. More specifically, we seek for a novel effective representation for large-scale satellite images, that would be generic, i.e., applicable for images worldwide and for a wide range of applications, and structure-preserving, i.e. best representing the meaningful objects in the image scene. To address this challenge, we plan to bridge the gap between advanced machine learning and geometric modeling tools to devise a multi-resolution vector-based representation, together with the methods for its effective generation and manipulation. Total budget: 225 KE for TITANE. The PhD thesis of Nicolas Girard is funded by this project which started in October 2017, for a total duration of 4 years.
Faults_R_GEMS: Properties of FAULTS, a key to Realistic Generic Earthquake Modeling and hazard Simulation
Participants: Lionel Matteo, Yuliya Tarabalka.
The goal of the project is to study the properties of seismic faults, using advanced math tools including learning approaches. The project is in collaboration with Geoazur lab (coordinator), Arizona State University, CALTECH, Ecole Centrale Paris, ENS Paris, ETH Zurich, Geosciences Montpellier, IFSTTAR, IPGP Paris, IRSN Fontenay-aux-Roses, LJAD Nice, UNAVCO Colorado and Pisa University. The PhD thesis of Lionel Matteo is funded by this project which started in October 2017, for a total duration of 4 years.
BIOM: Building Indoor and Outdoor Modeling
Participants: Muxingzi Li, Pierre Alliez, Florent Lafarge.
The BIOM project aims at automatic, simultaneous indoor and outdoor modelling of buildings from images and dense point clouds. We want to achieve a complete, geometrically accurate, semantically annotated but nonetheless lean 3D CAD representation of buildings and objects they contain in the form of a Building Information Models (BIM) that will help manage buildings in all their life cycle (renovation, simulation, deconstruction). The project is in collaboration with IGN (coordinator), Ecole des Ponts Paristech, CSTB and INSA-ICube. Total budget: 723 KE, 150 KE for TITANE. The PhD thesis of Muxingzi Li is funded by this project which started in February 2018, for a total duration of 4 years.
11 Dissemination
11.1 Promoting scientific activities
11.1.1 Scientific events: selection
Chair of conference program committees
- Florent Lafarge: Program Chair of the ISPRS congress 2020.
Member of the conference program committees
- Pierre Alliez: member of the advisory board for EUROGRAPHICS 2020.
- Pierre Alliez: member of the scientific committee for the SophIA Summit conference.
Reviewer
- Pierre Alliez: ACM SIGGRAPH, ACM SIGGRAPH Asia, Eurographics Symposium on Geometry processing, Eurographics annual conference.
- Gaetan Bahl: ISPRS congress
- Florent Lafarge: CVPR, ECCV, SIGGRAPH.
11.1.2 Journal
Member of the editorial boards
- Pierre Alliez: associate editor of the Computer Graphics Forum, Computer-Aided Geometric Design and Computer-Aided Design. He is also a member of the editorial board of the CGAL open source project.
- Florent Lafarge: associate editor of The Visual Computer and the Revue Française de Photogrammétrie et de Télédétection, and member of the Editorial Advisory Board of the ISPRS Journal of Photogrammetry and Remote Sensing.
Reviewer - reviewing activities
- Pierre Alliez: reviewer for Computer Graphics Forum and ACM Transactions on Graphics.
- Gaetan Bahl: International Journal of Computer Vision and the ISPRS Journal of Photogrammetry and Remote Sensing.
- Florent Lafarge: reviewer for the ISPRS Journal of Photogrammetry and Remote Sensing and IEEE Transactions on Pattern Analysis and Machine Intelligence.
- Muxingzi Li: The Visual Computer.
11.1.3 Invited talks
- Florent Lafarge gave an invited talk at the ECCV 2020 workshop "Holistic Scene Structures for 3D Vision".
11.1.4 Leadership within the scientific community
- Pierre Alliez is a member of the Steering Committees of the EUROGRAPHICS Symposium on Geometry Processing, EUROGRAPHICS Workshop on Graphics and Cultural Heritage and Executive Board Member for the Solid Modeling Association.
11.1.5 Scientific expertise
- Pierre Alliez was a reviewer for the ERC and ANR, and is a scientific advisory board member for the Bézout Labex in Paris (Models and algorithms: from the discrete to the continuous).
- Florent Lafarge was reviewer for the MESR (CIR and JEI expertises).
11.1.6 Research administration
- Pierre Alliez is Head of Science (VP for Science) of the Inria Sophia Antipolis center. In this role, he works with the executive team to advise and support the Institute's scientific strategy, to stimulate new research directions and the creation of project-teams, and to manage the scientific evaluation and renewal of the project-teams. This part-time position, in tandem with Fabien Gandon, runs until mid-2021.
- Pierre Alliez is a member of the scientific committee of the 3IA Côte d'Azur.
- Florent Lafarge is a member of the NICE committee. The main actions of the NICE committee are to verify the scientific aspects of the files of postdoctoral students, to give scientific opinions on candidates for national campaigns for postdoctoral stays, delegations, secondments but also for requests for invitations of long duration (more than 3 months).
11.2 Teaching - Supervision - Juries
11.2.1 Teaching
- Master: Pierre Alliez, Florent Lafarge and Gaétan Bahl, advanced machine learning, 21h, M2, Univ. Côte d'Azur, France.
- Master: Pierre Alliez and Florent Lafarge, 3D Meshes and Applications, 32h, M2, Ecole des Ponts ParisTech, France.
- Master: Florent Lafarge, Applied AI, 8h, M2, Univ. Côte d'Azur, France.
- Master: Pierre Alliez and Florent Lafarge, Interpolation numérique, 60h, M1, Univ. Côte d'Azur, France.
- Master: Florent Lafarge and Gaétan Bahl, Mathématiques pour la géométrie, 35h, M1, EFREI, France.
11.2.2 Supervision
- PhD defended September 22nd: Onur Tasar, Using deep learning approaches to devise an efficient representation for large-scale satellite images, since October 2017, Yuliya Tarabalka and Pierre Alliez.
- PhD defended October 16th: Nicolas Girard, How to structure satellite data, since November 2017, Yuliya Tarabalka.
- PhD in progress: Julien Vuillamy, city reconstruction from multi-sourced data, since April 2018, Pierre Alliez and Florent Lafarge.
- PhD in progress: Muxingzi Li, indoor reconstruction from a smartphone, since February 2018, Florent Lafarge.
- PhD in progress: Lionel Matteo: From Pleiades images to very high resolution topography in complex zones, since September 2017, Yuliya Tarabalka and Isabelle Manighetti.
- PhD in progress: Vincent Vadez, Geometric simplification of satellites for thermal simulation, since August 2018, Pierre Alliez.
- PhD in progress: Gaétan Bahl, low-power neural networks, since March 2019, Florent Lafarge.
- PhD in progress: Mulin Yu, remeshing urban-specific CAD formats, since November 2019, Florent Lafarge.
- PhD in progress: Tong Zhao, shape reconstruction, since November 2019, Pierre Alliez and Laurent Busé (from the Aromath Inria project-team), in collaboration with Jean-Marc Thiery and Tamy Boubekeur (Telecom ParisTech).
11.2.3 Juries
- Pierre Alliez was a PhD thesis reviewer for Adrien Poulenard (Ecole Polytechnique) and Shayan Nikoohemat (University Twente), and a PhD thesis committee member for Jean Feydy (ENS Cachan) and Simon Rodriguez (Inria Sophia Antipolis).
- Florent Lafarge was a PhD thesis reviewer for Marc Comino Trinidad (Universitat Politecnica de Catalunya) and Mohamed Boussaha (IGN), and a PhD committee member for Julien Philip (Inria Sophia Antipolis).
12 Scientific production
12.1 Major publications
- 1 inproceedings KIPPI: KInetic Polygonal Partitioning of Images IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Salt Lake City, United States June 2018
- 2 article Integer-Grid Maps for Reliable Quad Meshing ACM Transactions on Graphics 32 4 July 2013
- 3 articleOptimal Voronoi Tessellations with Hessian-based AnisotropyACM Transactions on GraphicsDecember 2016, 12
- 4 inproceedings Towards large-scale city reconstruction from satellites European Conference on Computer Vision (ECCV) Amsterdam, Netherlands October 2016
- 5 articleCurved Optimal Delaunay TriangulationACM Transactions on Graphics374August 2018, 16
- 6 articleIsotopic Approximation within a Tolerance VolumeACM Transactions on Graphics3442015, 12
- 7 articleVariance-Minimizing Transport Plans for Inter-surface MappingACM Transactions on Graphics362017, 14
- 8 articleSemantic Segmentation of 3D Textured Meshes for Urban Scene AnalysisISPRS Journal of Photogrammetry and Remote Sensing1232017, 124 - 139
- 9 articleSymmetry and Orbit Detection via Lie-Algebra VotingComputer Graphics ForumJune 2016, 12
- 10 articleLOD Generation for Urban ScenesACM Transactions on Graphics3432015, 15
12.2 Publications of the year
International journals
International peer-reviewed conferences
Conferences without proceedings
Doctoral dissertations and habilitation theses
Reports & preprints
Other scientific publications