3.1 Axis A: Augmented spatio-temporal perception of complex environments

The long-term objective of this research axis is to build accurate and composite models of large-scale environments that mix metric, topological and semantic information. Ensuring the consistency of these various representations during the robot exploration and merging/sharing observations acquired from different viewpoints by several collaborative robots or sensors attached to the infrastructure, are very difficult problems. This is particularly true when different sensing modalities are involved and when the environments are time-varying. A recent trend in Simultaneous Localization And Mapping is to augment low-level maps with semantic interpretation of their content. Indeed, the semantic level of abstraction is the key element that will allow us to build the robot’s environmental awareness (see Axis B). For example, the so-called semantic maps have already been used in mobile robot navigation, to improve path planning methods, mainly by providing the robot with the ability to deal with human-understandable targets. New studies to derive efficient algorithms for manipulating the hybrid representations (merging, sharing, updating, filtering) while preserving their consistency are needed for long-term navigation.

3.2 Axis B: Situation awareness for decision and planning

The long-term objective of this research axis is to design and develop a decision-making module that is able to (i) plan the mission of the robots (global planning), (ii) generate the sub-tasks (local objectives) necessary to accomplish the mission based on Situation Awareness and (iii) plan the robot paths and/or sets of actions to accomplish each subtask (local planning). Since we have to face uncertainties, the decision module must be able to react efficiently in real-time based on the available sensor information (on-board or attached to an IoT infrastructure) in order to guarantee the safety of humans and things. For some tasks, it is necessary to coordinate a multi-robots system (centralized strategy), while for other each robot evolves independently with its own decentralized strategy. In this context, Situation Awareness is at the heart of an autonomous system in order to feed the decision-making process, but also can be seen as a way to evaluate the performance of the global process of perception and interpretation in order to build a safe autonomous system. Situation Awareness is generally divided into three parts: perception of the elements in the environment (see Axis A), comprehension of the situation, and projection of future states (prediction and planning). When planning the mission of the robot, the decision-making module will first assume that the configuration of the multi-robot system is known in advance, for example one robot on the ground and two robots on the air. However, in our long-term objectives, the number of robots and their configurations may evolve according to the application objectives to be achieved, particularly in terms of performance, but also to take into account the dynamic evolution of the environment.

3.3 Axis C: Advanced multi-sensor control of autonomous multi-robot systems

The long-term objective of this research axis is to design multi-sensor (on-board or attached to an IoT infrastructure) based control of potentially multi-robots systems for tasks where the robots must navigate into a complex dynamic environment including the presence of humans. This implies that the controller design must explicitly deal not only with uncertainties and inaccuracies in the models of the environment and of the sensors, but also to consider constraints to deal with unexpected human behavior. To deal with uncertainties and inaccuracies in the model, two strategies will be investigated. The first strategy is to use Stochastic Control techniques that assume known probability distribution on the uncertainties. The second strategy is to use system identification and reinforcement learning techniques to deal with differences between the models and the real systems. To deal with unexpected human behavior, we will investigate Stochastic Model Predictive Control (MPC) techniques and Model Predictive Path Integral (MPPI) control techniques in order to anticipate future events and take optimal control actions accordingly. A particular emphasis will be given to the theoretical analysis (observability, controllability, stability and robustness) of the control laws.

4.1 Environment monitoring with a collaborative robotic system

The first application that we will consider concerns monitoring the environment using an autonomous multi-robots system composed by ground robots and aerial robots (see Figure 2). The ground robots will patrol following a planned trajectory and will collaborate with the aerial drones to perform tasks in structured (e.g. industrial sites), semi-structured (e.g. presence of bridges, dams, buildings) or unstructured environments (e.g. agricultural space, forest space, destroyed space). In order to provide a deported perception to the ground robots, an aerial drone will be in operation while the second one will be recharging its batteries on the ground vehicle. Coordinated and safe autonomous take-off and landing of the aerial drones will be a key factor to ensure the continuity of service for a long period of time. Such a multi-robot system can be used to localize survivors in case of disaster or rescue, to localize and track people or animals (for surveillance purpose), to follow the evolution of vegetation (or even invasion of insects or parasites), to follow evolution of structures (bridges, dams, buildings, electrical cables) and to control actions in the environment like for example in agriculture (fertilization, pollination, harvesting, ...), in forest (rescue), in land (planning firefighting). Successful achievement of such applications requires to build a representation of the environment and localize the robots in the map (see Axis A in section 3.1), to re-plan the tasks of each robot when unpredictable events occurs (see Axis B in section 3.2) and to control each robot to execute the tasks (see Axis C in section 3.3). Depending on the application field, the scale and the difficulty of the problems to be solved will be increasing. In the Smart Factories field, we have a relatively small size environment, mostly structured, with highly instrumented (sensors) and with the possibility to communicate. In the Smart Territories field, we have large semi-structured or unstructured environments that are not instrumented. To set up demonstrations of this application, we intend to collaborate with industrial partners and local institutions. For example, we plan to set up a collaboration with the Parc Naturel Régional des Prealpes d'Azur to monitor the evolution of fir trees infested by bark beetles.

4.2 Transportation of people and goods with autonomous connected vehicles

The second application that we will consider, concerns the transportation of people and goods with autonomous connected vehicles (see Figure 3). ACENTAURI will contribute to the development of Autonomous Connected Vehicles (e.g. Learning, Mapping, Localization, Navigation) and the associated services (e.g. towing, platooning, taxi). We will develop efficient algorithms to select on-line connected sensors coming from the infrastructure in order to extend and enhance the embedded perception of a connected autonomous vehicle. In cities, there exists situations where visibility is very bad for historical reason or simply occasionally because of traffic congestion, service delivery (trucks, buses) or roadworks. It exists also situation where danger are more important and where a connected system or intelligent infrastructure can help to enhance perception and then reduce the risk of accident (see Axis A in section 3.1). In ACENTAURI, we will also contribute to the development of assistance and service robotics by re-using the same technologies required in autonomous vehicles. By adding the social level in the representation of the environment, and using techniques of proactive and social navigation, we will offer the possibility of the robot to adapt its behavior in presence of humans (see Axis B in section 3.2). ACENTAURI will study sensing technology on SDVs (Self-Driving Vehicles) used for material handling to improve efficiency and safety as products are moved around Smart Factories. These types of robots have the ability to sense and avoid people, as well as unexpected obstructions in the course of doing its work (see Axis C in section 3.3). The ability to automatically avoid these common disruptions is a powerful advantage that keeps production running optimally. To set up demonstrations of this application, we will continue the collaboration with industrial partners (Renault) and with the Communauté d'Agglomération Sophia Antipolis (CASA). Experiments with 2 autonomous Renault Zoe cars will be carried out in a dedicated space lend by CASA. Moreover, we propose, with the help of the Inria Service d'Expérimentation et de Développement (SED), to set up a demonstration of an autonomous shuttle to transport people in the future extended Inria/UCA site.

5.1 Footprint of research activities

The main footprint of our research activities comes from travels and power consumption (computers and computer cluster). Concerning travels, after the limitation due to the COVID-19 pandemic, they have increased again but we make our best efforts to prioritize visioconferencing. Concerning power consumption, besides classical actions to reduce the waste of energy, our research focus on efficient optimization algorithms to minimize the computation time of computers onboard of our robotic platforms.

5.2 Impact of research results

We have planned to propose several projects related to the environmental challenges. We give below three examples of the most advanced projects that have been recently started.

The first two concerns the monitoring of forest, one in collaboration with the Parc Naturel Regional des Préalpes d'Azur, ONF and DRAAF (EPISUD Project) and one in collaboration with INRAE UR629 Ecologie des Forêts Méditerranéennes (see ROBFORISK project in section 10.4.6) .

The third concerns the autonomous vehicles in agricultural application in collaboration with INRAE Clermont-Ferrand in the context of the PEPR "Agrologie et numérique". We aim to develop robotic approaches for the realization of new cultural practices, capable of acting as a lever for agroecological practices (see NINSAR Project in section 10.4.4).

6.1 Team progression

This year that has been dedicated to the growth of our team to its nominal size. In particular we have:

• welcomed eight new members to our team, each bringing unique skills, experiences, and perspectives.
• kick off a new collaborative project 10.4.7.
• organized two workshops (one in France and one in Korea) in context of the AISENSE associated team with the AVELAB of KAIST in Korea.
• strengthened industrial collaborations by initiating two CIFRE PhD theses:
  • Thomas Campagnolo (NXP), "Embedded Machine Learning Solutions for Autonomous Navigation". Phd supervisors: Ezio Malis and Philippe Martinet.
  • Mohamed Mahmoud Ahmed Maloum (SAFRAN), "Inertial-vision SLAM using deep learning". Phd supervisors: Ezio Malis and Philippe Martinet.

We organized a two days team seminar in Marseille to foster scientific discussion and collaborations between team members (see Figure 4).

We have also continued a biweekly robotic seminar involving both Inria (HEPHAISTOS and ACENTAURI) and I3S (OSCAR, ROBOTVISION, ...) robotic teams in order to disseminate information about the latest advancements, trends, and research in robotics.

6.2 Awards

• Emmanuel Alao received the Best Student Paper Award for the paper "Multi-Risk Assessment and Management in the Presence of Personal Light Electric Vehicles" at the ICINCO (International Conference on Informatics in Control, Automation and Robotics) conference in November 2024.
• Mathilde Theunissen won the second prize of the jury for the work "Robustness Study of Optimal Geometries for Cooperative Multi-Robot Localization" at Journée des Jeunes Chercheuses et Jeunes Chercheurs en Robotique in November 2024.

7.1 New software

7.2 New platforms

Participants: Ezio Malis, Philippe Martinet, Nicolas Chleq, Pierre Joyet, Quentin Louvel.

7.2.1 ICAV platform

ICAV platform has been funded by PUV@SOPHIA project (CASA, PACA Region and state), self funding, Digital Reference Center from UCA, and Academy 1 from UCA. We have now two autonomous vehicles, one instrumented vehicle, many sensors (Real Time Kinematics GPS, Lidars, Cameras), Communications devices (C-V2X, IEEE 802.11p), and one standalone localization and mapping system.

ICAV platform is composed of (see Figure 5):

• ICAV1 is an old generation of ZOE. It has been bought fully robotized and intrumented. It is equipped with Velodyne Lidar VLP16, low cost Inetial Measurement Unit (IMU) and GPS, three cameras and one embedded computer.
• ICAV2 is a new generation of ZOE which has been instrumented and robotized in 2021. It is equipped with Velodyne Lidar VLP16, low cost IMU and GPS, three cameras, two solidstate Lidars RS-M1, one embedded computer and one NVIDIA Jetson AGX Xavier.
• ICAV3 will be instrumented with different LIDARS and multi cameras system (LADYBUG5+)
• A ground truth RTK system. An RTK GPS base station has been installed and a local server configured inside the Inria Center. Each vehicle is equipped with an RTK GPS receiver and connected to a local server in order to compute a centimeter localization accuracy.
• A standalone localization and mapping system. This system is composed of a Velodyne Lidar VLP16, low cost IMU and GPS, and one NVIDIA Jetson AGX Xavier.
• A communication system vehicle-to-everything (V2X) based on the technology C-V2X and IEEE 802.11p.
• Different lidar sensors (Ouster OS2-128, RS-LIDAR16, RS-LIDAR32, RS-Ruby), and one multi-cameras system (LADYBUG5+)

The main applications of this platform are:

• datasets acquisition
• localization, Mapping, Depth estimation, Semantization
• autonomous navigation (path following, parking, platooning, ...), proactive navigation in shared space
• situation awareness and decision making
• V2X communication
• autonomous landing of UAVs on the roof.

 

 

 

 

ICAV2 has been used by Maria Kabtoul in order to demonstrate the effectiveness of autonomous navigation of a car in a crowd.

 

Indoor autonomous mobile platform

The mobile robot platform has been funded by the MOBI-DEEP project in order to demonstrate autonomous navigation capabilities in encumbered and crowded environment. This platform is composed of (see Figure 6):

• one omnidirectional mobile robot (SCOOT MINI with mecanum wheels from AGIL-X)
• one NVIDIA Jetson AGX Xavier for deep learning algorithm implementation
• one general labtop
• one Robosense RS-LIDAR16
• one Ricoh Z1 360° camera
• one Sony RGB-D D455 camera

The main applications of this platform are:

• indoor datasets acquisition
• localization, Mapping, depth estimation, Semantization
• proactive navigation in shared space
• pedestrian detection and tracking.

This platform was used in MOBI-DEEP project for integration of different work from the consortium. It is used to demonstrate new results on social navigation.

7.2.2 E-Wheeled platform

E-WHEELED is an AMDT Inria project (2019-22) coordinated by Philippe Martinet. The aim is to provide mobility to things by implementing connectivity techniques. It makes available an Inria expert engineer (Nicolas Chleq) in ACENTAURI in order to demonstrate the Proof of Concept using a small size demonstrator (see Figure 7). Due to the COVID19, the project has been delayed.

Show image description

Two wheels

7.2.3 Moving Living Lab global platform

Moving Living Lab platform has been funded by the AgrifoodTEF project (H2020 project). It has been designed to perform physical field testing (Navigation algorithms, Monitoring of health and growth, Sensors and robots testing) and dataset acquisition in real agricultural sites.

Moving Living Lab platform is composed of different elements (see Figure 8):

• MLL is a moving laboratory. It has been fully equipped with a server, a RTK-GPS base, a 5G private network, WIFI, and three office desks. It has autonomy of energy and an electric generator. It has also a trailer in order to transport the robots.
• SUMMIT-HM is a customized and updated version of the Summit-XL offroad mobile robot from robotnik. It is an instrumented robots with many sensors (RTK GPS, Lidar, Camera, IMU Communications devices (WIFI, 5G)) and one embedded NVIDIA Jetson AGX Orin.
• VERSATYL is an UAV from Skydrone company. It has been customized with a payload instrumented with many sensors (RTK GPS, Lidar, Camera, IMU Communications devices (WIFI, 5G)) and one embedded NVIDIA Jetson AGX Orin.
• Matrice 300 RTK is an UAV from DJI company. It has been equipped with a multispectral camera and one embedded NVIDIA Jetson AGX Xavier.
• A ground truth RTK system. An RTK GPS base station has been installed and a local server configured inside the MLL. Each robot is equipped with an RTK GPS receiver and connected to a local server in order to compute a centimeter localization accuracy.

The main applications of this platform are:

• datasets acquisition
• localization, Mapping, Simultaneous Localization And Mapping (SLAM)
• autonomous navigation (path planning and tracking, Geofencing), proactive navigation in shared space.

 

 

 

 

 

7.2.4 UAV arena Dronix platform.

The UAV (Unmanned Aerial Vehicle) Arena (called Dronix) is a fixed and reconfigurable platform owned by ACENTAURI team at INRIA. It was cofunded by the European project AgrifoodTEF.

The volume of Dronix is 5 m x 6 m x 7 m. It is a specialized platform designed for the development, testing and demonstration of mobility algorithms for UAVs and AGRs in a controlled indoor environment. It can be considered as a fully controlled facility for preliminary testing of UAV and AGR functionalities.

Dronix platform is presented in Figure 9.

 

 

 

It is composed of different elements :

• Motion Capture Cameras Miqus M3 (resolution (1824 x 1088), normal mode (2MO, 340fps), High speed mode (0,5MO, 650fps))
• Dedicated Qualisys Motion Analysis System
• Multi object tracking with specific 3D markers

The main applications of this platform are:

• Data collection via UAVs and AGRs mounted with different sensors
• Testing and Validation of Different Sensors Calibration
• Building ground truth localization with 12 Cameras for a millimeter (mm) accuracy.
• Testing and Validation of Localization, Mapping, SLAM, and Navigation Algorithms for UAV, AGR, and their collaboration

Dronix is based on Qualisys Motion Capture and Tracking System and it uses 12 cameras Qualisys Miqus M3 to localize (estimation of the position and orientation) and track any moving object (equipped with markers) in a dedicated volume. The software named Qualisys Track manager is able to provide in high frequency localization of these objects with a 0.1mm accuracy and can be connected to a robotic system in real time as a ground truth source, and/or real time localization system.

8.1 Hybrid Dense Stereo Visual Odometry

Participants: Ziming Liu, Ezio Malis, Philippe Martinet.

Visual odometry is an important task for the localization of autonomous robots and several approaches have been proposed in the literature. We consider visual odometry approaches that use stereo images since the depth of the observed scene can be correctly estimated. These approaches do not suffer from the scale factor estimation problem of monocular visual odometry approaches that can only estimate the translation up to a scale factor. Traditional model-based visual odometry approaches are generally divided into two steps. Firstly, the depths of the observed scene are estimated from the disparity obtained by matching left and right images. Then, the depths are used to obtain the camera pose. The depths can be computed for selected features like in sparse methods or for all possible pixels in the image like in dense direct methods 1.

Recently, more and more end-to-end deep learning visual odometry approaches have been proposed, including supervised and unsupervised models. However, it has been shown that hybrid visual odometry approaches can achieve better results by combining a neural network for depth estimation with a model-based visual odometry approach. Moreover, recent works have shown that deep learning based approaches can perform much better than the traditional stereo matching approaches and provide more accurate depth estimation.

State-of-the-art supervised stereo depth estimation networks can be trained with the ground truth disparity (depth) maps. However, ground truth depth maps are extremely difficult (almost impossible for dense depth maps) to be obtained for all pixels in real and variable environments. Therefore, some works try to train a model on simulated environments and reduce its gap with the real world data. However, there are only few works focusing on unsupervised stereo matching. Besides, using the ground truth depth maps or unsupervised training, we can also build temporal images reconstruction loss with ground truth camera pose which is easy to be obtained. In previous works, we proposed hybrid visual odometry methods to achieve state-of-the-art performance by fusing both data-based deep learning networks and model-based localization approaches 67.

However, these methods also suffer from deep learning domain gap problems, which leads to an accuracy drop of the hybrid visual odometry approach when new type of data is considered. In 11, we explored a practical solution to this problem. Indeed, the deep learning network in the hybrid visual odometry predicts the stereo disparity with fixed searching space. However, the disparity distribution is unbalanced in stereo images acquired in different environments. We propose an adaptive network structure to overcome this problem. Secondly, the model-based localization module has a robust performance by online optimizing the camera pose in test data, which motivates us to introduce test-time training machine learning method for improving the data-based part of the hybrid visual odometry model.

Another problem we faced in our previous work concerned stereo matching, that is one of the low-level visual perception tasks. Currently, two-stage 2D-3D networks and three-stage recurrent networks dominate deep stereo matching. These methods build a cost volume with low-resolution stereo feature maps, which splits the network into a feature net and a matching net. However, the 2D feature map is not uncontrollable, and the low-resolution feature map has lost important matching information. To overcome these problems, we proposed in 20 the first one-stage 2D-3D deep stereo network, named StereoOne. It has an efficient module that builds a cost volume at image resolution in real-time. The feature extraction and matching are learned in a single 3D network. According to the experiments, the new network can easily surpass the 2D-3D network baseline and it can achieve competitive performance with the state-of-the-art.

8.2 Decision making for autonomous driving in urban environment

Participants: Monica Fossati, Philippe Martinet, Ezio Malis.

Recent advancements in autonomous vehicle (AV) technology have led to increasingly sophisticated systems, particularly for relatively simple environments like highways and rural roads. However, achieving full autonomy in urban settings remains challenging due to the dynamic and unpredictable behavior of road users. Our work aims to address these challenges by integrating high-definition (HD) maps, specifically those based on the Lanelet2 format, with real-time perception data to dynamically characterize the space around a road agent. This integration leverages the graph-based structure, semantic richness, and modularity of Lanelet2 maps to provide a comprehensive, context-aware representation of the environment. Our goal is to view the road from the user’s perspective, extracting navigation-relevant information to support adaptive and proactive decision-making, enhancing the vehicle’s situational awareness and ability to navigate complex urban scenarios safely and efficiently. This represents a significant step toward the development of autonomous vehicles capable of intelligent, context-driven behavior in real-world environments. In this aim, our work introduces key advancements in enhancing the situational awareness of autonomous vehicles:

• Improved Reachable Set Calculation integrating Lanelet2 maps with the current state and kinematic limits of vehicles to provide a more accurate prediction of potential movements at the lanelet level.
• Physical Rule Implementation leveraging the modularity of the Lanelet2 standard to estimate multiple reachable sets capable of describing and anticipating scenarios involving relaxed or violated traffic rules.
• Automatic Interaction Identification and Classification through the intersection of their reachable sets. This allows for identifying which agents could interact and how.

The proposed approach provides autonomous vehicles with a multi-level representation of the environment, enabling adaptive decision-making. It considers both traffic and physical rules, better describing human dynamics, and allows for proactive strategies and selective rule relaxation. This leads to a more robust understanding of the vehicle’s context in urban navigation.

8.3 Multi-Spectral Visual Servoing

Participants: Enrico Fiasché, Philippe Martinet, Ezio Malis.

The objective of this work is to propose a novel approach for Visual Servoing (VS) using a multispectral camera, where the number of data are more than three times that of a standard color camera. To meet real-time feasibility, the multispectral data captured by the camera is processed using a dimensionality reduction technique. Unlike conventional methods that select a subset of bands, our technique leverages the best information coming from all the bands of the multispectral camera. This fusion process sacrifices spectral resolution but keeping spatial resolution, crucial for precise robotic control in forests. The proposed approach is validated through simulations, demonstrating its efficacy in enabling timely and accurate VS in natural settings. By leveraging the full spectral information of the camera and preserving spatial details, our method offers a promising solution for VS in natural environment, surpassing the limitations of traditional band selection approaches 17.

8.4 Efficient and accurate closed-form solution to pose estimation from 3D correspondences

Participants: Ezio Malis.

Computing the pose from 3D data acquired in two different frames is of high importance for several robotic tasks like odometry, SLAM and place recognition. The pose is generally obtained by solving a least-squares problem given points-to-points, points-to-planes or points to lines correspondences. The non-linear least-squares problem can be solved by iterative optimization or, more efficiently, in closed-form by using solvers of polynomial systems.

In 8, a complete and accurate closed-form solution for a weighted least-squares problem has been studied. Adding weights for each correspondence allows to increase robustness to outliers. Contrary to existing methods, the proposed approach is complete since it is able to solve the problem in any non-degenerate case and it is accurate since it is guaranteed to find the global optimal estimate of the weighted least-squares problem. The key contribution of this work is the combination of a robust u-resultant solver with the quaternion parametrization of the rotation, that was not considered in previous works. Simulations and experiments on real data demonstrate the superior accuracy and robustness of the proposed algorithm compared to previous approaches.

During this year, we aimed to address the limitations and unanswered questions that emerged from earlier study and offer novel perspectives and methodologies to advance the field. Notably, one of the main contributing factors to the limited efficiency observed in the u-resultant approach was the failure to leverage the two-fold symmetry inherent in the quaternion parametrization, a missed opportunity that could have led to a more efficient solver. For these reasons, a new approach has been studied that shifts from utilizing the u-resultant method to adopting the h-resultant approach 12. The h-resultant approach consists in hiding one variable considering it as a parameter. Such a hidden variable approach has been used in the past to solve different problems in computer vision. Therefore, the key contribution of the current work is the design of a novel efficient h-resultant solver that exploits the two-fold symmetry of the quaternion parametrization, that was not considered in previous work.

8.5 Lidar Odometry

Participants: Matteo Azzini, Ezio Malis, Philippe Martinet.

In the field of lidar odometry for autonomous navigation, the Iterative Closest Point (ICP) algorithm is a prevalent choice for estimating robot motion by comparing point clouds. However, ICP accuracy is strictly dependent on the nature of the features involved, but also on the directional choice of the extraction and matching, either from the current to the reference point cloud or vice-versa. Point-to-line or point-to-plane correspondences have been proven to provide the more accurate odometry results. The matching is generally done in a mono-directional framework: extract the features (lines or planes) in the current point cloud and match them to points in the reference point cloud. In 15, we introduced a novel formulation, named Balanced ICP, that performs feature extraction (lines or planes) in both point clouds and consequent matching in both directions. Therefore, the cost function is designed to perform a simultaneous optimization of all available data balancing the noise and extraction errors. The experiments, conducted both on simulated and real data from the KITTI dataset, reveal that our method outperforms the classical mono-directional formulations, in terms of robustness, accuracy and stability.

8.6 Dense-direct visual-SLAM

Participants: Diego Navarro Tellez, Ezio Malis, Raphael Antoine (CEREMA), Philippe Martinet.

In the context of the ROADAI project, we proposed a comprehensive framework based on direct Visual Simultaneous Localization and Mapping (V-SLAM) to observe a vertical coastal cliff 21. The precise positioning of data measurements (such as ground-penetrating radar) is crucial for environmental observations. However, in GPS-denied environments near large structures, the GPS signal can be severely disrupted or even unavailable. To address this challenge, we focus on the accurate localization of drones using vision sensors and SLAM systems. Traditional SLAM approaches may lack robustness and precision, particularly when cameras lose perspective near structures. We propose a new framework that combines feature-based and direct methods to enhance localization precision and robustness. The proposed system operates in two phases: first, a SLAM phase utilizing a stereo camera to reconstruct the environment from a distance sufficient to benefit from a wide field of view; second, a localization phase employing a monocular camera. Experiments conducted in realistic simulated environments demonstrate the system's ability to achieve drone localization within 15-centimeter precision, surpassing existing state-of-the-art approaches.

8.7 Reliable Risk Assessment and Management in autonomous driving

Participants: Emmanuel Alao, Lounis Adouane [UTC Compiegne], Philippe Martinet.

Autonomous driving in urban scenarios has become more challenging due to the increase in Personal Light Electric Vehicles (PLEVs). PLEVs correspond mostly to electric devices such as gyropods and scooters. They exhibit varying velocity profiles as a result of their high acceleration capacity. Multiple hypotheses about their possible motion make autonomous driving very difficult, leading to the highly conservative behavior of most control algorithms. The work proposes in 13 to solve this problem by performing a continuous risk assessment using a Fusion of Predictive Inter-Distance Profile (F-PIDP). Then a stochastic MPC algorithm performs effective risk management using the F-PIDP while taking into account adaptive constraints. The advantages of the proposed approach are demonstrated through simulations of multiple scenarios.

To handle multiple multi-modal motions using a fusion of PIDPs (F-PIDP), the traditional Predictive Inter-Distance Profile (PIDP) risk assessment metric (Bellingard et al., 2023) is extended in 14. This approach accounts for the uncertainties in the various trajectories that PLEVs can follow on the road. A priority-based strategy is then developed to select the most dangerous agent. Then F-PIDP and Model Predictive Control (MPC) algorithm are employed for risk management, ensuring safe and reliable navigation. The efficiency of the proposed method is validated through several simulations.

8.8 Parameter estimation of the lateral vehicle dynamics

Participants: Fabien Lionti, Nicolas Gutowski [LERIA Angers], Sébastien Aubin [DGA], Philippe Martinet.

Estimating parameters for nonlinear dynamic systems is a significant challenge across numerous research areas and practical applications.

In 19, we present a novel two-step approach for estimating parameters that control the lateral dynamics of a vehicle, acknowledging the limitations and noise within the data. The methodology merges spline smoothing of system observations with a Bayesian framework for parameter estimation. The initial phase involves applying spline smoothing to the system state variable observations, effectively filtering out noise and achieving precise estimates of the state variables’ derivatives.

Consequently, this technique allows for the direct estimation of parameters from the differential equations characterizing the system’s dynamics, bypassing the need for labor-intensive integration procedures. The subsequent phase focuses on parameter estimation from the differential equation residuals, utilizing a Bayesian method known as likelihood-free ABCSMC.

This Bayesian strategy offers multiple advantages: it mitigates the impact of data scarcity by incorporating prior knowledge regarding the vehicle’s physical properties and enhances interpretability through the provision of a posterior distribution for the parameters likely responsible for the observed data. Employing this innovative method facilitates the robust estimation of parameters governing vehicle lateral dynamics, even in the presence of limited and noisy data.

8.9 A Novel 3D Model Update Framework for Long-Term Autonomy

Participants: Stefan Larsen, Ezio Malis, El Mustafa Mouaddib, Patrick Rives.

The ability to perform long-term robotic operations in dynamic environments remains a challenge in fields such as surveillance, agriculture and autonomous vehicles. For improved localization and monitoring over time, we proposed a novel model update framework using image-based 3D change localization and segmentation 18. Specifically, shallow image data is used to detect and locate significant geometric change areas in a pre-made 3D model. The main contribution of this paper is the ability to precisely segment and locate both new and missing objects from few observations, and to provide consistent model updates. The applied method for geometric change detection is robust to seasonal, viewpoint, and illumination differences that may occur between operations. Qualitative and quantitative tests with both our own and publicly available datasets show that the model update framework improves on previous methods and facilitates long-term localization.

8.10 Multi-robots localization and navigation for infrastructure monitoring

Participants: Mathilde Theunissen, Isabelle Fantoni, Ezio Malis, Philippe Martinet.

In the context of the ANR SAMURAI project, we studied the interest in leveraging the robot formation control to enforce the localization precision. This involves optimizing the shape of the formation based on localizability indices that depend on the sensors used by the robots and the environment. The first focus is on the definition and optimization of a localizability index applied to collaborative localization using Ultra-Wide-Band (UWB) antennas 22 .

8.11 Collaborative perception between autonomous vehicles and roadside units

Participants: Minh-Quan Dao, Ezio Malis, Vincent Fremont [LS2N, Nantes].

In the context of the ANR ANNAPOLIS project, we started to study how to increase the vehicle’s perception capacity both in terms of precision, measurement field of view and information semantics, through vehicle to intelligent infrastructure communication. Occlusion presents a significant challenge for safety-critical applications of autonomous driving 10. Collaborative perception has recently attracted a large research interest thanks to the ability to enhance the perception of autonomous vehicles via deep information fusion with intelligent roadside units (RSU), thus minimizing the impact of occlusion. While significant advancement has been made, the data-hungry nature of these methods creates a major hurdle for their real- world deployment, particularly due to the need for annotated RSU data. Manually annotating the vast amount of RSU data required for training is prohibitively expensive, given the sheer number of intersections and the effort involved in annotating point clouds. We address this challenge by devising a label-efficient object detection method for RSU based on unsupervised object discovery 16. Our work introduces two new modules: one for object discovery based on a spatial temporal aggregation of point clouds, and another for refinement. Furthermore, we demonstrate that fine-tuning on a small portion of annotated data allows our object discovery models to narrow the performance gap with, or even surpass, fully supervised models. Extensive experiments are carried out in simulated and real-world datasets to evaluate our method.

9.1 Bilateral contracts with industry

ACENTAURI is responsible for four research contracts.

10.1 International initiatives

10.2 International research visitors

10.3 European initiatives

10.4 National initiatives

11.1 Promoting scientific activities

11.2 Teaching - Supervision - Juries

11.3 Popularization

12.1 Major publications

• 1 articleA. I.Andrew I. Comport, E.Ezio Malis and P.Patrick Rives. Real-time Quadrifocal Visual Odometry.The International Journal of Robotics Research2922010, 245-266HALDOIback to text
• 2 inproceedingsE.Enrico Fiasché, P.Philippe Martinet and E.Ezio Malis. Towards autonomous robot navigation in human populated environments using an Universal SFM and parametrized MPC.IROS 2023 - IEEE/RSJ International Conference on Intelligent Robots and SystemsDetroit (MI), United States2023HAL
• 3 articleM.Maria Kabtoul, M.Manon Prédhumeau, A.Anne Spalanzani, J.Julie Dugdale and P.Philippe Martinet. How To Evaluate the Navigation of Autonomous Vehicles Around Pedestrians?IEEE Transactions on Intelligent Transportation SystemsOctober 2023, 1-11HALDOI
• 4 articleA.Ahmed Khalifa, O.Olivier Kermorgant, S.Salvador Dominguez and P.Philippe Martinet. Platooning of Car-like Vehicles in Urban Environments: Longitudinal Control Considering Actuator Dynamics, Time Delays, and Limited Communication Capabilities.IEEE Transactions on Control Systems TechnologyDecember 2020HALDOI
• 5 inproceedingsF.Fabien Lionti, N.Nicolas Gutowski, S.Sébastien Aubin and P.Philippe Martinet. Towards simulation of radio-frequency component with physics informed neural networks.CAID 2023 - 5e Conference on Artificial Intelligence for DefenseActes de la conférence CAID 2023Rennes, FranceNovember 2023HAL
• 6 inproceedingsZ.Ziming Liu, E.Ezio Malis and P.Philippe Martinet. A New Dense Hybrid Stereo Visual Odometry Approach.IROS 2022 - 2022 IEEE/RSJ International Conference on Intelligent Robots and SystemsKYOTO, JapanIEEEOctober 2022, 6998-7003HALDOIback to text
• 7 inproceedingsZ.Ziming Liu, E.Ezio Malis and P.Philippe Martinet. Multi-masks Generation for Increasing Robustness of Dense Direct Methods.ITSC 2023 - 26th IEEE International Conference on Intelligent Transportation SystemsBilbao, SpainSeptember 2023HALback to text
• 8 articleE.Ezio Malis. Complete closed-form and accurate solution to pose estimation from 3D correspondences.IEEE Robotics and Automation Letters83March 2023, 1786 - 1793HALDOIback to text
• 9 articleD.David Pérez-Morales, O.Olivier Kermorgant, S.Salvador Domínguez-Quijada and P.Philippe Martinet. Multi-Sensor-Based Predictive Control For Autonomous Parking.IEEE Transactions on Robotics382April 2022, 835-851HALDOI

12.2 Publications of the year