An overall architecture of our environment-modeling module with the inputs from heterogenous sensors is shown in Fig.  . The combined use of two lidars and stereo-vision helps mitigate uncertainty and allows for detection of partially occluded objects. The data processing includes the computation of probabilistic occupancy grids for each sensor and their subsequent fusion with the Bayesian Occupancy Filter (BOF). The output of the module is an estimation of the position, velocity and associated uncertainty of each observed object, which are used as input to the risk assessment module.

This architecture is implemented on our experimental platform, a Lexus LS600h car shown in Fig.  . The vehicle is equipped with a variety of sensors including two IBEO Lux lidars placed toward the edges of the front bumper, a TYZX stereo camera situated behind the windshield, and an Xsens MTi-G inertial sensor with GPS.

The stereo camera baseline is 22  $cm$, with a field of view of  ${62}^{\circ }$. Camera resolution is 512x320 pixels with a focal length of 410 pixels. Each lidar provides four layers of up to 200 impacts with a sampling period of 20  $ms$. The angular range is 100 ${}^{\circ }$, and the angular resolution is 0.5 ${}^{\circ }$. The on-board computer is equipped with 8GB of RAM, an Intel Xeon 3.4  $GHz$processor and an NVIDIA GeForce GTX 480 for GPU. The observed region is 40  $m$long by 40  $m$wide, with a maximum height of 2  $m$. Cell size of the occupancy grids is 0.2x0.2  $m$.

The Lexus experimental platform provides to acquire sensor data in real traffic environments: eight layers of laser scans, stereo images, IMU data (accelerations), velocity, GPS position, steering angle. The experiments are conducted in various road environements (country roads, downtown and highway), at different time of the day, with various driving situations (light traffic, dense traffic, traffic jams). The datasets are acquired online and are used for testing of our sensor fusion and risk assessment algorithms.

This is the follow up of the research activity started in 2009, when a self-calibration problem for a wheeled robot has been investigated. The main results achieved during that year were published in , and This calibration problem allows us to introduce a general framework able to deal with any estimation problem. This framework is based on a new theoretical concept, the concept of continuous symmetry. Detecting the continuous symmetries of a given system has a very practical importance. It allows us to detect an observable state whose components are non linear functions of the original non observable state. The general theory has been developed during the last two years. Preliminary results have been published in 2010 and a more complete version of these results, which include several extensions, has been published on Transaction on Robotics, in 2011 .

In 2011, this general framework has been extensively applied to investigate the problem of the fusion of visual and inertial data in the framework of the European project sFly. Special emphasis has been devoted to the structure from motion problem (SfM) when fusing these data. This problem has particular interest and has been investigated by many disciplines, both in the framework of computer science ( , , , and references therein) and in the framework of neuroscience and vision perception ( , and references therein). Even though prior work has answered the question of which are the observable modes, i.e. the states that can be determined by fusing visual and inertial measurements , , , the questions of how to compute these states in the absence of a prior, and of how many solutions are possible, were still unanswered. During 2011, we have derived, for the first time, a closed form solution to the SfM problem in this case, allowing the determination of the observable modes without the need for any prior knowledge. The proposed solution analytically expresses all the observable modes in terms of the visual and inertial measurements acquired during a given (short) time-interval allowing the determination of all the observable modes without the need for any prior knowledge. Additionally, we have shown that this problem can have a unique solution or two distinct solutions or infinite solutions depending on the trajectory, on the number of point-features and on the number of monocular images where the same point-features are seen. Our results are relevant in all the applications which need to solve the structure from motion problem with low-cost sensors and which do not demand any infrastructure. Typical examples are the emergent fields of space robotics , humanoid robotics and unmanned aerial navigation in urban-like environments , where the use of the GPS is often forbidden. Furthermore, our results could play an important role in neuroscience by providing a new insight on the process of vestibular and visual integration. To this regard, we remind the reader that the influence of extra retinal cues in depth perception has extensively been investigated in the last decades. In the case when this extra retinal cue is the motion parallax induced by self-motion relative to a stationary environment, the scale factor is provided by the head velocity , . The vast majority of these studies, consider the case when the head motion is active , . This prevents the possibility to understand the contribution of the vestibular signals because of efference copy generated by active self movement. However, a very recent study investigates this problem by performing trials with passive head movements . The conclusion of this study is that the combination of retinal image with vestibular signals can provide rudimentary ability to depth perception. Our findings could provide a new insight to this problem of depth perception since by combining retinal image with vestibular signals it is possible to determine the scale factor even without any knowledge about the initial speed. New trials would be necessary in order to verify whether a mechanism reproducing our closed form solution is present in humans and/or in other animals (especially the ones without binocular vision). Our findings also show that it is possible to easily distinguish linear acceleration from gravity. Specifically, our closed form solution perform this determination by a very simple matrix inversion. This problem has also been considered in neuroscience , . Our results could provide a new insight to this problem since they clearly characterize the conditions (type of motion, features layout) under which this determination can be performed.

Our results have been published in three conference papers , , and have been accepted for publication in transactions on robotics (a version is currently available as a technical report, ).

In parallel to this theoretical activity an experimental activity has started in order to experimentally validate our findings in the near future and to deploy our technologies to industrial partners. To this regard, a contact with the company Delta Drone in Grenoble has been established and a valorization contract with a SME in the field of civil drone applications is currently in preparation.

The grid-based environment representation is used for dynamic scene analysis in the Arosdyn project . The original idea behind the risk estimation approach developed in the e-Motion team consists in considering the possible behaviors of the vehicles in the scene. Indeed, with the classical TTC(time to collision)-based approach, the risk is estimated based on the prediction of the trajectory, considering the current state of the objects. This is only valid for very short term predictions, and in some cases it can result in a over-estimation of the collision risk. Understanding the intention of the other participants of the road scene allows a longer term, more precise prediction of trajectories.

Our approach is divided into two steps: behavior recognition and behavior realization. The behavior recognition aims at estimating the probability for a vehicle to perform one of its feasible behaviors. The behaviors are semantic representations of driving maneuvers (e.g. turn left, turn right, go straight, ...). The probability distribution over possible behaviors is obtained by inference using layered HMMs. Driving behavior realization is modeled as Gaussian Process (GP). This model allows us to obtain the probability distribution over the physical realization of the vehicle motion (i.e. trajectories) by assuming a usual driving, for a given behavior.

Finally, a complete probabilistic model of the possible future motion of the vehicle is given by the probability distribution over driving behaviors, and by the realization of these behaviors. The risk calculation is performed by sampling of the paths from the corresponding GP. The fraction of the samples in collision gives the risk of collision.

In 2011, we conducted some early experiments on sensor fusion, using real data acquired with our Lexus experimental vehicle . Moreover, the global framework of the Arosdyn project has been presented in .

We developed a generic object class recognition method. The method uses local image features and follows the part based detection approach. The state-of-the-art visual object class recognition systems operate with local descriptors and codebook representation of the objects. Various local features (e.g. gradient maps, edges) are used to create the descriptors. Then kernel based classifiers are commonly employed to classify the detected features in one of several object classes . The recognition of vehicles or pedestrians from sensors mounted on a moving platform is achieved by different approaches using various types of sensors, e.g. stereo camera, laser  . The approaches that perform data fusion from various sensors have proven to be the more robust in a variety of road conditions .

This work focuses on the development of an object class recognition system which follows the part based detection approach  . The system fuses intensity and depth information in a probabilistic framework. To train the system for a specific object class, a database of annotated with bounding boxes images of the class objects is required. Therefore, extending the system to recognize different object classes is straightforward. We apply our method to the problem of detecting vehicles by means of on-board sensors. Initially, depth information is used to find regions of interest. Additionally, the depth of each local feature is used to weight its contribution to the posterior of the object position in the corresponding scale. The votes are then accumulated in a 3d space-scale space and the possible detections are the local maxima in that space. Figure presents the steps of our approach.

The novelty of our approach is the fusion of depth and intensity information to form a probabilistic part-based detector. Using depth information is beneficial for the robustness of the approach, because we avoid including many noisy detections resulting from false matches between features of different scales. The method is tested with stereo video sequences captured in an urban environment. Fig. shows some example detections. The proposed method detects side-views of cars in various scales, in cases with partial occlusions, and under significant background clutter.

The work developed in this PhD is done in collaboration with Renault (CIFRE thesis) and concerns safety applications for cooperative vehicles.

In a few years, car manufacturers will start equipping vehicles with V2X communication devices, which will allow vehicles to share information with other vehicles and with roadside units using a dedicated communication channel. This new sensor on the car opens a whole new world of possibilities for Advanced Driver Assistance Systems (ADAS). In particular, the fact that the vehicle is able to “see” a car before it even enters the field-of-view of the driver allows for a better assistance in the tasks of perceiving, analyzing, predicting, and estimating the risk of a situation.

Early in the PhD we identified safety applications at road intersections as a relevant application domain for V2X technologies. The variety and complexity of scenes at road intersections makes reasoning and interpretation particularly difficult. On the other hand, intersections are a location of many accidents (they represent up to 50% of accidents in some countries), therefore reducing the accident rates in these areas would have a considerable impact of global traffic safety. We also identified the key issues (and challenges) to be 1) situation understanding and 2) risk assessment, to be carried out from incomplete models and uncertain data.

The focus of the year 2010 was on the first of these two problems. We developed a Bayesian Network that could estimate a driver's intended exit lane at an intersection based on the current state of the vehicle (position, orientation, turn signal state) and on contextual information extracted from the digital map. The idea was to use the information on the geometry of the road network and on the connectivity between lanes to build a statistical model of the relationship between the position and turn signal of a vehicle and the driver's intended exit lane. Initial results of this work were published in IEEE CIVTS'11 , then in IEEE IV'11 with a more thorough evaluation.

The objective of the work conducted in 2011 was twofold:

We proposed a probabilistic motion model for vehicles approaching and traversing an intersection that incorporates some knowledge about how the context (i.e. the traffic rules, the presence of other vehicles, the geometry and topology of the intersection) influences vehicle behavior. The distinctive features of our algorithm are:

This reasoning is implemented using a Bayesian filter which estimates the hidden variables M (maneuver intention), D (distance to intersection), H (intention to stop) and H' (necessity to halt) jointly for all the vehicles in the scene, using the position, speed and heading information shared between the vehicles via V2X communication. Inference on the hidden variables is carried out by a particle filter. The algorithm was described in an INRIA research report . In this report we showed by reasoning on theoretical scenarios that our assumption that drivers tend to respect priority rules should lead to improved situation assessment and risk assessment (see Fig.  ).

Recently, data has been collected at an intersection using the Renault demonstrator vehicles, so that our algorithm can be tested on real data. Preliminary results seem to confirm that the intuitions described in the research report were correct. A Graphical User Interface is in the process of being developed so that demonstrations of the system can be carried out live in the Renault demonstrator vehicles (see Fig.  ).

Navigation in large dynamic spaces has been adressed often using deterministic representations, fast updating and reactive avoidance strategies. However, probabilistic representations are much more informative and their use in mapping and prediction methods improves the quality of obtained results.

Since 2008 we have proposed a new concept to integrate a probabilistic collision risk function linking planning and navigation methods with the perception and the prediction of the dynamic environments . Moving obstacles are supposed to move along typical motion patterns represented by Gaussian Processes or Growing HMM. The likelihood of the obstacles' future trajectory and the probability of occupation are used to compute the risk of collision. The proposed planning algorithm, call Risk-RRT, is a sampling-based partial planner guided by the risk of collision. Results concerning this work were published in .

In 2011, our algorithms were integrated into an embedded sofware architecture for social aware navigation (see fig. ). For this purpose we started to migrate our algoritmhs to a new experimental plateform. Moreover, we adapted the code to the open source software called ROS (Robot Operating systems Willow Garage Inc., http:// www. ros. org) which offers tools to develop robot applications based in state of the art algorithms. Particularly, localization and visualization tools have been used. We have linked the control of our robotic wheelchair, the Risk-RRT planning and the social filter modules described in into the framework ROS as shown in figure . The main objective was to increase the visibility of our approach and develop common libraries with research groups in robotics. In 2011, in the scope of the AEN PAL project, we started a collaboration with the EPI Arobas and complementary developements have been put on the INRIA forge.

Next two sections are conducted under the french project PAL “Personally Assisted Living” with a goal to enhance the quality of living by providing more autonomy in the daily activities of the disabled.

The objectives of this work are to integrate the notion of comfortin the classical safe navigationmethods. If one consider that the navigation system transports a person, the integration of social conventions in the navigation strategy starts to be crucial. In this work, we propose to integrate the notions of personal space and interaction between people. We propose to enrich the knowlegde the robot has, with a representation of the social conventions. The robot must take into consideration interactions to avoid groups of people (even if passing through the group is the “best ”path for a conventionnal planning algorithm), or to join a group with a behavior close to the one of a human. To understand the behaviors of interaction between humans and the management of space, the works developed in the area of sociology to define some concepts as Personal space, o-spaceand F-formationsare used.

This is a useful tool for a robot to understand the intentions of the humans. It is well known that these measures are not stricts and that they change depending on age, culture and type of relationship but the categories proposed explain very well reactions like the uncomfortable sense of a stranger invading your intimate zone or the perception of somebody looking social interaction because he is entering to your social zone.

The first stage in order to achieve an integration of social concepts with robot navigation was to include estimations of the risk of disturbing personal space and interaction space in the general risk estimation. A strategy to detect interactions in the environment based in the velocity, position and orientation of humans was implemented.

In fig. we observe the results of the proposed integration, the robot (green rectangle) can use the detections of conversations (light ellipses) between humans (blue circles) for add more risk to paths that invade the space of conversations. When a conversation is detected, a bi-dimensional Gaussian $G$is created to represent the interaction space, also called o-space, the center of this space is approximated by taking into account the the participants' poses. Then, $G$is used to obtain an estimation of risk of disturbing by passing around the conversation. The navigation strategy is based on the Risk-RRT algorithm. Details of this approach were published in .

The elderly and the disabled are expected to benefit from the new technologies in the field of autonomous navigation robotics. Normal users of electric wheelchairs will also benefit from the development of more automatic functionalities bringing an extra driving comfort, especially during delicate maneuvers such as narrow door passages. This contribution is similar to the installation of driving assistance on a car. A simple improvement of the classical powered wheelchair can often diminish several difficulties of control.

Comfort defined as a state of ease and satisfaction of bodily wants, with freedom from pain and anxiety, has recently emerged as a design goal in autonomous navigation systems. Designers are becoming more aware of the importance of the user when scheming solution algorithms. The idea of comfort is especially important in the case of wheelchairs where the occupants are weak as result of their age or disease.

For any robot that is designed to transport people, the trajectory should be smooth and correspond to the user’s understanding as much as possible. Since human interpretation of the environment often differs from a robot’s interpretation, the decisions taken by the system might seem incomprehensible to a human observer. For example an autonomous vehicle could refuse to move forward due to some obstacle, while a human user would easily be able to move its way through. This undesirable behaviors may prove irritating and with time may lead to users stopping from using the system.

In 2011 we setup a robotic wheelchair as a trial platform. The wheelchair is a differential drive robot equiped with a SICK LMS-200 lidar to get 2D range information from the environment, odometry sensors, and a velocity controller we have also added a kinect sensor in order to perform some in the field of social interactions. Some basic functions can be executed including the mapping of the environment using a Rao-Blackwellized Particle Filter , localization using an Adaptive Monte Carlo Localization approach (AMCL) , global planning using an A* algorithm and local reactive planning using the Dynamic Window Algorithm .

Alongside we started working with the kinect sensor to detect and track people. Using the given tracking information, the wheelchair is able to follow a human located in front of it. This behavior is aimed to bring assistance not only to the user but also to the caregiver by allowing him to move without pushing the wheelchair. The technical implementation of the related approaches has been done on the basis of the ROS middleware due to easy integration with other opensourse robotics software which benefit sharing and testing developed software.

In 2012 we shall focus on the estimation of the user intentions by learning models of behavior. We'll then use these models to propose an adaptive autonomous navigation method that best answer the user needs.

This research has been carried out in the framework of the European project sFly. In recent years it is revealed more and more the importance of using multi-robot systems for security application, otherwise impossible to be performed by a single robot.

The main problem approached is the optimal surveillance coverage of an unknown and complex environment, i.e. finding the optimal deployment for the robots and the way to safely reach such configuration. The solution for the 2D case without obstacles is already known in literature . On the other hand, for the non-convex case, it is still a difficult problem. In we firstly proposed a possible strategy based on a combination of the repulsive potential field method and the Voronoi partition. Then, in the last two years we have mainly approached the coverage problem by using a new stochastic optimization method. This work is in collaboration with professor Elias Kosmatopoulos, from CERTH (Thessaloniki), and professor Lefteris Doitsidis, from TUC (Crete), partners in the sFly project.

The Kosmatopoulos's group has proposed a new adaptive stochastic optimization algorithm for a general class of multi-robot passive and active sensing applications  , . This method possesses the capability of being able to efficiently handle optimization problems for which an analytical form of the function to be optimized is unknown, but the function is available for measurement at each iteration of the algorithm employed to optimize it. As a result, it perfectly suits for multi-robot optimal coverage in non-convex environments, where the analytical form of the function to be optimized is unknown but the function is available for measurement (through the robots' sensors) for each multi-robot configuration.

The main results obtained for the 2D case by using this method has been published in , . We assume the robots are equipped with global positioning capabilities and visual sensors able to monitor the surrounding environment. The goal is to maximize the area monitored by the team, by identifying the best configuration of the team members. Moreover, in 2011, a distributed version of the algorithm was presented in . In multi-robot systems, a distributed approach is desirable for several fundamental reasons. The most important are failure of the central station and limited communication capabilities. The proposed approach has the following key advantages with respect to previous works:

The previous approach has been also extended for the more important and realistic 3D case. Working in collaboration with the ETHZ (Zurich), some simulations using real data, which were collected with the use of a miniature quadrotor helicopter specially designed for the needs of the European project sFly, have been performed (see fig. ). This work has lead to two joint publications with CERTH and TUC: one conference paper to present (CDC2011) and one journal papers under review, and two joint publications with CERTH, TUC and ETHZ: one conference paper ( ) and one journal papers under review.

In 2011, this approach has been combined with human aware navigation technics presented in section   .

In the next months, the algorithm will be implemented on real MAVs for the final demo of the project. This demo will include experimentation both in indoor and outdoor complex environments.

Finally, a new collaboration with professor Kosmatopoulos has recently begun. The objective of this work is to develop a new efficient and scalable algorithm for multi-robot active control to perform cooperative simultaneous localization and mapping (CSLAM) and target tracking. The main idea is to use a convex optimization algorithm based on Semi-Definite Programming and Sum-of-Squares polynomials. Preliminary simulation results are very promising and a journal paper is under preparation.

Suppose that we have a robot navigating in an unknown and complex environment where people are moving and interacting. In such scenario the respect of the humans' comfort becomes an important goal to achieve. The discomfort concept could be very general but we focus on the one mentioned before, i.e., the discomfort caused by disturbing one interaction or a personal space of humans. The approach here is to minimize the discomfort while the robot is navigating. As we cannot measure directly the value of discomfort, we can infer it by modeling the concepts presented before using simple equations and after by applying a method of optimization. We propose to exploit a new stochastic and adaptive optimization algorithm (CAO) . This method is very useful in particular when the analytical expression of the optimization function is unknown but numerical values are available for any state configuration. Furthermore, the proposed method can easily incorporate any dynamical and environmental constraints. To validate the performance of the proposed solution, several simulation results are provided.

In fig. the model for discomfort function is shown together with robot navigation. At each step the robot randomly generate configurations in the environment and selects the one that takes it closer to the goal while minimizing values for the discomfort function of humans in the environment, this is repeated until goal is reached. Several executions of proposed approach in different scenarios can be observed in fig.

The details of this approach have been submitted to ICRA2012.

The problem addressed in this work is the autonomous replacement of a human player. It is the continuation of last year's work on the same topic as well as a follow-up of previous E-Motion Ph.D Ronan Le Hy . This year, we focused on real-time strategy (RTS) games, in which the players have to build an economy, advance technology, produce and control an army to kill the opponents. From a research point of view, multi-player games are interesting because they stand for a good in-between of the real world and simulations. The world is finite and simulated (no sensors problems) but we didn't wrote the simulation and the other players are humans (or advanced robots in the case of AI competitions).

This year's research work focused on plan recognition from noisy and incomplete observations. Previous plan recognition works in multiplayer games were mainly based on planning and case-based reasoning (CBR) , , , or HMMs . CBR allows for taking domain knowledge into account easily while not dealing efficiently with uncertainty/incompleteness of information, HMMs deal with uncertainty quite well but domain knowledge is harder to structure. We found different ways to decompose the joint $P(Observatio{n}_{1:N},Pla{n}_{1:M})$which allows for tractable and robust inference. For instance with the help of intermediate variables which can be derived from domain knowledge (as we did) or found automatically (e.g. cross-validation on a HMM). Particularly, we were able to structure dependencies between domain knowledge extracted variables using coherence variables. We then learn the parameters of such joint distributions from data. Supervised (labeled), and semi-supervised learning (when we label automatically from clustering) have led to a publication at CIG (IEEE) 2011 and unsupervised learning (using only raw game data) led to a publication at AIIDE (AAAI) 2011 .

On top of the research/evaluation implementation, we also implemented it in our StarCraft: Broodwar's bot implementation BroodwarBotQ. With this bot, we took part in AIIDE and CIG conferences AI tournaments placing respectively 9th (out of 18) and 4th (out of 10). We also published last year's result on multiple units control in real-time engagements (see ) at CIG (IEEE) 2011 . As optimal micro-management is almost always intractable (P-space) in real situations, we considered each unit as a Bayesian sensory motor robot which makes a fusion of its sensory inputs about the world, the enemy units, but also its allies (without explicit communication for less complexity) and higher level directions. So the units only take short term decision on where to go and who to attack, higher level planning is done at a squad (and then army) level and given as a sensory input. Results in micro-management tournaments are state of the art. In the more general case, they could be improved by reinforcement learning of the models parameters.

We are now working on concurrent goals resources attribution, still in the context of incomplete knowledge about the opponent. We are also working on correlating low-level observations (effects) and high-level inferences (causes) about the enemy strategy to be able to predict its future behavior.

A central issue in speech science concerns the nature of representations and processes involved in communication. The search for phoneme or syllable specific invariants led to three major sets of approaches: motor, auditory and perceptuo-motor theories, which have been widely argued for and against. The debate appears to be stagnating. This work is based on the belief that mathematical modeling of these theories could provide breakthroughs. More precisely, it is proposed that casting these theories into a single, unified mathematical framework would be the most efficient way of comparing the theories and their properties in a systematic manner.

Bayesian modeling provides a mathematical framework that precisely allows such comparisons. The same tool, namely probabilities, can be used both for defining the models and for comparing them. Moreover, the use of a unified framework implies that common hypotheses would have common mathematical translations. This helps toward more principled studies of the competing theories.

Following this integrative approach, the motor, auditory and perceptuo-motor theories are thus cast into one unifying Bayesian framework in which they all appear as instances of various questions asked to one probabilistic communication model. This allows to compare these theories through quantitative testing in various paradigms. The work is aimed at understanding the differences in the predictions given by the different theories, and from these predictions to suggest experiments involving human subjects.

The model was used first to work on purely theoretical simulations aimed at studying with diverse paradigms the decrease in the performances predicted by the different theories due to communication noise. It was then used to work on plosive syllables production and perception, thanks to VLAM, a vocal tract simulation tool, which allows to map articulatory parameters to acoustic signals.