The project of the Auctus team is to design the collaborative robotics cells of the future.
The robotics community still tends to separate the cognitive (HRI) and physical (pHRI) aspects of human/robot interaction. One of the main challenges is to characterize the task as well as biomechanical, physiological and cognitive capacities of humans in the form of physical constraints or objectives for the design of cobotized workstations. This design is understood in a large sense: the choice of the robot's architecture (cobot, exoskeleton, etc.), the dimensional design (human/robot workspace, trajectory calculation, etc.), the coupling mode (comanipulation, teleoperation, etc.) and control. The approach then requires the contributions of the human and social sciences to be considered in the same way as those of exact sciences. The topics considered are broad, ranging from cognitive sciences, ergonomics, human factors, biomechanics and robotics.
The first challenge is to evaluate the hardship at work, the well-being of the operators and, further upstream, their cognitive state which impacts their sensorimotor strategy for performing a task. In industry, the ergonomic analysis of the task is carried out by an ergonomist based on direct but often ad hoc observations. However, the context is changing: the digitization of factories, through the installation of on-site sensors, allows longitudinal observation of machines and humans. The information available can thus allow us to rethink the way in which the evaluation of activities is carried out. Currently, an emerging subdomain named ergonomic robotics adapts the available ergonomic evaluation criteria (RULA, REBA, etc.). However, they are related to the (quasi-static) posture of the operator, which limits the understanding of human motor strategies over a long period of time. Similarly, kinematic or biomechanical analysis may tend to see humans as a high-performance machine to be optimized. This may make sense for a top-level athlete, but repeating actions in the industry over a day, months or years of work means that a temporary change of posture, possibly poorly rated according to usual ergonomic criteria, can in fact be a good long-term strategy. These questions make a direct link between motor and cognitive aspects that can be reflected in particular strategies as the fatigue or the expertise (manual and cognitive). This approach has not been widely explored in robotics to determine the right criteria to adapt the behavior of a cobot.
The second challenge is to define a methodology to link the analysis of the task and the human movements it induces to the robot design. Indeed, as we have been able to verify on several occasions in the context of industrial projects, between the ergonomist, expert in task analysis and psychology, and the robotician, expert in mechanics, control and computer science, there is a significant conceptual distance that makes it very difficult to analyze needs and define the specifications of the technical solution. To fill these methodological gaps, it is necessary, on the basis of case studies, to better define the notion of tasks in the context of a human/robot coupling and to establish a typology of this type of interaction by taking into account, with as much details as possible, the different physical and cognitive constraints and their potential psychological, organizational or ethical impacts.
The third challenge is related to the need to think about the control laws of collaborative robots in terms of human/robot coupling. The effectiveness of this coupling requires an ability to predict future human actions. This prediction should make the interaction more intuitive but also aims at an optimal coupling from the point of view of “slow” phenomena such as fatigue. The major challenge is therefore to move from reactive to predictive control laws, integrating a human prediction model, both in terms of movement strategies and decision strategies. Beyond the great computational complexity of predictive approaches, obtaining prediction models is an ambitious challenge. It is indeed necessary to learn models that are quite complex in terms of the physical realities they can account for and quite simple from a computational point of view.
The purpose of this axis is to provide metrics to assess human behavior. We place ourselves here from the point of view of the human being and more precisely of the industrial operator. We assume the following working hypotheses: the operator's task and environmental conditions are known and circumscribed; the operator is trained in the task, production tools and safety instructions; the task is repeated with more or less frequent intervals. We focus our proposals on assessing:
In the industrial context, the fields that best answer these questions are work ergonomics and cognitive sciences. Scientists typically work on 4 axes: physiological/biomechanical, cognitive, psychological and sociological. More specifically, we focus on biomechanical, cognitive and psychological aspects, as described by the ANACT 18, 19. The aim here is to translate these factors into metrics, optimality criteria or constraints in order to implement them in our methodologies for analysis, design and control of the collaborative robot.
To understand our desired contributions in robotics, we must review the current state of ergonomic workstation evaluation, particularly at the biomechanical level. The ergonomist evaluates the gesture through the observation of workstations and, generally, through questionnaires. This requires long periods of field observation, followed by analyses based on ergonomic grids (e.g. RULA 35, REBA 25, LUBA 30, OWAS 29, ROSA 51,...). Until then, the use of more complex measurement systems was reserved for laboratories, particularly biomechanical laboratories. The appearance of inexpensive sensors such as IMUs (Inertial Measurement Units) or RGB-D cameras makes it possible to consider a digitalized, and therefore objective, observation of the gesture, postures and more generally of human movement. Thanks to these sensors, which are more or less intrusive, it is now possible to permanently install observation systems on production lines. This completely changes paradigms and opens the door to longitudinal observations. It should be noted that this is comparable to the evolution of maintenance, which becomes predictive.
On the strength of this new paradigm, ergonomic robotics has recently taken an interest in this type of evaluation to adapt the robot's movements in order to reduce ergonomic risk scores. This approach complements the more traditional approaches that only consider the performance of the action produced by the human in interaction with the robot. However, we must go further. Indeed, the ergonomic criteria are based on the principle that the comfort positions are distant from the human joints articular limits. In addition, the notation must be compatible with an observation of the human being through the eye of the ergonomist. In practice, evaluations are inaccurate and subjective 54. Moreover, they are made for quasi-static human positions without taking into account the evolution of the person's physical, physiological and psychological state. The repetition of gestures, the solicitation of muscles and joints is one of the questions that must complete these analyses. One of the methods used by ergonomists to limit biomechanical exposures is to increase variations in motor stress by rotating tasks 52. However, this type of extrinsic method is not always possible in the industrial context 33.
One of Auctus' objectives is to show how, through a cobot, the operator's environment can be varied to encourage more appropriate motor strategies. To do so, we must focus on a field of biomechanics that studies the intrinsic variability of the motor system allowed by the joint redundancy of the human body. This motor variability refers to the natural alternation of postures, movements and muscle activity observed in the individual to respond to a requested task 52. This natural variation leads to differences between the motor coordinates used by individuals, which evokes the notion of motor strategy 26.
As shown by the cognitive dimension of ergonomics (see above), we believe that some of these motor strategies are a physically quantifiable reflection of the operator's cognitive state. For example, fatigue 48 and its anticipation or the manual expertise (dexterous and cognitive) of the operator which allows him to anticipate his movements over long periods of time in order to preserve his body, his performance and his pain.
How can we observe, understand and quantify these human motor strategies to better design and control the behavior of the cobotic assistant? When we study the systems of equations considered (kinematic, static, dynamic, musculoskeletal), several problems appear and explain our methodological choices:
The idea is to start from a description of redundant workspaces (geometric, static, dynamic...). To do this, we use set theory approaches, based on interval analysis 3, 41, which allow us to respond to the uncertainties and variability issues previously mentioned. In addition, one of the advantages of these techniques is that they allow the results to be certified, which is essential to address safety issues. Some members of the team has already achieved success in mechanical design for performance certification and robot design 37. The adaptation of these approaches allows us to obtain a mapping of ergonomic and efficient movements in which we can project the operators' motor strategies and thus define a metric quantifying the sensorimotor commands chosen with regard to the cognitive criteria studied.
It is therefore necessary to:
The originality and contribution of the methodology is to allow an analysis taking into account in the same model the measurement uncertainties (important for on-site use of analytical equipment), the variability of tasks and trajectories, and the physiological characteristics of the operators.
Other avenues of research are being explored, particularly around the inverse optimal control 42 which allows us to project human movement on the basis of performance indices and thus to offer a possible interpretation in the analysis of behaviors.
We also use automatic classification techniques: 1) to propose cognitive models that will be learned experimentally 2) for segmentation or motion recognition, for example by testing Reservoir Computing 27 approaches.
Thanks to the progress made in recent years in the field of p-HRI (Physical Human-Robot Interaction), robotic systems are beginning to operate in the same workspace as humans, which is profoundly changing industrial issues and allowing a wide variety of human-robot coupling solutions to be considered to perform the same task 20. Different types of interactions exist. They can be classified in different ways: according to the degree of autonomy of the robot and its proximity to the user 24 with particularities for “wearable robots” 22, 23, or for collaborative robotics 53, or according to the role of the human being 49. From a cognitive point of view, classifications are more concerned with autonomy, the complexity of information processing and the type of communication and representation of the human being by the robot 40, 55.
We proposed a classification of cobotic systems according to the configuration of the schema of interactions between humans, robots and the environment 38, 47.
The parameters of the coupling being numerous and complex, the determination of the most appropriate type of coupling for a type of problem is an open problem 43, 2, 39, 34. The traditional approach consists in trying to identify and classify the various possible options and to select the one that seems most relevant with regard to the feasibility, efficiency, budget envelope and acceptability of the operator. One of the main objectives of our research project is to define a typology of cobots or cobotic systems in order to specify the methodology for developing the best solution: what are the criteria for defining the best robotic architecture, what type of coupling, what autonomy of the robot, what role for the operator, what risks for the human, what overall performance? These are the key issues that need to be addressed. To meet this methodological need, we propose an approach guided by experience on use cases obtained thanks to our industrial partners.
Task analysis and human behavior modelling, discussed in the previous sections, should help to characterize the different types of coupling and interaction modalities, their advantages and disadvantages, in order to assist in the decision-making process. One of the ideas we would like to develop is to try to break down the task into a sequence of elementary gestures corresponding to simple motor actions performed in a clearly identified context and to evaluate for each of them the degree of feasibility in automatic mode or in robot assistance mode. The assessment must take into account a large number of parameters that relate to physical interactions, human-robot communication, reliability and human factors, including acceptability and impact on the valuation or devaluation of the operator's work. Concerning the evaluation of human factors, we have already begun to work on the subject within the more general framework of human systems interactions by operating Bayesian networks, drawing inspiration from the work of 21, 46.
The adoption of assessment criteria for a single domain (e. g. robotics or ergonomics) cannot guarantee that the performance of this coupling will be maximized. From design to evaluation, cross-effects must be constantly considered:
In addition to purely physical assistance, some cobotic systems are designed to assist the operator in his decision-making. The issues of trust, acceptance, sharing of representations and co-construction of a shared awareness of the situation are then to be addressed 50.
Is it necessary to cobotize, robotize or assist the human being? Which mechanical architecture meets the task challenges (a serial cobot, a specific mechanism, an exoskeleton)? What type of interaction (H/R cohabitation, comanipulation, teleoperation)? These questions are the first requests from our industrial partners. For the moment, we have few comprehensive methodological answers to provide them. Choosing a collaborative robot architecture is a difficult problem 31. It is all the more when the questions are approached from both a cognitive ergonomics, boimechanical and robotics perspectives. There are indeed major methodological and conceptual differences in these areas. It is therefore necessary to bridge these representational gaps and to propose an approach that takes into consideration the expectations of the robotician to model and formalize the general properties of a cobotic system as well as those of the ergonomist to define the expectations in terms of an assistance tool.
To do this, we propose a user-centered design approach, with a particular focus on human-system interactions. From a methodological point of view, this requires first of all the development of a structured experimental approach aimed at characterizing the task to be carried out through a “system” analysis but also at capturing the physical markers of its realization: movements and efforts required, ergonomic stress. This characterization must be done through the prism of the systematic study of the exchange of information (and their nature) by humans in their performance of the considered task. On the basis of these analyses, the main challenge is to define a decision support tool for the choice of the robotic architecture and for the specifications of the role assigned to the robot and the operator as well as their interactions.
The evolution of the chosen methodology is for the moment empirical, based on the user cases regularly treated in the team (see sections on contracts and partnerships).
It can be summarized for the moment as:
In addition, the different human and task analyses take advantage of the different expertise available within the team. We would like to gradually introduce the evaluation criteria presented above. Indeed, the team has already worked on the current dominant approach: the use of a virtual human to design the cobotic cell through virtual tools 1. However, the very large dimensions of the problems treated (modelling of the body's dofs and the constraints applied to it) makes it difficult to carry out a certified analysis. We then choose to go through the calculation of the body's workspace, representing its different performances, which is not yet done in this field. The idea here is to apply set theory approaches, using interval analysis already discussed in section 3.1.2. The goal is then to extend to intervals the constraints played in virtual reality during the simulation. This would allow the operator to check his trajectories and scenarios not only for a single case study but also for sets of cases. For example, it can be verified that, regardless of the bounded sets of simulated operator physiologies, the physical constraints of a simulated trajectory are not violated. Thus, the assisted design tools certify cases of use as a whole. Moreover, the intersection between the human and robot workspaces provides the necessary constraints to certify the feasibility of a task. This allows us to better design a cobotic system to integrate physical constraints. In the same way, we will look for ways in which human cognitive markers can be included in this approach.
Thus, we summarize here the contributions of the other research axes, from the analysis of human behavior in its environment for an identified task, to the choice of a mechanical architecture, via an evaluation of torque and interactions. All the previous analyses provide design constraints. This methodological approach is perfectly integrated into an Appropriate Design approach used for the dimensional design of robots, again based on interval analysis. Indeed, to the desired performance of the human-robot couple in relation to a task, it is sufficient to add the constraints limiting the difficulty of the operator's gesture as described above. The challenges are then the change of scale in models that symbiotically consider the human-robot pair, the uncertain, flexible and uncontrollable nature of human behavior and the many evaluation indices needed to describe them.
The control laws of collaborative robots from the major robot manufacturers differ little or not at all from the existing control laws in the field of conventional industrial robotics. Security is managed a posteriori, as an exception, by a security PLC / PC. It is therefore not an intrinsic property of the controller. This quite strongly restricts the possibilities of physical interaction 1 and collaboration and leads to sub-optimal operation of the robotic system. It is difficult in this context to envision real human-robot collaboration. Collaborative operation requires, in this case, a control calculation that integrates safety and ergonomics as a priori constraints.
The control of truly collaborative robots in an industrial context is, from our point of view, underpinned by two main issues. The first is related to the macroscopic adaptation of the robot's behaviour according to the phases of the production process. The second is related to the fine adaptation of the degree and/or nature of the robot's assistance according to the ergonomic state of the operator. If this second problem is part of a historical dynamic in robotics that consists in placing safety constraints, particularly those related to the presence of a human being, at the heart of the control problem 2436, 28, it is not approached from the more subtle point of view of ergonomics where the objective cannot be translated only in terms of human life or death, but rather in terms of long-term respect for their physical and mental integrity. Thus, the simple and progressive adoption by a human operator of the collaborative robot intended to assist him in his gesture requires a self-adaptation in the time of the command. This self-adaptation is a fairly new subject in the literature 44, 45.
At the macroscopic level, the task plan to be performed for a given industrial operation can be represented by a finite state machine. In order not to increase the human's cognitive load by explicitly asking him to manage transitions for the robot, we propose to develop a decision algorithm to ensure discrete transitions from one task (and the associated assistance mode) to another based on an online estimate of the current state of the human-robot couple. The associated scientific challenge requires establishing a link between the robot's involvement and a given working situation. To do so, we propose an incremental approach to learning this complex relationship. The first stage of this work will consist in identifying the general and relevant control variables to conduct this learning in an efficient and reusable way, regardless of the particular method of calculating the control action. Physically realistic simulations and real word experiments will be used to feed this learning process.
In order to handle mode transitions, we propose to explore the richness of the multi-tasking control formalism under constraints 32 in order to ensure a continuous transition from one control mode to another while guaranteeing compliance with a certain number of control constraints. Some of these constraints are based on ergonomic specifications and are dependent on the state of the robot and of the human operator, which, by nature, is difficult to predict accurately. We propose to exploit the interval analysis paradigm to efficiently formulate ergonomic constraints robust to the various existing uncertainties.
Purely discrete or reactive adaptation of the control law would make no sense given the slow dynamics of certain physiological phenomena such as fatigue. Thus, we propose to formulate the control problem as a predictive problem where the impact of the control decision at a time
The planned developments require both an approach to modelling human sensorimotor behaviour, particularly in terms of accommodating fatigue via motor variability, and validating related models in an experimental framework based on observation of movement and quantification of ergonomic performance. Experimental developments must also focus on the validation of proposed control approaches in concrete contexts. To begin with, the Woobot project related to gesture assistance for carpenters (Nassim Benhahib's thesis) and a collaboration currently being set up with Safran on assistance to operators in shrink-wrapping tasks (manual knotting) in aeronautics are rich enough background elements to support the research conducted. Collaborative research projects with PSA will also soon provide a larger set of contexts in which the proposed research can be validated.
The 4th industrial revolution (factory 4.0) is characterized by the integration of digital technologies into the production process, in order to meet the challenge of customizing services and products. This agility requires making manufacturing and maintenance lines flexible and versatile. This capacity for adaptation is the characteristic of the human being, which puts him at the center of the production apparatus. However, this can no longer be done at the expense of their health and well-being. How then can we reconcile the enhancement of our manual and analytical expertise, the ever desired increase in productivity and manufacturing quality, while reducing the hardship at work? Collaborative robotics, which we are seeking to build, is one of the central solutions to meet this societal challenge. By assisting humans in their most dangerous and painful tasks, it complements or replaces them in their phases of physical and cognitive fragility.
More generally, we are interested in workstation cobotization, in the manufacturing and assembly industry but also in the construction and craft industries. The application areas are related to regional needs in aeronautics, including maintenance, water and waste treatment. In most of these cases, it is possible to define the tasks, evaluate the stakes and added value of our work.
In january 2020, the start-up "Touch sensity" was created from researches carried out within the Auctus project team on innovative technologies of sensitive and connected materials (http://
In the framework of the WooBot project which aims at securing carpentry tasks by collaborative robots, Nassim Benhabib, Vincent Padois and David Daney received the 2020 IEEE ICRA Best Paper Award in Automation for their contribution entitled "Securing Industrial Operators with Collaborative Robots: Simulation and Experimental Validation for a Carpentry task" 10.
The paper of Jean-Marc Salotti (Salotti, JM. Minimum Number of Settlers for Survival on Another Planet. Sci Rep 10, 9700 (2020). https://
In october 2020, Nasser Rezzoug was elected at the board of directors of the "Société de Biomécanique". (https://
WoobotCtrl is a set of ROS packages that provide control of a collaborative robot with 7 degrees of freedom (Franka's PANDA) in order to reproduce the cutting forces experienced by an operator during a wood shaping task.
Developed in C++, WoobotCtrl also manages the real-time acquisition and synchronization of data from multiple sensors (Kinect, force sensors, robot sensors). As well as the pre-shrinking of the data.
A set of tools to control a robot with the OROCOS toolchain software.
Implemented tools: a library using KDL to modelize the robot , a wrapper for qpOASES , a Cartesian trajectory generator using KDL , a cartesian PID controller , a library to publish OROCOS port to ROS topics , a driver for the Panda Franka Emika robot , a Gazebo plugin , a set of tool to launch the scripts for the panda
Opensim is a motion modelling and simulation tool, mainly used in biomechanics. It addresses a number of issues related to the manipulation of biological bodies (calculation of muscular forces, precise modelling of musculoskeletal models based on experimental data, management of data from motion capture systems).
A number of robotics projects linking robots and humans could benefit from the functionalities of this tool for modelling the upper body of a human being in real time. This is why the objective of this project is to combine some of the functionalities of Opensim with ROS packages dedicated to the modelling, control and analysis of the movement of polyarticulated robotic systems (torque_qp, Pinocchio).
One of the paths currently being explored is to use OpenSim and ROS packages together for the same model of a human being. As the modeling format of Opensim models (.osim) is not the same as the one classically used by the ROS packages (.urdf), it is first necessary to convert the .osim format to .urdf. A library is being created for this purpose.
Once this library is completed, it will be possible to create the framework allowing the use of OpenSim functionalities to the packages used for the management of polyarticulated systems in real time, in particular for motion capture problems.
In 2020, Auctus has pursued his effort of developping its experimental platform of collaborative Robotics (Arcol).
The Arcol platform provides technical support for the short, medium and long term experimental developments carried out within the framework of Auctus' scientific and dissemination activities. These technological developments are essentially software related in the context of human motion capture and real-time control of collaborative robots. Arcol aims at easing their implementation, deployment, documentation and support.
New developpements in 2020 are mostly related to:
Within the framework of the continuation of the HARRY² project funded by the COVR initiative (H2020 program) and by Région Nouvelle Aquitaine, impact testing procedures have been evaluated and a report analysing the obtained results has been submitted as second Milestone of the project. This Milestone was validated by the COVR consortium and a promotional video summarizing the progress of the project has been released2.
The work related to the first milestone has been published in 12. This work is being extended to online trajectory generation in Cartesian space using a Model Predictive Control formulation. This formulation allows the definition of a trajectory taking into account the robot limits in Cartesian space over a time horizon. The objective is to combine this algorithm with interval analysis algorithms allowing to determine the robot capacities over a time horizon given its current configuration. These algorithms will take into account the robot physical capabilities i.e. joint limits, joint velocity limits, etc. and send back the Cartesian acceleration limits over a time horizon as a function of the current configuration. These limits can be directly inserted in the MPC problem. This will allow to generate trajectories online that respect the robot true capabilities. Additional constraints could then be added such as obstacle avoidance.
Related publications:
12
A robot low level control architecture requires a good handling of the real time communication between each components. The ROS middleware is not really adapted for such usage. Therefore, a new control architecture has been developed around the Orocos toolchain software. This control architecture allows to define trajectories in Cartesian space, compute the robot model, update it online, compute the desired command and send it to a Franka Emika Panda robot. It also interfaces with the Gazebo simulation environment. This architecture is modular and works with components. The communication between each component is strictly enforce and true real time synchronization is achieved.
Ellipsoid-based manipulability measures are often used to characterize the force/velocity task-space capabilities of robots. While computationally simple, this approach largely approximates and underestimates the true capabilities. Force/velocity polytopes appear to be a more appropriate representation to characterize robot's task-space capabilities. However, due to the computational complexity of the associated vertex search problem, the polytope approach is mostly restricted to offline use, e.g. as a tool aiding robot mechanical design, robot placement in work-space and offline trajectory planning.Therefore a new on-line polytope vertex search algorithm is proposed, exploiting the parallelotop geometry of actuator constraints. The proposed algorithm significantly reduces the complexity and computation time of the vertex search problem in comparison to commonly used algorithms. The algorithm is intended to be used with redundant serial robotic manipulators and opens many doors for using this force/velocity capacity polytopes for robot control and human-robot interaction. The algorithm execution time for 7DOF Franka Amika Panda and for 6DOF Universal Robots UR5 robots is under 3 and 2 milliseconds respectively. In order to highlight the on-line capability of the proposed algorithm and its potential for robot control, a challenging experiment with two collaborating Franka Emika Panda robots, carrying a load of 12 kilograms, was developed. In this experiment, the load distribution is adapted on-line, as a function of the configuration dependant task-space force capability of each robot, in order to avoid, as much as possible, the saturation of their capacity.
Related publications:
17
To obtain a safe and effective shared workspace between humans and robots, safety needs to be taken into account both at the design phase and at control algorithm. While concentrating in the control law, collaborative robot solutions ignore its capacity of detecting human presence and interaction such as interrupters that completely turn off the system.
The objective of this project is to work on these principles to avoid the classic restrained static security zones, synthesizing a dynamic representation of the shared workspace to take advantage of state of the art control laws, allowing a fluid collaboration between human-robot. This dynamic synthesis requires knowledge of the robots state (geometric, cinematic and cognitive, such as fatigue, expertise and conscience of the situation), its tasks, capacities, state of humans that surround it and their tasks. Furthermore, it needs to achieve a formal and provable online algorithm that correctly estimates the state of the human and guarantee a safe shared workspace tackling ambitious scientific questions poorly addressed in literature.
An industrial collaboration has been initiated with PSA Group on the synthesis and dynamic analysis of shared workspaces for safety in collaborative robotics. A CIFRE PhD thesis has been approved by ANRT and started on April 2020.
The first stage of the project is focused on the design of a reactive robot control scheme that adapts to its environment to ensure safety of surrounding human operators without sacrificing performance. The controller needs to account for the robot's dynamic braking capabilities and the actuator's constraints. Constructing on the previous work by AUCTUS 12, the current effort is directed towards improving the online estimation of the robot's capabilities to adapt its behaviour as operators get closer/farther to it.
In this work, a methodology for analyzing the gesture of an industrial operator is developed in order to design a collaborative robotics assistance adaptable to the needs of the worker. Wood milling is chosen as a pilot task due to its importance in carpentry.
In order to analyze the operator's gesture during wood shaping, a physical simulation of the task was reproduced. Where, a collaborative robot reproduces the cutting forces exerted on the operator.
Force, posture and motion sensors are used to measure the quantities of interest related to performance, safety and comfort criteria. We hypothesize that the evolution of these multiple criteria according to the operator's experience gives a clear indication of the operator's level of expertise.
In our opinion, the needs of an industrial operator vary according to his level of expertise. Identifying this level of expertise would allow us to design more efficient and acceptable robotic assistance.
Two quantifiable assessment criteria have been established for each aspect of task completion (performance, safety and comfort). By observing their evolution at different levels of experience for multiple users . We noticed that a trend was emerging. This helps to analyze the preferences of an operator and to observe which aspects of the task completion are favored as the expertise is acquired.
Related publications:
10
In the context of reducing the risk of musculoskeletal disorders (MSDs) in industry, the question of motor variability (VM) of industrial operators has to be raised. Indeed, VM could promote the distribution of effort and fatigue in the body through variations in motor strategies during movement. By optimising this VM, it may be possible to spare the muscles, joints and skeleton of the operators. A project, carried out through the PhD thesis of Raphaël Bousigues in collaboration between INRIA Bordeaux, INRIA Nancy and INRS, aims at verifying a set of hypotheses, such as, first of all, the fundamental existence of VM. Reproducing a repetitive manual task with a same operator allows to observe the kinematics of his arm and to quantify the presence and evolution of motor variability through the various repetitions of the task. The experimental protocol of this experiment is in the process of writing, but a simulation of a human arm performing a repetitive task has already been put into practice. Once the first experimental data will have been acquired, it will be possible to focus on the origin of VM and on the conditions influencing it. An experimental framework, entitled Mover, has been submitted to the COERLE for the experimetal work related to this work.
The estimation of human motion from sensors that can be used in an ecological environment is an important issue being it for home assistance for frail people or for human/robot interaction in industrial contexts. We are continuing our work on data fusion from RGB-D sensors using extended Kalman filters. The original approach uses a biomechanical model of the person to obtain anthropomorphically constrained joint angles to make their estimation physically coherent. In addition, we propose a method for the optimal adjustment of the covariance matrices of the extended Kalman filter. The proposed approach was tested with six healthy subjects performing 4 rehabilitation tasks. The accuracy of the joint estimates was evaluated with a reference stereophotogrammetric system. Our results show that an affordable RGB-D sensor can be used for simple home rehabilitation when using a constrained biomechanical model. This work has led to an accepted article in MDPI Sensors: Physically Consistent Whole-Body Kinematics Assessment Based on an RGB-D Sensor. Application to Simple Rehabilitation Exercises.
In a second step, we compared the joint centre estimates obtained with the new Kinect 3 (Azure Kinect) sensor, the Kinect 2 (Kinect for Windows) and a reference stereophotogrammetric system. Regardless of the system used, we have shown that our algorithm improves the body tracker data. This study also shows the importance of defining good heuristics to merge the data according to the body tracking operation. This study was accepted in International Conference on Activity and Behavior Computing and presented by Jessica Colombel in Kitakyushu, Japan as Markerless 3D Human Pose Tracking in the Wild with fusion of Multiple Depth Cameras: Comparative Experimental Study with Kinect 2 and 3. This article will also be a chapter book in Activity and Behavior Computing by Springer.
Related publications:
14
The aim of the work is to find ways of representing human movement in order to extract meaningful physical and cognitive information. After the realization of a state of the art on human movement, several methods are compared: principal component analysis (PCA), Fourier series decomposition and inverse optimal control. These methods are used on a signal comprising all the angles of a walking human being. PCA makes it possible to understand the correlations between the different angles during the trajectory. Fourier series decomposition methods are used for a harmonic analysis of the signal. Finally, inverse optimal control sets up a modeling of the engine control to highlight qualitative properties characteristic of the whole motion. These three methods are tested, combined and compared on data from the EWalk database3 in order to test emotion recognition based on these decompositions and simple classifiers. An in-depth study of the inverse optimal control method has been carried out in order to better understand its properties and to develop easier methods of resolution. This study led to a robust resolution of the inverse optimal control based on SVD and orthogonal projection.
In the Portage project, one of the modus operandi of using ashared robotized platform is to allow the remote operation, by a human, of heavy loads. During this year, our task was to analyze the human-system interactions and to provide the technical specifications of the system. More precisely, the objective was to define the uses (including the maintenance task), the interaction modalities for 1 to 3 people and the control interfaces of the platform, paying attention to safety and contingencies. As this task is closely linked to the task of empowerment, particular attention was paid to the control or regaining of control by the operator following the abandonment of a task that could theoretically be performed autonomously. The evaluation of the scenarios makes it possible to predetermine the number of operators assigned to the task.
We have contributed on three main topics :
We analyzed industrial robotics accidents from the INRS database EPICEA. We adopted an approach based on interactions in a robotic workspace, coupled with Endsley’s model of Situation Awareness (SA) to identify factors that potentially degrade operators’ SA. We analyzed 45 industrial accident reports involving robotic. Our analysis showed that SA demons were mainly associated with the perception level of SA. Additionally, SA demons are generally involved in human-robot interaction and human-environment interaction. We have also been able to describe five patterns of SA demon occurrence. At the end, we could propose recommendations regarding workspace design.
In the context of the CIFRE Orange PhD work by Nicolas Simonazzi under the supervision of Jean-Marc Salotti and with the objective of analyzing and identifying emotions during interactions with a chatbot, a first experiment was conducted. It involved a user, the use of a smartphone, viewing videos and asking questions about the content of the video and the feeling of the user just after the answer. The collected data were numerous: the accuracy of the answers to the questions, the emotional feeling (choice of emoticons by the user) as well as the real-time measurements of the accelerometer of the smartphone. An analysis of the data was carried out with Russell's relatively simple emotional model as an explanatory framework based on two variables, the positive or negative valence of the emotion, and the degree of excitement. The experimental results showed that there was a slight correlation between the valence indicated by the user, the accuracy of the answers to the questions and the accelerations of the smartphone. However, it was hoped that the videos would have an impact on the valence, because their content had an intrinsic valence, but it proved impossible to find a correlation with the valence indicated by the user, probably due to a lack of the user engagement and also because of the focus on the questions that followed and the accuracy of the answers. A new experimental protocol is currently being studied with a priori more impactful videos (likely to produce an emotion with a greater degree of excitement).
Related publications:
15
In collaboration with the Research Groups in Intelligent Machines, (National School of Engineers of Sfax ENIS, University Sfax, Tunisia), we seek to apply modeling trajectory formalisms to identify the state of a subject or operator from hand movement tracing (writing, drawing or hand trajectory). Nowadays, motion capture (kinect, graphical tablets e.g.) associated with appropriate signal processing allow an objective analysis of these trajectories, 1) for documenting the effect of ageing and highlighting the neural, motor and sensory effects, 2) to develop new clinical tests for neurodegenerative disease early detection, 3) for more engineering-oriented applications such as signature automatic recognition, authentication for forensics, 4) to assess the state of an operator during hand movement execution (fatigue). The proposed model (the beta elliptic model) is in a phase of validation for detecting changes in hand movement during ellpse drawing on a database of 99 subjects with age ranging frm 19 to 85 years.
The global objective of this project is to develop a semi-autonomous carrier dedicated to the transport of heavy structures in industrial factories. The Auctus team has been assigned the role of task analysis and human systems interactions analysis in order to determine the best interface, to improve ergonomics, to reduce risks and to account for acceptability. A postdoctoral student, Charles Fage, has been recruited for the first year of the study (10/2019-10/2020). A 2-years contract (2019-2021) has been signed with AKKA Technologies as part of a consortium, which included two other companies, IIDRE and Ez-Wheel, and another research team from IMS laboratory. The total amount of the grant from Akka to Auctus is 110 kEuros.
Project in a nutshell:
In 2019, a contract was signed with Orange for a PhD on emotions detection in chatbot interactions. The PhD student is Nicolas Simonazzi under the supervision of Jean-Marc Salotti. The objective is analyzing and identifying emotions during interactions with a chatbot. For the company, an important issue is indeed to be able to adapt the response of the chatbot on the website of the customer service. The study was further on focused on the use of smartphones and the identification of emotions based on accelerometers and move patterns. The associated grant was 10 kEuros per year during the 3-years period of the PhD.
Project in a nutshell:
The LiChIE project (funded by BPI) aims to design a constellation of mini-satellites for optical Earth observation. Among many other topics, this requires to rethink the way sattelites are being produced in order to ease this highly complex process. There is actually an unprecedented economical and societal demand for robots that can be used both as advanced and easily programmable tools for automatizing complex industrial operations in contexts where human expertise is a key factor to success and as assistive devices for alleviating the physical and cognitive stress induced by such industrial task. Unfortunately, the discrepancy between the expectations related to idealized versions of such systems and the actual abilities of existing so-called collaborative robots is large. Beyond the limitations of existing systems, especially from a safety point of view, there are very few methodological tools that can actually be used to quantify physical and cognitive stress. There is also a lack of formal approaches that can be used to quantify the contribution of collaborative robots to the realization of industrial tasks by expert operators. Of course, in the state-of-the-art, existing works in that domain do consider some aspects of the current state of the operator in order to propose an appropriate robot behaviour. One of their conceptual limitations is to consider an a priori defined human-robot collaboration scenario where the expertise of the human operator is of importance but limited to a single operation. The consideration of larger varieties of tasks is rarely considered and, when it is, only a strict separation of the tasks to be achieved by each member of the human-robot dyad is considered. In this project, we propose to develop a coupled model of human-robot physical abilities that does not make any a priori with respect to the type of assistance. This requires to develop a parameterisable generic model of the potential physical link and implied constraints between the human operator and the robot. This model should allow to describe the task to be achieved by the human alone or using a collaborative robot through different interaction modalities. Online simulation of these scenarios coupled with ergonomic and performance indicators should both allow for the discrete choice of the right assistance mode given the task currently being achieved as well as for the continuous modulation of the robot behaviour.
Project in a nutshell:
The objective of this project is to work on the required principles to avoid the classic restrained static security zones, synthesizing a dynamic representation of the shared workspace to take advantage of state of the art control laws, allowing a fluid collaboration between human-robot. This dynamic synthesis requires knowledge of the robots state (geometric, cinematic and cognitive, such as fatigue, expertise and situation awareness), its tasks, capacities, state of humans that surround it and their tasks. Furthermore, it needs to achieve a formal and provable online algorithm that correctly estimates the state of the human and guarantee a safe shared workspace tackling ambitious scientific questions poorly addressed in literature.
Project in a nutshell:
Joshua Pickard, University of New Brunswick and Eigen Innovations inc and Milan Hladik, Charles University
Research topic of the collaboration: Set-Based Approaches for the Reliable Computation of Robot and Human Capabilities. Publications related to this collaboration: 2 in preparations.
Alessandro Saccon, Assistant Professor in nonlinear control and robotics at the Mechanical Engineering Department, Eindhoven University of Technology (TU/e)
Research topic of the collaboration: Modelling and prediction of robots impacting their environment Publications related to this collaboration: 1 in preparation Student projects: co-advising of 2 M.Sc. students projects.
Clint Hansen, Christian-Albrechts-Universität zu Kiel Department Universitätsklinikum Schleswig - Holstein
Research topic of the collaboration: We are working together on algorithmic assessment of locomotion by means of inertial sensors.
Houcine Boubaker, Thameur Dhieb, Research Groups in Intelligent Machines, National School of Engineers of Sfax ENIS, University Sfax, Tunisia: We are working on modeling of hand-written tracing.
Research topic of the collaboration: The objective is to propose signal processing tools to assess the state of a subject from manuscript production and hands moves (age, pathologies, tiredness, ...).
Olfa Jemaa, Sami Bennour, University of Sousse and Lotfi Romdhane, American University of Sharjah
Research topic of the collaboration: the individual musculo-tendon forces for ergonomic assessment
HARRY2
The objective of the HARRY2 project is to attain more advanced workspace sharing capabilities through fully exploiting the collaborative possibilities defined by ISO TS 15066. This is achieved by:
- Developing PLC software and motion controllers using robot-agnostic industrially-rated components to ease and standardize the development of safe robotic applications with workspace sharing.
- Integrating state-of-the-art energy-based control algorithms using these industrial hardware components, so that safety is no longer treated as an exception but considered as a constraint when computing the control solution in real-time.
- Enabling the use of high-level and intuitive teaching interfaces reducing robot programming time and difficulty.
- Developing a systematic and practical methodology for quantitative safety evaluation.
Project in a nutshell:
ANR PACBOT
The general objective of the project is to design a semi-autonomous cobotic system for assistance, able to choose, synchronize and coordinate tasks distributed between humans and robots by adapting to different types of variability in professional gestures, all by anticipating dangerous situations. The orchestration of tasks between man and robot is difficult because it must answer the question of the distribution of roles within the couple according to physical and decision-making skills and constraints as well as the consequences of their interactions. However, we cannot put the two actors at the sae level: the robot has to adapt his actions to the work of an operator and, more precisely, to his motor and cognitive strategies that materialize through the quantifiable variability of professional actions. On the other hand, the very interest of the robot is to assist the operator in his phases of fragility while preserving his physical and mental integrity, in particular considering that human error is inherent in operator action. These considerations are, for Pacbot, the conditions necessary for the joint achievement of efficient work.
Partners of the ANR project:
General information:
MIELS, in partnership with Solvay, Laboratoire du Futur (Pessac)
The main objective of the MIELS project is to develop innovative strategies to characterize and develop neoteric, non-toxic solvents through strategies that will enable to grasp the enormous quantity of required experimental tests all in insuring an absolute safety of the manipulator. For this purpose, we intend to work on two complementary routes, the development of solvent characterization methodologies and the integration of a cobotic approach in solvent handling and evaluation, with the ambition of merging these developments at the end of the project in order to draw as much synergy as possible. This project is built around teams with complementary competencies to achieve these objectives. UMR LOF and Solvay LOF have great experience acquired over several years of research the fields of solvent evaluation and robotics, whereas Auctus INRIA team has a strong expertise in collaborative robotics. By combining our competencies and expertise, the MIELS project aims to merge all these fields in order to expand 1) the fundamental study of solvents and their characterization techniques including theoretical techniques, in particular for green solvents and 2) the development of the use of cobotics, in collaboration with Auctus INRIA team, for increasing the efficacy and safety of laboratory workers in industry, in particular those working on characterization of solvents.
Project in a nutshell:
WOOBOT
The main goal of the work is to develop a design methodology (control and physical architecture) of collaborative robotic systems dedicated to assisting and securing operator expert gestures in the context of industrial applications involving strong physical interaction between a human, a tool and a manipulated object. The collaborative system should improve ergonomics and safety while maintaining the level of industrial performance and preserving the user’s know-how. The application context is the carpentry, in particular wood shaping.
Future work will include a more complex human model in order to quantify the impact of potential assistance architectures according to relevant criteria of industrial performances, ergonomics/safety and user expertise.
This project is conducted by Nassim Benhabib, Vincent Padois and David Daney. With the collaboration of BTP CFA de la Gironde. This allows to provide realistic information of the task studied and to collect the feedback from the craftsmen.
Project in a nutshell: