The DEFROST team aims to address the open problem of control and modelling methods for deformable robots by answering the following challenges:
Providing numerical methods and software support to reach the real-time constraint needed by robotic systems: the numerical solutions for the differential equations governing the deformation generate tens of thousands degrees of freedom, which is three orders of magnitude of what is frequently considered in classical methods of robotic modelling and control.
Integrating deformation models in the control methods of soft robot: In soft-robotics, sensing, actuation and motion are coupled by the deformations. Deformable models must be placed at the heart of the control algorithm design.
Investigating predictable interaction models with soft-tissues and parameter estimation by visual feedback from medical imaging: On the contrary to many cases in surgical robotics, the contact of the soft robot with the anatomy is permitted and it creates additional deformations on the robot.
Our research crosses different disciplines: numerical mechanics, control design, robotics, optimisation methods, clinical applications. Our organisation aims at facilitating the team work and cross-fertilisation of research results in the group. We have three objectives (1, 2 and 3) that correspond to the main scientific challenges. In addition, we have two transversal objectives that are also highly challenging: the development of a high performance software support for the project (objective 4) and the validation tools and protocols for the models and methods (objective 5).
The objective is to find concrete numerical solutions to the challenge of modelling soft robots with strong real-time constraints. To solve continuum mechanics equations, we will start our research with real-time FEM or equivalent methods that were developed for soft-tissue simulation. We will extend the functionalities to account for the needs of a soft-robotic system:
Coupling with other physical phenomenons that govern the activity of sensors and actuators (hydraulic, pneumatic, electro-active polymers, shape-memory alloys...).
Fulfill the new computational time constraints (harder than surgical simulation for training) and find better tradeoff between cost and precision of numerical solvers using reduced-order modelling techniques with error control.
Exploring interactive and semi-automatic optimisation methods for design based on obtained solution for fast computation on soft robot models.
The focus of this objective is on obtaining a generic methodology for soft robot feedback control. Several steps are needed to design a model based control from FEM approach:
The fundamental question of the kinematic link between actuators, sensors, effectors and contacts using the most reduced mathematical space must be carefully addressed. We need to find efficient algorithms for real-time projection of non-linear FEM models in order to pose the control problem using the only relevant parameters of the motion control.
Intuitive remote control is obtained when the user directly controls the effector motion. To add this functionality, we need to obtain real-time inverse models of the soft robots by optimisation. Several criteria will be combined in this optimisation: effector motion control, structural stiffness of the robot, reduce intensity of the contact with the environment...
Investigating closed-loop approaches using sensor feedback: as sensors cannot monitor all points of the deformable structure, the information provided will only be partial. We will need additional algorithms based on the FEM model to obtain the best possible treatment of the information. The final objective of these models and algorithms is to have robust and efficient feedback control strategies for soft robots. One of the main challenge here is to ensure / prove stability in closed-loop.
Even if the inherent mechanical compliance of soft robots makes them more safe, robust and particularly adapted to interaction with fragile environments, the contact forces need to be controlled by:
Setting up real-time modelling and the control methods needed to pilot the forces that the robot im- poses on its environment and to control the robot deformations imposed by its environment. Note that if an operative task requires to apply forces on the surrounding structures, the robot must be anchored to other structures or structurally rigidified.
Providing mechanics models of the environment that include the uncertainties on the geometry and on the mechanical properties, and are capable of being readjusted in real-time.
Using the visual feedback of the robot behavior to adapt dynamically the models. The observation provided in the image coupled with an inverse accurate model of the robot could transform the soft robot into sensor: as the robot deforms with the contact of the surroundings, we could retrieve some missing parameters of the environment by a smart monitoring of the robot deformations.
Expected research results of this project are numerical methods and algorithms that require high-performance computing and suitability with robotic applications. There is no existing software support for such development. We propose to develop our own software, in a suite split into three applications:
The first one will facilitate the design of deformable robots by an easy passage from CAD software (for the design of the robot) to the FEM based simulation
The second one is an anticipative clinical simulator. The aim is to co-design the robotic assistance with the physicians, thanks to a realistic simulation of the procedure or the robotic assistance. This will facilitate the work of reflection on new clinical approaches prior any manufacturing
The third one is the control design software. It will provide the real-time solutions for soft robot control developed in the project.
The implementation of experimental validation is a key challenge for the project. On one side, we need to validate the model and control algorithms using concrete test case example in order to improve the modelling and to demonstrate the concrete feasibility of our methods. On the other side, concrete applications will also feed the reflexions on the objectives of the scientific program.
We will build our own experimental soft robots for the validation of objectives 2 and 3 when there is no existing « turn-key » solution. Designing and making our own soft robots, even if only for validation, will help the setting-up of adequate models.
For the validation of objective 4, we will develop « anatomical soft robot »: soft robot with the shape of organs, equipped with sensors (to measure the contact forces) and actuators (to be able to stiffen the walls and recreate natural motion of soft-tissues). We will progressively increase the level of realism of this novel validation set-up to come closer to the anatomical properties.
Robotics in the manufacturing industry is already highly diffused and is one of the ways put forward to maintain the level of competitiveness of companies based in France and to avoid relocation in cheap labor countries. Yet, in France, it is considered that the level of robotization is insufficient compared to Germany, for instance. One of the challenge is the high investment cost for buying robotic arms. In the recent years, it has led the development of « generic » and « flexible » (but rigid) robotic solution that can be produced in series. But their applicability to specific tasks is still challenging or too costly. With the development of 3D
printing, we can imagine the development of a complete opposite strategy: a « task-specific » design of robots. Given a task that needs to be performed by a deformable robot: we would optimize the shape of its structure to create the set of desired motion. An second important aspect is the reduction of the manufacturing cost:
It is often anticipated that the cost of deformable robots will be low compared to classical rigid robotics. The robot could be built on one piece using rapid prototyping or 3D printers and be more adapted for collaborative work with operators. In this area, using soft materials are particularly convenient as they provide a mass/carried load ratio several orders higher than traditional robots, highly decreasing the kinetic energy and so increasing the motion speed allowed in presence of humans. Moreover, the technology allows more efficient and ergonomic wearable robotic devices, opening the options for exo-skeletons. This remains to be put in place, but it can open new perspectives in robotic applications. A last remarkable property of soft robots is their adaptability to fragile or tortuous environment. For some
particular industry (chemistry, food industry...) this could also be an advantage compared to existing rigid solutions. For instance, the German company https://
This year (2017), we have started to work more concretely on this domain, with a research contrat with TDR company on design of deformable graspers.
The personal and service robotics are considered as an important source of economic expansion in the coming years. The potential applications are numerous and particularly include the challenge of finding robotic solutions for active and healthy aging at home. We plan to develop functional orthosis for which it is better not to have a rigid exoskeleton that is particularly not comfortable. These orthosis will be ideally personalised for each patient and built using rapid prototyping.
On this topic, the place of our team will be to provide algorithms for controlling the robots.
We will find some partners to build these robots that would fall in the category of « wearable robots ». With this thematic we also connect with a strong pole of excellence of the region on intelligent textile (see http://
Robots have a long history with entertainment and arts where https://
Key Scientific results: - First closed looped control of a soft robot (Thesis of Thor Bieze and Paper of Zhongkai Zhang at IROS) - Inverse model of soft robots with contact handling (Paper of Eulalie Coevoet for RA-L Letters at ICRA)
Christian Duriez took part in a show on France Culture: https://
One of the team's robot was shown on France 24: http://
Le Point http://
France inter: https://
Gang Zheng as researcher.
Félix Vanneste and Erwan Douaille as engineers.
Thomas Morzadec and Stefan Escaida Navarro as postdoctoral researchers.
Marwa Mohammed Alaa Eldean Eldiwiny as PhD student.
Vascular neurosurgery simulation based on SOFA Framework
Keywords: Simulation - Health - Computer-assisted surgery
Participants: Christian Duriez, Eulalie Coevoet, Jérémie Dequidt and Laurent Thines
Partners: Université de Lille - CHRU Lille
Contact: Christian Duriez
Simulation Open Framework Architecture
Keywords: Real time - Multi-physics simulation - Medical applications
Functional Description: SOFA is an Open Source framework primarily targeted at real-time simulation, with an emphasis on medical simulation. It is mostly intended for the research community to help develop new algorithms, but can also be used as an efficient prototyping tool. Based on an advanced software architecture, it allows : the creation of complex and evolving simulations by combining new algorithms with algorithms already included in SOFA, the modification of most parameters of the simulation (deformable behavior, surface representation, solver, constraints, collision algorithm, etc. ) by simply editing an XML file, the building of complex models from simpler ones using a scene-graph description, the efficient simulation of the dynamics of interacting objects using abstract equation solvers, the reuse and easy comparison of a variety of available methods.
Participants: Christian Duriez, François Faure, Hervé Delingette and Stéphane Cotin
Partner: IGG
Contact: Stéphane Cotin
SoftRobots plugin for Sofa
Keywords: Numerical simulations - Problem inverse - Soft robotics
Functional Description: This framework is based on a mechanical modeling of the robot elements combined with fast real-time direct/inverse FEM solvers. The keypoint of our approach is that the same modeling is used for interactive simulation of its behavior and interactive control of the fabricated robots.
Participants: Christian Duriez, Olivier Goury, Jérémie Dequidt, Damien Marchal, Eulalie Coevoet, Erwan Douaille and Félix Vanneste
Contact: Christian Duriez
Publication in RA-Letter 2017 (proceeding ICRA 2017): Optimization-based inverse model of soft robots with contact handling, E. Coevoet, A. Escande, C. Duriez. Abstract. This paper presents a physically-based algorithm to interactively simulate and control the motion of soft robots interacting with their environment. We use the Finite Element Method (FEM) to simulate the non-linear deformation of the soft structure, its actuators, and surroundings, and propose a control method relying on a quadratic optimization to find the inverse of the model. The novelty of this work is that the deformations due to contacts, including self-collisions, are taken into account in the optimization process. We propose a dedicated and efficient solver to handle the linear complementarity constraints introduced by the contacts. Thus, the method allows interactive transfer of the motion of soft robots from their task space to their actuator space while interacting with their surrounding. The method is generic and tested on several numerical examples and on a real cable-driven soft robot. http://
Publication at ICRA 2017: Real-time simulation of hydraulic components for interactive control of soft robots, A. Rodrìguez, E. Coevoet, C. Duriez. Abstract. In this work we propose a new method for online motion planning in the task-space for hydraulic actuated soft robots. Our solution relies on the interactive resolution of an inverse kinematics problem, that takes into account the properties (mass, stiffness) of the deformable material used to build the robot. An accurate modeling of the mechanical behavior of hydraulic components is based on a novel GPU parallel method for the real-time computation of fluid weight distribution. The efficiency of the method is further increased by a novel GPU parallel leveraging mechanism. Our complete solution has been integrated within the open-source SOFA framework. In our results, we validate our simulation with a fabricated silicone cylinder and we demonstrate the usage of our approach for direct control of hydraulic soft robots. https://
Publication at IROS 2017: Visual Servoing Control of Soft Robots based on Finite Element Model, Zhongkai Zhang, Thor Morales Bieze, Jeremie Dequidt, Alexandre Kruszewski, Christian Duriez. Abstract. In this paper, we propose a strategy for the control of soft robots with visual tracking and simulation-based predictor. A kinematic model of soft robots is obtained thanks to the Finite Element Method (FEM) computed in real-time. The FEM allows to obtain a prediction of the Jacobian matrix of the robot. This allows a first control of the robot, in the actuator space. Then, a second control strategy based on the feedback of infrared cameras is developed to obtain a correction of the effector position. The robust stability of this closed-loop system is obtained based on Lyapunov stability theory. Otherwise, to deal with the problem of image features (the marker points placed on the end effector of soft robot) loss, a switched control strategy is proposed to combine both the open-loop controller and the closed-loop controller. Finally, experiments on a parallel soft robot driven by four cables are conducted and show the effectiveness of these methods for the real-time control of soft robots. https://
Simulation results have been presented at the International Federation of Automatic Control (IFAC) World Congress in Toulouse in July. We presented a control strategy build on a reduced order dynamical model of a soft robot and showed that it allows the user to make the studied soft robot converge faster to a desired position without oscillations. This paper was illustrated with simulation experiments. https://
Publication in Advanced Robotics 2017: Software toolkit for modeling, simulation and control of soft robots, E. Coevoet, T. Morales-Bieze, F. Largilliere, Z. Zhang, M. Thieffry, M. Sanz-Lopez, B. Carrez, D. Marchal, O. Goury, J. Dequidt, C. Duriez. Abstract. The technological differences between traditional robotics and soft robotics have an impact on all of the modeling tools generally in use, including direct kinematics and inverse models, Jacobians, and dynamics. Due to the lack of precise modeling and control methods for soft robots, the promising concepts of using such design for complex applications (medicine, assistance, domestic robotics...) cannot be practically implemented. This paper presents a first unified software framework dedicated to modeling, simulation and control of soft robots. The framework relies on continuum mechanics for modeling the robotic parts and boundary conditions like actuators or contacts using a unified representation based on Lagrange multipliers. It enables the digital robot to be simulated in its environment using a direct model. The model can also be inverted online using an optimization-based method which allows to control the physical robots in the task space. To demonstrate the effectiveness of the approach, we present various soft robots scenarios including ones where the robot is interacting with its environment. The software has been built on top of SOFA, an open-source framework for deformable online simulation and is available at https://
Thomas Morzadec and Erwan Douaille are working on new approaches to model the soft robots' shape. They are working under the supervision of Damien Marchal and Christian Duriez. During this year, they developped a modelling kernel based on distance field function and integrate it into Sofa. The geometrical representation offer several interesting properties that are: compactness and generative. For usability the distance functions are described using the python language then transpiled into more efficient languages (glsl and cython). This geometric kernel is planned for integration into Sofa. On top of this framework shape optimization are now investigated using genetic algorithms.
TDR group is a robotics integrator specialized on optimizing production chains, usually multiplexing robots to perform several activities. Hence, their interest in graspers and the time invested in this activity has been growing within the last years. To improve this aspect, we have been developing together a concept of "universal grasper", based on soft robotics technology and capable of grasping an object with an arbitrary shape, and partially misplaced or misoriented. The prototype developed complies with the specifications and allows for scalability, with flexibility between grasping force and shape tolerance, and the ability for replacing objects without the need of an external vision system. Relying in SOFA for physical simulation, we could validate the different prototypes proposed, put in place test scenarios and put in place a design tool to test generic, application-specific prototypes. A patent redaction is ongoing.
Olivier Goury was hired as a postdoctoral researcher by the "Réhabilitation chirurgicale mini-invasive et robotisée de l'audition" to collaborate with the DEFROST team on the simulation of Cochlear Implant surgery. The contract stopped since Olivier has been recruited as a Research scientist. The collaboration with Inserm has been continued since, with the hiring of Piyush Jain as an engineer.
Tremplin ERC Christian Duriez recieved a ANR grant "tremplin ERC" (150k€) given the result obtained last year on the ERC proposal (evaluated at "grade A"). The project has allowed to allocate new resources on the developments that were presented in this ERC.
CO2DMod Control-Oriented Data-Driven Modeling of Complex System. The goal of this project was to propose Data-Driven Modelling technique (model reduction as well as model identification) that provides an Uncertainty Certificate (UC). The goal of these certificates are (i) to guarantee that the models obtained from data are good enough for control, (ii) to help the user determine the class of controller design problem the model is tuned for. Unfortunately, the project has not been funded. It was resubmitted this year with hopefully a better outcome.
ROBOCOP ROBOtization of COchlear imPlant. ROBOCOP aims at creating a new prototype of cochlear implant, and robotize (i.e. actuate and control) its insertion process to facilitate the work of surgeon, to increase the success ratio, and to decrease the probability of trauma. Partnership with IEMN (Institute of Electronics, Microelectronics and Nanotechnology), OTICON Medical and UMRS-1159 at Inserm. This project was submitted in 2017 and we are awaiting the answer from the ANR.
Program: FEDER
Project acronym: COMOROS
Project title: Control of deformable robots for surgery
Duration april 2017 to march 2020 (in two phases)
Coordinator: C. Duriez
Abstract: Surgical procedures are often carried out using instruments made of stiff materials that interact with delicate biological tissues such as internal organs, blood vessel walls and small cavities. This incompatibility of stiffness is one of the sources of danger in many surgical procedures. The use of robots made of soft materials, also called soft robots, would limit such risks by reducing contact pressures and stress concentrations. Their intrinsic deformability would also increase the ability to manoeuvre in confined spaces. However, the promising concept of using soft robots for surgical procedures cannot be practically implemented, due to the lack of precise modelling and control methods for soft robots. This scientific obstacle, identified as a pending issue by major surveys in this field, becomes particularly challenging when interacting with an environment as complex as the human anatomy. Drawing on our background in soft tissue simulation, contact models, surgical applications and soft robotics, our ambition in this project is to:
Develop accurate and generic numerical methods for continuum mechanics, adapted to strong real-time constraints in order to demonstrate the ability to model soft mechatronics systems.
Reconsider parametrization methodologies of digital models of the patient anatomy through the observation of mechanical interactions with soft robots via embedded sensors and medical imaging.
Rethink motion generation and teleoperation control with force feedback so as to be compatible with the large number of degrees of freedom of soft robots and be based on accurate, rapidly-computed deformable models and interaction models.
The project also targets the development of software with the required performance and features, as well as the experimental validation of models and methods using prototypes in realistic environments.
Université Libre Brussels, Pr. Terwagne, Pr.Massar and Mr Tillema
Artificial Intelligence Algorithms for the control of soft robots, based on the simulation (associated north-european team 2016-2017)
University of Luxembourg: Pr Bordas
Model order reduction and topological changes (journal in 2013 & submission of a proposal in 2017)
King’s college: Pr Liu
Robotic catheter navigation and control and soft surgcial robotics (conference publication at IROS in 2017)
Margaret Koehler is a PhD student in Mechanical Engineering from the Collaborative Haptics and Robotics in Medicine (CHARM) Lab at Stanford University, led by Allison Okamura. Her PhD topic is "Design and Control of Soft Haptic Devices." Supported by a Chateaubriand Fellowship in partnership with Inria, she is conducting a 6-month research internship with the DEFROST team from September 2017 through February 2018. Her internship is part of a year-long collaboration between the DEFROST team and the CHARM Lab. In 2018, Christian Duriez will spend six months as a visiting scholar in the CHARM Lab to continue this exchange. The collaboration focuses on the development of a soft haptic device using the SOFA framework and soft robot control methods developed by the DEFROST team for design and control.
Christian Duriez: World Haptic Conference 2017, Pacific Graphics 2017, Soft robotics conference 2018.
Alexandre Kruszewski:
ICRA
ECC
ACC
CDC
MED
IFAC World Congress
ICRA
Olivier Goury:
IEEE Transactions on Automation Science and Engineering
Damien Marchal:
Reviewer for VRIPHYS 2017 conference and the Computer & Graphics journal.
Christian Duriez:
ICRA
PacificGraphics
Vriphys
Christian Duriez: IEEE Transaction on Haptics
Kruszewki:
IEEE TCST
IEEE TFS
Fuzzy Sets and Systems
Automatica
Christian Duriez:
SORO (Soft Robotic Journal)
IEEE Transactions on Robotics
IEEE Robotic and Automation Letters
Workshop on Computer Animation, Feb 2017, La Barbade
Mechatronic Conference, Nov 2017, Paris
Lagadic Seminar, Nov 2017, Rennes
Kruszewski: IDEX project (Univ. Strasbourg)
Duriez:
ANR project
EPSRC project (UK)
Habilitation evaluation (University of Luxembourg)
IDEX project (Univ. Strasbourg)
A. Kruszewski and C. Duriez are members of the Laboratory council (CRIStAL)
C.Duriez is vice-president of the "Commission des Emplois de Recherche" of Lille
Ingénieur : Kruszewski, Automatique 1, 54h, niveau L3, Centrale Lille, France
Ingénieur : Kruszewski, Contrôle des systèmes dynamiques, 12h, niveau L4, Centrale Lille, France
Ingénieur : Kruszewski, Drone, 16h, niveau M1, Centrale Lille, France
Ingénieur : Kruszewski, Systèmes embarqués, 8h, niveau M1, Centrale Lille, France
Ingénieur : Kruszewski, Conception d'un système embarqué, 18h, niveau L3, Centrale Lille, France
Ingénieur : Kruszewski, Automatique 2, 12h, niveau M1, Centrale Lille, France
Ingénieur : Kruszewski, Automatique, 36h, niveau L3, Centrale Lille, France
Ingénieur : Kruszewski, Projets, 50h, niveau L3-M1, Centrale Lille, France
Ingénieur : Kruszewski, Projet recherche, 12h, niveau M1, Centrale Lille, France
Master : Damien Marchal, Reality-Virtual et Interaction, 24h, M2, University of Lille, France
Doctorat : Duriez, Modeling and Simulation, 2H, Summer school on surgical simulation, Montpelier, France
Master : Duriez, Interactive simulation, 24H, M2, University of Lille, France
HdR: Kruszewski, Are polytopic control design methods suitable for the next robotic control challenges?, Université de Lille, 12/12/2017
PhD: Bieze, Contribution à la modélisation cinématique et au contrôle de manipulateurs déformables, fondée sur la mécanique numérique, Université de Lille, 24/10/2017
PhD in progress: Largilière, Conception de nouvelles interface déformables, contrôlés par une simulation éléments-finis temps-réel, 01/10/2013, C. Duriez, L. Grisoni
PhD in progress: Thieffry, Modélisation et contrôle de robots déformables à grande vitesse, 01/09/2016, C. Duriez, T.M. Guerra, A. Kruszewski
PhD in progress: Zhang, New methods of visual servoing for soft-robots, 01/10/2015, C. Duriez, J. Dequidt, A. Kruszewski
PhD in progress: Coevoet, Méthodes d’optimisation pour la robotique déformable, 01/05/2017, C. Duriez
PhD in progress: Eldiwiny, Modélisation, Simulation et Design de micro robots déformables, 01/09/2017, C. Duriez
Benoit Le Gouis, C. Duriez président du jury, Rennes, soutenance novembre 2017
Renato Torres, C. Duriez rapporteur, Paris, soutenance prévue en janvier 2018
Mario Sanz Lopez, Olivier Goury and Christian Duriez participated in "Fête de la science: Opération Chercheurs itinérants", which involves giving scientific lectures in middle and high schools. Olivier Goury and Bruno Carrez participated in the Numéric'elle event, which involved sensibilising female middle school students to pursue a carreer in the digital world. Mario Sanz Lopez installed a new demonstrator on Soft Robotics at Plateau Euratechnologies.
Our team has been highlighted in the medias this year: https://