The DEFROST team aims to address the open problem of control and modelling methods for deformable robots by answering the following challenges:
Providing numerical methods and software support to reach the real- time constraint needed by robotic systems: the numerical solutions for the differential equations governing the deformation generate tens of thousands degrees of freedom, which is three orders of magnitude of what is frequently considered in classical methods of robotic modelling and control.
Integrating deformation models in the control methods of soft-robot: In soft-robotics, sensing, actuation and motion are coupled by the deformations. Deformable models must be placed at the heart of the control algorithm design.
Investigating predictable interaction models with soft-tissues and pa- rameter estimation by visual feedback from medical imaging: On the contrary to many cases in surgical robotics, the contact of the soft-ro- bot with the anatomy is permitted and it creates additional deformation on the robot.
Our research crosses different disciplines: numerical mechanics, control design, robotics, optimisation methods, clinical applications. Our organisation aims at facilitating the team work and cross- fertilisation of research results in the group. We have three objectives (1, 2 and 3) that correspond to the main scientific challenges. In addition, we have two transversal objectives that are also highly challenging: the development of a high performance software support for the project (objective 4) and the validation tools and protocols for the models and methods (objective 5).
The objective is to find concrete numerical solutions to the challenge of modelling soft robots with strong real-time constraints. To solve continuum mechanics equations, we will start our research with real-time FEM or equivalent methods that were developed for soft-tissue simulation. We will extend the functionalities to account for the needs of a soft-robotic system:
Coupling with other physical phenomenons that govern the activity of sensors and actuators (hydraulic, pneumatic, electro-active polymers, shape-memory alloys...).
Fulfil the new computational time constraints (harder than surgical simulation for training) and find better tradeoff between cost and precision of numerical solvers using reduced-order modelling techniques with error control.
Exploring interactive and semi-automatic optimisation methods for design based on obtained solution for fast computation on soft-robot models.
The focus of this objective is on obtaining a generic methodology for soft-robot feedback control. Several steps are needed to design a model based control from FEM approach:
The fundamental question of the kinematic link between actuators, sensors, effectors and contacts us- ing the most reduced mathematical space must be carefully addressed. We need to find efficient algo- rithms for real-time projection of non-linear FEM models in order to pose the control problem using the only relevant parameters of the motion control.
Intuitive remote control is obtained when the user directly controls the effector motion. To add this functionality, we need to obtain real-time inverse models of the soft-robots by optimisation. Several criteria will be combined in this optimisation: effector motion control, structural stiffness of the robot, reduce intensity of the contact with the environment...
Investigating closed-loop approaches using sensor feedback: as sensors cannot monitor all points of the deformable structure, the information provided will only be partial. We will need additional algo- rithms based on the FEM model to obtain the best possible treatment of the information. The final ob- jective of these models and algorithms is to have robust and efficient feedback control strategies for soft-robots. One of the main challenge here is to ensure / prove stability in closed-loop.
Even if the inherent mechanical compliance of soft-robots makes them more safe, robust and particularly adapted to interaction with frag- ile environments, the contact forces need to be controlled by:
Setting up real-time modelling and the control methods needed to pilot the forces that the robot im- poses on its environment and to control the robot deformations imposed by its environment. Note that if an operative task requires to apply forces on the surrounding structures, the robot must be anchored to other structures or structurally rigidified.
Providing mechanics models of the environment that include the uncertainties on the geometry and on the mechanical properties, and are capable of being readjusted in real-time.
Using the visual feedback of the robot behaviour to adapt dynamically the models. The observation provided in the image coupled with an inverse accurate model of the robot could transform the soft robot into sensor: as the robot deforms with the contact of the surroundings, we could retrieve some missing parameters of the environment by a smart monitoring of the robot deformations.
Expected research results of this project are numerical methods and algorithms that require high-performance computing and suitability with robotic applications. There is no existing software support for such development. We propose to develop our own software, in a suite split into three applications:
The first one will facilitate the design of deformable robots by an easy passage from CAD software (for the design of the robot) to the FEM based simulation
The second one is an anticipative clinical simulator. The aim is to co-design the robotic assistance with the physicians, thanks to a realistic simulation of the procedure or the robotic assistance. This will facilitate the work of reflection on new clinical approaches prior any manufacturing
The third one is the control design software. It will provide the real-time solutions for soft-robot control developed in the project.
The implementation of experimental valida- tion is a key challenge for the project. On one side, we need to validate the model and control algorithms using concrete test case example in order to improve the modelling and to demonstrate the concrete feasibility of our methods. On the other side, concrete applications will also feed the reflexions on the objec- tives of the scientific program.
We will build our own experimental soft-robots for the validation of objective 2 and 3 when there is no existing « turn-key » solution. Designing and making our own soft-robots, even if only for validation, will help the setting-up of adequate models.
For the validation of objective 4, we will develop « anatomical soft robot »: soft-robot with the shape of organs, equipped with sensors (to measure the contact forces) and actuators (to be able to stiffen the walls and recreate natural motion of soft-tissues). We will progressively increase the level of realism of this novel validation set-up to come closer to the anatomical properties.
Surgical procedures are often carried out using instruments made from stiff materials that interact with delicate biological tissues such as internal organs, blood vessel walls and small cavities. This is one of the source of danger for many surgical procedures. Soft-robotics open up new perspectives in minimally invasive approaches. Thanks to the highly deformability of their structure, similar to organic materials, and their motion, created by deformation in the same way as the muscles in living animals, they offer many advantage for surgical applications. Recent work anticipates that their compliant nature and their large number of degrees of freedom will provide key surgical positive outcomes:
Improving the capacity of access with security to the fragile parts of the anatomy by applying less pressure to the anatomical walls
Easy maneuvering through soft and confined spaces allowing new Minimally Invasive Surgery approaches.
These positive outcomes are expected given the properties of soft-robot. In a recent state-of-the art reports on soft robotics, surgery in the list of killer applications of soft-robotics. However, the lack of existing methodology for modeling and control remains an obstacle to be proved by a practical implementation. Given our background on surgical simulations: soft tissue and tool/tissues contact models we are particularly well positioned to address the challenge of using soft-robots in surgery.
Robotics in the manufacturing industry is already highly diffused and is one of the ways put forward to maintain the level of competitiveness of companies based in France and to avoid relocation in cheap labor countries. Yet, in France, it is considered that the level of robotization is insufficient compared to Germany for instance. One of the challenge is the high investment cost for buying robotic arms. In the recent years, it has led the development of « generic » and « flexible » (but rigid) robotic solution that can be produced in series. But their applicability to specific tasks is still challenging or too costly. The manufacturing of deformable robots could be very low compared to classical rigid robotics. Moreover, with the development of 3D printing, we can imagine the development of a complete opposite strategy: a « task-specific » design of robots. Given a task that need to be performed by a deformable robot: we would optimize the shape of its structure to create the set of desired motion (see in Challenge2: Exploring interactive and semi-automatic optimisation methods for design). An other remarkable property of soft-robots is their adaptability to fragile or tortuous environment. For some particular industry, this could also be an advantage compared to existing rigid solutions.
The personal and service robotics are considered as an important challenge for industry in the coming years. The potential applications are numerous and particularly include the challenge of finding robotic solutions for active and healthy ageing at home. We plan to develop functional orthosis for which it is better not to have a rigid exoskeleton that are particularly not comfortable. These orthosis will be ideally personalised for each patient and built using rapid prototyping. Again the low manufacturing price and the robustness of deformable robots could be key advantages for this particular market. On this topic, the place of our team will be to provide algorithms for controlling the robots. We need to find some partners to build these wearable robots. Our team will also propose innovative technology for robotic games: we are currently working on a new technique of control for deformable puppets. If the project succeeds, a user will be able to build his/her own puppet with a 3D printer and control it with a Kinect. Finally, an other direction for the transfer of our research towards society is art: soft-robotics seems a source of inspiration for artists. This year, we have been collaborating with the art school Le Fresnoy based at Tourcoing (near our Lab) and the result had a good impact for the visibility of our team. We may also collaborate in the close future with IRCAM in the context of the transversal project Inria-ART led by Arshia Cont and Laurent Grisoni.
Inverse deformable model in real-time by quadratic programming optimization
We have published the formulation of an inverse deformable model that we can compute in real time in the form of quadratic problem under equality and inequality constraints. After the projection of the deformable model in the reduced space of unknown parameters, we get an extremely compact formulation of the problem to be optimized. The quadratic formulation allows to write the problem with the conditions Karush-Kuhn-Tucker (KKT) and thus have certainties about the uniqueness and optimality of a solution. This formulation was used in image registration project for adaptive radiotherapy (study published in the International journal of computer assisted radiology and surgery) and also to calculate the inverse model of a deformable robot (study published in the conference ICRA 2015).
Deformable robots with vertebras
We proposed a for generic modeling method suitable for manipulator arm composed of a successive series of deformable portion (inter-vertebrae) and rigid (vertebrae). This method is very computationally efficient and compatible with real-time. These manipulators have a very large number of degrees of freedom. Our approach is to make a domain decomposition from a FEM model on inter-vertebrae and pre-compute a condensation of the model on the vertebrae to drastically reduce the complexity of the model used online. Condensed models are assembled for the global model of the robot. We have demonstrated in an article published in the ICRA 2015 conference that this model allowed to pilot the flexible robot CBHA developed by Festo. Furthermore, we have used this model to propose a new manipulator arm design called FETCH to the competition website Robotic Toolkits Harvard University. We had the 2nd place ex-aequo with 4 other teams.
SOFA
The work we have done and published around our simulation platform SOFA allowed us to get the price "Dirk Bartz Prize for Visual Computing in Medicine 2015," a biennial competition organized by the Eurographics conference. This award recognizes the significant contributions in computer graphics have an impact in the field of medicine. Parallel to that price, a consortium was created to SOFA (https://
Eurographics Dirk Bartz Prize for Visual Computing in Medicine.
Simulation Open Framework Architecture
Keywords: Physical simulation - Health - Biomechanics - GPU - Computer-assisted surgery
Functional Description
SOFA is an Open Source framework primarily targeted at real-time simulation, with an emphasis on medical simulation. It is mostly intended for the research community to help develop new algorithms, but can also be used as an efficient prototyping tool. Based on an advanced software architecture, it allows : the creation of complex and evolving simulations by combining new algorithms with algorithms already included in SOFA, the modification of most parameters of the simulation (deformable behavior, surface representation, solver, constraints, collision algorithm, etc. ) by simply editing an XML file, the building of complex models from simpler ones using a scene-graph description, the efficient simulation of the dynamics of interacting objects using abstract equation solvers, the reuse and easy comparison of a variety of available methods.
Participants: Stéphane Cotin and Hervé Delingette
Partner: IGG
Contact: Stéphane Cotin
Our contribution consists in a new framework to simulate and control soft robots. This framework is based on a mechanical modeling of the robot elements combined with fast real-time direct/inverse FEM solvers. The keypoint of our approach is that the same modeling is used for interactive simulation of its behavior and interactive control of the fabricated robots. This plugin is being developped in the ADT project SORBET.
Keywords: Simulation - Soft-Robot - Inverse models - Finite Element Method - Quadratic Programmings
Participants: Eulalie Coevoet, Olivier Goury, Frédéric Largillière, Bruno Carrez, Damien Marchal, Jérémie Dequidt and Christian Duriez
Contact: Eulalie Coevoet and Christian Duriez
Vascular neurosurgery simulation based on SOFA Framework
Keywords: Simulation - Health - Computer-assisted surgery
Participants: Christian Duriez, Eulalie Coevoet, Laurent Thines and Jérémie Dequidt
Partners: Université de Lille - CHRU Lille
Contact: Christian Duriez
In this work, we propose a new formulation of contact and friction laws, in the context of virtual graping. The work allows to reduce the number of contact and friction constraints, using volume interpenetration measure, instead of interpenetration distance. The work has been conducted in collaboration with Antony Talvas and Maud Machal (Inria Hybrid Team, Rennes) and Miguel Otaduy (URJC Madrid). It has been presented at the conference IEEE VR and published in the journal TVCG .
We have reached an important milestone with this work: we have merge two important research tracks of these last years: On one hand, haptic rendering of friction contact between deformable objects ; on the ohter hand, real-time simulation of hyperelastic objects (particularly to simulate soft-tissues). This work has been conducted in collaboration with Hadrien Courtecuisse (Inria team Mimesis) and Hervé Delingette (Inria team Asclepios)
In this work, we propose to recover the 3D shape and to augment elastic objects with self-occlusions handling, using only single view images. Shape recovery from a monocular video sequence is an underconstrained problem and many approaches have been proposed to enforce constraints and resolve the ambiguities. State-of-the art solutions enforce smoothness or geometric constraints, consider specific deformation properties such as inextensibility or resort to shading constraints. We propose a real-time method that uses a mechanical model and that is able to handle highly elastic objects. The problem is formulated as an energy minimization problem accounting for a non-linear elastic model constrained by external image points acquired from a monocular camera. This method prevents us from formulating restrictive assumptions and specific constraint terms in the minimization. In addition, we propose to handle self-occluded regions thanks to the ability of mechanical models to provide appropriate predictions of the shape. This result has been published in the journal TVCG and has been extended to handle cutable objects and has been published as a SIGGRAPH poster .
Finite Element analysis can provide accurate deformable models for soft-robots. However, using such models is very difficult in a real-time system of control. In this study, we introduce a generic solution that enables a high-rate control and that is compatible with strong real-time constraints. From a Finite Element analysis, computed at low rate, an inverse model of the robot outputs the setpoint values for the actuator in order to obtain a desired trajectory. This inverse problem uses a QP (quadratic-programming) algorithm based on the equations set by the Finite Element Method. To improve the update rate performances, we propose an asynchronous simulation framework that provides a better trade-off between the deformation accuracy and the computational burden. Complex computations such as accurate FEM deformations are done at low frequency while the control is performed at high frequency with strong real-time constraints. The two simulation loops (high frequency and low frequency loops) are mechanically coupled in order to guarantee mechanical accuracy of the system over time. Finally, the validity of the multi-rate simulation is discussed based on measurements of the evolution in the QP matrix and an experimental validation is conducted to validate the correctness of the high-rate inverse model on a real robot.
This study focuses on a new method dedicated to the modeling and control of Continuum Robots, based on the Finite Element Method (FEM) using quasi-static assumption. The modeling relies on a discretization of the continuum robots using 6 DoFs Frames along the structure of the robot that is compatible with the modeling of a sequence of rigid vertebras. When the robot's structure relies on rods with constant sections, internal forces are computed with beam elements, placed between two adjacent frames, that applies forces and torques. In the opposite, when the robot is composed of a complex shape deformable backbone separated by the rigid vertebras, a domain decomposition strategy is used to obtain an equivalent stiffness between two vertebras using volumetric FEM. In both cases, for solving the whole robot model and inverting it in real-time, the numerical method takes advantage of the serial nature of continuum robots, using a Block-Tri-Diagonal solver. The factor of improvement in the computation time reaches several order of magnitude compared to a classical FEM model, while keeping a good precision. The method has also been implemented and tested on a real pneumatic CBHA trunk designed by Festo Robotics and some complementarity examples have been generated numerically.
IDeaS (ANR JCJC): this is a project targeted at per-operative guidance for interventional radiology procedures. Our main goal is to provide effective solutions for the two main drawbacks of interventional radiology procedures, namely: reduce radiation exposure and provide a fully 3D and interactive visual feedback during the procedure. To do so, our project relies on an original combination of computer vision algorithms and interactive physics-based medical simulation. Defrost is involved with Magrit, MIMESIS and Nancy Hospital.
Partner 1: King's College, Robotics Dept (UK)
Soft robot modeling and control using pneumatic and hydraulic technology
Christian Duriez is member of the WorldHaptics 2015 Conference Editorial Board and Vriphys 2015
Jérémie Dequidt has been member of the program committee of AFIG 2016
Christian Duriez has been reviewer for SIGGRAPH 2015, Vriphys 2015, ISBI 2015.
Jeremie Dequidt has been reviewer for MICCAI 2015, Haptics Symposium 2016
Damien Marchal was reviewer for IEEE 3D User Interface 2016 (3DUI 2016), VriPhys 2015, "la Conférence francophone sur l'Interaction Homme-Machine" (IHM 2015), World Haptics Conference (WHC 2015)
Alexandre Kruszewski was reviewer for American Control Conference 2016, Eurpoean Control Conference 2016, World Haptics Conference 2015, International Conference on Fuzzy Systems 2015.
Christian Duriez has recieved an award from the journal IEEE transactions on haptics for outstanding review activities
Christian Duriez has been reviewer for the following journals: Computer Graphics Forum, IEEE Transaction on Automation Science and Engineering, Mechanism and Machine Theory, IEEE Transactions on Haptics, IEEE Transactions on Visualization and computer Graphics
Jeremie Dequidt was reviewer for the following journals: IEEE Transactions on Haptics, Computers and Graphics, Computer Graphics and Applications, Annals of Biomedical Engineering, REFIG
Alexandre Kruszewski was reviewer for International Journal of Systems Science, Fuzzy sets and systems (2 papers), IEEE Transaction on Fuzzy systems (4 papers), Control Engineering Practice.
Christian Duriez has been invited for a talk on the summer school on surgical robotics (Montpellier) and for the Autumn school on soft robotics and surgery (Torino).
Christian Duriez is one of the founder and scientific advisor of the startup InSimo (www.
Christian Duriez was reviewer for the founding program AIRR (Nord-Pas de Calais) and Ecos-Sud (France-South America)
Master : Christian Duriez, Simulation interactive basé sur la physique, niveau M2, université de Lille, France
Master : Jérémie Dequidt is in charge of the "Fillière Systèmes Communicants (IMA)" at Polytech Lille, France
Master : Jérémie is in charge of the transversal modules, which are common to all students M2 at Polytech Lille, France
Licence : Alexandre Kruszewski is in charge of Teaching Automation in the "Cursus Iteem" (L1 to M2) at Centrale Lille, France
Master : Alexandre Kruszewski, Advanced Automatic, Centrale Lille, France
Master : Damien Marchal, “Introduction to Unity 3D” Master 2 IVI de l’UFR d’IEEA, université de Lille, France
Master : Damien Marchal, “Introduction to Unity 3D” professional training SUDES, université de Lille, France
PhD: Julien Bosman, Physically-based 6DOF Nodes deformable models: Application to connective tissues simulation and soft-robots control, Université de Lille, 27 Novembre 2015, encadrée par C.Duriez
PhD: Raymundo Márquez, «New control and observation schemes based on Takagi-Sugeno models», Université de Valencienne, 12 Novembre 2015, encadrée par A. Kruszewski
PhD in progress: Frederick Largillière, Control and Modelling of Interfaces based on Soft-robots, Unviersité de Lille, advisors: C. Duriez and L.Grisoni (3d year)
PhD in progress: Thor Morales, Feedback Control and Design of Soft-robots manipulators, Unviersité de Lille, advisors: C. Duriez and R.Merzouki (3d year)
PhD in progress: Zhongkai Zhang, Real-time control of deformable robot using visual servoing. Encadrement: J. Dequidt et C.Duriez (1st year)
Christian Duriez was reviewer for the HdR of Pierre Joli (Université d'Evry)
Christian Duriez was member of the committee for the position "Chargés de Recherche" (Inria Lille)
Presentation of a deformable robot in the Senate on 02/11/2015 at the "journée du numérique".
A popular article on the work of the team was simultaneously broadcast on the BIT Blog Site lemonde.fr and the Interstices website.
We worked with the Fresnoy (National Studio of Contemporary Arts) on the work of art: Exo-Biota artist Jonathan Pepe. This work was presented at the Panorama exhibition. It will be exhibited at the Palais de Tokyo in 2017. This project was the subject of an article on the humanoide.fr site.
At the Science Festival, our team has greatly contributed to the itinerant scholars operation. Scientists and engineers have moved in colleges and high school in the region to present their work. This year, 15 one hour lessons were conducted in front of 430 students in secondary schools (http://
Participation in various operations Chti'Code. Chti'Code is an action of the University of Lille 1, which aims to promoting computer education in primary schools. Damien Marchal intervened with 4 CM2 classes for a total of 20h. https://
In march 2015 Presentation of the work of Defrost students ISN Terminal specialty by Eulalie Coevoet and Jeremie Dequidt.
J.Dequidt made an invited presentation and was a member of the jury of robotics competition organized by the High School Baggio (SI @ Baggio): https://
J.Dequidt participated in the organization of the Robotics Cup of Primary Schools Polytech Lille. http://
J.Dequidt was a member of the European competition Android application creation students, supported by Google. https://
A reporter did a paper about our team, posted on the site rue89 (over 11,000 visits to the article): http://
Defrost research works have been presented to students (Terminale ISN) by E. Coevoet and J. Dequidt in march 2015.