The Mint team focuses on gestural interaction, i.e. the use of gesture for human-computer interaction (HCI). The New Oxford American Dictionary defines gesture as a movement of part of the body, especially a hand or the head, to express an idea or meaning. In the particular context of HCI, we are more specifically interested in movements that a computing system can sense and respond to. A gesture can thus be seen as a function of time into a set of sensed dimensions that might include but are not limited to positional information (the pressure exerted on a contact surface being an example of non-positional dimension).
Simple pointing gestures have long been supported by interactive graphics systems and the advent of robust and affordable sensing technologies has somewhat broadened their use of gestures. Swiping, rotating and pinching gestures are now commonly supported on touch-sensitive devices, for example. Yet the expressive power of the available gestures remains limited. The increasing diversity and complexity of computer-supported activities calls for more powerful gestural interactions. Our goal is to foster the emergence of these new interactions, to further broaden the use of gesture by supporting more complex operations. We are developing the scientific and technical foundations required to facilitate the design, implementation and evaluation of these interactions. Our interests include:
gestures captured using held, worn or touched objects or contactless perceptual technologies;
transfer functions possibly used during the capture process;
computational representations of the captured gestures;
methods for characterising and recognising them;
feedback mechanisms, and more particularly haptic ones;
tools to facilitate the design and implementation of tactile and gestural interaction techniques;
evaluation methods to assess the usability of these techniques.
The scientific approach that we follow considers user interfaces as means, not an end: our focus is not on interfaces, but on interaction considered as a phenomenon between a person and a computing system . We observe this phenomenon in order to understand it, i.e. describe it and possibly explain it, and we look for ways to significantly improve it. HCI borrows its methods from various disciplines, including Computer Science, Psychology, Ethnography and Design. Participatory design methods can help determine users' problems and needs and generate new ideas, for example . Rapid and iterative prototyping techniques allow to decide between alternative solutions . Controlled studies based on experimental or quasi-experimental designs can then be used to evaluate the chosen solutions . One of the main difficulties of HCI research is the doubly changing nature of the studied phenomenon: people can both adapt to the system and at the same time adapt it for their own specific purposes . As these purposes are usually difficult to anticipate, we regularly create new versions of the systems we develop to take into account new theoretical and empirical knowledge. We also seek to integrate this knowledge in theoretical frameworks and software tools to disseminate it.
Whatever is the interface, user provides some curves, defined over time, to the application. The curves constitute a gesture (positional information, yet may also include pressure). Depending on the hardware input, such a gesture may be either continuous (e.g. data-glove), or not (e.g. multi-touch screens). User gesture can be multi-variate (several fingers captured at the same time, combined into a single gesture, possibly involving two hands, maybe more in the context of co-located collaboration), that we would like, at higher-level, to be structured in time from simple elements in order to create specific command combinations. One of the scientific foundations of the research project is an algorithmic and numerical study of gesture, which we classify into three points:
clustering, that takes into account intrinsic structure of gesture (multi-finger/multi-hand/multi-user aspects), as a lower-level treatment for further use of gesture by application;
recognition, that identifies some semantic from gesture, that can be further used for application control (as command input). We consider in this topic multi-finger gestures, two-handed gestures, gesture for collaboration, on which very few has been done so far to our knowledge. On the contrary, in the case of single gesture case (i.e. one single point moving over time in a continuous manner), numerous studies have been proposed in the current literature, and interestingly, are of interest in several communities: HMM , Dynamic Time Warping are well-known methods for computer-vision community, and hand-writing recognition. In the computer graphics community, statistical classification using geometric descriptors has previously been used ; in the Human-Computer interaction community, some simple (and easy to implement) methods have been proposed, that provide a very good compromise between technical complexity and practical efficiency .
mapping to application, that studies how to link gesture inputs to application. This ranges from transfer function that is classically involved in pointing tasks , to the question to know how to link gesture analysis and recognition to the algorithmic of application content, with specific reference examples.
We ground our activity on the topic of numerical algorithm, expertise that has been previously achieved by team members in the physical simulation community (within which we think that aspects such as elastic deformation energies evaluation, simulation of rigid bodies composed of unstructured particles, constraint-based animation... will bring up interesting and novel insights within HCI community).
Our scientific approach in the design and control of haptic devices is focused on the interaction forces between the user and the device. We search of controlling them, as precisely as possible. This leads to different designs compared to other systems which control the deformation instead. The research is carried out in three steps:
identification: we measure the forces which occur during the exploration of a real object, for example a surface for tactile purposes. We then analyse the record to deduce the key components – on user's point of view – of the interaction forces.
design: we propose new designs of haptic devices, based on our knowledge of the key components of the interaction forces. For example, coupling tactile and kinesthetic feedback is a promising design to achieve a good simulation of actual surfaces. Our goal is to find designs which lead to compact systems, and which can stand close to a computer in a desktop environment.
control: we have to supply the device with the good electrical signals to accurately output the good forces.
EVITA is a tactile feedback tablet, produced by Hap2U SME company, based in grenoble. It is presented at CES in january 2017, the SME has been awarded a CES innovation award. This device is issued from a strong collaboration with MINT group. Eric Vezzoli PhD thesis, contributed significantly to this device. EVITA is a very generic interaction device, and several projects are currently being discussed for understanding the fields of applications of this device. It is also, in particular, the hardware support for our haptic book for children, described below, that is our second highlight for this raweb.
The first digital book augmented with a high fidelity feedback has been released in October 2016. Based on a scenario and illustrations made by Dominique Maes - an artist from Belgium - this haptic book was presented for the first time during "la nuit des bibliothèques" in Lille. The popularity of this project as well as its possible social outcomes were underlined in a paper in a national magazine ("Science et Avenir", November 2016)
MINT played an active role in the "Oeuvres et recherches" project (http://
The ControllAR project, started in 2016, investigates the appropriation of visual feedback on control surfaces for multimedia production systems. It has already yielded many results. The system and results of a study on electronic musicians were presented both as a paper and as a demo at the ACM Internation conference on Surfaces and Spaces (ISS 16) where it received a best demo award. The software was released and is available at http://
Best demo award for ControllAR : appropriation of visual feedback on control surfaces @ ACM International Conference on Interactive Surfaces and Spaces (ISS 16).
Best work in progress at Eurohaptics 2016 for the work The human perception of transient frictional modulation, David Gueorguiev, Eric Vezzoli, André Mouraux, Betty Semail, Jean-Louis Thonnard
SME Hap2U had a "CES innovation award", based on the collaboration that MINT group has with them (E-vita tactile feedback tablet) at CES (january 2017).
Functional Description
ControllAR is a novel system that facilitates the appropriation of rich visual feedback on control surfaces through remixing of graphical user interfaces and augmented reality display.
Contact: Florent Berthaut
Participants: Nicolas Bremard and Laurent Grisoni
Contact: Laurent Grisoni
Functional Description
Revil is an application for building and manipulating 3D SceneGraphs for Mixed-Reality Artistic Performances. It is based on OpenGL/GLSL(glfw, glm), OpenNI2, FLTK and is entirely controllable via OpenSoundControl messages. It relies on the approach of revealing virtual content in the physical space by intersecting it with performers and spectator's bodies and props.
Contact: Florent Berthaut
Functional Description
TaxtelOSC is a software wich uses the concept of taxtel to reproduce rich and dense tactile feedback on
Contact: Laurent Grisoni
Florent Berthaut, Alex Jones
Despite the development of touchscreens, many expert systems for working with digital multimedia content, such as in music composition and performance, video editing or visual performance, still rely on control surfaces. This can be due to the accuracy and appropriateness of their sensors, the haptic feedback that they offer, and most importantly the way they can be adapted to the specific subset of gestures and tasks that users need to perform. On the other hand, visual feedback on controllers remains limited and/or fixed, preventing similar personalizing. In this paper, we propose ControllAR, a novel system that facilitates the appropriation of rich visual feedback on control surfaces through remixing of graphical user interfaces and augmented reality display. We then use our system to study current and potential appropriation of visual feedback in the case of digital musical instruments and derive guidelines for designers and developers.
Hanaë Rateau, Yosra Rekik, Laurent Grisoni, Joaquim Jorge
We present an interaction technique combining tactile actions and Midair pointing to access out-of-reach content on large displays without the need to walk across the display. Users can start through a Touch gesture on the display surface and finish Midair by pointing to push content away or inversely to retrieve a content. The technique takes advantage of wellknown semantics of pointing in human-to-human interaction. These, coupled with the semantics of proximal relations and deictic proxemics make the proposed technique very powerful as it leverages on well-understood human-human interaction modalities. Experimental results show this technique to outperform direct tactile interaction on dragging tasks. From our experience we derive four guidelines for interaction with large-scale displays.
Sofiane Ghenna, Christophe Giraud-Audine, Michel Amberg, Frédéric Giraud, Betty Lemaire-Semail
In this study, we develop and implement a method for superimposing two vibration modes in order to produce different tactile stimuli on two fingers located in different positions. The tactile stimulation is based on the squeeze film effect which decreases the friction between a fingertip and a vibrating plate.
Experimental test have been conducted on a 1D tactile device. They show that it is possible to continuously control the friction on two fingers moving independently. Then, we developed the design of a 2D device based on the same principle, which gives rise to the design of a two fingers tactile display. Evaluations were conducted using a modal analysis with experimental validation.
Farzan Kalantari, Laurent Grisoni, Frédéric Giraud, Yosra Rekik
Tactile devices with ultrasonic vibrations (based on squeeze film effect) using piezoelectric actuators are one of the existing haptic feedback technologies. In this study we have performed two psychophysical experiments on an ultrasonic haptic tablet, in order to find the minimum size of a tactile element on which all the users are able to perfectly identify different types of textures. Our results show that the spatial resolution of the tactile element on haptic touchscreen actually varies, depending on the number and types of tactile feedback information. A first experiment exhibits three different tactile textures, chosen as being easily recognized by users. We use these textures in a second experiment, and evaluate minimal spatial area on which the chosen set of textures can be recognized. Among other, we find the minimal size depends on the texture nature.
Florent Berthaut, Luke Dahl, Patricia Plénacoste
Orchestras of Digital Musical Instruments (DMIs) enable new musical collaboration possibilities, extending those of acoustic and electric orchestras. However the creation and development of these orchestras remain constrained. In fact, each new musical collaboration system or orchestra piece relies on a fixed number of musicians, a fixed set of instruments (often only one), and a fixed subset of possible modes of collaboration. In this paper, we describe a unified framework that enables the design of Digital Orchestras with potentially different DMIs and an expand-able set of collaboration modes. It relies on research done on analysis and classification of traditional and digital orchestras, on research in Collaborative Virtual Environments, and on interviews of musicians and composers. The BOEUF framework consists of a classification of modes of collaboration and a set of components for modelling digital orchestras. Integrating this framework into DMIs will enable advanced musical collaboration modes to be used in any digital orchestra, including spontaneous jam sessions.
Current work on this project consists in the implementation of BOEUF in the PureData programming language and in the study of its impact on musical collaboration during short improvised jam sessions.
Hap2U SME is licenced two patents of MINT team.
The aim of this project is to create the first digital book, enhanced with haptic feedback, in order to anticipate the integration of this technology into everyday products. This project adresses technological issues, like programming haptic content in a multimedia software, and design issues to understand how the haptic feedback is perceived by the users.
Stimtac is a book, and could thus be presented to non-specialists users and to a wide public during presentations, demos and foru. The scenario and the illustrations were made by Dominique Maes, a belgium artist, who did the digital book "bleu de toi" among other things. The Public Library of Lille is a partner of this project and allows us to meet the public.
This project has been granted 8Keuros from IRCICA.
The purpose of this project is twofold. It aims at designing and implementing hardware solutions for tactile feedback based on programmable friction. It also aims at developing the knowledge and software tools required to use these new technologies for human-computer interaction. Grant for MINT is balanced on 272 keuro handled at University for L2EP, and 220 Keuros for Inria.
Partners: STMicroelectronics, CEA/LETI, Orange Labs, CNRS, EASii IC, MENAPIC and ALPHAUI.
Competitive clusters involved: Minalogic, Cap Digital and MAUD.
3 Meuros project, co-funded by ERDF for the developpement of a pluri-disciplinary project on ICT-based tools for understanding human perception of visual contents. Laurent Grisoni is member of the lead group of this project, and animates an axis devoted to art-sciences and technologies collaborations.
Funds: 4 Meuros (validated at national level, funded by Region), and 1 Meuro additional funding provided by ERDF.
Subject: ICT tools for mediation and access to knowledge.
Lead: University of Lille, University of Artois. Laurent Grisoni is co-lead of this project.
290 Keuros for MINT. Funding for two 18 months contracts and 24 months of post-doc.
Subject: rationalized process for industrial use of holographic displays.
MINT contribution: anamorphic software tools for holographics displays, and study of interactive aspects, including collaborative activities. This project has been prematurily stopped by french government.
Art/science Inria internal network gathering projects interested in collaborating with artists.
Inria teams involved: MuTANT (paris), Imagine (grenoble), Flowers, Potioc (Bordeaux), Hybrid, MimeTic (Rennes). This initiative shall take advantage of an agreement between Inria and french ministry of culture, signed early december 2016.
This regional project, funded by ERDF, led by lille school of architecture, aims at understanding in which way 3D printing may be interesting for the building economy. partners: école d'architecture de Lille, Inria, ecole centrale de lille, télécome Lille 1, Ecole des mines de douai.
INESC-ID: collaboration with Joaquim Jorge (Talaria paper, published at ISS'16)
Collaboration with Mrad UofT (paper published, harvesting energy)
Dr. Luke Dahl (University of Virginia) for the BOEUF project
Masaya Takasaki and Masayuki Hara (University of Saitama, Japan) 22nd of january
Masaya Takasaki has also been visiting Professor at University lille1 (April, 18th - April 30th)
visiting PhD student from University of Chile: Orlando Errazo (nov 2015-jan 2016). One publication currently on submission.
Laurent Grisoni : PC for VISIGRAPP (IEEE InfoViz Art Track), Computer Graphics International (CGI, computer graphics), MOCO (international workshop on gesture), ISEA (art-science), GRAPP (computer graphics)
Florent Berthaut: Reviewer for ACM CHI Conference and NIME conference
Laurent Grisoni: Eurohaptics, ACM UIST, ACM CHI
Frédéric Giraud is Associate Editor of IEEE Transactions on Haptics
Florent Berthaut: Reviewer for IEEE Multimedia
Laurent Grisoni: Computer & Graphics,
Licence: Florent Berthaut, Spreadsheets et VBA programming, LEA L3 TCI (22h), Web programming, LEA L3, Université Lille 3, France
Licence: Frédéric Giraud, Physique pour le génie électrique (30h), Université Lille 1
Master: Christophe Giraud-Audine: Control of electrical machines (30h), Power Electronics (30h), signal processing (30h), niveau M1, ENSAM, France
Master: Florent Berthaut, Web programming, LEA M1 TSM (16,5h) and LEA M1 RICI (48h), Database, LEA M1 ANI (22h), Université Lille 3, France
Master : Frédéric Giraud, Control of electrical machines (30h), Power Electronics (40h) niveau M1, Université Lille1, France
Master : Laurent Grisoni, NIHM : nouvelles Interactions Homme-Machine, (6h), niveau M2, Université Lille 1, France
Master: Laurent Grisoni, représentation et compression de données (24h), introduction à la programmation (38h), cryptographie (8h), Ecole Polytech'lille (dept IMA)
Master: Laurent Grisoni: gestion de projet en Co-design interdisciplinaire, Master Sciences et Culture du Visuel, Université de Lille Sciences Humaines et Sociales, Master Sciences et Cultures du Visuel (12h)
Master : Laurent Grisoni, IHM et Interface à Gestes, (24h), niveau M2 (IMA5), Polytech Lille, France
PhD : Ehsan Enferad, Modélisation et Commande d’une Interface Tactile à Stimulation Hybride par Modulation de Friction et Retournement Temporel, nov. 2015, F. Giraud, C. Giraud-Audine
PhD in progress: Cagan Arslan, Fusion de données pour l'interaction homme-machine, oct. 2015, L. Grisoni/J. Martinet
PhD in progress: Farzan Kalantari, Interaction sur dispositif à retour tactile et kinesthésique, oct. 2014, L. Grisoni, F. Giraud
PhD Sofiane Ghenna, Contrôle multimodal d'acionneurs piézo-électriques pour applications tactiles, F.Giraud, C.Giraud-Audine, University Lille1, 30th November 2016
PhD Thomas Sednaoui, Tactile feedback integration on mobile communication devices, B Semail, F Casset, University Lille1, 14th dec. 2016
PhD : Eric Vezzoli, Tactile feedback devices: friction control and texture generation, University Lille1, 22 Sept 2016, B. Lemaire-Semail, F. Giraud
PhD in progress: Hanae Rateau, l'interaction esquissée, oct. 2012, L. Grisoni
PhD in progress: Vincent Gouezou: L'architecte et ses outils, au travers de l'histoire et dans sa relation actuelle au numérique, oct. 2014, L. Grisoni 25% (with F. Vermandel, architect, Lille school of architecture)
PhD in progress: Charlotte Planckeel, Le sens de la lacunae en archéologie de l’Âge du bronze, archéologie et outils numériques, L. Grisoni (25%, with A. Lehoerff, Lille 3, archeologist)
PhD in progress: Olivier Capra, Interaction de présentation, oct. 2016, L. Grisoni, F. Berthaut
Futur en Seine (Paris, 9-10 june 2016; http://
Nuit des bibliothèques (Lille, 15th october 2016, http://
Forum du CNRS (Lille, 19th November 2016, http://
Nuit des partenaires (Lille, 29th november 2016, https://
Salon Humanités connectées, Toulouse, dec. 2016.
Rencontres Inria Entreprises (25 novembre 2016), imaginarium.
journées RV en haut de france, 24 novembre 2016, imaginarium.
tables ronde journées régionales de l'innovation, amiens, 20 novembre 2016.
BOEUF project workshops. Introduction to PureData and digital orchestras @ La condition publique (Roubaix) and l'Imaginarium (Tourcoing) http://