The overall objective of Potioc is to open 3D digital worlds
Printing, photography, Internet, are examples of technologies that have changed the way people communicate together, understand, create, or get entertainment. At the beginning, these technologies were dedicated to expert users, then they have become available to all, contributing to the fulfillment of everyone. Today, 3D digital worlds could play a similar role, but they tend to be fully exploited only by a limited number of highly trained people. Our motivation is to democratize such 3D technologies for favoring communication, understanding, creation, or entertainment for everyone.
For example, in the scope of learning, it can be difficult for a child to understand how molecules can be assembled together from a drawing on a piece of paper, as classically done in school. We can presume that, with a 3D visualization of the molecules, the understanding of the data can be improved. Beyond a 3D visualization, it will be particularly important for the child to manipulate the molecules to understand how they fit together. Indeed, perception and action are extremely linked, as it has been shown by Berthoz. By being active in the process of interactive visualization, the implication of the child will be increased and, consequently, the learning process will be favored. Such an example requires the completion of complex interaction tasks for which there is no standard solution. Numerous research questions are thus linked to such a challenge, and this is what we want to address in Potioc.
To reach this goal, we focus on the interaction aspect, which we consider as the main barrier for increasing the adoption of 3D digital worlds by all. Indeed, without a proper interface, users would not be able or would not want to use 3D digital worlds. We address 3D interaction tasks such as navigation in 3D environments, manipulation of 3D objects, exploration of volumes, etc. The conception of relevant user interfaces for completing such tasks implies tackling at the same time motor, perceptive, and cognitive aspects, at different levels: hardware, software, and at the user level. In Potioc, we follow a multi-disciplinary approach where our research and development works are arranged according to the following three main axes:
Exploring and enhancing input/output interaction space
Designing targeted interaction techniques
Understanding and assessing user interaction
We do not focus on one specific technology. Instead, we take advantage of the wide input and output space that is available today, with a special interest for emerging technologies. In particular, we consider both technologies that are able to detect motor activities, i.e., physical actions of the user (e.g., joysticks, multitouch surfaces and 3D spatial technologies) and interfaces measuring and interpreting the physiological signals of the user, i.e., non-motor activities of the user. This notably includes Brain-Computer Interfaces (BCI)
Based on this rich interaction space, we design interaction techniques dedicated to the completion of 3D interaction tasks. In this project part, we put aside hardware considerations to concentrate on the software part of the UIs. Such interaction techniques target users that are not expert in 3D interaction. In particular, our target users are the general public, specialists with no expertize in 3D, and people with impairments.
Finally, a large part of the project is dedicated to human factors for understanding and for evaluating user interaction. In particular, we conduct experiments a-priori to guide our developments and a-posteriori to assess them. We also explore passive BCI to better evaluate 3D interaction.
Previous research works on 3D UI tended to be driven by criteria of speed, efficiency or precision for industrial purposes. These criteria, which are very important in domains where profitability is crucial, may not be the most fundamental ones in our project where the approach is to increase the intrinsic motivation of non-expert users. Therefore, in Potioc, we will emphasize criteria such as appeal and enjoyment, which have been scarcely taken into account in the past. Indeed, we believe that non-expert users need pleasant and enjoyable interfaces to adopt 3D digital worlds, in the scope of education, art, assistive technologies, or entertainment.
Potioc has organized IHM 2013, the French conference on Human-Computer Interaction, in cooperation with the AFIHM association. This conference gathered about 125 participants. A dedicated collection has been created on HAL.
A tutorial about interaction techniques for 3D environments was presented at Eurographics 2013 and Web3D 2013 .
The design of new user interfaces is a complex process that requires tackling research challenges at different levels. First, at a technological level, the input and output interaction space is becoming richer and richer. We will explore the new input/output modalities offered by such a technological evolution, and we will contribute to extend these modalities for the purpose of our main objective, which is to make 3D digital worlds available to all. Then, we will concentrate on the design of good interaction techniques that rely on such input/output modalities, and that are dedicated to the population targeted by this project, i.e. general public, specialists which are not 3D experts, and people with impairments. Finally, a large part of our work will be dedicated to the understanding and the assessment of user interaction. In particular, we will conduct user studies to guide the design of hardware and software UI, to evaluate them, and to better understand how a user interacts with 3D environments.
These three levels, input/output modalities, interaction techniques, and human factors will be the three main research directions of Potioc. Of course, they are extremely linked, and they cannot be studied independently, one after the other. In particular, user studies will follow the design process of hardware/software user interfaces from the beginning to the end, and both hardware and software exploration will be interdependent. The design of a new 3D user interface will thus require some work at different levels, as illustrated in Figure . All members of Potioc will contribute in each of these research directions.
The Potioc project-team will be widely oriented towards new innovative input and output modalities, even though standard approaches based on keyboard/mouse and standard screens will not be excluded. This includes motor-based interfaces, and physiological interfaces like BCI, as well as stereoscopic display and augmented reality setups. These technologies may have a great potential for opening 3D digital worlds to everyone, if they are correctly exploited.
We will explore various input/output modalities. Of course, we will not explore all of them at the same time, but we do not want to set an agenda either, for focusing on one of them. For a given need fed by end-users, we will choose among the various input/output modalities the ones that have the biggest potential. In the following paragraphs, we explain in more details the research challenges we will focus on to benefit from the existing and upcoming technologies.
There is a wide number of sensors that can detect users' activity. Beyond the mouse that detects x and y movements in the plane, various sensors are dedicated to the detection of 3D movements, pressure, brain and physiological activity, and so on. These sensors provide information that may be very rich, either to detect command intent from the user, or to estimate and understand the user's state in real-time, but that is difficutly exploitable as it. Hence, a major challenge here is to extract the relevant information from the noisy raw data provided by the sensor.
An example, and important research topic in Potioc, is the analysis of brain signals for the design of BCI. Indeed, brain signals are usually measured by EEG, such EEG signals being very noisy, complex and non-stationary. Moreover, for BCI-based applications, they need to be processed and analyzed in real-time. Finally, EEG signals exhibit large inter-user differences and there are usually few examples of EEG signals available to tune the BCI to a given user (we cannot ask the user to perform thousands of time the same mental task just to collect examples). As such, appropriate signal processing algorithms must be designed in order to robustly identify EEG patterns reflecting the user's intention. The research challenges are thus to design algorithms with high performances (in terms of rate of correctly identified user's state) anytime, anywhere, that are fully automatic and with minimal or no calibration time. In other words, we must design BCI that are convenient, comfortable and efficient enough so that they can be accepted and used by the end-user. Indeed, most users, in particular healthy users in the general public are used to highly convenient and efficient input devices (e.g., a simple mouse) and would not easily tolerate systems with a lower performance. Achieving this would make BCI good enough to be usable outside laboratories, e.g., for video gamers or patients. This will also make BCI valuable and reliable evaluation tools, e.g., to understand users' state during a given task. To address these challenges, pattern recognition and machine learning techniques are often used in order to find the optimal signal processing parameters. Similar approaches may contribute to the analysis of signals coming from other input devices than BCI. An example is the exploitation of depth cameras, where we need to find relevant information from noisy signals. Other emerging technologies will require similar attention, where the goal will be to transform an unstructured raw signal into a set of higher level descriptors that can be used as input parameters for controlling interaction techniques.
Similarly to the input side, the feedback provided to the user through various output modalities will be explored in Potioc. Beyond the standard screens that are commonly used, we will explore various displays. In particular, in the scope of visual restitution, we will notably focus on large screens and tables, mobile setups and projection on real objects, and stereoscopic visualization. The challenge here will be to conceive good visual metaphors dedicated to these unconventional output devices in order to maximize the attractiveness and the pleasure linked to the use of these technologies.
For example, we will investigate the use of stereoscopic displays for extending the current visualization approaches. Indeed, stereoscopic visualization has been little explored outside complex VR setups dedicated to professional users and 3DTV. We believe that this modality may be very interesting for non-expert users, in wider contexts. To reach this goal, we will thus concentrate on new visual metaphors that benefit from stereoscopic visualization, and we will explore how, when, and where stereoscopy may be used.
Depending on the targeted interaction tasks, we may also investigate various additional output modalities such as tangible interaction, audio displays, and so on. In any case, our approach will be the same: understanding how new perceptive modalities may push the frontier of our current interactive systems.
In addition to the exploration and the exploitation of existing input and output modalities for enhancing interaction with 3D content, we may also contribute to extend the current input/output interaction space by building new interactive systems. This will be done by combining hardware components, or by collaborating with mechanics/electronics specialists.
In the previous section, we focused on the input/output interaction space, which is closely related to hardware components. In this part, we focus on the design of interaction techniques, which we define here as the means through which a user will complete an interaction task in a given interaction space. Even if this is naturally also linked to the underlying hardware components, the research conducted in this axis of the project will mainly concern software developments.
Similar to the input/output interaction space, the design of interaction techniques requires focusing on both the motor and the sensory components. Thus, in our 3D spatial context, the challenges will be to find good mappings between the available input and the DOF that need to be controlled in the 3D environment, and to provide relevant feedback to users so that they can understand well what they are doing.
The design of interaction techniques should be strongly guided by the targeted end-users. For example, a 3D UI dedicated to an expert user will not suit a novice user, and the converse is also true. In Potioc, where the final goal is to open 3D digital worlds to anyone, we will concentrate on the general public, specialists that are not 3D experts, and people with impairments.
3D UIs have mainly been designed for professional use. For example, modeling tools require expertize to be used correctly and, consequently, they exclude the general public from the process of creating 3D content. Similarly, immersive technologies have been dedicated to professional users for a long time. Therefore, immersive 3D interaction techniques have generally been thought for trained users, and they may not fit well with a general public context. In Potioc, an important motivation will be to re-invent 3D UIs to adapt them to the general public. This motivation will guide us towards new approaches that have been little explored until now. In particular, to reach our objective, we will give a strong importance to the following criteria:
Intuitiveness: a very short learning curve is required.
Enjoyability: this is needed to motivate novice users in the complex process of interaction with 3D content.
Robustness: the UIs should support untrained users that may potentially interact with unpredictable actions.
In addition, we will keep connected with societal and technological factors surrounding the general public. For example, [multi]touch-screens have become very popular these past few years, and everyone tends to be familiar with a standard gesture vocabulary (e.g. pinch gestures and flicking gestures). We will rely on these commonly acquired ways-of-interact to optimize the acceptability of the 3D UIs we will design. In this part of the project the challenge will be to conceive 3D UIs that offer a high degree of interactivity, while ensuring an easy access to technology, as well as a wide adherence.
General public will be one of the main targets of Potioc for the design of 3D UIs. However, we do not exclude specialists, who have little experience with 3D interaction. These specialists can be for example artists, archaeologists, or architects. In any case, we are convinced that 3D digital worlds could benefit to such categories of users if we propose dedicated 3D UIs that allow them to better understand, communicate, or create, with their respective skills. Because such specialists will gain expertize while interacting with 3D content, it will be necessary to design 3D UIs that can adapt to their evolving level of expertize. In particular, the UIs should be easy to use and attractive enough to encourage new users. At the same time, they should provide advanced features that the specialist can discover while gaining expertize.
While the general public has been only scarcely considered as a potential target audience for 3D digital worlds, another category of users is even more neglected: people with impairments. Indeed, such people, in particular those with motor impairments, are unable to use classical input devices, since they have been designed for healthy users. People with motor impairment have to use dedicated input devices, adapted to their disabilities, such as a single switch. Since such input devices usually have much fewer degrees of freedom than classical devices, it is necessary to come up with appropriate interaction techniques in order to efficiently use this limited number of DOF to still enable the user to perform complex tasks in the 3D environment. In Potioc, our focus will be on the use of BCI to enable motor impaired users to interact with 3D environment for learning, creation and entertainment. Indeed, BCI enable a user to interact without any motor movement.
The exploration of the input/output interaction space, and the design of new interaction techniques, are strongly linked with human factors, which will be the third research axis of the Potioc project. Indeed, to guide the developments described in the previous sections, we first need to well understand users' motor and cognitive skills for the completion of 3D interaction tasks. This will be explored thanks to a-priori experiments. In order to evaluate our hardware and software interfaces, we will conduct a-posteriori user studies. Finally, we will explore new approaches for a real-time cognitive analysis of the performance and the experience of a user interacting with a 3D environment.
The main challenge in this part of the project will be to design good experimental protocols that will allow us to finely analyze various parameters for improving our interfaces. In 2D, there exist many standard protocols and prediction laws for evaluating UIs (e.g. Fitts law and ISO 9241). This is not the case in 3D. Consequently, a special care must be taken when evaluating interaction in 3D spatial contexts.
In addition to the standard experiments we will conduct in our lab, we will conduct large scale experiments thanks to the strong collaboration we have with the center for the widespread diffusion of scientific culture, Cap Sciences (see Collaboration section). With such kind of experiments, we will be able to test hundreds of participants of various ages, gender, or level of expertize that we will be able to track thanks to the Navinum system
Before designing 3D UIs, it is important to understand what a user is good at, and what may cause difficulties. This is true at a motor level, as well as a cognitive level. For example, are users able to coordinate the movements of several fingers on a touchscreen at the same time, or are they able to finely control the quantity of force applied on it while moving their hand? Similarly, are the users able to mentally predict a 3D rotation, and how many levels of depth are they able to distinguish when visualizing stereoscopic images? To answer these questions, we will conduct preliminary studies.
Our research in that direction will guide our developments for the other research axes described above. For example, it will be interesting to explore touch-based 3D UIs that take into account several levels of force if we see that this parameter can be easily handled by users. On the other hand, if the results of a-priori tests show that this input cannot be easily controlled, then we will not push forward that direction.
The members of Potioc have already conducted such kinds of experiments, and we will continue our work in that direction. For some investigations, we will collaborate with psychologists and experts in cognitive science (see Collaborations section) to explore in more depth motor and cognitive human skills.
A-priori studies will allow us to understand how users tend to ”naturally” interact to complete 3D interaction tasks, and to understand which feedbacks are the best suited. This will be a first answer to our global quest of providing pleasant interfaces. Indeed, this will allow us to adapt the UIs to the users, and not the opposite. This should enhance the global acceptability and motivation of users facing a new interactive system.
In Potioc, we will conceive new hardware and software interfaces. To validate these UIs, and to improve them, we will conduct user experiments, as classically done in the field of HCI. This is a standard methodology that we currently follow (see Bibliography). We will do this in our lab, and in Cap Sciences.
Beyond the standard evaluation criteria that are based on performance for speed, accuracy, coordination, and so on, we will also consider other criteria that are more relevant for the Potioc project. Indeed, we will give a great importance to enjoyability, pleasure of use, accessibility, and so on. Consequently, we will need to redefine the standard way to evaluate UIs. Once again, our relationship with Cap Sciences will help us in such investigations. The use of questionnaires will be a way to better understand how an interface should be designed to reach a successful use. In addition, we will observe and analyze how visitors tend to interact with various interfaces we will propose. For example, we will collect information like the time spent on a given interactive system or the number of smiles recorded during an interaction process. The identification of good criteria to use for the evaluation of a popular 3D UI will be one of the research directions of our team.
Conducting such a-posteriori studies, in particular with experts of mediation, with new criteria of success, will be a second answer to our goal of evaluating the pleasure linked to the use of 3D UIs.
Classically, the user's subjective preferences for a given 3D UI are assessed using questionnaires. While these questionnaires provide important information, this is only a partial, biased, a-posteriori/a-priori measure, since they are collected before or after the 3D interaction process. When questionnaires are administered during 3D interaction, this interrupts and disturbs the user, hence biasing the evaluation. Moreover, while evaluating performance and usefulness is now well described and understood, evaluating the user's experience and thus the system usability appears as much more difficult, with a lack of systematic and standard approaches. Ideally, we would like to measure the user response and subjective experience while he/she is using the 3D UI, i.e., in real-time and without interrupting him/her, in order to precisely identify the UI pros and cons. Questionnaires cannot provide such a measure.
Fortunately, it has been recently shown that BCI could be used in a passive way, to monitor the user's mental state. More precisely, recent results suggested that appropriately processed EEG signals could provide information about mental states such as error perception, attention or mental workload. As such, BCI are emerging as a new tool to monitor a user's mental state and brain responses to various stimuli, in real-time. In the Potioc project, we propose a completely new way to evaluate 3DUI: rather than relying only on questionnaires to estimate the user's subjective experience , we propose to exploit passive BCI to estimate the user's mental state in real-time, without interrupting nor disturbing him or her, while he/she is using the 3DUI. In particular, we aim at measuring and processing EEG and other biosignals (e.g., pulse, galvanic skin response, electromyogram) in real-time in order to estimate mental states such as interaction error potentials or workload/attention levels, among others. This will be used to finely identify how intuitive, easy-to-use and (ideally) enjoyable any given 3D UI is. More specifically, it will allow us to identify how, when and where the UI has flaws. Because the analysis will occur in real-time, we will potentially be able to modify the interface while the user is interacting. This should lead to a better understanding of 3D interaction. The work that will be achieved in this area could potentially also be useful for 2D interface design. However, since Potioc's main target is 3DUI, we will naturally focus the real-time cognitive evaluations on 3D contexts, with specific targets such as depth perception, or perception of 3D rotations.
This real-time cognitive analysis will be a third answer to reach the objectives of Potioc, which are to open 3D digital worlds to everyone by increasing the pleasure of use.
Since our project aims at providing 3D digital worlds to all, including the general public, to stimulate understanding, learning, communication and creation, our scope of applications will naturally be the following one:
Culture and education: We are convinced that a 3D digital world is a powerful media that may contribute to enhance understanding processes. For example, a museum would benefit from new 3D user interfaces allowing visitors to better understand complex content. Similarly, at school, this media has an extraordinary potential for enhancing learning. For example, a child being able to navigate in archaeological sites, or to manipulate by himself 3D molecules as described previously, will probably understand and learn things while having pleasure in interacting with the content.
Art: We believe that 3D digital worlds may stimulate creativity, too. Our first investigations with music and drawings have shown that this media opens new possibilities for creation. The challenge here will be to design good interfaces that will allow artists to explore new dimensions. The user may be an experienced artist, or a three years old child who would express his creativity through tools that go beyond papers and pens.
Assistive technologies: People suffering from motor or cognitive impairments are one of the target populations of the Potioc project. Indeed, we believe that new interfaces that exploit 3D digital worlds may help people to overcome their disabilities. For example, someone with very reduced motor capabilities could benefit from BCI to explore a virtual museum, or a children having difficulties for concentrating may benefit from new 3D interactive systems.
Entertainment: The objective of Potioc is to open 3D digital worlds to everyone by designing innovative interfaces driven by enjoyment. Consequently, the entertainment industry will be an obvious application domain where Potioc can contribute. This can be in the scope of video games, entertainment parks, Web and TV of the future, applications for mobile devices, and so on.
Naturally, we will not necessarily address all these applications, and certainly not all at the same time. These areas just define the applicative scope of our work. As an example, much of our current work is targeted at artistic and entertainment applications, with VR-based musical performances, augmented paper-based drawing or BCI-based video games. We are also currently starting to conduct research on digital cities, in order to provide ordinary citizens suitable tools and UI to explore 3D content related to their city, such as 3D maps, information about population density or sound nuisance, among other. It should also be noted that our work might find applicative connexions outside these main application domains and benefit to a large range of academic and industrial areas, with which we could build relationships. For example, in the scope of medicine, new and easy to adopt user interfaces designed in Potioc could prove valuable for medical professionals as well, to better access and interact with biological 3D content (e.g., X-rays or MRI scans).
As part of our research work on BCI, we contribute to the development of the OpenViBE
As part of the research on Virtual Reality for Musical Performance, notably the Drile system, various software pieces are being developped and made available to the community. These software pieces are the following:
Pure-Data external to access data from the Virtual Reality Peripheral Network : https://
As part of the research on the "Villes transparentes" project in collaboration with Mappy (Solocal group) and Vectuel - VirtuelCity, we develop a software platform dedicated to the design and evaluation of innovative interaction techniques for the navigation in urban 3D environments. This code is not publicly available yet.
As part of his thesis work, Jérémy Laviole has developed a software suite for PapARt : Paper Augmented Reality Toolkit.
This suite enables the calibration of depth cameras such as the Kinect with a planar surface and with a videoprojection. It also enables the detection of finger touch on the planar surface.
This system can be extended to 3D objects. Many external devices are compatible, such as pen tablets (Wacom) and LEAP Motion. The code runs on Linux, Mac OS and Windows, and is made for Processing
This study aimed at characterizing today's most common interaction techniques for street-level navigation in 3D digital cities, for mobile touch devices, in terms of their efficiency and usability. To do so, we conducted a user study, where we compared target selection (Go-To), rate control (Joystick), position control, and stroke-based control navigation metaphors (see Figure ). The results suggest that users performed best with the Go-To interaction technique. The subjective comments showed a preference of novices towards Go-To and expert users towards the Joystick technique. This work has been accepted for publication at the 3DUI 2014 conference .
As part of this project on Navigation in 3D digital cities, the Potioc group also built a tutorial about interaction techniques for 3D environments. It was presented at Eurographics 2013 and Web3D 2013 . The goal of this work is to provide an up-to-date state-of-the-art of this topic to the community.
We developed tools that enable precise interactive projection on pieces of paper. The sheets of paper are tracked by a camera while the user's inputs (e.g., touch and hovering events) are detected by a Kinect. The paper acts as a screen, its image coming from an overhead projector. The focus of this work is to use such tools to assist the creation of physical drawings and painting. In this context we propose Digital Construction Lines (DCL), in opposition with physical construction lines. Traditionally, the structure of a physical drawing can be created with construction lines which are light pencil strokes. These strokes are then erased during the drawing process. With DCL, it is not required to erase the construction lines anymore. Furthermore, it is possible to create construction lines on fragile material like a canvas for waterpainting or on fresh paint. It also enables construction lines on a dark canvas. In addition to these projection advantages, it is possible to create these DCL interactively and directly onto the support. Consequently, the DCL complement the physical ones during the creation process.
We investigated in a user study if the DCL could effectively replace the physical construction lines, and compared the performance (speed, cleanliness) between the two kinds of construction lines. In this user study we also evaluated the quality and usability of projection of thin lines in a fully controlled environment with a low-cost setup. The study showed that DCL could effectively replace physical construction lines, even though it might not be desirable. The study also showed that the drawing experience was as pleasant with projection, and with the usual tools. The feedback about the quality of tracking and projection was also positive. The only negative evaluation concerned the size of the projection area, which was limited by the resolution of the projector. This work was published as part of Jérémy Laviole's PhD thesis .
PapARt was also used as part of a museum exhibition on the Lascaux caves, together with other 3D UI from Potioc. This exhibitation has provided us with the opportunity to experiment with touch-based interfaces for manipulating 3D virtual objects. We targeted three tasks: observing rare objects with Cubtile, reassembling object fragments with Toucheo, and reproducing artwork with PapARt (see Figure ). These exhibitions allowed us to experiment our systems in real conditions. It led to a Living Lab, where the visitors can test our devices.
We have developped Rouages , a mixed-reality display system associated with a 3D visualization application. Rouages reveals the mechanisms of digital musical instruments in two ways. First, by amplifying musicians’ gestures with virtual extensions of the sensors. Second, by representing the sound components with 3D shapes and specific behaviors and by showing the impact of musicians’ gestures on these components. In addition, we have explored new setups to enhance collaboration between musicians using our VR-based instruments. This is illustrated in Figure .
As part of the SIMCA FUI project, the POTIOC team focuses on the design and evaluation of a gateway driving simulator (see Figure ), to teach drivers how to drive an airport gateway in virtual reality, i.e., in a safe and cost- effective environment. We conducted a comprehensive user study to assess the impact of various parameters on user performances. This study allowed us to provide a set of recommendations for the design of an actual simulator.
While recent research on Brain-Computer Interfaces (BCI) has highlighted their potential for many applications, they remain barely used outside laboratories due to a lack of robustness. Spontaneous BCI (i.e., mental imagery-based BCI) often rely on mutual learning efforts by the user and the machine, with BCI users learning to produce stable EEG patterns (spontaneous BCI control being widely acknowledged as a skill) while the computer learns to automatically recognize these EEG patterns, using signal processing. Most research so far was focused on signal processing, mostly neglecting the human in the loop. However, how well the user masters the BCI skill is also a key element explaining BCI robustness. Unfortunately, despite the importance of BCI training protocols, they have been scarcely studied so far, and used mostly unchanged for years. In our work, we advocate that current human training approaches for spontaneous BCI are most likely inappropriate. We notably studied instructional design literature in order to identify the key requirements and guidelines for a successful training procedure that promotes a good and efficient skill learning. This literature study highlighted that current spontaneous BCI user training procedures satisfy very few of these requirements and hence are likely to be suboptimal. We therefore identified the flaws in BCI training protocols according to instructional design principles. We also proposed new research directions that are theoretically expected to address some of these flaws and to help users learn the BCI skill more efficiently. This work has been published in the Frontiers in Neuroscience journal .
On a related topic, together with colleagues from Inria Rennes (A. Lécuyer and L. Bonnet, Hybrid team) we explored the design and evaluation of multiuser BCI applications, notably to see their impact on user training and performance. We created a multiuser videogame called BrainArena in which two users can play a simple football game by means of two BCIs. They can score goals on the left or right side of the screen by simply imagining left or right hand movements (see Figure ). To add another interesting element, the gamers can play in a collaborative manner (their two mental activities are combined to score in the same goal), or in a competitive manner (the gamers must push the ball in opposite directions). Interestingly enough, our results showed that compared to a single player version of the same BCI-game, collaborative multiplayer BCI-gaming increased the motivation and performance of the most skilled of each player pairs, while leaving the performance of the other players unchanged, hence proving a useful tool to improve BCI training. This work has been published in the IEEE Transactions on Computational Intelligence and AI in Games journal .
Stress is a major societal issue with negative impacts on health and economy. Physiological computing offers a continuous, direct, and unobtrusive method for stress level assessment and computer-assisted stress management. However, stress is a complex construct and its physiology can vary depending on its source, for example cognitive load or social evaluation. To study the feasibility of physiology-based load-invariant psychosocial stress-detection, we designed a stress-induction protocol able to independently vary the relevant types of psychophysiological activity: mental and psychosocial stress. In , , we validate the efficacy of our protocol to induce psychosocial and mental stress. Our participants (N=24) had to perform a cognitive task associated with two workload conditions (low/high mental stress), in two contexts (low/high psychosocial stress), during which we recorded subject’s self-reports, behavior, physiology and neurophysiology. Questionnaires showed that the subjectively perceived level of anxiety varied with the psychosocial stress induction, while perceived arousal and mental effort levels vary with mental stress induction. Behavior and physiology corroborated the validity of our protocol further. Heart rate and skin conductance globally increased after psychosocial stress induction relative to the non-stressful condition. Moreover, we demonstrated that higher workload tasks (mental stress) led to decrease in performance and a marked increase of heart rate.
Based on this protocol, we also explored the effect of stress on workload estimation. Workload estimation from electroencephalographic signals (EEG) offers a highly sensitive tool to adapt the human-computer interaction to the user state. To create systems that reliably work in the complexity of the real world, a robustness against contextual changes (e.g., mood), has to be achieved. To study the resilience of state-of-the-art EEG-based workload classification against stress, we test the capability of the workload classifier to generalize across affective contexts (stress/non-stress). We show that the classifier is able to transfer between affective contexts, though performance suffers. However, cross-context training is a simple and powerful remedy allowing the extraction of features more resilient to task-unrelated variations in signal characteristics, leading to a performance comparable to within-context training and testing.
Evaluating human-computer interaction is essential as a broadening population uses machines, sometimes in sensitive contexts. However, traditional evaluation methods may fail to combine real-time measures, an objective approach and data contextualization. We presented a review seeking how neuroimaging techniques can respond to such needs. We focused on electroencephalography (EEG), as it could be handled effectively during a dedicated evaluation phase. We identified workload, attention, vigilance, fatigue, error recognition, emotions, engagement, flow and immersion as being recognizable by EEG. We find that workload, attention and emotions assessments would benefit the most from EEG. Moreover, we advocate to study further error recognition through neuroimaging to enhance usability and increase user experience. This review paper was published in the proceeding of the Physiological Computing Systems (PhyCS) conference .
Along this line of research, we also explored whether it was possible to assess the zone of comfort in stereoscopic displays using electroencephalography. Indeed, the conflict between vergence (eye movement) and accommodation (crystalline lens deformation) occurs with every stereoscopic display. It could cause important stress outside the "zone of comfort", when stereoscopic effect is too strong. This conflict has already been studied using questionnaires, during viewing sessions of several minutes. We built an experimental protocol (see Figure ) which compares two different comfort conditions using electroencephalography over short viewing sequences. Analyses showed significant differences both in event-related potentials and in frequency bands power. By extending our protocol it should be possible to study at the same time comfort and depth perception, having a better understanding of stereoscopy.
Potioc has strong relationships with Cap Sciences.
FUI SIMCA 2000:
duration: 2011-2013
partners: Oktal, ENAC (Ecole Nationale de l'Aviation Civile), Toulouse-Blagnac airport, Air France, CGx AERO in SYS
website: https://
PIA ville numérique "Villes transparentes":
duration: 2012-2014
partners: Pages Jaunes/Mappy, Vectuel/Virtuelcity
Inria ADT OpenViBE-NT:
duration: 2012-2014
partners: Inria teams Hybrid, Neurosys and Athena
website: http://
Inria Project Lab BCI-LIFT:
partners: Inria team Athena (Inria Sophia-Antipolis), Inria team Hybrid (Inria Rennes), Inria team Neurosys (Inria Nancy), LITIS (Université de Rouen), Inria team DEMAR (Inria Sophia-Antipolis), Inria team MINT (Inria Lille), DyCOG (INSERM Lyon)
Project around BCI in the evaluation process, with collaboration just starting (first meeting with all the partners in October 2013)
Collaboration with the University of Bristol, BIG (UK):
University of Bristol, Bristol Interaction and Graphics (BIG) group, UK (Head: Pr. Sriram Subramanian)
We have strong relationships with Sriram Subramanian. This has led to joint paper publications, numerous visits and a co-supervision of a PhD thesis (Camille Jeunet)
Bordeaux Idex project "Conception de Système d'interfaces cerveau-ordinateur prenant en compte les facteurs humains afin d'optimiser l'apprentissage de l'utilisateur:
Bordeaux Idex funding for international PhD project
partners: Bordeaux Segalen University (Handicap & Système nerveux team), Bristol University (BIG team)
duration: October 2013 - September 2016
LIRA Stress and Relaxation project:
Program: Inria - Philips - Fraunhofer partnership
Project title: LIfe-style Research Association, Lifestyle Management: Stress and Relaxation
Coordinator: Frederic Alexandre
Other partners: Philips (Netherlands), Fraunhofer (Germany), Inria teams Hybrid and Mimetic
Abstract: The Stress and Relaxation project aims at offering services to a user, at home or at work, to help this user evaluate and control his level of stress
Pr. Gerwin Schalk (Schalk Lab, Wadsworth center, NY, USA), Pr. Jonathan Brumberg (Kansas University, USA), Dr. Cuntai Guan (I2R, Singapore).
Collaboration in the context of a project around the analysis of ElectroCorticoGraphic (ECoG) brain signals in order to decode speech related information from them.
Multidisciplinary project involving experts from ECoG signals (Gerwin Schalk), speech neuroscience (Jonathan Brumberg) and speech recognition (Cuntai Guan)
This year, the Potioc team has hosted three PhD students
Nicoletta Caramia, University de Pavia, Italy (Avril-July 2013)
Asier Marzo, Universidad Pública de Navarra, Spain (August-November 2013)
Flavio Bertini, University of Bologna, Italy (December 2013-February 2014)
Editoring : ACM Journal of Computer and Cultural Heritage - Special issue on "Interacting with the Past" (M. Hachet)
Conference organization (all Potioc team) and general chairing (M. Hachet): IHM 2013
Workshop organization: "affective Brain-Computer Interface (aBCI) workshop" at ACII, 2013 (C. Mühl)
Workshop organization: "Brain-Computer Interface workshop in Bordeaux", co-organized with the g.tec R&D company, March 2013 (F. Lotte)
Workshop organization: "Teachning the BCI skill: feedback and human training approaches", International BCI meeting 2013, Asilomar, CA, USA, co-organized with R. Scherer (TU Graz) and A. Lécuyer (Inria Rennes), June 2013 (F. Lotte)
Steering committee : IEEE 3DUI (M. Hachet)
Program committees : IEE VR 2013 (M. Hachet), ISIS3D (M. Hachet), Web3D 2013 (J. Jankowski), aBCI workshop @ ACII 2013 (C. Mühl, F. Lotte), ACII 2013 (F. Lotte), PRNI 2013 (F. Lotte), IEEE SSCI-CCMB 2013 (F. Lotte)
Reviewing for conferences: ACM CHI 2013 (M. Hachet, F. Lotte), ACM CHI 2014 (M. Hachet, F. Lotte, J. Jankowski, C. Mühl, J. Laviole), ACM UIST 2013 (M. Hachet, J. Laviole), INTERACT 2013 (J. Jankowski, F. Lotte), Web3D 2013 (J. Jankowski), SUI 2013 (J. Jankowski), MobileHCI 2013 (J. Jankowski), CSCW 2014 (J. Jankowski), 3DUI 2014 (J. Jankowski, J. Laviole), ISMAR 2013 (J. Laviole), ITS 2013 (R. Gervais), PG 2013 (R. Gervais), ACII 2013 (F. Lotte), aBCI Workshop @ ACII 2013 (J. Frey, C. Mühl, F. Lotte), IHM 2013 (F. Lotte), PRNI 2013 (F. Lotte), IEEE SSCI-CCMB 2013 (F. Lotte)
Reviewing for journals: ACM Journal on Computing and Cultural Heritage (J. Jankowski, R. Gervais, J. Frey), GeoInformatica (J. Jankowski), Technique et Science Informatique (TSI) (R. Gervais), IEEE Transactions on Affective Computing (C. Mühl), Brain-Computer Interfaces (F. Lotte), Frontiers in Neuroprosthetics (F. Lotte), IEEE Transactions on Biomedical Engineering (F. Lotte), IEEE Transactions on Human Machine Systems (F. Lotte), International Journal of Neural Systems (F. Lotte), Journal of Neural Engineering (F. Lotte), NeuroImage (F. Lotte), Presence (F. Lotte), IEEE Reviews in Biomedical Engineering (F. Lotte)
Reviewing for book chapters: Handbook of Affective Computing @Oxford (C. Mühl)
ANR : Revue de programme CONTINT 2013 (M. Hachet)
Teaching at University of Bordeaux I:
Master: Jérémy Laviole, Réalité Virtuelle immersive, Projet Étude et Développement, Master 2, University of Bordeaux I, France
Bachelor: Jérémy Frey, supervision of 3 programming projects during "semestre rebondir", 30h, 1st year, University of Bordeaux 1, France
Bachelor: Jérémy Frey, réseau et projet de programmation 2, 35h, 3rd year, University of Bordeaux 1, France
Teaching at University of Bordeaux II:
Master: Martin Hachet, Réalité Virtuelle, 24h eqtd, M2 Sciences Cognitives, Université Bordeaux Segalen, France
Master: Fabien Lotte, Réalité Virtuelle, 12h eqtd, M2 Sciences Cognitives, Université Bordeaux Segalen, France
Bachelor: Camille Jeunet, Knowledge and Representations, 18h, 3rd Year, Cognitive Sciences, Université Bordeaux Segalen, France
Teaching at Enseirb-Matmeca:
Master: Martin Hachet, Réalité Virtuelle, 12h eqtd, 3rd Year, Enseirb-Matmeca, IPB, France
Master: Fabien Lotte, Réalité Virtuelle, 2h eqtd, 3rd Year, Enseirb-Matmeca, IPB, France
Master: Jérémy Laviole, Virtual Reality, 9h, 3rd Year, Enseirb-Matmeca, France
Master: Jérémy Laviole, Object Tracking Project, 8h, 3rd Year, Enseirb-Matmeca, France
Bachelor: Renaud Gervais, Medias numeriques, 4h, Enseirb-Matmeca, France
Other teaching activities:
Master: Jérémy Laviole, Augmented Reality Course + TP, 16h, 3rd Year, ENJMIN, France
PhD: Jérémy Laviole, "Interaction en Réalité Augmentée Spatiale pour le Dessin Physique", Université Bordeaux 1, Defended December 5th, 2013, Martin Hachet
PhD in progress: Renaud Gervais, "Organic User Interfaces", started December 1st, 2012, Martin Hachet
PhD in progress: Jérémy Frey, "Assesing 3DUI with passive Brain-Computer Interfaces", started October 1st, 2012, Fabien Lotte and Martin Hachet
PhD in progress: Camille Jeunet, "Human Learning for Brain-Computer Interfaces", started October 1st, 2013, Fabien Lotte, Martin Hachet, Bernard N'kaoua and Sriram Subramanian.
Master 2: Camille Jeunet, Master 2 from University of Bordeaux 2, affective BCI, February-June 2013 (C. Mühl, F. Lotte)
Engineer (Master 2): Pierre-Marie Plans, Enseirb-Matmeca 3rd year, Collaborative virtual reality, February-June 2013 (M. Hachet)
Master 1: Leonard Pommereau, Master 1 from University of Bordeaux 2, EEG and stereoscopy, April-June 2013 (F. Lotte, M. Hachet)
PhD Jury: Prashant Arvind-Pala, Université Bordeaux Segalen, 18 Dec. 2013 (M. Hachet)
PhD Jury: Javier Mauricio Antelis, University of Zaragoza, Spain, June 2013 (F. Lotte)
PhD Jury: Mohit Kumar Goel, Ecole Polytechnique Fédérale de Lausanne (EPFL), Switzerland, June 2013 (F. Lotte)
PhD Jury for half-PhD (after 1.5 year of PhD work): Raphaelle N. Roy, CEA-Leti/Gipsa-Lab, Grenoble, France (F. Lotte)
External PhD Review: Adam Westerski - Universidad Politecnica de Madrid (J. Jankowski)
The members of Potioc have published several popularization articles targeted to the general public:
"Quand la réalité virtuelle rencontre les surfaces tactiles" (http://
"Arrêtez de regarder la 3D, prenez-en le contrôle !" (http://
"Grâce au numérique, on peut lire dans les pensées" (https://
" Interaction Hommes-Machines : une histoire de 30 ans" (http://
They gave the following popularization talks:
"L'activité cérébrale pilote directement l'ordinateur : présentation de l'interface cerveau-ordinateur (Brain-Computer interface)", conference "Demain les objets sont connectés !", event "Semaine Digitale 2013" in Bordeaux (J. Frey)
Talk introducing BCI and follow-up debate around augmented human, conference "L'homme « augmenté » : notre avenir est-il « cyborg » ?", event "Nancy Renaissance 2013" in Nancy (J. Frey)
"Utiliser son cerveau pour contrôler une machine", Cité des sciences, Paris (F. Lotte)
"Le cerveau au commandes", conference as part of the "Cervorama" exhibition, Cap-Sciences, Bordeaux (F. Lotte)
"Interfaces Cerveau-Ordinateur et Homme Augmenté", conférence "Neurotechnologies: vers un homme augmenté?", Bordeaux (F. Lotte)
They also demonstrated their work to the general public during various events:
Portes ouvertes Université Bordeaux1, demonstrations de PapARt (J. Laviole)
Cap Sciences - expo permanente, PapARt - development (J. Laviole)
Cap Sciences - Évènement IHM 2013, demonstration de PapARt (J. Laviole)
BCI demo for highschool students following "informatique et sciences du numérique" courses (J. Frey, A. Cellard, F. Lotte)
BCI demo for the general public during "La nuit des chercheurs" in Talence (F. Lotte, A. Cellard)
Finally, they also participated to the following events
Aquitec 2013, for promoting research jobs (J. Frey)
Film-debate, "ExistenZ : faut-il avoir peur de la réalité virtuelle ?", University of Bordeaux 1 (J. Frey)
Interview for a highschool class for their video project on science (A. Cellard)
Participation to “visages des sciences 2013” (postcard and video interview) (A. Cellard)
Interview (video and paper) for the SoNews online journal (local Inria Bordeaux newspaper) (A. Cellard)