2.3.1 Industrial Contracts

• Toyota: (Action Recognition System):

  This project run from the 1st of August 2013 up to 2023. It aimed at detecting critical situations in the daily life of older adults living home alone. The system is intended to work with a Partner Robot (to send real-time information to the robot for assisted living) to better interact with older adults. The funding was 106 Keuros for the 1st period and more for the following years.

• Thales: This contract is a CIFRE PhD grant and runs from September 2018 until September 2021 within the French national initiative SafeCity. The main goal is to analyze faces and events in the invisible spectrum (i.e., low energy infrared waves, as well as ultraviolet waves). In this context models will be developed to efficiently extract identity, as well as event - information. This models will be employed in a school environment, with a goal of pseudo-anonymized identification, as well as event-detection. Expected challenges have to do with limited colorimetry and lower contrasts.
• Kontron: This contract is a CIFRE PhD grant and runs from April 2018 until April 2021 to embed CNN based people tracker within a video-camera.
• ESI: This contract is a CIFRE PhD grant and runs from September 2018 until March 2022 to develop a novel Re-Identification algorithm which can be easily set-up with low interaction.

3.2.1 Introduction

Our main goal in perception is to develop vision algorithms able to address the large variety of conditions characterizing real world scenes in terms of sensor conditions, hardware requirements, lighting conditions, physical objects, and application objectives. We have also several issues related to perception which combine machine learning and perception techniques: learning people appearance, parameters for system control and shape statistics.

3.2.2 Appearance Models and People Tracking

An important issue is to detect in real-time physical objects from perceptual features and predefined 3D models. It requires finding a good balance between efficient methods and precise spatio-temporal models. Many improvements and analysis need to be performed in order to tackle the large range of people detection scenarios.

Appearance models. In particular, we study the temporal variation of the features characterizing the appearance of a human. This task could be achieved by clustering potential candidates depending on their position and their reliability. This task can provide any people tracking algorithms with reliable features allowing for instance to (1) better track people or their body parts during occlusion, or to (2) model people appearance for re-identification purposes in mono and multi-camera networks, which is still an open issue. The underlying challenge of the person re-identification problem arises from significant differences in illumination, pose and camera parameters. The re-identification approaches have two aspects: (1) establishing correspondences between body parts and (2) generating signatures that are invariant to different color responses. As we have already several descriptors which are color invariant, we now focus more on aligning two people detection and on finding their corresponding body parts. Having detected body parts, the approach can handle pose variations. Further, different body parts might have different influence on finding the correct match among a whole gallery dataset. Thus, the re-identification approaches have to search for matching strategies. As the results of the re-identification are always given as the ranking list, re-identification focuses on learning to rank. "Learning to rank" is a type of machine learning problem, in which the goal is to automatically construct a ranking model from a training data.

Therefore, we work on information fusion to handle perceptual features coming from various sensors (several cameras covering a large scale area or heterogeneous sensors capturing more or less precise and rich information). New 3D RGB-D sensors are also investigated, to help in getting an accurate segmentation for specific scene conditions.

Long term tracking. For activity recognition we need robust and coherent object tracking over long periods of time (often several hours in video surveillance and several days in healthcare). To guarantee the long term coherence of tracked objects, spatio-temporal reasoning is required. Modeling and managing the uncertainty of these processes is also an open issue. In Stars we propose to add a reasoning layer to a classical Bayesian framework modeling the uncertainty of the tracked objects. This reasoning layer can take into account the a priori knowledge of the scene for outlier elimination and long-term coherency checking.

Controlling system parameters. Another research direction is to manage a library of video processing programs. We are building a perception library by selecting robust algorithms for feature extraction, by insuring they work efficiently with real time constraints and by formalizing their conditions of use within a program supervision model. In the case of video cameras, at least two problems are still open: robust image segmentation and meaningful feature extraction. For these issues, we are developing new learning techniques.

3.3.1 Introduction

Due to the recent development of high processing units, such as GPU, this is now possible to extract meaningful features directly from videos (e.g. video volume) to recognize reliably short actions. Action Recognition benefits also greatly from the huge progress made recently in Machine Learning (e.g. Deep Learning), especially for the study of human behavior. For instance, Action Recognition enables to measure objectively the behavior of humans by extracting powerful features characterizing their everyday activities, their emotion, eating habits and lifestyle, by learning models from a large number of data from a variety of sensors, to improve and optimize for example, the quality of life of people suffering from behavior disorders. However, Smart Homes and Partner Robots have been well advertised but remain laboratory prototypes, due to the poor capability of automated systems to perceive and reason about their environment. A hard problem is for an automated system to cope 24/7 with the variety and complexity of the real world. Another challenge is to extract people fine gestures and subtle facial expressions to better analyze behavior disorders, such as anxiety or apathy. Taking advantage of what is currently studied for self-driving cars or smart retails, there is a large avenue to design ambitious approaches for the healthcare domain. In particular, the advance made with Deep Learning algorithms has already enabled to recognize complex activities, such as cooking interactions with instruments, and from this analysis to differentiate healthy people from the ones suffering from dementia.

To address these issues, we propose to tackle several challenges:

3.3.2 Action recognition in the wild

The current Deep Learning techniques are mostly developed to work on few clipped videos, which have been recorded with students performing a limited set of predefined actions in front of a camera with high resolution. However, real life scenarios include actions performed in a spontaneous manner by older people (including people interactions with their environment or with other people), from different viewpoints, with varying framerate, partially occluded by furniture at different locations within an apartment depicted through long untrimmed videos. Therefore, a new dedicated dataset should be collected in a real-world setting to become a public benchmark video dataset and to design novel algorithms for ADL activity recognition. A special attention should be taken to anonymize the videos.

3.3.3 Attention mechanisms for action recognition

Activities of Daily Living (ADL) and video-surveillance activities are different from internet activities (e.g. Sports, Movies, YouTube), as they may have very similar context (e.g. same background kitchen) with high intra-variation (different people performing the same action in different manners), but in the same time low inter-variation, similar ways to perform two different actions (e.g. eating and drinking a glass of water). Consequently, fine-grained actions are badly recognized. So, we will design novel attention mechanisms for action recognition, for the algorithm being able to focus on a discriminative part of the person conducting the action. For instance, we will study attention algorithms, which could focus on the most appropriate body parts (e.g. full body, right hand). In particular, we plan to design a soft mechanism, learning the attention weights directly on the feature map of a 3DconvNet, a powerful convolutional network, which takes as input a batch of videos.

3.3.4 Action detection for untrimmed videos

Many approaches have been proposed to solve the problem of action recognition in short clipped 2D videos, which achieved impressive results with hand-crafted and deep features. However, these approaches cannot address real life situations, where cameras provide online and continuous video streams in applications such as robotics, video surveillance, and smart-homes. Here comes the importance of action detection to help recognizing and localizing each action happening in long videos. Action detection can be defined as the ability to localize starting and ending of each human action happening in the video, in addition to recognizing each action label. There have been few action detection algorithms designed for untrimmed videos, which are based on either sliding window, temporal pooling or frame-based labeling. However, their performance is too low to address real-word datasets. A first task consists in benchmarking the already published approaches to study their limitations on novel untrimmed video datasets, recorded following real-world settings. A second task could be to propose a new mechanism to improve either 1) the temporal pooling directly from the 3DconvNet architecture using for instance Temporal Convolution Networks (TCNs) or 2) frame-based labeling with a clustering technique (e.g. using Fisher Vectors) to discover the sub-activities of interest.

3.3.5 View invariant action recognition

The performance of current approaches strongly relies on the used camera angle: enforcing that the camera angle used in testing is the same (or extremely close to) as the camera angle used in training, is necessary for the approach performs well. On the contrary, the performance drops when a different camera view-point is used. Therefore, we aim at improving the performance of action recognition algorithms by relying on 3D human pose information. For the extraction of the 3D pose information, several open-source algorithms can be used, such as openpose or videopose3D (from CMU or Facebook research, https://github.com/CMU-Perceptual-Computing-Lab/openpose. Also, other algorithms extracting 3d meshes can be used. To generate extra views, Generative Adversial Network (GAN) can be used together with the 3D human pose information to complete the training dataset from the missing view.

3.3.6 Uncertainty and action recognition

Another challenge is to combine the short-term actions recognized by powerful Deep Learning techniques with long-term activities defined by constraint-based descriptions and linked to user interest. To realize this objective, we have to compute the uncertainty (i.e. likelihood or confidence), with which the short-term actions are inferred. This research direction is linked to the next one, to Semantic Activity Recognition.

3.4.1 Introduction

Semantic activity recognition is a complex process where information is abstracted through four levels: signal (e.g. pixel, sound), perceptual features, physical objects and activities. The signal and the feature levels are characterized by strong noise, ambiguous, corrupted and missing data. The whole process of scene understanding consists in analyzing this information to bring forth pertinent insight of the scene and its dynamics while handling the low level noise. Moreover, to obtain a semantic abstraction, building activity models is a crucial point. A still open issue consists in determining whether these models should be given a priori or learned. Another challenge consists in organizing this knowledge in order to capitalize experience, share it with others and update it along with experimentation. To face this challenge, tools in knowledge engineering such as machine learning or ontology are needed.

Thus we work along the following research axes: high level understanding (to recognize the activities of physical objects based on high level activity models), learning (how to learn the models needed for activity recognition) and activity recognition and discrete event systems.

3.4.2 High Level Understanding

A challenging research axis is to recognize subjective activities of physical objects (i.e. human beings, animals, vehicles) based on a priori models and objective perceptual measures (e.g. robust and coherent object tracks).

To reach this goal, we have defined original activity recognition algorithms and activity models. Activity recognition algorithms include the computation of spatio-temporal relationships between physical objects. All the possible relationships may correspond to activities of interest and all have to be explored in an efficient way. The variety of these activities, generally called video events, is huge and depends on their spatial and temporal granularity, on the number of physical objects involved in the events, and on the event complexity (number of components constituting the event).

Concerning the modeling of activities, we are working towards two directions: the uncertainty management for representing probability distributions and knowledge acquisition facilities based on ontological engineering techniques. For the first direction, we are investigating classical statistical techniques and logical approaches. For the second direction, we built a language for video event modeling and a visual concept ontology (including color, texture and spatial concepts) to be extended with temporal concepts (motion, trajectories, events ...) and other perceptual concepts (physiological sensor concepts ...).

3.4.3 Learning for Activity Recognition

Given the difficulty of building an activity recognition system with a priori knowledge for a new application, we study how machine learning techniques can automate building or completing models at the perception level and at the understanding level.

At the understanding level, we are learning primitive event detectors. This can be done for example by learning visual concept detectors using SVMs (Support Vector Machines) with perceptual feature samples. An open question is how far can we go in weakly supervised learning for each type of perceptual concept (i.e. leveraging the human annotation task). A second direction is to learn typical composite event models for frequent activities using trajectory clustering or data mining techniques. We name composite event a particular combination of several primitive events.

3.4.4 Activity Recognition and Discrete Event Systems

The previous research axes are unavoidable to cope with the semantic interpretations. However they tend to let aside the pure event driven aspects of scenario recognition. These aspects have been studied for a long time at a theoretical level and led to methods and tools that may bring extra value to activity recognition, the most important being the possibility of formal analysis, verification and validation.

We have thus started to specify a formal model to define, analyze, simulate, and prove scenarios. This model deals with both absolute time (to be realistic and efficient in the analysis phase) and logical time (to benefit from well-known mathematical models providing re-usability, easy extension, and verification). Our purpose is to offer a generic tool to express and recognize activities associated with a concrete language to specify activities in the form of a set of scenarios with temporal constraints. The theoretical foundations and the tools being shared with Software Engineering aspects.

The results of the research performed in perception and semantic activity recognition (first and second research directions) produce new techniques for scene understanding and contribute to specify the needs for new software architectures (third research direction).

Domaine : Video Analytics

Our experience in video analytic (also referred to as visual surveillance) is a strong basis which ensures both a precise view of the research topics to develop and a network of industrial partners ranging from end-users, integrators and software editors to provide data, objectives, evaluation and funding.

For instance, the Keeneo start-up was created in July 2005 for the industrialization and exploitation of Orion and Pulsar results in video analytic (VSIP library, which was a previous version of SUP). Keeneo has been bought by Digital Barriers in August 2011 and is now independent from Inria. However, Stars continues to maintain a close cooperation with Keeneo for impact analysis of SUP and for exploitation of new results.

Moreover new challenges are arising from the visual surveillance community. For instance, people detection and tracking in a crowded environment are still open issues despite the high competition on these topics. Also detecting abnormal activities may require to discover rare events from very large video data bases often characterized by noise or incomplete data.

Domaine : Healthcare Monitoring

Since 2011, we have initiated a strategic partnership (called CobTek) with Nice hospital (CHU Nice, Prof P. Robert) to start ambitious research activities dedicated to healthcare monitoring and to assistive technologies. These new studies address the analysis of more complex spatio-temporal activities (e.g. complex interactions, long term activities).

4.1.1 Research

To achieve this objective, several topics need to be tackled. These topics can be summarized within two points: finer activity description and longitudinal experimentation. Finer activity description is needed for instance, to discriminate the activities (e.g. sitting, walking, eating) of Alzheimer patients from the ones of healthy older people. It is essential to be able to pre-diagnose dementia and to provide a better and more specialized care. Longer analysis is required when people monitoring aims at measuring the evolution of patient behavioral disorders. Setting up such long experimentation with dementia people has never been tried before but is necessary to have real-world validation. This is one of the challenge of the European FP7 project Dem@Care where several patient homes should be monitored over several months.

For this domain, a goal for Stars is to allow people with dementia to continue living in a self-sufficient manner in their own homes or residential centers, away from a hospital, as well as to allow clinicians and caregivers remotely provide effective care and management. For all this to become possible, comprehensive monitoring of the daily life of the person with dementia is deemed necessary, since caregivers and clinicians will need a comprehensive view of the person's daily activities, behavioral patterns, lifestyle, as well as changes in them, indicating the progression of their condition.

4.1.2 Ethical and Acceptability Issues

The development and ultimate use of novel assistive technologies by a vulnerable user group such as individuals with dementia, and the assessment methodologies planned by Stars are not free of ethical, or even legal concerns, even if many studies have shown how these Information and Communication Technologies (ICT) can be useful and well accepted by older people with or without impairments. Thus one goal of Stars team is to design the right technologies that can provide the appropriate information to the medical carers while preserving people privacy. Moreover, Stars will pay particular attention to ethical, acceptability, legal and privacy concerns that may arise, addressing them in a professional way following the corresponding established EU and national laws and regulations, especially when outside France. Now, Stars can benefit from the support of the COERLE (Comité Opérationnel d'Evaluation des Risques Légaux et Ethiques) to help it to respect ethical policies in its applications.

As presented in 2, Stars aims at designing cognitive vision systems with perceptual capabilities to monitor efficiently people activities. As a matter of fact, vision sensors can be seen as intrusive ones, even if no images are acquired or transmitted (only meta-data describing activities need to be collected). Therefore new communication paradigms and other sensors (e.g. accelerometers, RFID, and new sensors to come in the future) are also envisaged to provide the most appropriate services to the observed people, while preserving their privacy. To better understand ethical issues, Stars members are already involved in several ethical organizations. For instance, F. Brémond has been a member of the ODEGAM - “Commission Ethique et Droit” (a local association in Nice area for ethical issues related to older people) from 2010 to 2011 and a member of the French scientific council for the national seminar on “La maladie d'Alzheimer et les nouvelles technologies - Enjeux éthiques et questions de société” in 2011. This council has in particular proposed a chart and guidelines for conducting researches with dementia patients.

For addressing the acceptability issues, focus groups and HMI (Human Machine Interaction) experts, are consulted on the most adequate range of mechanisms to interact and display information to older people.

7.1.1 SUP

• Name: Scene Understanding Platform
• Keywords: Activity recognition, 3D, Dynamic scene
• Functional Description: SUP is a software platform for perceiving, analyzing and interpreting a 3D dynamic scene observed through a network of sensors. It encompasses algorithms allowing for the modeling of interesting activities for users to enable their recognition in real-world applications requiring high-throughput.
• URL: https://team.inria.fr/stars/software
• Contact: François Brémond
• Participants: Etienne Corvée, François Brémond, Hung Nguyen, Vasanth Bathrinarayanan
• Partners: CEA, CHU Nice, USC Californie, Université de Hamburg, I2R

7.1.2 VISEVAL

• Functional Description: ViSEval is a software dedicated to the evaluation and visualization of video processing algorithm outputs. The evaluation of video processing algorithm results is an important step in video analysis research. In video processing, we identify 4 different tasks to evaluate: detection, classification and tracking of physical objects of interest and event recognition.
• URL: http://www-sop.inria.fr/teams/pulsar/EvaluationTool/ViSEvAl_Description.html
• Contact: François Brémond
• Participants: Bernard Boulay, François Brémond

Perception for Activity Recognition

Participants: François Brémond, Antitza Dantcheva, Juan Diego Gonzales Zuniga, Ujjwal Ujjwal, Hao Chen, David Anghelone, Monique Thonnat.

The new results for perception for activity recognition are:

• Deep Learning applied to Embedded Systems for People Tracking (see 8.2)
• Joint Detection and Tracking of Pedestrians in Real-time (see 8.3)
• Enhancing Diversity in Teacher-Student Networks via Asymmetric branches for Unsupervised Person Re-identification (see 8.4)
• Beyond the Visible - A survey on cross-spectral face recognition (see 8.5)
• Selective Spatio-Temporal Aggregation Based Pose Refinement System (see 8.6)
• Tattoo Fusion emotion recognition through a Tattoo-based wearable and multimodal Fusion (see 8.7)
• G${}^3$AN: Disentangling Appearance and Motion for Video Generation (see 8.8)
• Comparing 3DCNN approaches for detecting deepfakes (see 8.9)
• Demographic Bias in Biometrics:A Survey on an Emerging Challenge (see 8.10)

Action Recognition

Participants: François Brémond, Antitza Dantcheva, Srijan Das, Rui Dai, Jen-Cheng Hou, Monique Thonnat.

The new results for action recognition are:

• Spatio-Temporal Attention Mechanism for Activity Recognition (see 8.11)
• VPN: Learning Video-Pose Embedding for Activities of Daily Living (see 8.12)
• PDAN: Pyramid Dilated Attention Network for Action Detection (see 8.13)
• TSU: Toyota Smarthome Untrimmed (see 8.14)
• Quantified Analysis for Epileptic Seizure Videos (see 8.15)
• Apathy Classification by Exploiting Task Relatedness (see 8.16)
• A weakly supervised learning technique for classifying facial expressions (see 8.17)
• Semi-supervised emotion recognition using inconsistently annotated data (see 8.18)

Semantic Activity Recognition

Participants: François Brémond, Elisabetta De Maria, Antitza Dantcheva, Srijan Das, Daniel Gaffé, Thibaud L'Yvonnet, Sabine Moisan, Jean-Paul Rigault, Yaohui Wang, Alexandra König, Michal Balazia, Philippe Robert, Monique Thonnat.

For this research axis, the contributions are:

• Probabilistic Model Checking for Activity Recognition in Medical Serious Games (see 8.19)
• MePheSTO – Digital Phenotyping 4 Psychiatric Disorders from Social Interaction (see 8.20)
• DeepSPA - Early detection of cognitive disorders such as dementia on the basis of speech analysis (see 8.21)
• Activis (see 8.22)
• E-Santé (see 8.23)
• An Investigative Study on Face Uniqueness (see 8.24)
• Using Artificial Intelligence for Diagnosis of Psychiatric Disorders (see 8.25)

8.2.1 Online Joint Detection and Tracking

In people tracking, we are questioning the main paradigm that is tracking-by-detection which heavily relies on the performance of the underlying detection method. This requires access to a highly accurate and robust people detector. On the other hand, few frameworks attempt detect and track people jointly. Our intent is to perform people tracking online and jointly with detection.

We are trying to determinate a manner in which a single model can both perform detection and tracking simultaneously. Along these lines, we experimented with a variation of I3D on the Posetrack data set that takes an input of 8 frames in order to create heatmaps along multiple frames as seen in Figure 4. Giving that the data of Posetrack or MOT cannot train a network as I3D, we are doing the pretraining with the synthetic JTA-Dataset.

This work is inspired by the less common methods of tracking-by-tracks and tracking-by-tracklets. Both 43 and 44 generate multi-frame bounding box tuplet proposals and extract detection scores and features with a CNN and LSTM, respectively. Recent researches improve object detection by applying optical flow to propagate scores between frames.

Another method we implemented is by using the detections of previous frames as proposal for the data association, it only uses the IOU between two objects as a distance metric. This approach is simple and efficient assuming the objects do not move drastically. An improved method increases the performance by using a siamese network to conserve identity across frames and predictions for death and birth of tracks.

8.2.2 OpenVINO and ROCm

Regarding embedded hardware, we focus on enlarging both implementation and experimentation of two specific frameworks: OpenVINO and ROCm.

OpenVINO allows us to transfer deep learning models into Myriad and KeemBay chips, taking advantage of their capacity to compute multiple operations without the need of much power consumption. We have thoroughly tested their power consumption under different scenarios as well as implemented many qualitative algorithms with these two platforms, Figure 5 shows the Watt consumption and frame rate of the most popular backbone networks, making it viable to use on embedded applications with a reasonable 25FPS.

For ROCm, we have used the approach of 42 to optimize the compiler execution for a variety of CNN features and filters using a substitute GPU with similar computation capability as Nvidia but still remaining a low branch consumption around 15 Watts.

8.3.1 Problem Statement

One of the fundamental ideas of “Joint Detection and Tracking” is to eliminate the period of waiting by the tracker. One other aspect of this problem is to jointly train the detector and tracker together to achieve better global optimization. End-to-end training in modern deep learning systems is preferred as it leads to multiple components of a computer vision system learning in parallel. This has been shown to result in considerable improvements in moden computer vision systems most importantly by reducing cases where early components perform worse resulting in a compromised performance by the whole system.

Another important problem worked upon has been to balance the speed accuracy trade-offs in pedestrian detection 36. This is especially important in autonomous vehicles where highly accurate detections at a high throughput are desired in real-time. Towards, this we choose a novel technique of selecting only the most relevant samples for inference which decrease computational costs by nearly $97\%$ and provide highly accurate pedestrian detection at a high throughput of 32 FPS. This comprises the state-of-art performance in pedestrian detection and is easily deployable in real-life scenarios.

8.3.2 Work Summary

We propose a joint approach to detection and tracking which is end-to-end trainable (see Figure6). Our approach is described in figure 6. Our approach first detects targets using a FPN based object detector called FCOS. These detections are further processed by a graph attention network (GAT). The performance of our method is comparable to state-of-art approaches and is outlined in table 1.

We also summarize our results on pedestrian detection with both speed and accuracy measurements in table 2.

8.6.1 Results

The quantitative evaluation results with pose ground-truth (Tab. 3) and with action ground-truth (Tab. 4) demonstrate the effectiveness of the proposed method.

8.15.1 A Multimodal Approach for Seizure Classification with Knowledge Distillation

In clinical epilepsy practice, a common diagnostic challenge is distinguishing between patients with epileptic seizures and those with psychogenic non-epileptic seizures. Accurate diagnosis is important in order to implement appropriate clinical management, and often relies on visual analysis of seizures recorded on video in the hospital setting. In this work, we propose a multimodal approach with knowledge distillation to classify these two types of seizures. As shown in Figure 16, the proposed framework is based on deep learning models and utilizes multimodal information from keypoints and appearance from both body and face. Inspired by recent success of graph convolutional networks (GCNs) in various tasks, we take the detected keypoints through time as spatio-temporal graph and train it with an adaptive GCN method (AGCN) to model the spatio-temporal dynamics throughout the seizure event. Besides, we regularize the keypoint features with complementary information from the appearance stream by imposing a knowledge distillation mechanism (KD). We demonstrate the effectiveness of our approach by conducting experiments on real-world seizure videos and compare it with several baseline methods from related works. We performed experiments on real patients with 61 seizures. The experiments are conducted by seizure-wise cross validation, and with the proposed model, the performances of the F1-score and the accuracy are 0.94 and 0.92, respectively, as shown in Table 6. This work is currently under review.

8.15.2 Neural correlates of rhythmic rocking in prefrontal seizures

In our previous study 22, we analyzed several prefrontal seizures with rhythmic rocking features based on their head movement trajectories. In collaboration with researchers from Aix Marseille University, neural correlates was found on the same study cases by analyzing the frequencies between EEG signals and patients' rocking movement. This work has been published in 26.

Preliminary findings

Similar results were obtained on the several cognitive test measures when comparing the remote to the face-to-face administration method. Picture naming task (r=,893), semantic verbal fluency (animals) (r=,861) and total word recall (r=,657) showed the strongest correlations. Acceptability of the tool was relatively high with preference for certain participants even in the remote method for more convenience.

User experience

Based on the System Usability Scale (SUS), a questionnaire was developed and administered to all participants in order to assess their level of satisfaction and comfort during the video conference-based testing administration. Moreover, several conversations between the research nurse; the patients and the psychologists revealed how the participants experienced this new method. We did a first preliminary analysis of 21 of the acceptability questionnaires with rather encouraging results. Overall, 16 out of 21 participants rated that they were satisfied with the experience of the video-conference based assessment. Most participants find the system easy to use. When we ask what type of method the participants prefer 41,7

Some more qualitative comments were:

• ”Strangely, I feel more free in front of the computer to express myself”
• “It still feels intimate and personal but this might depend on the clinician”
• “For me it is not different than face to face and I'm used to it.”
• “Not rather pleasantly surprised, puts a distance that is rather facilitating”

Overall, results support the feasibility and reliability of remote cognitive testing through administration via a telemedicine tool. These systems can be used for remote disease monitoring, enabling patients to be assessed in their own homes and improve utilization of expert assessors allowing them to conduct neurocognitive testing remotely.

Demonstration of the effectiveness of this technology may later make it possible to diffuse its use across all rural areas (‘medical deserts’) in France and thus to improve the early diagnosis of neurodegenerative pathologies, while providing data crucial for basic research. Ultimately, it will lead to an improvement of health care access and care of isolated seniors in these regions. Furthermore, recruitment, onboarding and monitoring of potential candidates in these regions for clinical trials will be facilitated.

Related results can be found in the publications 23, 27, 20.

8.22.1 Work Description

Dataset

Initially we collect raw data including approximately 50 videos for Activis project. Each video is 40-50 mins long. Currently, we have 25 total action classes as shown in the figure 23. Based on these action labels we clipped the videos, achieving 586 videos in total, this includes a total of 1001945 frames. At the moment we have an unbalanced dataset, which will be improved later with more data collection and annotations.

8.22.2 Preprocessing

Before training a good action classifier, the data needs to be preprocessed. Our preprocessing steps involve pose estimation, bounding box generation, tracking and extraction. Currently, we perform person extraction on few clips from the data to perform certain tests. Firstly, we perform 2D and 3D pose estimation using LCRNet9 45 architecture. Bounding boxes were created based on detected poses. These bounding boxes were used to track each person within the video. We used SORT 46 algorithm to track each person (in our case clinician and kid) in each clip. Since, we were interested to use both the clinician and kid for training, both of the persons were extracted.

Toyota

Toyota is working with Stars on action recognition software to be integrated on their robot platform. This project aims at detecting critical situations in the daily life of older adults alone at home. This will require not only recognition of ADLs but also an evaluation of the way and timing in which they are being carried out. The system we want to develop is intended to help them and their relatives to feel more comfortable because they know that potential dangerous situations will be detected and reported to caregivers if necessary. The system is intended to work with a Partner Robot - HSR - (to send real-time information to the robot) to better interact with the older adult.

Thales

Thales and Inria jointly explore facial analysis in the invisible spectrum. Among the different spectra low energy infrared waves, as well as ultraviolet waves will be studied. In this context following tasks will be included: We are designing a model to extract biometric features from the acquired data. Analysis of the data related to contours, shape, etc. will be performed. Current methodology cannot be adopted, since colorimetry in the invisible spectrum is more restricted with less diffuse variations and is less nuanced. Then facial recognition will be performed in the invisible spectrum. Expected challenges have to do with limited colorimetry and lower contrasts. In addition to the first milestone (face recognition in the invisible spectrum), there are two other major milestones: 2. Implementation of such a face recognition system, to be tested at the passage of the access portal to a school. 3. Pseudo-anonymized identification within a school (outdoor courtyards, interior buildings). Combining biometrics in the invisible spectra and anonymisation within an established group requires removing certain additional barriers that are specific to biometrics but also the use of statistical methods associated with biometrics. This pseudo-anonymized identification must also incorporate elements of information provided by the proposed electronic school IDs.

Kontron

Kontron has a collaboration with Stars, which runs from April 2018 until July 2021 to embed CNN based people tracker within a video-camera. Their system uses Intel VPU modules, such as Myriad X (MA2485), based on OpenVino library.

European System Integration

The company ESI (European System Integration) has a collaboration with Stars, which runs from September 2018 until March 2022 to develop a novel Re-Identification algorithm which can be easily set-up with low interaction for videosurveillance applications. ESI provides software solutions for remote monitoring stations, remote assistance, video surveillance, and call centers. It was created in 1999 and ESI is a leader in the French remote monitoring market. Nowadays, ensuring the safety of goods and people is a major problem. For this reason, surveillance technologies are attracting growing interest and their objectives are constantly evolving: it is now a question of automating surveillance systems and helping video surveillance operators in order to limit interventions and staff. One of the current difficulties is the human processing of video, as the multiplication of video streams makes it difficult to understand meaningful events. It is therefore necessary to give video surveillance operators suitable tools to assist them with tasks that can be automated. The integration of video analytics modules will allow surveillance technologies to gain in efficiency and precision. In recent times, deep learning techniques have been made possible by the advent of GPU processors, which offer significant processing possibilities. This leads to the development of automatic video processing.

Fantastic Sourcing

Fantastic Sourcing is a French SME specialized in micro-electronics, it develops e-health technologies. Fantastic Sourcing is collaborating with Stars through the UCA Solitaria project, by providing their Nodeus system. Nodeus is a IoT (Internet of Things) system for home support for the elderly, which consists of a set of small sensors (without video cameras) to collect precious data on the habits of isolated people. Solitaria project performs a multi-sensor activity analysis for monitoring and safety of older and isolated people. With the increase of the ageing population in Europe and in the rest of the world, keeping elderly people at home, in their usual environment, as long as possible, becomes a priority and a challenge of modern society. A system for monitoring activities and alerting in case of danger, in permanent connection with a device (an application on a phone, a surveillance system ...) to warn relatives (family, neighbours, friends ...) of isolated people still living in their natural environment could save lives and avoid incidents that cause or worsen the loss of autonomy. In this R$\&$D project, we propose to study a solution allowing the use of a set of innovative heterogeneous sensors in order to: 1) detect emergencies (falls, crises, etc.) and call relatives (neighbours, family, etc.); 2) detect, over short or longer predefined.

Nively - WITA SRL

Nively is a French SME specialized in e-health technologies, it develops position and activity monitoring of activities of daily living platforms based on video technology. Nively’s mission is to use technological tools to put people back at the center of their interests, with their emotions, identity and behavior. Nively is collaborating with Stars through the UCA Solitaria project, by providing their MentorAge system. This software allows the monitoring of elderly people in nursing homes in order to detect all the abnormal events in the lives of residents (falls, runaways, strolls, etc.). Nively’s technology is based on RGBD video sensors (Kinects type) and a software platform for event detection and data visualization. Nively is also in charge of Software distribution for the ANR Activis project. This project is based on an objective quantification of the atypical behaviors on which the diagnosis of autism is based, with medical (diagnostic assistance and evaluation of therapeutic programs) and computer scientific (by allowing a more objective description of atypical behaviors in autism) objectives. This quantification requires video analysis of the behavior of people with autism. In particular, we propose to explore the issues related to the analysis of ocular movement, gestures and posture to characterize the behavior of a child with autism. Thus, Nively will add autistic behavior analysis software to its product range.

ARECO

ARECO is a French SME specialized in the field of nebulization and misting technologies. It manufactures and sells nebulization devices and dynamic display processes. In this study, an algorithm will be designed for the analysis of customer behaviors when approaching the Fruits & Vegetables department. The analysis should be done on an offline video tape in the evening to store the analyzed interactions. The algorithm will be able to extract the recognized actions and the corresponding time information from a video and store them in a separate file.

LiChIE Project

The LiChIE project (Lion Chaine Image Elargie) is conducted in collaboration with AirBus and BPI to found nine topics including six on the theme of In-flight imagery and three on the robotics theme for the assembly of satellites. The two topics involving STARS are :

• Mohammed Guermal's PhD thesis on Visual Understanding of Activities for an improved collaboration between humans and robots. He began on December 1, 2020.
• Farhood Negin post-doctoral studies on detection and tracking of vehicles from satellite videos and abnormal activity detection. He started in Oct 2020 for 2 years.

10.1.1 Inria associate team not involved in an IIL

10.2.1 FP7 & H2020 Projects

10.2.2 Collaborations in European programs, except FP7 and H2020

10.3.1 ANR

10.3.2 FUI

Solitaria - Multi-sensor activity monitoring

• Title: Solitaria
• Duration: October 2019 - April 2020.
• Coordinator: Idex UCA
• Partners:
  • Inria (Stars): technical partner and project coordinator
  • CoBTek,
  • Fantastic Sourcing.
• Inria contact: François Brémond (STARS)
• Summary: Solitaria is a UCA project, which performs a multi-sensor activity analysis for monitoring and safety of older and isolated people. With the increase of the ageing population in Europe and in the rest of the world, keeping elderly people at home, in their usual environment, as long as possible, becomes a priority and a challenge of modern society. A system for monitoring activities and alerting in case of danger, in permanent connection with a device (an application on a phone, a surveillance system ...) to warn relatives (family, neighbours, friends ...) of isolated people still living in their natural environment could save lives and avoid incidents that cause or worsen the loss of autonomy. In this R&D project, we propose to study a solution allowing the use of a set of innovative heterogeneous sensors in order to:
  • 1) detect emergencies (falls, crises, etc.) and call relatives (neighbours, family, etc.);
  • 2) detect, over short or longer predefined periods, behavioural changes in the elderly through an intelligent analysis of data from sensors.
• Fantastic Sourcing is collaborating with Stars through the UCA Solitaria project, by providing their Nodeus system. Nodeus is a IoT (Internet of Things) system for home support for the elderly, which consists of a set of small sensors (without video cameras) to collect precious data on the habits of isolated people. Fantastic Sourcing is a French SME specialized in micro-electronics, it develops e-health technologies.

11.1.1 Scientific events: organisation

11.1.2 Scientific events: selection

11.1.3 Journal

11.1.4 Invited talks

François Brémond gave an invited talk at IPAS 2020 (https://ipas.ieee.tn/) IEEE International Conference on Image Processing, Applications and Systems.

11.1.5 Scientific expertise

• Antitza Dantcheva was member of the evaluation committee of ANR AAPG 2020 - Comité Sécurité Globale et Cybersécurité.
• Antitza Dantcheva was in the Technical Activities Committee of IEEE Biometrics Council.
• Antitza Dantcheva served in the EURASIP Biomedical Image & Signal Analytics (BISA) special area team.
• Srijan Das was mentor for the Emerging Technology Business Incubator (ETBI) Led by NIT Rourkela, a platform envisaged transforming the start-up ecosystem of the region.
• Monique Thonnat was expert for international evaluation: CONACYT Ciencia de Frontera 2019 (Mexico) and ECOS-Sud Chili 2020 (Chili).

11.1.6 Research administration

Monique Thonnat is member of the scientific board of ENPC, Ecole Nationale des Ponts et Chaussées.

11.2.1 Teaching

• François Brémond organized and lectured the Master MSc Data Science and Artificial Intelligence (Computer Vision and Deep Learning) 30h class at Université Côte d'Azur in 2020 and 2021. Web-site: http://www-sop.inria.fr/members/Francois.Bremond/MSclass/deepLearningWinterSchool/UCA_master/index.html
• Antitza Dantcheva taught 2 classes at Polytech Nice Sophia - Univ Côte d'Azur (Applied Artificial Intelligence, Master 2) in Oct.2020.
• David Anghelone was Teaching assistant at Polytech Nice Sophia - Univ Côte d'Azur (Applied Artificial Intelligence, Master 2) in Oct.2020

11.2.2 Supervision

• PhD: Srijan DAS "Spatio-temporal Attention Mechanisms for Activity Recognition" (defended 1rst of October 2020) supervisors Monique Thonnat and François Brémond.
• PhD in progress: Jen-Cheng HOU "Quantified Analysis for Seizure Videos", supervisor Monique Thonnat.
• PhD in progress: Rui Dai, 2019, "Action Detection for Untrimmed Videos based on Deep Neural Networks", François Brémond.
• PhD in progress: Abdorrahim Bahrami, "Modelling and verifying dynamical properties of biological neural networks in Coq", Elisabetta De Maria.
• PhD in progress: Thibaud L'Yvonnet, "Relations between human behaviour models and brain models - Application to serious games", 2018, Sabine Moisan and Elisabetta De Maria.
• PhD in progress: Hao Chen, 2019, “People Re-identification using Deep Learning methods”, $70\%$ François Brémond and $30\%$ Benoit Lagadec (Fellowship CIFRE - ESI).
• PhD in progress: Yaohui Wang, "Learning to generate human videos", supervisor Antitza Dantcheva and François Brémond.
• PhD in progress: David Anghelone, "Computer Vision and Deep Learning applied to Facial analysis in the invisible spectra", supervisor Antitza Dantcheva.

11.2.3 Juries

François Brémond was reviewer of the HDR (habilitation) jury of Stéphane Herbin, ONERA, 6 Jul 2020.

• François Brémond was in following PhD committees.
  • Cristiano Massaroni, Sapienza University, 16 February 2020 (Reviewer)
  • Konstantinos Papadopoulos, Université du Luxembourg, 20th April 2020
  • Lucas Pascal, Eurecom, 12 June 2020 (comité de suivi de thèse)
  • Florent Jousse, Université Côte d'Azur, 27 May 2020 (comité de suivi de thèse)
  • Melissa Sanabria, Université Côte d'Azur, 20 July 2020 (comité de suivi de thèse)
  • Magali Payne, Université Côte d'Azur, 4 September 2020 (comité de suivi de thèse)
  • Claire Labit Bonis, LAAS (comité de suivi de thèse)
  • Nicolas Girard, Université Côte d'Azur, 16 October 2020 (Président)
  • Thibault Blanc-Beyne, IRIT, 9 November 2020 (Reviewer)
  • Hani Javan Hemmat, Eindhoven University of Technology, 24 November 2020 (Reviewer)
  • Renato BAPTISTA, Université du Luxembourg, 13th January 2021
• Monique Thonnat was in the following PhD committees.
  • Nicolas Foulquier, Université Bretagne Occidentale, 26th of February 2020 (Reviewer)
  • Adrien Malaisé, Université de Lorraine, 7th of July 2020 (Reviewer)
  • Francois Lasson, ENIB, 5th of October 2020 (Reviewer)
  • Srijan Das, Université Côte d'Azur, 1st of October 2020 (Examiner)
• Antitza Dantcheva was in the comité de suivi de thèse of following PhD students.
  • Santiago Smith Silva, Université Côte d'Azur, 19th March 2020 and 1st December 2020
  • Julien Aubert, Eurecom, 7th May 2020

International journals

• 16 article AnamariaA. Bogdan, ValeriaV. Manera, AlexandraA. Koenig and RenaudR. David. Pharmacologic Approaches for the Management of Apathy in Neurodegenerative Disorders Frontiers in Pharmacology 10 January 2020
  HAL DOI
• 17 articleElisabettaE. De Maria, AbdorrahimA. Bahrami, ThibaudT. L'yvonnet, AmyA. Felty, DanielD. Gaffé, AnnieA. Ressouche and FranckF. Grammont. On the Use of Formal Methods to Model and Verify Neuronal ArchetypesFrontiers of Computer ScienceDecember 2020, 40
  HALback to text
• 18 article PawelP. Drozdowski, ChristianC. Rathgeb, AntitzaA. Dantcheva, NaserN. Damer and ChristophC. Busch. Demographic Bias in Biometrics: A Survey on an Emerging Challenge IEEE Transactions on Technology and Society May 2020
  HAL DOI back to text
• 19 articleLauraL. Ferrari, KirillK. Keller, BernhardB. Burtscher and FrancescoF. Greco. Temporary tattoo as unconventional substrate for conformable and transferable electronics on skin and beyondMultifunctional Materials33September 2020, 46
  HALDOIback to text
• 20 articleLindaL. Francis, KathrynK. Lively, AlexandraA. König and JesseJ. Hoey. The Affective Self: Perseverance of Self-Sentiments in Late-Life DementiaSocial Psychology Quarterly832June 2020, 152-173
  HALDOIback to text
• 21 article S LS. Happy, AntitzaA. Dantcheva and Francois FF. Bremond. Expression Recognition with Deep Features Extracted from Holistic and Part-based Models Image and Vision Computing September 2020
  HAL back to text
• 22 articleJen-ChengJ.-C. Hou, MoniqueM. Thonnat, RaoulR. Huys, FabriceF. Bartolomei and AileenA. McGonigal. Rhythmic rocking stereotypies in frontal lobe seizures: A quantified video studyNeurophysiologie Clinique/Clinical Neurophysiology502April 2020, 75 - 80
  HALDOIback to text
• 23 articleAnastasiosA. Karakostas, AlexandraA. König, Carlos FernandoC. Crispim-Junior, Francois FF. Bremond, AlexandreA. Derreumaux, IouliettaI. Lazarou, IoannisI. Kompatsiaris, MagdaM. Tsolaki and PhilippeP. Robert. A French–Greek Cross-Site Comparison Study of the Use of Automatic Video Analyses for the Assessment of Autonomy in Dementia PatientsBiosensors109September 2020, 103
  HALDOIback to text
• 24 article AlexandraA. König, RadiaR. Zeghari, RachidR. Guerchouche, NicklasN. Linz, Inez HGBI. Ramakers, PascaleP. Lemoine, VincentV. Bultingaire and PhilippeP. Robert. Validation of a telemedicine tool for patient monitoring in clinical dementia trials Alzheimer's & Dementia: Diagnosis, Assessment & Disease Monitoring 16 S7 December 2020
  HAL DOI
• 25 articleValeriaV. Manera, SharonS. Abrahams, LuisL. Agüera-Ortiz, FrancoisF. Bremond, RenaudR. David, KaciK. Fairchild, AurianeA. Gros, CécileC. Hanon, MasudM. Husain, AlexandraA. König, PatriciaP. Lockwood, MaribelM. Pino, RatkoR. Radakovic, GabrielG. Robert, AndreaA. Slachevsky, FlorindoF. Stella, AnaïsA. Tribouillard, Pietro DavideP. Trimarchi, FransF. Verhey, JeromeJ. Yesavage, RadiaR. Zeghari and PhilippeP. Robert. Recommendations for the Nonpharmacological Treatment of Apathy in Brain DisordersAmerican Journal of Geriatric Psychiatry284April 2020, 410-420
  HALDOI
• 26 article ArnaudA. Zalta, Jen-ChengJ.-C. Hou, MoniqueM. Thonnat, FabriceF. Bartolomei, BenjaminB. Morillon and AileenA. McGonigal. Neural correlates of rhythmic rocking in prefrontal seizures Neurophysiologie Clinique/Clinical Neurophysiology September 2020
  HAL DOI back to text
• 27 articleRadiaR. Zeghari, ValeriaV. Manera, RoxaneR. Fabre, RachidR. Guerchouche, AlexandraA. König, Minh KhueM. Phan Tran and PhilippeP. Robert. The “Interest Game”: A Ludic Application to Improve Apathy Assessment in Patients with Neurocognitive DisordersJournal of Alzheimer's Disease742March 2020, 669-677
  HALDOIback to text

International peer-reviewed conferences

• 28 inproceedings MichalM. Balazia, S LS. Happy, Francois FF. Bremond and AntitzaA. Dantcheva. How Unique Is a Face: An Investigative Study ICPR 2020 - 25th International Conference on Pattern Recognition Milan, Italy January 2021
  HAL back to text
• 29 inproceedings HaoH. Chen, BenoitB. Lagadec and Francois FF. Bremond. Enhancing Diversity in Teacher-Student Networks via Asymmetric branches for Unsupervised Person Re-identification WACV 2021 – IEEE Winter Conference on Applications of Computer Vision Virtual, United States January 2021
  HAL
• 30 inproceedings HaoH. Chen, BenoitB. Lagadec and Francois FF. Bremond. Learning Discriminative and Generalizable Representations by Spatial-Channel Partition for Person Re-Identification WACV 2020 - IEEE Winter Conference on Applications of Computer Vision Snowmass Village, United States March 2020
  HAL back to text
• 31 inproceedings RuiR. Dai, SrijanS. Das, LucaL. Minciullo, LorenzoL. Garattoni, GianpieroG. Francesca and Francois FF. Bremond. PDAN: Pyramid Dilated Attention Network for Action Detection WACV 2021 - Winter Conference on Applications of Computer Vision 2021 Waikoloa, United States January 2021
  HAL back to text
• 32 inproceedings SrijanS. Das, SauravS. Sharma, RuiR. Dai, Francois FF. Bremond and MoniqueM. Thonnat. VPN: Learning Video-Pose Embedding for Activities of Daily Living ECCV 2020 - 16th European Conference on Computer Vision Glasgow (Virtual), United Kingdom August 2020
  HAL back to text
• 33 inproceedings SrijanS. Das, MoniqueM. Thonnat and Francois FF. Bremond. Looking deeper into Time for Activities of Daily Living Recognition WACV 2020 - IEEE Winter Conference on Applications of Computer Vision Snowmass village, Colorado, United States March 2020
  HAL back to text
• 34 inproceedings S LS. Happy, AntitzaA. Dantcheva and Francois FF. Bremond. Semi-supervised Emotion Recognition using Inconsistently Annotated Data FG 2020 - 15th IEEE International Conference on Automatic Face and Gesture Recognition Buenos Aires / Virtual, Argentina November 2020
  HAL back to text
• 35 inproceedings S LS. Happy, AntitzaA. Dantcheva, AbhijitA. Das, Francois FF. Bremond, RadiaR. Zeghari and PhilippeP. Robert. Apathy Classification by Exploiting Task Relatedness FG 2020 - 15th IEEE International Conference on Automatic Face and Gesture Recognition Buenos Aires / Virtual, Argentina November 2020
  HAL back to text
• 36 inproceedings UjjwalU. Ujjwal, AzizA. Dziri, BertrandB. Leroy and Francois FF. Bremond. A One-and-Half Stage Pedestrian Detector WACV 2020 - IEEE Winter Conference on Applications of Computer Vision Snowmass Village, United States March 2020
  HAL back to text
• 37 inproceedings YaohuiY. Wang, PiotrP. Bilinski, Francois FF. Bremond and AntitzaA. Dantcheva. G3AN: Disentangling Appearance and Motion for Video Generation CVPR 2020 - IEEE Conference on Computer Vision and Pattern Recognition Seattle / Virtual, United States June 2020
  HAL back to text
• 38 inproceedings YaohuiY. Wang, PiotrP. Bilinski, Francois FF. Bremond and AntitzaA. Dantcheva. ImaGINator: Conditional Spatio-Temporal GAN for Video Generation WACV 2020 - Winter Conference on Applications of Computer Vision Snowmass Village, United States March 2020
  HAL back to text
• 39 inproceedings YaohuiY. Wang and AntitzaA. Dantcheva. A video is worth more than 1000 lies. Comparing 3DCNN approaches for detecting deepfakes FG 2020 - 15th IEEE International Conference on Automatic Face and Gesture Recognition Buenos Aires / Virtual, Argentina November 2020
  HAL back to text
• 40 inproceedings DiD. Yang, RuiR. Dai, YaohuiY. Wang, RupayanR. Mallick, LucaL. Minciullo, GianpieroG. Francesca and Francois FF. Bremond. Selective Spatio-Temporal Aggregation Based Pose Refinement System: Towards Understanding Human Activities in Real-World Videos IEEE Winter Conference on Applications of Computer Vision 2021 Virtual, United States January 2021
  HAL back to text

Conferences without proceedings

Doctoral dissertations and habilitation theses

• 41 thesis SrijanS. Das. Spatio-Temporal Attention Mechanism for Activity Recognition Université Côte d’Azur October 2020
  HAL back to text