2025Activity reportProject-Team​STARS

2.1 Presentation​​

The STARS research project-team​​​‌ focuses on the design​ of computer vision methods​‌ for real-time understanding of​​ social interactions observed by​​​‌ sensors. Our objective is​ to propose new algorithms​‌ to analyze the behavior​​ of people suffering from​​​‌ behavioral disorders, in order​ to improve their quality​‌ of life. We study​​ long-term spatio-temporal interactions performed​​​‌ by humans in their​ natural environment. We address​‌ this challenge by proposing​​ novel deep-learning architectures to​​​‌ model behavioral traits such​ as facial expression, gaze,​‌ gestures, body behavior, and​​ body language. To cope​​​‌ with the limited amount​ of available data and​‌ the privacy issues of​​ medical data, we propose​​​‌ data generation for data​ augmentation and anonymization. Another​‌ important challenge is to​​ make the link between​​​‌ collected data, medical diagnosis,​ and ultimately treatments. To​‌ validate our research we​​ work closely with our​​​‌ clinical partners, in particular​ those of the Nice​‌ Hospital.

2.2 Motivation

Deep​​ learning techniques are highly​​​‌ successful for simple action​ recognition, nevertheless several important​‌ challenges remain in activity​​ recognition in general and​​​‌ specifically for our target​ medical application domain.

To​‌ validate our research, we​​ work closely with our​​​‌ clinical partners. We have​ a strategic partnership named​‌ CoBTek with the clinicians​​ of Nice Hospital (CHU​​​‌ Nice) to study the​ impact of video understanding​‌ approaches for cognitive disorders.​​ This partnership started in​​​‌ January 2012 and has​ evolved to a University​‌ Côte d'Azur team and​​ joint work with monthly​​​‌ regular meetings between STARS​ and the clinicians of​‌ Institut Claude Pompidou (ICP),​​ Lenval, and Pasteur hospitals.​​​‌ The two directors of​ CoBTek are François Brémond​‌ and Florence Askenazy (PU-PH)​​ at Lenval. Our objective​​​‌ to deepen research in​ social interaction is motivated​‌ by the needs of​​ our clinician partners. A​​​‌ typical use-case of social​ interactions observed by sensors​‌ appears in the clinical​​ assessments of psychiatric patients,​​ such as people suffering​​​‌ from conditions like major‌ depression, bipolar disorder, or‌​‌ schizophrenia 25. In​​ these clinical assessments, interactions​​​‌ between the patient and‌ the clinician are recorded‌​‌ with multi-modalities, i.e., with​​ video, audio, and physiological​​​‌ sensors. The goal is‌ to extract digital markers‌​‌ (defined by formal interaction​​ models), which are indicators​​​‌ that can characterize a‌ digital phenotype. The digital‌​‌ markers are automatically extracted​​ from the recorded data​​​‌ and the digital phenotypes‌ could lead to a‌​‌ treatment improving the patient's​​ behavioral disorders.

2.3​​ Social interaction understanding: a​​​‌ challenging task

The major​ challenge in semantic interpretation​‌ of dynamic scenes is​​ to bridge the gap​​​‌ between the task dependent​ interpretation of data and​‌ the flood of measures​​ provided by sensors. The​​​‌ problems we address range​ from physical object detection,​‌ activity understanding, activity learning​​ to vision system design​​​‌ and evaluation. The two​ principal classes of human​‌ activities we focus on​​ are assistance to older​​​‌ adults and video analytics.​

Typical examples of complex​‌ activity are shown in​​ Figure 1 and Figure​​​‌ 2 for a homecare​ application (See Toyota Smarthome​‌ Dataset here). In​​ this example, the duration​​ of the monitoring of​​​‌ an older person apartment‌ could last several months.‌​‌ The activities involve interactions​​ between the observed person​​​‌ and several pieces of‌ equipment. The application goal‌​‌ is to recognize the​​ everyday activities at home​​​‌ through formal activity models‌ and data captured by‌​‌ a network of sensors​​ embedded in the apartment.​​​‌ Here typical services include‌ an objective assessment of‌​‌ the frailty level of​​ the observed person to​​​‌ be able to provide‌ a more personalized care‌​‌ and to monitor the​​ effectiveness of a prescribed​​​‌ therapy. The assessment of‌ the frailty level is‌​‌ performed by an Activity​​ Recognition System which transmits​​​‌ a textual report (containing‌ only meta-data) to the‌​‌ general practitioner who follows​​ the older person. Thanks​​​‌ to the recognized activities,‌ the quality of life‌​‌ of the observed people​​ can thus be improved​​​‌ and their personal information‌ can be preserved.

The‌​‌ image presents a study​​ of activity patterns in​​​‌ different environments: the dining‌ room, kitchen, and living‌​‌ room, marked as C1-C7.​​ It includes pie charts​​​‌ showing distribution of activity‌ duration (short, medium, long)‌​‌ and temporal variability (low,​​ medium, high). There are​​​‌ bar graphs displaying the‌ frequency of various activities‌​‌ and their respective environments.​​ The bottom section uses​​​‌ colors to represent where‌ and how often each‌​‌ activity occurs. (Description generated​​ at January 15th, 2026​​​‌ by Albert AI with‌ the model Mistral-Small-3.2-24B)

Homecare monitoring:‌​‌ the annotation of a​​ composed activity "Cook", captured​​​‌ by a video camera‌

The ultimate goal‌​‌ is for cognitive systems​​ to perceive and understand​​​‌ their environment to be‌ able to provide appropriate‌​‌ services to a potential​​ user. An important step​​​‌ is to propose a‌ computational representation of people‌​‌ activities to adapt these​​ services to them. Up​​​‌ to now, the most‌ effective sensors have been‌​‌ video cameras due to​​ the rich information they​​​‌ can provide on the‌ observed environment. These sensors‌​‌ are currently perceived as​​ intrusive ones. A key​​​‌ issue is to capture‌ the pertinent raw data‌​‌ for adapting the services​​ to the people while​​​‌ preserving their privacy. We‌ study different solutions including‌​‌ of course the local​​ processing of the data​​​‌ without transmission of images‌ and the utilization of‌​‌ new compact sensors developed​​ for interaction (also called​​​‌ RGB-Depth sensors, an example‌ being the Kinect) or‌​‌ networks of small non-visual​​ sensors.

2.4 International and​​​‌ Industrial Cooperation

Our work‌ has been applied in‌​‌ the context of more​​ than 10 European projects​​​‌ such as COFRIEND, ADVISOR,‌ SERKET, CARETAKER, VANAHEIM, SUPPORT,‌​‌ DEM@CARE, VICOMO, EIT Health.​​

We had or have​​​‌ industrial collaborations in several‌ domains: transportation (CCI Airport‌​‌ Toulouse Blagnac, SNCF, Inrets,​​ Alstom, Ratp, Toyota, GTT​​​‌ (Italy), banking (Crédit Agricole‌ Bank Corporation, Eurotelis and‌​‌ Ciel), security (Thales R&T​​ FR, Thales Security Syst,​​​‌ EADS, Sagem, Bertin, Alcatel,‌ Keeneo), multimedia (Thales Communications),‌​‌ civil engineering (Centre Scientifique​​​‌ et Technique du Bâtiment​ (CSTB)), computer industry (BULL),​‌ software industry (AKKA), hardware​​ industry (ST-Microelectronics) and health​​​‌ industry (Philips, Link Care​ Services, Vistek).

We have​‌ international cooperations with research​​ centers such as Reading​​​‌ University (UK), Idiap (Switzerland),​ Multitel (Belgium), National Cheng​‌ Kung University (Taiwan), National​​ Taiwan University (Taiwan), University​​​‌ of Southern California (USA),​ University of South Florida​‌ (USA), Michigan State University​​ (USA), Chinese Academy of​​​‌ Sciences (China), IIIT Delhi​ (India), Hochschule Darmstadt (Germany),​‌ Fraunhofer Institute for Computer​​ Graphics Research IGD (Germany).​​​‌

3.1 Axis​​ 1: Human Interaction Recognition​​​‌

Participants: François Brémond,​ Michal Balazia, Antitza​‌ Dantcheva, Monique Thonnat​​.

3.2​​ Axis 2: Data Generation​​​‌ for Augmentation and Anonymization‌

Participants: Antitza Dantcheva,‌​‌ François Brémond.

4.1 Medical Applications

Our​​ main motivation as explained​​​‌ before is to help​ clinicians to diagnose, monitor​‌ and provide pertinent treatment​​ to patients with behavior​​​‌ disorders. The applications we​ target are not general​‌ medical diseases but the​​ ones related to the​​​‌ brain and more precisely​ to psychiatric disorders. These​‌ disorders can appear very​​ early in the life​​​‌ of the patient (for​ instance autism spectrum disorder​‌ 4), they can​​ concern adults (depression, bipolar,​​​‌ schizophrenia 25) or​ the elderly (for instance​‌ Alzheimer disease). We have​​ been working for the​​​‌ elderly patients since the​ creation of the CoBTek​‌ joint team in January​​ 2012. More recently, we​​​‌ have extended our study​ to the two other​‌ categories of age. Now​​ we have some clinical​​​‌ trials within these three​ categories of patients.

4.2​‌ Other Applications

    Sport applications:​​ Sport is an interesting​​ application domain for human​​​‌ activity understanding for three‌ reasons. First, data are‌​‌ often publicly available, so​​ with less ethical concerns​​​‌ than medical ones. Moreover,‌ many data have been‌​‌ recorded and annotated to​​ be part of international​​​‌ challenges Website Challenges.‌ Second, human activities are‌​‌ complex at the level​​ of individuals, of a​​​‌ team and along time.‌ Third, many companies are‌​‌ interested to fund research​​ to advance the field​​​‌ of human activity understanding‌ for sport. For instance,‌​‌ we have a collaboration​​ with a local company,​​​‌ Fairvision (see Fairvision website‌ on football games).

Security‌​‌ applications: The interest and​​ investment in vision-based security​​​‌ systems is large and‌ rapidly growing and is‌​‌ fueled by applications ranging​​ from autonomous vehicles to​​​‌ personalization of customer service.‌ Accordingly, numerous companies, military‌​‌ and public organizations are​​ interested in research in​​​‌ this context.

4.3 Ethical‌ and Acceptability Issues

The‌​‌ development and ultimate use​​ of novel assistive technologies​​​‌ by a vulnerable user‌ group such as individuals‌​‌ with dementia, and the​​ assessment methodologies planned by​​​‌ STARS are not free‌ of ethical, or even‌​‌ legal concerns, even if​​ many studies have shown​​​‌ how these Information and‌ Communication Technologies (ICT) can‌​‌ be useful and well​​ accepted by older people​​​‌ with or without impairments.‌ Thus, one goal of‌​‌ STARS team is to​​ design the right technologies​​​‌ that can provide the‌ appropriate information to the‌​‌ medical carers while preserving​​ people privacy. Moreover, STARS​​​‌ pay particular attention to‌ ethical, acceptability, legal and‌​‌ privacy concerns that may​​ arise, addressing them in​​​‌ a professional way following‌ the corresponding established EU‌​‌ and national laws and​​ regulations, especially when outside​​​‌ France. STARS can also‌ benefit from the support‌​‌ of the COERLE (Comité​​ Opérationnel d'Evaluation des Risques​​​‌ Légaux et Ethiques) to‌ help it to respect‌​‌ ethical policies in its​​ applications.

As presented in​​​‌ Section 2, STARS‌ aims at designing cognitive‌​‌ vision systems with perceptual​​ capabilities to efficiently monitor​​​‌ people activities. As a‌ matter of fact, vision‌​‌ sensors can be seen​​ as intrusive ones, even​​​‌ if no images are‌ acquired or transmitted (only‌​‌ meta-data describing activities need​​ to be collected). Therefore,​​​‌ new communication paradigms and‌ other sensors (e.g. accelerometers,‌​‌ RFID (Radio Frequency Identification),​​ and new sensors to​​​‌ come in the future)‌ are also envisaged to‌​‌ provide the most appropriate​​ services to the observed​​​‌ people, while preserving their‌ privacy. To better understand‌​‌ ethical issues, STARS members​​ are already involved in​​​‌ several ethical organizations.

For‌ addressing the acceptability issues,‌​‌ focus groups and HMI​​ (Human Machine Interaction) experts​​​‌ are consulted on the‌ most adequate range of‌​‌ mechanisms to interact and​​ display information to older​​​‌ people.

5.1 Footprint‌​‌ of research activities

We​​ have limited our travels​​​‌ by reducing our physical‌ participation to conferences and‌​‌ to international collaborations.

5.2​​ Impact of research results​​​‌

We have been involved‌ for many years in‌​‌ promoting public transportation by​​ improving safety onboard and​​​‌ in station. Moreover, we‌ have been working on‌​‌ pedestrian detection for self-driving​​​‌ cars, which will help​ also reducing the number​‌ of individual cars.

6.1 Awards

• Antitza Dantcheva​ was appointed 3IA chair.​‌
• Monique Thonnat has been​​ nominated Coordinatrice Alpes Maritimes​​​‌ for the foundation FUAE​ Fondation Un Avenir Ensemble​‌ of Grande Chancellerie de​​ la Legion d'Honneur (​​​‌Website Fondation). The​ objective is to promote​‌ social mobility by offering​​ recipients of national honors​​​‌ the opportunity to mentor​ deserving and motivated students​‌ from high school to​​ higher education and entry​​​‌ into working life.

6.2​ Major results

• A first​‌ work has consisted of​​ releasing novel tracking algorithms​​​‌ that can reliably track​ people through a video​‌ stream. These algorithms can​​ combine bounding box detection​​​‌ with pixel mask to​ significantly improve the quality​‌ of tracking and to​​ be able to track​​​‌ people on a long-term​ basis.
• During this period,​‌ several novel activity recognition​​ algorithms have also been​​​‌ designed for Activities of​ Daily Living (ADLs) in​‌ real-world settings. These algorithms​​ got the best performances​​​‌ on all relevant action​ datasets. Previously, these algorithms​‌ were built in more​​ or less supervised settings.​​​‌ Thus, we have proposed​ new algorithms for action​‌ detection with a weakly​​ supervised setting with only​​​‌ video-level labels. These algorithms​ can reliably detect specific​‌ events with their time​​ of occurrence within untrimmed​​​‌ videos.
• We have also​ improved the quality and​‌ the capacity of action​​ recognition algorithms by processing​​​‌ long videos with a​ duration of more than​‌ 10 minutes. For that,​​ we have designed new​​​‌ adapters that can be​ plugged into strong video​‌ backbones and thus necessitate​​ only retraining the adapters,​​​‌ which reduces the training​ time and enables a​‌ training process with videos​​ of a much longer​​​‌ duration.
• We have also​ designed novel algorithms for​‌ video action anticipation that​​ can detect some possible​​​‌ events after having observed​ only a limited amount​‌ of normal video streams.​​
• All these algorithms have​​​‌ been successfully evaluated on​ the main international benchmarks​‌ and also on video​​ datasets depicting patients with​​​‌ cognitive disorders in order​ to help doctors to​‌ better monitor their patients.​​

7.1​ Open data

We have​‌ provided two benchmark datasets.​​

Human​​ Interaction Recognition

Participants: François​​​‌ Brémond, Antitza Dantcheva‌, Michal Balazia,‌​‌ Monique Thonnat, Baptiste​​ Chopin, Di Yang​​​‌, Abid Ali,‌ Olivier Huynh, Tomasz‌​‌ Stanczyk, Sanya Sinha​​, Mohammed Guermal,​​​‌ Tanay Agrawal, Snehashis‌ Majhi, Aglind Reka‌​‌.

The new results​​ for Human Interaction Recognition​​​‌ are:

• No Train Yet‌ Gain: Towards Generic Multi-Object‌​‌ Tracking in Sports and​​ Beyond (see 8.1)​​​‌
• Does Re-ID Really Help‌ in Multi-Object Tracking? (see‌​‌ 8.2)
• CM3T: Framework​​ for Efficient Multimodal Learning​​​‌ for Inhomogeneous Interaction Datasets‌ (see 8.3)
• Are‌​‌ Attention Maps Richer than​​ we Imagined for Action​​​‌ Recognition? (see 8.4)‌
• Scaling Action Detection: AdaTAD++‌​‌ with Transformer-Enhanced Temporal-Spatial Adaptation​​ (see 8.5)
• SKI​​​‌ Models: SKeleton Induced Vision-Language‌ Embeddings for Understanding Activities‌​‌ of Daily Living (see​​ 8.6)
• LLAVIDAL :​​​‌ A Large LAnguage VIsion‌ Model for Daily Activities‌​‌ of Living (see 8.7​​)
• Human-Centric Video Understanding:​​​‌ From Single-Modality to Multi-Modal‌ Learning (see 8.8)‌​‌
• B-MoE: A Body-Part-Aware Mixture-of-Experts​​ “All Parts Matter” Approach​​​‌ to Micro-Action Recognition (see‌ 8.9)
• Loose Social-Interaction‌​‌ Recognition in Real-world Therapy​​ Scenarios (see 8.10)​​​‌
• Just Dance with π!,‌ A Poly-modal Inductor for‌​‌ Weakly-supervised Video Anomaly Detection​​ (see 8.11)
• Mixture​​​‌ of Experts Guided by‌ Gaussian Splatters Matters: A‌​‌ new Approach to Weakly-Supervised​​ Video Anomaly Detection (see​​​‌ 8.12)
• Denoise, Divide,‌ Distill, and Predict (‌​‌${𝒟}^3𝒫$):​​ Towards Forecasting Long-horizon Real-world​​​‌ Anomaly from Normalcy (see‌ 8.13)
• Not All‌​‌ Blends Are Equal: The​​ BLEMORE Dataset of Blended​​​‌ Emotion Expressions with Relative‌ Salience Annotations (see 8.14‌​‌)
• The INEMO Dataset:​​ A Multimodal Benchmark of​​​‌ Physiological and Behavioral Responses‌ to Social Media and‌​‌ Film Stimuli (see 8.15​​​‌)
• EEG Classification with​ Limited Data: A Deep​‌ Clustering Approach. (see 8.16​​)
• MEPHESTO: Multimodal Phenotyping​​​‌ of Psychiatric Disorders from​ Social Interaction (see 8.17​‌)
• MultiMediate'25: Cross-Cultural Multi-domain​​ Engagement Estimation (see 8.18​​​‌)
• Stress Estimation in​ Dancers for Injury Prevention​‌ (see 8.19)
• Emotion​​ Recognition using Deep Learning​​​‌ (see 8.20)
• Identifying​ Surgical Instruments in Pedagogical​‌ Cataract Surgery Videos through​​ an Optimized Aggregation Network​​​‌ (see 8.21)
• TBDM:​ Temporal Boundary Distillation Module​‌ for Surgical Gesture Segmentation​​ (see 8.22)
• Effective​​​‌ Video Feature Extraction for​ Training and Comprehension: Human-Centered​‌ Multimodal Video (see 8.23​​)

Data Generation for​​​‌ Augmentation and Anonymization

Participants:​ François Brémond, Antitza​‌ Dantcheva, Baptiste Chopin​​, Nabyl Quignon,​​​‌ Charbel Yahchouchi, Anil​ Egin, Michal Balazia​‌, Di Yang,​​ Valeriya Strizhkova.

The​​​‌ new results for Data​ Generation for Augmentation and​‌ Anonymization are:

• Rotation-Induced Centroid​​ Shift in Latent Space​​​‌ (see 8.24)
• Dual​ Volume Skeleton-Guided 3D Face​‌ Reconstruction from Sparse Views​​ (see 8.25)
• Turbo​​​‌ Learning: 3D Face Reconstruction​ with Mesh Re-Projection and​‌ Re-Identification Consistency (see 8.26​​)
• THEval. Evaluation Framework​​​‌ for Talking Head Video​ Generation (see 8.27)​‌
• Beyond Real versus Fake​​ Towards Intent-Aware Video Analysis​​​‌ (see 8.28)
• AI​ killed the video star.​‌ Audio-driven diffusion model for​​ expressive talking head generation​​​‌ (see 8.29)
• LIA-X:​ Interpretable Latent Portrait Animator​‌ (see 8.30)
• Simplicity-Bias-Aware​​ Adaptation of Foundation Models​​​‌ for Deepfake Detection (see​ 8.31)
• Now You​‌ See Me, Now You​​ Don't: A Unified Framework​​​‌ for Expression Consistent Anonymization​ in Talking Head Videos​‌ (see 8.32)
• Beyond​​ the visible: A survey​​​‌ on cross-spectral face recognition​ (see 8.33)

8.1​‌ No Train Yet Gain:​​ Towards Generic Multi-Object Tracking​​​‌ in Sports and Beyond​

Participants: Tomasz Stanczyk,​‌ Seongro Yoon, Francois​​ Bremond.

We proposed​​​‌ McByte 46, a​ novel tracking-by-detection framework that​‌ enhances multi-object tracking (MOT)​​ by integrating temporally propagated​​​‌ segmentation masks as an​ additional association cue. The​‌ key objective was to​​ improve robustness and generalization​​​‌ in challenging sports scenarios​ - characterized by fast​‌ motion, occlusions, blur, and​​ camera shifts - without​​​‌ requiring any training or​ per-sequence parameter tuning.

Starting​‌ from a strong ByteTrack-based​​ baseline, we designed a​​​‌ pipeline that combines Kalman​ filter motion prediction, IoU-based​‌ matching, and a pre-trained​​ mask temporal propagation model.​​​‌ The propagated masks are​ not used blindly; instead,​‌ we introduced regulated policies​​ that activate mask-based guidance​​​‌ only in well-defined situations​ - namely ambiguity (multiple​‌ plausible associations) and isolation​​ (failure of IoU-based matching).​​​‌ This controlled fusion ensures​ that the mask cue​‌ strengthens association decisions while​​ avoiding instability caused by​​​‌ unreliable mask predictions.

Figure​ 3 illustrates the full​‌ tracking pipeline, showing how​​ bounding-box predictions, detections, and​​​‌ temporally propagated masks are​ jointly integrated into a​‌ unified association cost matrix​​ solved via Hungarian matching.​​​‌ This design allows McByte​ to preserve the strengths​‌ of tracking-by-detection while benefiting​​ from the spatial coherence​​​‌ provided by mask propagation.​

The image illustrates a​‌ tracking process in video​​ analysis. It begins with​​ tracklet boxes from the​​​‌ previous frame (t-1) processed‌ through a Kalman filter‌​‌ to predict tracklet boxes​​ for the current frame​​​‌ (t). In parallel, an‌ object detector identifies detection‌​‌ boxes in the current​​ frame. These predictions and​​​‌ detections are matched using‌ Hungarian matching assignment to‌​‌ update tracklets. Masks from​​ frame t-1 are propagated​​​‌ temporally and matched to‌ detections using IoU-based and‌​‌ mask-enhanced matching to ensure​​ accurate tracking of objects​​​‌ across frames, shown in‌ a basketball scene with‌​‌ two players. (Description generated​​ at January 19th, 2026​​​‌ by Albert AI with‌ the model Mistral-Small-3.2-24B)

We conducted extensive‌ ablation studies to analyze‌​‌ the impact of each​​ design choice, demonstrating that​​​‌ uncontrolled use of masks‌ can degrade performance, whereas‌​‌ carefully gated mask usage​​ yields consistent gains. Qualitative​​​‌ results further show McByte’s‌ ability to maintain identities‌​‌ through heavy occlusions and​​ motion blur. In particular,​​​‌ Fig. 4 highlights challenging‌ football scenarios where McByte‌​‌ successfully preserves tracklets that​​ baseline methods fail to​​​‌ maintain due to abrupt‌ camera motion and degraded‌​‌ visual quality.

We evaluated​​ McByte on four diverse​​​‌ datasets - SportsMOT, DanceTrack,‌ SoccerNet-tracking 2022, and MOT17‌​‌ - using standard MOT​​ metrics (HOTA, IDF1, MOTA).​​​‌ Across all benchmarks, McByte‌ consistently outperformed strong tracking-by-detection‌​‌ baselines, especially in sports​​ datasets, while remaining competitive​​​‌ on pedestrian tracking. Importantly,‌ these improvements were achieved‌​‌ without training, dataset-specific tuning,​​ or additional annotations, demonstrating​​​‌ the method’s generality and‌ practical value.

Overall, this‌​‌ work introduces a generic,​​ training-free MOT framework that​​​‌ bridges the gap between‌ detection-based and mask-based tracking,‌​‌ offering a robust solution​​ applicable across sports and​​​‌ non-sports domains.

Baseline	McByte‌

The image depicts a‌​‌ soccer match in the​​ Barclays Premier League between​​​‌ Liverpool (LIV) and Manchester‌ United (MU) with a‌​‌ score of 1-2. Liverpool​​ is playing with 10​​​‌ men due to a‌ red card to Gerrard‌​‌ at the 46th minute.​​ The time shown is​​​‌ 86:15. The scene focuses‌ on the goal area,‌​‌ where the goalkeeper and​​ other players are positioned.​​​‌ Several players are highlighted‌ with numbers such as‌​‌ 160, 131, 112, 106,​​ 132, 177, and 144.​​​‌ Arrows point to the‌ goalkeeper and two other‌​‌ players near the goalpost,​​ indicating their positions and​​​‌ possible actions. (Description generated‌ at January 19th, 2026‌​‌ by Albert AI with​​ the model Mistral-Small-3.2-24B)

8.2 Does​​​‌ Re-ID Really Help in‌ Multi-Object Tracking?

Participants: Tomasz‌​‌ Stanczyk, Francois Bremond​​.

We conducted a​​​‌ systematic and critical analysis‌ of the role of‌​‌ person re-identification (re-ID) in​​ multi-object tracking (MOT) 49​​​‌. While re-ID is‌ widely assumed to improve‌​‌ association quality, its actual​​ contribution in practical tracking​​​‌ pipelines remains unclear. Our‌ goal was to rigorously‌​‌ evaluate when, how, and​​ to what extent re-ID​​​‌ genuinely benefits MOT performance.‌

We focused our study‌​‌ on the widely used​​​‌ BoT-SORT tracking framework and​ evaluated multiple re-ID configurations,​‌ including re-ID trained on​​ the target dataset, re-ID​​​‌ trained on external datasets,​ and a strong generic​‌ re-ID model. Experiments were​​ conducted on the MOT17​​​‌ validation set, using both​ ground-truth detections and realistic​‌ detector outputs to disentangle​​ the effects of detection​​​‌ quality from appearance-based association.​

Beyond standard tracking evaluations,​‌ we introduced a custom​​ re-ID assessment protocol tailored​​​‌ to tracking. This protocol​ directly measures correct and​‌ incorrect inter-frame matches produced​​ by re-ID, enabling a​​​‌ deeper understanding of re-ID​ behavior in realistic tracking​‌ scenarios. We analyzed cosine​​ distance distributions, match accuracy,​​​‌ and failure modes across​ sequences with varying crowd​‌ density, occlusion patterns, and​​ bounding-box sizes.

Our results​​​‌ show that re-ID often​ provides only marginal gains​‌ and, in several scenarios,​​ can even degrade tracking​​​‌ performance, especially when bounding​ boxes are small, heavily​‌ occluded, or visually ambiguous.​​ We further demonstrated that​​​‌ tuning re-ID similarity thresholds​ is non-trivial and highly​‌ sequence-dependent, undermining the robustness​​ and general applicability of​​​‌ re-ID-based association.

To mitigate​ these issues, we explored​‌ constraints on re-ID usage,​​ such as filtering based​​​‌ on occlusion level and​ minimum bounding-box size. While​‌ these constraints reduced incorrect​​ matches in isolation, their​​​‌ impact on full tracking​ performance remained limited and​‌ inconsistent across sequences.

Overall,​​ this work provides evidence-based​​​‌ insight into the limitations​ of re-ID in MOT​‌ and challenges the assumption​​ that stronger re-ID models​​​‌ automatically lead to better​ tracking. We conclude that​‌ re-ID is not a​​ universally reliable solution for​​​‌ improving MOT and that​ its effectiveness is strongly​‌ conditioned on scene characteristics,​​ detection quality, and careful​​​‌ integration into the tracking​ pipeline.

This study offers​‌ practical guidance for both​​ researchers and practitioners, encouraging​​​‌ more critical and context-aware​ use of re-ID in​‌ future MOT systems.

8.3​​ CM3T: Framework for Efficient​​​‌ Multimodal Learning for Inhomogeneous​ Interaction Datasets

Participants: Tanay​‌ Agrawal, Mohammed Guermal​​, Michal Balazia,​​​‌ Francois Bremond.

Challenges​ in cross-learning involve inhomogeneous​‌ or even inadequate amount​​ of training data and​​​‌ lack of resources for​ retraining large pretrained models.​‌ Inspired by transfer learning​​ techniques in NLP (i.e.,​​​‌ natural language processing), adapters​ and prefix tuning, we​‌ present a new model-agnostic​​ plugin architecture for cross-learning,​​​‌ called CM3T 36,​ that adapts transformer-based models​‌ to new or missing​​ information (see Figure 5​​​‌). We introduce two​ adapter blocks: multi-head vision​‌ adapters for transfer learning​​ and cross-attention adapters for​​​‌ multimodal learning. Training becomes​ substantially efficient as the​‌ backbone and other plugins​​ do not need to​​​‌ be fine-tuned along with​ these additions.

Backbones pretrained​‌ using self-supervised learning provide​​ good general features, thus​​​‌ all methods of fine-tuning​ work well. In the​‌ case of supervised pretraining,​​ adapters fail to perform​​​‌ well (in red) and​ CM3T is introduced to​‌ solve this (in green).​​

Comparative and ablation​​​‌ studies on three datasets‌ Epic-Kitchens-100, MPIIGroupInteraction and UDIVA‌​‌ v0.5 show efficacy of​​ this framework on different​​​‌ recording settings and tasks.‌ With only 12.8% trainable‌​‌ parameters compared to the​​ backbone to process video​​​‌ input and only 22.3%‌ trainable parameters for two‌​‌ additional modalities, we achieve​​ comparable and even better​​​‌ results than the state-of-the-art.‌ CM3T has no specific‌​‌ requirements for training or​​ pretraining and is a​​​‌ step towards bridging the‌ gap between a general‌​‌ model and specific practical​​ applications of video classification.​​​‌

8.4 Are Attention Maps‌ Richer than we Imagined‌​‌ for Action Recognition?

Participants:​​ Tanay Agrawal, Abid​​​‌ Ali, Francois Bremond‌.

Deep learning models‌​‌ are becoming more general​​ and robust by the​​​‌ day. Specifically, image foundation‌ models have recently shown‌​‌ exponential growth. We introduce​​ a way to exploit​​​‌ this growth in the‌ field of video classification.‌​‌ The basic idea here​​ is that if we​​​‌ have a good understanding‌ of space, we should‌​‌ not require complicated spatio-temporal​​ processing. We introduce the​​​‌ Attention Map (AM) flow,‌ a way to identify‌​‌ the location of local​​ changes between two frames​​​‌ in a video, without‌ adding additional parameters specifically‌​‌ for it. We utilize​​ adapters, which have been​​​‌ growing in popularity in‌ the field of parameter-efficient‌​‌ transfer learning. These help​​ us incorporate AM flow​​​‌ in a pretrained image‌ model without the need‌​‌ of fine-tuning it. With​​ just these changes and​​​‌ minimal temporal processing, an‌ image model is able‌​‌ to achieve state-of-the-art results​​ on popular action recognition​​​‌ datasets with low training‌ time and requiring minimal‌​‌ pretraining. This work explores​​ the theory behind this​​​‌ idea and the intricacies‌ involved. Through relevant experiments,‌​‌ we show the efficacy​​ of this method and​​​‌ discuss various ideas to‌ take this work forward.‌​‌ We use kinetics-400, something-something​​ v2, and the Toyota​​​‌ SmartHome datasets and achieve‌ state-of-the-art or comparable results.‌​‌ We also show that​​ video models suffer from​​​‌ extensive pretraining on multiple‌ datasets and a large‌​‌ training time, but our​​ work answers these problems.​​​‌

This work has been‌ published at WACV 2025‌​‌ 35.

8.5 Scaling​​ Action Detection: AdaTAD++ with​​​‌ Transformer-Enhanced Temporal-Spatial Adaptation

Participants:‌ Tanay Agrawal, Abid‌​‌ Ali, Francois Bremond​​.

Temporal Action Detection​​​‌ (TAD) is essential for‌ analyzing long-form videos by‌​‌ identifying and segmenting actions​​ within untrimmed sequences. While​​​‌ recent innovations like Temporal‌ Informative Adapters (TIA) have‌​‌ improved resolution, memory constraints​​ still limit large video​​​‌ processing. To address this‌ issue, we introduce AdaTAD++,‌​‌ an enhanced framework that​​ decouples temporal and spatial​​​‌ processing within adapters, organizing‌ them into independently trainable‌​‌ modules. Our novel two-step​​ training strategy first optimizes​​​‌ for high temporal and‌ low spatial resolution, then‌​‌ vice versa, allows the​​ model to utilize both​​​‌ high spatial and temporal‌ resolutions during inference, while‌​‌ maintaining training efficiency. Additionally,​​ we incorporate a more​​​‌ sophisticated temporal module capable‌ of capturing long-range dependencies‌​‌ more effectively than previous​​​‌ methods. Experiments on benchmark​ datasets, including ActivityNet-1.3, THUMOS14,​‌ and EPIC-Kitchens 100, demonstrate​​ that AdaTAD++ achieves state-of-the-art​​​‌ performance. We also explore​ various adapter configurations, discussing​‌ their trade-offs regarding resource​​ constraints and performance, providing​​​‌ valuable insights into their​ optimal application.

This work​‌ has been published at​​ ICCV 2025 38.​​​‌

8.6 SKI Models: SKeleton​ Induced Vision-Language Embeddings for​‌ Understanding Activities of Daily​​ Living

Participants: Arkaprava Sinha​​​‌, Dominick Reilly,​ Francois Bremond, Srijan​‌ Das.

The introduction​​ of vision-language models like​​​‌ CLIP has enabled the​ development of foundational video​‌ models capable of generalizing​​ to unseen videos and​​​‌ human actions. However, these​ models are typically trained​‌ on web videos, which​​ often fail to capture​​​‌ the challenges present in​ Activities of Daily Living​‌ (ADL) videos. Existing works​​ address ADL-specific challenges, such​​​‌ as similar appearances, subtle​ motion patterns, and multiple​‌ viewpoints, by combining 3D​​ skeletons and RGB videos.​​​‌ However, these approaches are​ not integrated with language,​‌ limiting their ability to​​ generalize to unseen action​​​‌ classes. In this paper,​ we introduce SKI models,​‌ which integrate 3D skeletons​​ into the vision-language embedding​​​‌ space. SKI models leverage​ a skeleton language model,​‌ SkeletonCLIP, to infuse skeleton​​ information into Vision Language​​​‌ Models (VLMs) and Large​ Vision Language Models (LVLMs)​‌ through collaborative training. Notably,​​ SKI models do not​​​‌ require skeleton data during​ inference, enhancing their robustness​‌ for real-world applications. The​​ effectiveness of SKI models​​​‌ is validated on three​ popular ADL datasets for​‌ zero-shot action recognition and​​ video caption generation tasks.​​​‌ Our code is available​ at this github Github​‌ page.

This work​​ has been published at​​​‌ AAAI 2025 45.​

8.7 LLAVIDAL : A​‌ Large LAnguage VIsion Model​​ for Daily Activities of​​​‌ Living

Participants: Dominick Reilly​, Francois Bremond,​‌ Srijan Das.

Current​​ Large Language Vision Models​​​‌ (LLVMs) trained on web​ videos perform well in​‌ general video understanding but​​ struggle with fine-grained details,​​​‌ complex human object interactions​ (HOI), and view-invariant representation​‌ learning essential for Activities​​ of Daily Living (ADL).​​​‌ This limitation stems from​ a lack of specialized​‌ ADL video instruction-tuning datasets​​ and insufficient modality integration​​​‌ to capture discriminative action​ representations. To address this,​‌ we propose a semi-automated​​ framework for curating ADL​​​‌ datasets, creating ADL-X, a​ multiview, multimodal RGBS (i.e.,​‌ RGB and Segmentation) instruction-tuning​​ dataset. Additionally, we introduce​​​‌ LLAVIDAL, an LLVM integrating​ videos, 3D skeletons, and​‌ HOIs to model ADL's​​ complex spatiotemporal relationships. For​​​‌ training LLAVIDAL a simple​ joint alignment of all​‌ modalities yields suboptimal results;​​ thus, we propose a​​​‌ Multimodal Progressive (MMPro) training​ strategy, incorporating modalities in​‌ stages following a curriculum.​​ We also establish ADL​​​‌ MCQ and video description​ benchmarks to assess LLVM​‌ performance in ADL tasks.​​ Trained on ADL-X, LLAVIDAL​​​‌ achieves state-of-the-art (SOTA) performance​ across ADL benchmarks.

This​‌ work has been published​​ at CVPR 2025 43​​​‌.

8.8 Human-Centric Video​ Understanding: From Single-Modality to​‌ Multi-Modal Learning

Participants: Mahmoud​​ Ali, Di Yang​​​‌, Francois Bremond.​

General pipeline of MoVie​‌ for action detection

Human action recognition is​​​‌ an active research field‌ with significant contributions to‌​‌ applications such as home-care​​ monitoring, human-computer interaction, and​​​‌ game control. However, recognizing‌ human activities in real-world‌​‌ videos remains challenging, especially​​ when learning effective video​​​‌ representations with a high‌ expressive power to represent‌​‌ human spatio-temporal motion, view-invariant​​ actions, complex composable actions,​​​‌ etc. To address this‌ challenge, we made three‌​‌ contributions toward learning effective​​ representations that can be​​​‌ applied and evaluated in‌ real-world human action classification,‌​‌ retrieval, prediction, detection, and​​ segmentation tasks by transfer​​​‌ learning.

The first contribution‌ (single modality): we improve‌​‌ the generalizability of human​​ skeleton motion representation models​​​‌ under the skeleton-only modality.‌ We introduce two novel‌​‌ self-supervised learning frameworks based​​ on contrastive learning to​​​‌ learn robust and transferable‌ skeleton representations without relying‌​‌ on action labels. By​​ exploiting the inherent spatio-temporal​​​‌ structure of human skeleton‌ sequences, our approach encourages‌​‌ discriminative motion representations through​​ instance-level and temporal consistency​​​‌ objectives. Extensive evaluations demonstrate‌ that the proposed frameworks‌​‌ improve performance across diverse​​ downstream tasks and scenarios,​​​‌ bridging the gap between‌ controlled 3D laboratory datasets‌​‌ (e.g., NTU-RGB-D) and challenging​​ 2D real-world datasets (e.g.,​​​‌ SmartHome), highlighting the strength‌ of SSL (i.e., Self-Supervised‌​‌ Learning) for skeleton-based motion​​ understanding.

The second contribution​​​‌ (two modality): Despite the‌ effectiveness of skeleton-based models‌​‌ in capturing spatial and​​ temporal dynamics, they struggle​​​‌ to recognize fine-grained actions.‌ In particular, they fail‌​‌ to distinguish between semantically​​ similar actions, such as​​​‌ "drinking from a cup"‌ versus "drinking from a‌​‌ bottle", as these models​​ lack access to object-centric​​​‌ and semantic information. To‌ address this, we propose‌​‌ MoVie as shown in​​ Fig. 6, a​​​‌ motion-augmented framework designed to‌ improve real-world human action‌​‌ detection by integrating skeleton​​ motion features with visual​​​‌ information through the Motion-Vision‌ Mixer and incorporating history-aware‌​‌ temporal modeling.

Overview of​​ the framework

The third contribution​​ (multi modality): our previous​​​‌ works show that VLFMs​ (i.e., Vision-Language Foundation Models)​‌ are still far away​​ from satisfactory performance in​​​‌ all evaluated tasks, particularly​ in densely labeled and​‌ long video datasets, such​​ as the fine-grained activities​​​‌ in complex and real-world​ scenarios. As shown in​‌ Fig. 7, we​​ introduce our proposed Transferable​​​‌ skeleton MOtion Representation learning​ architecture (T-MOR) based on​‌ a contrastive motion-video-language pre-training​​ strategy. The pre-trained skeleton​​​‌ model is effective for​ both action classification, segmentation​‌ and zero-shot action recognition​​ tasks.

Overall, this work​​​‌ contributes to the field​ of human-centric video understanding​‌ by proposing novel methods​​ for skeleton-based action representation​​​‌ learning and general RGB​ video representation learning. Such​‌ representations benefit both action​​ classification and segmentation tasks.​​​‌

8.9 B-MoE: A Body-Part-Aware​ Mixture-of-Experts “All Parts Matter”​‌ Approach to Micro-Action Recognition​​

Participants: Aglind Reka,​​​‌ Nishit Poddar, Diana​ Borza, Snehashis Majhi​‌, Michal Balazia,​​ Francois Bremond.

Micro-action​​​‌ recognition (MAR) presents unique​ challenges due to the​‌ inherently subtle, fleeting, and​​ ambiguous nature of micro-actions.​​​‌ Unlike conventional actions, which​ are often clearly distinguishable,​‌ micro-actions, such as a​​ slight nod, a subtle​​​‌ shift in posture, or​ a brief glance are​‌ characterized by their fine-grained​​ motion and short duration.​​​‌ These movements often overlap​ in meaning and arise​‌ from reflexes or situational​​ cues, making them difficult​​​‌ to interpret and classify.​ Additionally, micro-actions are influenced​‌ by environmental and social​​ factors, further complicating their​​​‌ recognition.

A significant issue​ in current approaches is​‌ the failure to account​​ for the structured nature​​​‌ of human motion. Micro-actions​ often originate from specific​‌ body parts, such as​​ the head, torso, or​​​‌ limbs, and follow a​ consistent body-to-action hierarchy. However,​‌ most existing models treat​​ these actions as flat​​​‌ categories, overlooking the spatial​ dependencies between body regions.​‌ This oversight leads to​​ difficulties in isolating informative​​​‌ signals from background noise​ and differentiating between highly​‌ similar micro-movements within the​​ same body region. Another​​​‌ challenge lies in the​ imbalance and variability of​‌ micro-action datasets. Datasets like​​ MA-52, SocialGesture, and MPII-GroupInteraction​​​‌ capture a wide range​ of human movements, from​‌ short, dynamic gestures to​​ long, static postures. This​​​‌ variability in temporal scale​ and class frequency makes​‌ it challenging for models​​ to capture rare yet​​​‌ distinctive motion patterns, which​ are characteristic of micro-actions.​‌

To address these challenges,​​ we introduce B-MoE, a​​​‌ body-part-aware Mixture-of-Experts framework (see​ Figure 8) designed​‌ to explicitly model the​​ structured nature of human​​​‌ motion. B-MoE specializes in​ analyzing motions from localized​‌ body regions such as​​ the head, torso, upper​​​‌ limbs, and lower limbs,​ allowing the model to​‌ focus on subtle movements​​ and discriminative cues within​​​‌ each region. By doing​ so, B-MoE suppresses background​‌ interference and enhances the​​ detection of fine-grained motion​​ cues, improving the ability​​​‌ to differentiate between ambiguous‌ action classes. Central to‌​‌ B-MoE is the Macro–Micro​​ Motion Encoder (M3E) as​​​‌ shown in Figure 9‌, a lightweight yet‌​‌ powerful backbone that captures​​ both long-range contextual structure​​​‌ and fine-grained local motion.‌ This dual capability enables‌​‌ the model to effectively​​ recognize both prolonged poses​​​‌ and rapid micro-movements. A‌ cross-attention routing mechanism further‌​‌ enhances the framework by​​ dynamically selecting and fusing​​​‌ informative region-wise semantic cues,‌ as shown in Figure‌​‌ 10, which are​​ then integrated with global​​​‌ motion features. Through this‌ approach, B-MoE effectively addresses‌​‌ the core challenges of​​ MAR subtlety, ambiguity, and​​​‌ class imbalance by amplifying‌ fine local cues, suppressing‌​‌ irrelevant regions, and providing​​ complementary semantic and motion​​​‌ evidence. This work was‌ submitted to CVPR 2026.‌​‌

The image depicts a​​ flowchart of a machine​​​‌ learning model. It takes‌ video input and processes‌​‌ it through multiple branches:​​ semantic branches for different​​​‌ body parts (head, body,‌ upper limb, lower limb),‌​‌ a semantic encoder, and​​ a motion encoder. The​​​‌ semantic branches are frozen,‌ while the experts, which‌​‌ are learnable components, adapt.​​ The outputs from these​​​‌ branches and encoders converge‌ and pass through a‌​‌ series of modules including​​ cross-attention mechanisms, a transformer,​​​‌ and a multi-layer perceptron‌ (MLP). The process aims‌​‌ to learn and integrate​​ semantic and motion features​​​‌ for analysis. (Description generated‌ at January 15th, 2026‌​‌ by Albert AI with​​ the model Mistral-Small-3.2-24B)

The​​​‌ image depicts a neural‌ network architecture consisting of‌​‌ an SGP layer and​​ a semantic embedding alignment​​​‌ mechanism. The SGP layer‌ includes components like MHSA‌​‌ (Multi-Head Self Attention), ConvC​​ (Convolution), and fully connected​​​‌ (FC) layers. The process‌ starts with input (T,‌​‌ D) through MHSA, followed​​ by several convolutional and​​​‌ pooling operations. The semantic‌ embedding alignment, used only‌​‌ during pre-training, aligns word​​ embeddings through a TAN​​​‌ module with fully connected‌ layers, minimizing the embedding‌​‌ loss. (Description generated at​​ January 16th, 2026 by​​​‌ Albert AI with the‌ model Mistral-Small-3.2-24B)

The image depicts‌​‌ a flowchart of a​​ video processing system. It​​​‌ begins with a video‌ input of a person‌​‌ sitting. The video frame​​ (T, C, H, W)​​​‌ is first processed by‌ a module called "Sapiens,"‌​‌ likely for segmentation, producing​​ segmentation maps (T, H,​​​‌ W). These maps are‌ combined with another input,‌​‌ represented by a bone​​​‌ structure image, which then​ goes to a "Semantic​‌ Encoder." The output is​​ a feature representation (T/16,​​​‌ 1408) (8 x 1408).​ Snowflake icons indicate the​‌ use of frozen parameters​​ or pre-trained weights in​​​‌ these modules. (Description generated​ at January 16th, 2026​‌ by Albert AI with​​ the model Mistral-Small-3.2-24B)

Our extensive​​​‌ experiments on three socially​ contextual micro-action benchmarks (MA-52,​‌ MPII-GI, and SocialGesture) demonstrate​​ significant improvements, with notable​​​‌ gains in F1macro accuracy​ of +4.32%, +3.35%, and​‌ +1.17%, respectively. These results​​ highlight B-MoE’s robustness in​​​‌ handling class imbalance and​ its superior performance in​‌ recognizing subtle and ambiguous​​ actions recognition of ambiguous,​​​‌ underrepresented, and low-amplitude actions.​

8.10 Loose Social-Interaction Recognition​‌ in Real-world Therapy Scenarios​​

Participants: Abid Ali,​​​‌ Monique Thonnat, Francois​ Bremond.

The computer​‌ vision community has explored​​ dyadic interactions for atomic​​​‌ actions such as pushing,​ carrying-object, etc. However, with​‌ the advancement in deep​​ learning models, there is​​​‌ a need to explore​ more complex dyadic situations​‌ such as loose interactions.​​ These are interactions where​​​‌ two people perform certain​ atomic activities to complete​‌ a global action irrespective​​ of temporal synchronization and​​​‌ physical engagement, like cooking-together​ for example. Analyzing these​‌ types of dyadic-interactions has​​ several useful applications in​​​‌ the medical domain for​ social-skills development and mental​‌ health diagnosis. To achieve​​ this, we propose a​​​‌ novel dual-path architecture to​ capture the loose interaction​‌ between two individuals. Our​​ model learns global abstract​​​‌ features from each stream​ via a CNNs backbone​‌ and fuses them using​​ a new Global-Layer-Attention module​​​‌ based on a cross-attention​ strategy. We evaluate our​‌ model on real-world autism​​ diagnoses such as our​​​‌ Loose-Interaction dataset, and the​ publicly available Autism dataset​‌ for loose interactions. Our​​ network achieves baseline results​​​‌ on the Loose-Interaction and​ SOTA results on the​‌ Autism datasets. Moreover, we​​ study different social interactions​​​‌ by experimenting on a​ publicly available dataset i.e.​‌ NTU-RGB+D (interactive classes from​​ both NTU-60 and NTU-120).​​​‌ We have found that​ different interactions require different​‌ network designs. We also​​ compare a slightly different​​​‌ version of our method​ by incorporating time information​‌ to address tight interactions​​ achieving SOTA results.

This​​​‌ work has been published​ at WACV 2025 37​‌.

8.11 Just Dance​​ with π!, A Poly-modal​​​‌ Inductor for Weakly-supervised Video​ Anomaly Detection

Participants: Snehashis​‌ Majhi, Giacomo D’amicantonio​​, Antitza Dantcheva Ali​​​‌, Francois Bremond.​

Weakly-supervised methods for video​‌ anomaly detection (VAD) are​​ conventionally based merely on​​​‌ RGB spatiotemporal features, which​ continues to limit their​‌ reliability in real-world scenarios.​​ This is because RGB-features​​​‌ are not sufficiently distinctive​ in setting apart categories​‌ such as shoplifting from​​ visually similar events. Therefore,​​​‌ towards robust complex real-world​ VAD, it is essential​‌ to augment RGB spatio-temporal​​ features with additional modalities.​​ Motivated by this, we​​​‌ introduce the Poly-modal Induced‌ framework for VAD: “PI-VAD”‌​‌ (or π-VAD), a novel​​ approach that augments RGB​​​‌ representations by five additional‌ modalities. Specifically, the modalities‌​‌ include sensitivity to fine-grained​​ motion (Pose), three-dimensional scene​​​‌ and entity representation (Depth),‌ surrounding objects (Panoptic masks),‌​‌ global motion (optical flow),​​ as well as language​​​‌ cues (VLM). Each modality‌ represents an axis of‌​‌ a polygon, streamlined to​​ add salient cues to​​​‌ RGB. π-VAD includes two‌ plug-in modules, namely the‌​‌ Pseudo-modality Generation module and​​ the Cross Modal Induction​​​‌ module, which generate modality-specific‌ prototypical representations and, thereby,‌​‌ induce multi-modal information into​​ RGB cues. These modules​​​‌ operate by performing anomaly-aware‌ auxiliary tasks and necessitate‌​‌ five modality backbones –​​ only during training. Notably,​​​‌ π-VAD achieves state-of-the-art accuracy‌ on three prominent VAD‌​‌ datasets encompassing real-world scenarios,​​ without requiring the computational​​​‌ overhead of five modality‌ backbones at inference.

This‌​‌ work has been published​​ at CVPR 2025 40​​​‌.

8.12 Mixture of‌ Experts Guided by Gaussian‌​‌ Splatters Matters: A new​​ Approach to Weakly-Supervised Video​​​‌ Anomaly Detection

Participants: Snehashis‌ Majhi, Giacomo D’Amicantonio‌​‌, Dantcheva Antitza,​​ Francois Bremond.

We​​​‌ identify one of the‌ main issues in the‌​‌ formulation of the Weakly-supervised​​ video anomaly detection (WSVAD)​​​‌ task. Multi-instance learning (MIL)‌ strikes a balance between‌​‌ fully supervised methods, which​​ exhibit good performance but​​​‌ require costly data annotation,‌ and unsupervised methods, which‌​‌ do not require manual​​ annotations but generally result​​​‌ in worse performance. The‌ core idea of MIL‌​‌ is to create bags​​ containing positive and negative​​​‌ data samples (i.e.‌, normal and abnormal‌​‌ videos), labeled only at​​ the video-level. During training,​​​‌ the model assigns a‌ score between 0 and‌​‌ 1 to each snippet,​​ with 0 indicating a​​​‌ normal snippet and 1‌ indicating an abnormal snippet.‌​‌ The highest-scoring samples in​​ the normal bag are​​​‌ guided towards 0, allowing‌ the model to learn‌​‌ most normal scenarios correctly.​​ On the other hand,​​​‌ the highest-scoring negative samples‌ are pushed towards 1.‌​‌ This leads the model​​ to be supervised, and​​​‌ therefore learn few and‌ specific instances of anomalous‌​‌ events, ignoring useful information​​ contained in neighboring snippets.​​​‌ Over time, this approach‌ has proved to be‌​‌ powerful but insufficient to​​ train a model to​​​‌ correctly capture the secondary‌ and specific attributes of‌​‌ different anomalous classes. In​​ recent works, different auxiliary​​​‌ objectives are identified as‌ priors for the VAD‌​‌ task to optimize the​​ training process.

Overview of​​​‌ the GS-MoE architecture

To address this​​​‌ issue, we propose to​ model the anomalies in​‌ a video as Gaussian​​ distributions (see Fig. 11​​​‌), rendering multiple Gaussian​ kernels in correspondence with​‌ peaks detected along the​​ temporal dimension of the​​​‌ scores estimated for abnormal​ videos. This technique, called​‌ Temporal Gaussian Splatting (TGS),​​ creates a more complete​​​‌ representation of an anomalous​ event over time, including​‌ snippets of the anomaly​​ with lower abnormal scores​​​‌ in the training objective.​ The Gaussian kernels are​‌ extracted from the abnormal​​ scores produced by the​​​‌ model.

An additional challenge​ is related to the​‌ intrinsic differences between abnormal​​ classes. Under the MIL​​​‌ paradigm, the models are​ trained to learn the​‌ difference between normal and​​ abnormal videos, while the​​​‌ specific differences between anomalous​ classes are overlooked. As​‌ a result, these methods​​ mainly focus on coarse-level​​​‌ representations of anomalies that​ allow us to distinguish​‌ between normal and abnormal​​ events, but ignore the​​​‌ fine-grained category-specific cues. Therefore,​ the more salient anomalies​‌ (i.e., such​​ as an explosion) are​​​‌ likely to be easily​ detected, while subtle anomalies​‌ (i.e., shoplifting)​​ are more likely to​​​‌ be confused with normal​ events. This constitutes a​‌ major limitation of most​​ recent methods based on​​​‌ WSVAD. We address this​ issue via a Mixture-of-Expert​‌ (MoE) architecture, in which​​ each expert is trained​​​‌ to model a single​ anomaly class, enhancing the​‌ specific attributes of each​​ anomaly class that are​​​‌ often overlooked. To further​ leverage the correlations and​‌ differences between anomalies, a​​ gate model mediates between​​​‌ the predictions of each​ expert and the more​‌ coarse-level anomalous features to​​ learn potential interactions between​​​‌ anomalies.

This work has​ been published at ICCV​‌ 2025 48.

8.13​​ Denoise, Divide, Distill, and​​​‌ Predict (D3¶): Towards Forecasting​ Long-horizon Real-world Anomaly from​‌ Normalcy

Participants: Quentin Merilleau​​, Snehashis Majhi,​​​‌ Dantcheva Antitza, Francois​ Bremond.

Forecasting abnormal​‌ human behavior (AHB) in​​ unconstrained real-world environments is​​​‌ critical for enabling proactive​ safety interventions 42.​‌ Unlike short-term anomaly detection,​​ long-horizon forecasting offers a​​​‌ vital reaction window but​ remains underexplored due to​‌ three core challenges: (i)​​ noisy, complex human–agent interactions;​​​‌ (ii) weak temporal coupling​ between normal observations and​‌ distant anomalies; and (iii)​​ data scarcity limiting the​​​‌ scalability of autoregressive models.​ To address these, we​‌ propose (Denoise, Divide, Distill,​​ and Predict) displayed in​​​‌ Fig. 12, a​ novel encoder–decoder framework that​‌ bridges denoised pasts with​​ distilled autoregressive futures, which​​​‌ has been accepted for​ publication in WACV 2026.​‌ Our Differential Past Encoder​​ (DiPE) disentangles scene-level and​​​‌ object-level dynamics via differential​ attention, suppressing irrelevant interactions​‌ and enhancing discriminative cues.​​ The Distilled Future Auto-Regressive​​​‌ Decoder (D-FAD) adopts a​ divide-and-conquer strategy, segmenting future​‌ queries into temporal chunks​​ for sequential prediction, while​​​‌ leveraging distillation to balance​ robustness and latency. We​‌ validate our approach on​​ the AHB-F benchmark, the​​​‌ only dataset dedicated to​ abnormal behavior forecasting, and​‌ further integrate D-FAD with​​ several state-of-the-art methods. In​​​‌ all cases, our framework​ consistently outperforms prior work​‌ in both forecasting accuracy​​ and computational efficiency.

The​​ image compares traditional video​​​‌ anomaly detection with a‌ new video anomaly anticipation‌​‌ method. In the traditional​​ method, labeled "Video Anomaly​​​‌ Detection," anomalies are identified‌ in the current frame‌​‌ by analyzing past frames​​ and classifying future frames​​​‌ as normal or anomalous.‌ The new method, "Our‌​‌ Video Anomaly Anticipation," not​​ only detects anomalies in​​​‌ the current frame but‌ also predicts future anomalies‌​‌ by utilizing both short-term​​ (1-3 seconds) and long-term​​​‌ (4-8 seconds) anticipations. The‌ diagram is divided into‌​‌ sections showing offline and​​ online processes. Offline training​​​‌ involves multiple frames, while‌ online detection and anticipation‌​‌ use a reference model​​ (Dref) to predict anomalies.​​​‌ The overall aim is‌ to enhance early detection‌​‌ of irregular events in​​ videos. (Description generated at​​​‌ January 15th, 2026 by‌ Albert AI with the‌​‌ model Mistral-Small-3.2-24B)

8.14 Not‌ All Blends Are Equal:‌​‌ The BLEMORE Dataset of​​ Blended Emotion Expressions with​​​‌ Relative Salience Annotations

Participants:‌ Michal Balazia, Teimuraz‌​‌ Saghinadze, Francois Bremond​​.

(Both paper and​​​‌ competition are accepted at‌ FG 2026)

Humans often‌​‌ experience not just a​​ single basic emotion at​​​‌ a time, but rather‌ a blend of several‌​‌ emotions with varying salience.​​ Despite the importance of​​​‌ such blended emotions, most‌ video-based emotion recognition approaches‌​‌ are designed to recognize​​ single emotions only. The​​​‌ few approaches that have‌ attempted to recognize blended‌​‌ emotions typically cannot assess​​ the relative salience of​​​‌ the emotions within a‌ blend. This limitation largely‌​‌ stems from the lack​​ of datasets containing a​​​‌ substantial number of blended‌ emotion samples annotated with‌​‌ relative salience. To address​​ this shortcoming, we introduce​​​‌ BLEMORE, a novel dataset‌ for multimodal (video, audio)‌​‌ BLended EMOtion​​ REcognition (see Figure​​​‌ 13) that includes‌ information on the relative‌​‌ salience of each emotion​​ within a blend.

Examples​​​‌ of stills from the‌ video recordings

The image‌​‌ shows a sequence of​​ four photos featuring a​​​‌ person with dark hair‌ tied back, wearing a‌​‌ black shirt. The photos​​ progressively depict increasing expressions​​​‌ of emotion, starting from‌ a neutral expression and‌​‌ moving through stages of​​ surprise or excitement. The​​​‌ person's mouth opens wider‌ and their facial muscles‌​‌ tense more in each​​ subsequent photo. The background​​​‌ is a plain, neutral‌ gray. (Description generated at‌​‌ January 15th, 2026 by​​​‌ Albert AI with the​ model Mistral-Small-3.2-24B)

BLEMORE comprises over​​ 3,000 clips from 58​​​‌ actors, performing 6 basic​ emotions (anger, disgust, fear,​‌ happiness, sadness, and neutral)​​ and 10 distinct blends​​​‌ consisting of all pairwise​ combinations of anger, disgust,​‌ fear, happiness, and sadness.​​ All pairwise combinations (see​​​‌ Figure 14) were​ further conveyed with three​‌ different blend conditions:

• 50/50​​ = same amount of​​​‌ both emotions (e.g. 50/50​ happiness-sadness where both happiness​‌ and sadness are expressed​​ in equal proportions)
• 70/30​​​‌ = the first emotion​ is more salient than​‌ the second emotion (e.g.​​ 70/30 happiness-sadness conveys mainly​​​‌ happiness blended with a​ tinge of sadness)
• 30/70​‌ = the second emotion​​ is more salient than​​​‌ the first emotion (e.g.​ 30/70 happiness-sadness conveys mainly​‌ sadness blended with a​​ tinge of happiness)

Structure​​​‌ of the BLEMORE full​ dataset

Using this dataset,​‌ we conduct extensive evaluations​​ of state-of-the-art video classification​​​‌ approaches on two blended​ emotion prediction tasks: (1)​‌ predicting the presence of​​ emotions in a given​​​‌ sample, and (2) predicting​ the relative salience of​‌ emotions in a blend.​​ Our results show that​​​‌ unimodal classifiers achieve up​ to 29% presence accuracy​‌ and 13% salience accuracy​​ on the validation set,​​​‌ while multimodal methods yield​ clear improvements, with ImageBind+WavLM​‌ reaching 35% presence accuracy​​ and HiCMAE 18% salience​​​‌ accuracy. On the held-out​ test set, the best​‌ models achieve 33% presence​​ accuracy (VideoMAEv2+HuBERT) and 18%​​​‌ salience accuracy (HiCMAE).

BLEMORE​ dataset is also the​‌ basis of BLEMORE competition​​ where participants develop systems​​​‌ to predict the emotions​ present in each recording​‌ and the relative salience​​ of each emotion. To​​​‌ support participation, we provide​ training data with labels,​‌ test data without labels,​​ pre-extracted audio-visual feature embeddings,​​​‌ and baseline unimodal and​ multimodal classification results. The​‌ competition offers the first​​ comprehensive platform for evaluating​​​‌ blended emotion recognition and​ aims to stimulate methodological​‌ innovation in multimodal affective​​ computing.

8.15 The INEMO​​​‌ Dataset: A Multimodal Benchmark​ of Physiological and Behavioral​‌ Responses to Social Media​​ and Film Stimuli

Participants:​​​‌ Wenxin Xiong, Valeriya​ Strizhkova, Aowen Shi​‌, Michal Balazia,​​ Laura Ferrari, Francois​​​‌ Bremond.

The INEMO​ dataset is a multimodal​‌ benchmark designed to study​​ emotional and behavioral responses​​​‌ to influencer-style social media​ videos and emotion calibration​‌ film clips. As shown​​ in Figures 15 and​​​‌ 16, participants complete​ two tasks (Influencer and​‌ Calibration), in which they​​ watch short video clips​​​‌ and then rate their​ emotions using 1–9 Self-Assessment​‌ Manikin (SAM) scales for​​ valence and arousal, as​​​‌ well as provide preference​ judgments about the videos.​‌ During these sessions, multiple​​ synchronized modalities are recorded,​​​‌ including facial video, electrocardiography​ (ECG), electrodermal activity (EDA),​‌ eye tracking and screen​​ activity, all time-aligned and​​ stored in a structured​​​‌ metadata format organized by‌ participant, task and modality.‌​‌ This design makes INEMO​​ directly usable for machine​​​‌ learning and deep learning‌ models and positions it‌​‌ as a bridge between​​ traditional lab-based affective datasets​​​‌ and more realistic social‌ media scenarios.

The image‌​‌ depicts an experiment protocol​​ with two tasks. Task​​​‌ 1 involves watching influencer‌ video clips in three‌​‌ sets, each with three​​ videos, and using the​​​‌ Self-Assessment Manikin (SAM) to‌ gauge reactions. Participants also‌​‌ rank preferences for individuals​​ and others. Task 2​​​‌ includes emotion calibration with‌ videos evoking different emotions‌​‌ (amusement, tenderness, sadness, disgust,​​ fear), followed by SAM​​​‌ assessments. The process ends‌ with questionnaires. SAM measures‌​‌ valence (negative to positive)​​ and arousal (calm to​​​‌ exciting). (Description generated at‌ January 15th, 2026 by‌​‌ Albert AI with the​​ model Mistral-Small-3.2-24B)

To evaluate the‌​‌ dataset and illustrate its​​ potential for multimodal emotion​​​‌ recognition, classical machine learning‌ models (SVM, Random Forest,‌​‌ Gradient Boosting) were trained​​ on handcrafted features extracted​​​‌ from ECG and EDA,‌ with and without video‌​‌ features, and compared to​​ a multimodal MVP-based (i.e.,​​​‌ Multimodal for Video and‌ Physio) baseline that jointly‌​‌ integrates ECG, EDA and​​ facial video. The best​​​‌ results are obtained with‌ a Gradient Boosting model‌​‌ using the combined ECG+EDA+Video​​ configuration, reaching weighted F1-scores​​​‌ of about 0.78 for‌ valence and 0.76 for‌​‌ arousal, and accuracies up​​ to 0.80 for valence​​​‌ and 0.70 for arousal.‌ These results confirm that‌​‌ the INEMO signals are​​ informative and that the​​​‌ associated classification tasks are‌ learnable, while still leaving‌​‌ room for more advanced​​ multimodal modeling approaches.

8.16​​​‌ EEG Classification with Limited‌ Data: A Deep Clustering‌​‌ Approach.

Participants: Mohsen Tabejamaat​​, Farhood Negin,​​​‌ Francois Bremond.

The‌ computer vision community has‌​‌ explored dyadic interactions for​​​‌ atomic actions such as​ pushing, carrying-object, etc. However,​‌ with the advancement in​​ deep learning models, there​​​‌ is a need to​ explore more complex dyadic​‌ situations such as loose​​ interactions. These are interactions​​​‌ where two people perform​ certain atomic activities to​‌ complete a global action​​ irrespective of temporal synchronization​​​‌ and physical engagement, like​ cooking-together for example. Analyzing​‌ these types of dyadic-interactions​​ has several useful applications​​​‌ in the medical domain​ for social-skills development and​‌ mental health diagnosis. To​​ achieve this, we propose​​​‌ a novel dual-path architecture​ to capture the loose​‌ interaction between two individuals.​​ Our model learns global​​​‌ abstract features from each​ stream via a CNNs​‌ backbone and fuses them​​ using a new Global-Layer-Attention​​​‌ module based on a​ cross-attention strategy. We evaluate​‌ our model on real-world​​ autism diagnoses such as​​​‌ our Loose-Interaction dataset, and​ the publicly available Autism​‌ dataset for loose interactions.​​ Our network achieves baseline​​​‌ results on the Loose-Interaction​ and SOTA results on​‌ the Autism datasets. Moreover,​​ we study different social​​​‌ interactions by experimenting on​ a publicly available dataset​‌ i.e. NTU-RGB+D (interactive classes​​ from both NTU-60 and​​​‌ NTU-120). We have found​ that different interactions require​‌ different network designs. We​​ also compare a slightly​​​‌ different version of our​ method by incorporating time​‌ information to address tight​​ interactions achieving SOTA results.​​​‌

This work has been​ published in Pattern Recognition​‌ 2025 34.

8.17​​ MEPHESTO: Multimodal Phenotyping of​​​‌ Psychiatric Disorders from Social​ Interaction

Participants: Michal Balazia​‌, Aowen Shi,​​ Miriana Russo, Francois​​​‌ Bremond.

Identifying objective​ and reliable markers to​‌ tailor diagnosis and treatment​​ of psychiatric patients remains​​​‌ a challenge, as conditions​ like major depression, bipolar​‌ disorder, or schizophrenia are​​ qualified by complex behavior​​​‌ observations or subjective self-reports​ instead of easily measurable​‌ somatic features. Recent progress​​ in computer vision, speech​​​‌ processing and machine learning​ has enabled detailed and​‌ objective characterization of human​​ behavior in social interactions.​​​‌ However, the application of​ these technologies to personalized​‌ psychiatry is limited due​​ to the lack of​​​‌ sufficiently large corpora that​ combine multimodal measurements with​‌ longitudinal assessments of patients​​ covering more than a​​​‌ single disorder. Our multi-centre,​ multi-disorder longitudinal corpus creation​‌ effort MEPHESTO is designed​​ to develop and validate​​​‌ novel multimodal markers for​ psychiatric conditions. MEPHESTO consists​‌ of multimodal audio, video,​​ and physiological recordings as​​​‌ well as clinical assessments​ of psychiatric patients covering​‌ a six-week main study​​ period as well as​​​‌ several follow-up recordings spread​ across twelve months.

Diagnoses​‌ include schizophrenia, depression and​​ bipolar disorder. Dataset does​​​‌ not include control subjects.​ Each patient is contributing​‌ with 1–8 videos, roughly​​ 5.5 videos on average.​​​‌ In addition to video,​ the recordings include patients'​‌ and clinicians' biosignals electrodermal​​ activity (EDA), blood volume​​​‌ pulse (BVP), inter-beat interval​ (IBI), heart rate, temperature,​‌ and accelerometer. Videos are​​ recorded by Azure Kinect​​​‌ and biosignals by Empatica.​ People do not wear​‌ face masks while being​​ recorded, although to minimize​​​‌ the transmission of COVID-19​ there is a large​‌ transparent plexi-glass. Dataset is​​ confidential, but many patients​​ agreed to publish their​​​‌ raw or anonymized data‌ for research purposes. Figure‌​‌ 17 shows a screenshot​​ from a mock recording.​​​‌

The image shows two‌ people sitting in different‌​‌ rooms. Each person has​​ a set of physiological​​​‌ data displayed below them.‌ The data includes temperature,‌​‌ EDA (electrodermal activity), BVP​​ (blood volume pulse), ACC​​​‌ (accelerometer), and HR (heart‌ rate). The person on‌​‌ the left is seated​​ near a window and​​​‌ wears a black and‌ white striped shirt. The‌​‌ person on the right​​ sits near a bookshelf​​​‌ and wears a black‌ top. (Description generated at‌​‌ January 15th, 2026 by​​ Albert AI with the​​​‌ model Mistral-Small-3.2-24B)

This​​ year, we have made​​​‌ three major contributions regarding‌ therapeutic alliance, recognizing depression‌​‌ and schizophrenia, and detecting​​ childhood trauma from speech.​​​‌ These contributions are explained‌ in detail in the‌​‌ subsections below.

8.17.1 Contextualized​​ Synchrony for Therapeutic Alliance​​​‌

Non-verbal behavioral synchrony has‌ been widely studied as‌​‌ an indicator of relational​​ dynamics in clinical interactions​​​‌ and has been shown‌ to exhibit weak to‌​‌ moderate associations with therapeutic​​ alliance (TA). However, most​​​‌ existing synchrony measures are‌ computed in a content-agnostic‌​‌ manner, implicitly assuming that​​ synchrony occurring at different​​​‌ moments of an interaction‌ contributes equally to the‌​‌ development of the therapeutic​​ relationship. This work is​​​‌ motivated by the hypothesis‌ that the relational meaning‌​‌ of synchrony is context-dependent,​​ and that linguistic content​​​‌ may play a critical‌ role in determining when‌​‌ non-verbal coordination is most​​ relevant to therapeutic alliance.​​​‌ In our setting, TA‌ is assessed at the‌​‌ end of each session​​ via a seven-item patient​​​‌ questionnaire capturing liking, perceived‌ helpfulness, feeling understood and‌​‌ supported, and ease of​​ sharing personal information, with​​​‌ the global TA score‌ obtained by averaging item‌​‌ responses. By integrating semantic​​ information derived from spoken​​​‌ language with non-verbal synchrony‌ measures, this study aims‌​‌ to move beyond global,​​ uniform synchrony metrics toward​​​‌ a more fine-grained, context-sensitive‌ understanding of therapist–patient interaction‌​‌ dynamics. Non-verbal synchrony was​​ computed at the window​​​‌ level using Motion Energy‌ Analysis (MEA, see Figure‌​‌ 18 for an example​​ of patient–therapist MEA time​​​‌ series) and a cross-correlation‌ framework applied to the‌​‌ continuous motion energy time​​ series of patient and​​​‌ therapist.

The image is‌ a line graph titled‌​‌ "Motion Energy Analysis (MEA):​​ Patient vs. Therapist." It​​​‌ depicts standardized motion energy‌ on the y-axis versus‌​‌ time on the x-axis.​​ The graph compares the​​​‌ motion energy of a‌ patient and a therapist,‌​‌ represented by blue and​​ orange lines, respectively. The​​​‌ patient's motion energy shows‌ smaller, more frequent fluctuations,‌​‌ while the therapist's energy​​ exhibits larger, more sporadic​​​‌ peaks. Both lines show‌ significant activity at the‌​‌ beginning and end of​​ the time period. (Description​​​‌ generated at January 15th,‌ 2026 by Albert AI‌​‌ with the model Mistral-Small-3.2-24B)​​​‌

We evaluate all​ models by predicting session-level​‌ TA scores and using​​ Pearson’s correlation coefficient $\mathrm{r‌}$ between predicted and observed​ TA as the primary​‌ outcome measure, computed in​​ a session-level cross-validation setting.​​​‌ We first replicated a​ stable baseline association between​‌ global MEA synchrony and​​ patient-reported TA, with a​​​‌ content-agnostic aggregation over all​ windows yielding a correlation​‌ of approximately $r\approx 0.22$.​​​‌ Building on this foundation,​ transcript data were processed​‌ into semantic embeddings and​​ temporally aligned with synchrony​​​‌ windows, enabling a multimodal​ representation in which textual​‌ context modulates how window-level​​ synchrony is aggregated over​​​‌ time. In the current​ implementation, not all MEA​‌ windows have a corresponding​​ text segment, so windows​​​‌ without aligned transcripts are​ ignored when applying text-informed​‌ weighting. Evaluating a uniform​​ (all-ones) aggregation under this​​​‌ constraint leads to a​ reduced MEA-TA association of​‌ $r\approx 0.13$, compared to​​​‌ the $r\approx \mathrm{0}.22$ obtained when​‌ all MEA windows are​​ used. Within this constrained​​​‌ evaluation setting, however, our​ text-informed weighting scheme increases​‌ the correlation to $\mathrm{r}\approx 0.\mathrm{18‌}$, suggesting that linguistic​ information helps to highlight​‌ synchrony segments that are​​ more informative about alliance.​​​‌ While the overall performance​ of this preliminary implementation​‌ does not yet surpass​​ the full-window MEA baseline,​​​‌ the results support the​ view that synchrony is​‌ not uniformly informative throughout​​ an interaction and highlight​​​‌ the potential of window-level,​ context-aware multimodal modeling combined​‌ with improved textual coverage​​ for capturing subtle relational​​​‌ processes in therapeutic settings.​

8.17.2 Psychiatric Diagnosis Classification​‌ through Temporal Behavioral Analysis​​

This sub-project focuses on​​​‌ automated psychiatric diagnosis through​ multimodal behavioral analysis of​‌ clinical interview videos, with​​ the objective of distinguishing​​​‌ between depression and schizophrenia.​ We utilize a portion​‌ of the MEPHESTO dataset​​ of 34 patients: 25​​​‌ with depression and 9​ with schizophrenia. The dataset​‌ includes manual behavioral annotations​​ provided by expert clinical​​​‌ annotators who labeled over​ 3000 video segments with​‌ observable behaviors. The implemented​​ system (see Figure 19​​​‌) follows a 7-stage​ pipeline: (1) input data​‌ acquisition from MEPHISTO with​​ pre-annotated transcriptions, (2) low-level​​​‌ extraction using OpenFace 3.0​ (8 Action Units: AU01,​‌ AU02, AU04, AU06, AU07,​​ AU12, AU14, AU45 +​​​‌ gaze + head pose​ + 8 emotions), MediaPipe​‌ holistic (33 pose, 42​​ hand, 468 face landmarks),​​​‌ and Whisper for speech​ (1,842 features/frame), (3) temporal​‌ alignment with frame-level synchronization​​ (±1 frame precision, 33ms),​​​‌ (4) multi-scale windowing (5s,​ 10s, 30s windows, 50%​‌ overlap) extracting 188 features​​ across 24,588 windows, (5)​​​‌ temporal variability aggregation computing​ 6 statistics per feature​‌ (mean, standard deviation, coefficient​​ of variation, minimum, maximum,​​​‌ range), (6) feature selection​ via ANOVA F-test selecting​‌ top 20 features (70%​​ speech-based, 30% visual), and​​​‌ (7) classification with random​ forest using leave-one-out cross-validation​‌ across 13 tested methods.​​

The image depicts a​​​‌ process for diagnosing psychiatric​ conditions (Depression vs. Schizophrenia)​‌ using a baseline random​​ forest model with feature​​ fusion. It involves extracting​​​‌ multi-modal features from patient‌ interview videos using three‌​‌ pipelines: OpenFace 3.0 for​​ facial actions and gaze,​​​‌ MediaPipe for body and‌ hand movements, and Whisper‌​‌ with speech analysis for​​ speech features. These features​​​‌ are temporally windowed, fused,‌ and statistically analyzed. Feature‌​‌ selection is performed using​​ ANOVA F-test, reducing the​​​‌ dataset to 20 features.‌ A random forest classifier‌​‌ is trained and validated​​ using leave-one-out cross-validation, achieving​​​‌ 94.1% accuracy. A confusion‌ matrix and top feature‌​‌ importance are displayed, highlighting​​ the most influential features​​​‌ for diagnosis. (Description generated‌ at January 15th, 2026‌​‌ by Albert AI with​​ the model Mistral-Small-3.2-24B)

Random forest achieves​​​‌ 94.1% accuracy with only‌ two schizophrenia patients misclassified.‌​‌ Top discriminative feature is​​ the standard deviation of​​​‌ patient's incomplete utterances. During‌ our experiments, we found‌​‌ that temporal variability is​​ the critical discriminative marker,​​​‌ that speech features dominate‌ (70%) in the top-20‌​‌ features, that feature fusion​​ outperforms modality separation, and​​​‌ that traditional machine learning‌ beats deep learning on‌​‌ small datasets. In the​​ future, we are going​​​‌ to focus on temporal‌ trauma detection in the‌​‌ long untrimmed clinical interviews.​​

8.18 MultiMediate'25: Cross-Cultural​‌ Multi-domain Engagement Estimation

Participants:​​ Michal Balazia, Francois​​​‌ Bremond.

Estimating momentary​ conversational engagement is central​‌ to assistive, socially aware​​ AI systems, yet models​​​‌ are typically trained and​ evaluated within a single​‌ domain, limiting real-world robustness.​​ The MultiMediate'25 challenge 47​​​‌ advances engagement estimation to​ more challenging, cross-cultural, and​‌ multi-domain settings. Building on​​ prior challenge editions, we​​​‌ expand beyond NOXI and​ MPIIGroupInteraction (see Figure 20​‌) as the sole​​ training source by introducing​​​‌ NOXI-J, a new multilingual​ corpus covering Japanese and​‌ Chinese interactions, enabling both​​ training and evaluation in​​​‌ diverse linguistic contexts. Although​ NOXI-J conceptually extends NOXI,​‌ we treat it as​​ a distinct domain because​​​‌ linguistic, cultural, capture, and​ annotation differences induce measurable​‌ distribution shifts. MultiMediate'25 continues​​ all previously defined tasks​​​‌ and creates another task:​ Cross-cultural Multi-domain Engagement Estimation.​‌

In this work, we​​ present new annotations, precomputed​​​‌ multi-modal features (visual, vocal,​ and verbal), baseline evaluations,​‌ and an analysis of​​ the best performing challenge​​​‌ solutions. Beyond accuracy, we​ quantify fairness using conditional​‌ demographic disparity for gender​​ and language. Our baselines​​​‌ confirm strong in-domain performance​ (e.g., paralinguistic eGeMAPS and​‌ video-transformer features) and reveal​​ notable cross-domain drops, underscoring​​​‌ the challenge of cultural,​ linguistic, and interactional shifts.​‌ Fairness analyses indicate generally​​ small discrepancies for our​​​‌ baselines. We observe the​ largest disparities for the​‌ proposed challenge solutions on​​ the Chinese language test​​​‌ set. All annotations, features,​ code, and leaderboards are​‌ made publicly available to​​ foster sustained progress on​​​‌ robust and fair engagement​ estimation.

The image consists​‌ of three side-by-side photos​​ of a man in​​​‌ different poses. In the​ first photo, he is​‌ talking on a phone,​​ in the second he​​​‌ is standing relaxed, and​ in the third he​‌ is gesturing with his​​ hands. Below each photo,​​​‌ there is a graphical​ representation indicating some form​‌ of measured data, possibly​​ volume or sound levels.​​​‌ Each graph has a​ red vertical line and​‌ a yellow-shaded area with​​ varying black line patterns.​​​‌ (Description generated at January​ 14th, 2026 by Albert​‌ AI with the model​​ Mistral-Small-3.2-24B)

The image consists​​​‌ of three side-by-side photos​ of a man in​‌ different poses. In the​​ first photo, he is​​​‌ talking on a phone,​ in the second he​‌ is standing relaxed, and​​ in the third he​​​‌ is gesturing with his​ hands. Below each photo,​‌ there is a graphical​​ representation indicating some form​​​‌ of measured data, possibly​ volume or sound levels.​‌ Each graph has a​​ red vertical line and​​​‌ a yellow-shaded area with​ varying black line patterns.​‌ (Description generated at January​​ 14th, 2026 by Albert​​​‌ AI with the model​ Mistral-Small-3.2-24B)

As​​ training datasets, we provide​​​‌ NOXI and NOXI-J to‌ our participants. NOXI is‌​‌ a corpus of dyadic,​​ screen-mediated face-to-face interactions in​​​‌ an expert-novice knowledge sharing‌ context. In a session,‌​‌ one participant assumes the​​ role of an expert​​​‌ and the other participant‌ the role of a‌​‌ novice. NOXI includes interactions​​ recorded at three locations​​​‌ (France, Germany and UK),‌ spoken in seven languages‌​‌ (English, French, German, Spanish,​​ Indonesian, Arabic and Italian),​​​‌ discussing a wide range‌ of topics. The languages‌​‌ Indonesian, Arabic, Spanish, and​​ Italian serve as an​​​‌ out-of-domain evaluation set. NOXI‌ is extended by NOXI-J‌​‌ consisting of 66 dyadic​​ interactions and over 16​​​‌ hours of material using‌ the same setup as‌​‌ original NOXI. NOXI-J features​​ 48 interactions in Japanese​​​‌ with native Japanese speakers‌ and 18 interactions in‌​‌ Chinese with Chinese native​​ speakers. See Table 1​​​‌ for the train-validation-test split.‌

The​​ task is frame-wise prediction​​​‌ of each interlocutor's engagement‌ on a continuous scale‌​‌ $[0,\mathrm{1}]$. Accuracy is​​​‌ measured with the Concordance‌ Correlation Coefficient (CCC), ranging‌​‌ from $-1$ to​​ $+1$. Participants​​​‌ are free to use‌ the provided labeled data‌​‌ for training and validation​​ and undergo in-domain and​​​‌ out-of-domain evaluations on NoXI,‌ NoXI-J, NoXI (Additional Languages),‌​‌ and MPIIGroupInteraction. We provide​​ a multi-modal set of​​​‌ precomputed features to participants.‌ From the audio signal,‌​‌ we provide transcripts generated​​ with the Whisper model.​​​‌ Additionally, we supply GeMAPS‌ features along with wav2vec‌​‌ 2.0 embeddings. From the​​ video, we provide the​​​‌ backbone embeddings of Video‌ Swin Transformer, DINOv2, CLIP‌​‌ and VideoMAEv2 and the​​ outputs of OpenFace and​​​‌ OpenPose to cover facial‌ as well as body‌​‌ behaviors.

8.19 Stress Estimation​​ in Dancers for Injury​​​‌ Prevention

Participants: Dian-Wei Lai‌, Quentin Merilleau,‌​‌ Aowen Shi, Francois​​ Bremond.

Detecting stress​​​‌ in dancers is important,‌ as high stress levels‌​‌ are often related to​​ fatigue and injuries, which​​​‌ can negatively affect both‌ performance and health. However,‌​‌ stress detection itself is​​ not an easy task.​​​‌ This becomes even more‌ challenging when using indirect‌​‌ and non-invasive data such​​ as video. Although video​​​‌ is one of the‌ most commonly available modalities,‌​‌ extracting reliable stress information​​ from it remains highly​​​‌ challenging.

In this work,‌ we investigate automatic stress‌​‌ estimation from dance videos​​ using a small, weakly​​​‌ labeled dataset collected from‌ professional dancers at Université‌​‌ Côte d’Azur. Each dancer​​​‌ performs the same dance​ under three different difficulty​‌ levels and in different​​ scenes. The dataset currently​​​‌ includes 84 dancers, with​ two camera views (front​‌ view and diagonal view).​​ Each video is approximately​​​‌ 1 to 2 minutes​ long. Data collection is​‌ still ongoing to further​​ enrich the dataset and​​​‌ improve the reliability of​ the stress score distributions,​‌ PDF and CDF curves​​ in Figure 21.​​​‌

The image shows a​ study with 84 dancers​‌ performing three exercises of​​ varying difficulty: Easy (Exercise​​​‌ 1), Intermediate (Exercise 2),​ and Hard (Exercise 3).​‌ The right side displays​​ histograms and cumulative distribution​​​‌ functions (CDFs) of stress​ levels assessed by judges​‌ for each exercise. The​​ histograms show the distribution​​​‌ of stress scores, while​ the CDFs illustrate the​‌ cumulative probabilities of these​​ scores. Stress levels appear​​​‌ to increase with the​ difficulty of the exercise.​‌ (Description generated at January​​ 15th, 2026 by Albert​​​‌ AI with the model​ Mistral-Small-3.2-24B)

To obtain meaningful​‌ results with limited data,​​ we leverage pretrained models​​​‌ trained on large-scale video​ and motion datasets to​‌ improve feature representations. We​​ then study the contribution​​​‌ of different visual modalities,​ including RGB, skeleton poses​‌ extracted using different methods​​ with richer joint (see​​​‌ Figure 22) and​ hand motion information, depth,​‌ and optical flow. By​​ analyzing each modality separately​​​‌ and in combination, we​ aim to build a​‌ robust multi-modal pipeline for​​ stress estimation and to​​​‌ identify which modalities and​ movement cues are most​‌ informative for effective stress​​ prediction.

The image contains​​​‌ a series of four​ photos showing a person​‌ in a dynamic pose,​​ with different colored lines​​​‌ and dots overlaid on​ the person's body. These​‌ lines and dots appear​​ to represent a pose​​​‌ estimation or skeletal tracking​ system, mapping key points​‌ such as the head,​​ shoulders, elbows, wrists, hips,​​​‌ knees, and ankles. The​ person is dressed in​‌ a long-sleeved shirt and​​ pants. The background shows​​​‌ a room with a​ partially visible doorway and​‌ paintings on the wall.​​ (Description generated at January​​​‌ 15th, 2026 by Albert​ AI with the model​‌ Mistral-Small-3.2-24B)

8.20​ Emotion recognition using Deep​‌ Learning

Participants: Valeriya Strizhkova​​, Antitza Dantcheva,​​​‌ Francois Bremond.

Understanding​ human emotions is crucial​‌ in healthcare, human-robot interaction,​​ and marketing. Despite the​​​‌ progress in emotion recognition​ from one modality, such​‌ as a facial video​​ and a sequence of​​​‌ physiological signals, it is​ still challenging to improve​‌ by combining multiple modalities.​​ Moreover, it is difficult​​​‌ to recognize emotions in​ long sequential data, such​‌ as long videos, although​​ most real-world videos of​​ people expressing emotions are​​​‌ long. Existing emotion datasets‌ are limited in volume‌​‌ and quality, making it​​ difficult to develop an​​​‌ effective deep learning-based emotion‌ recognition system. An effective‌​‌ real-world emotion understanding system​​ should be able to​​​‌ recognize emotions from long‌ videos synchronized with multiple‌​‌ modalities. In this thesis​​ 52, we focus​​​‌ on multimodal emotion recognition‌ from long videos synchronized‌​‌ with physiological signals. Specifically,​​ multimodal emotion methods face​​​‌ three main challenges: (a)‌ learning the emotion representation,‌​‌ (b) learning the representation​​ of fine-grained emotions, as​​​‌ well as (c) combining‌ modalities to predict emotions.‌​‌ In this thesis, we​​ first introduce two large​​​‌ behavior analysis datasets: INEMO‌ and StressID. INEMO is‌​‌ a multimodal dataset designed​​ to facilitate emotion recognition​​​‌ from watching social media‌ videos. StressID is a‌​‌ multimodal dataset designed for​​ stress identification. Secondly, we​​​‌ propose two pre-training techniques‌ for facial expression recognition:‌​‌ (1) supervised pre-training on​​ synthetic data generated by​​​‌ our video generation method‌ and (2) self-supervised pre-training‌​‌ on multi-view videos. We​​ show that the proposed​​​‌ pre-training techniques allow us‌ to get rich facial‌​‌ representations, which allow us​​ to improve fine-grained emotion​​​‌ recognition accuracy. Thirdly, we‌ tackle the problem of‌​‌ emotion recognition from multiple​​ modalities. We propose a​​​‌ framework for multimodal fusion‌ of videos and physiological‌​‌ signals to predict emotions.​​ This framework consists of​​​‌ mainly two steps: (1)‌ extracting features from long‌​‌ raw videos and physiological​​ signals; (2) fusing extracted​​​‌ features to predict emotions‌ using a cross-modality approach‌​‌ based on attention mechanism.​​ Our methods leverage the​​​‌ additional modalities resulting in‌ better emotion recognition performance.‌​‌ Our methods have been​​ extensively evaluated on various​​​‌ emotion recognition benchmarks. The‌ proposed methods outperform previous‌​‌ methods, significantly pushing emotion​​ recognition to real-world deployments.​​​‌

8.21 Identifying Surgical Instruments‌ in Pedagogical Cataract Surgery‌​‌ Videos through an Optimized​​ Aggregation Network

Participants: Sanya​​​‌ Sinha, Michal Balazia‌, Francois Bremond.‌​‌

Instructional cataract surgery videos​​ are crucial for ophthalmologists​​​‌ and trainees to observe‌ surgical details repeatedly. In‌​‌ 44, we present​​ a deep learning model​​​‌ for real-time identification of‌ surgical instruments in these‌​‌ videos, using a custom​​ dataset scraped from open-access​​​‌ sources. Inspired by the‌ architecture of YOLOv9, the‌​‌ model employs a Programmable​​ Gradient Information (PGI) mechanism​​​‌ and a novel Generally-Optimized‌ Efficient Layer Aggregation Network‌​‌ (Go-ELAN) to address the​​ information bottleneck problem, enhancing​​​‌ Minimum Average Precision (mAP)‌ at higher Non-Maximum Suppression‌​‌ Intersection over Union (NMS​​ IoU) scores.

Go-ELAN YOLOv9​​​‌ Architecture (see Figure 23‌) contains an auxiliary‌​‌ block which works on​​ the Programmable Gradient Information​​​‌ (PGI) concept by creating‌ an auxiliary reverse branch‌​‌ for enabling reliable gradient​​ calculation by avoiding potential​​​‌ semantic loss. The GELAN‌ block in the backbone‌​‌ feature extractor is replaced​​ by the Go-ELAN block​​​‌ proposed in this paper.‌ The Spatial Pyramid Pooling‌​‌ block SPPELAN removes the​​ fixed size limitation of​​​‌ the backbone. The ADown‌ block downsamples the generated‌​‌ feature maps to target​​ sizes. The CBLinear blocks​​​‌ extract higher level features‌ from the images, and‌​‌ the CBFuse block fuses​​​‌ these extracted features. The​ Neck combines the acquired​‌ features and the Head​​ predicts the final bounding​​​‌ bound outputs with their​ respective probabilities.

The image​‌ depicts a neural network​​ architecture with sections labeled​​​‌ Auxiliary, Backbone, Neck, and​ Head. The Backbone starts​‌ with a Silence layer,​​ followed by convolutional (Conv)​​​‌ layers, multiple Go-ELAN blocks,​ and ADown layers, ending​‌ with an SPPELAN block.​​ The Auxiliary section mirrors​​​‌ parts of the Backbone​ with convolutional layers, Go-ELAN​‌ blocks, and ADown layers,​​ and includes CBFuse and​​​‌ CBLinear blocks connecting to​ the Backbone. Both the​‌ Auxiliary and Backbone sections​​ feed into the Neck,​​​‌ which connects to the​ final Head section, consisting​‌ of Detect blocks for​​ making predictions. Connections between​​​‌ blocks are indicated by​ arrows showing data flow.​‌ (Description generated at January​​ 15th, 2026 by Albert​​​‌ AI with the model​ Mistral-Small-3.2-24B)

Our Go-ELAN YOLOv9 model,​​​‌ evaluated against YOLO v5,​ v7, v8, v9 vanilla,​‌ Laptool and DETR, achieves​​ a superior mAP of​​​‌ 73.74 at IoU 0.5​ on a dataset of​‌ 615 images with 10​​ instrument classes, demonstrating the​​​‌ effectiveness of the proposed​ model. To illustrate the​‌ visual and qualitative superiority​​ of our model, we​​​‌ have compared 12 ground-truth​ images with their respective​‌ model predictions in Figure​​ 24.

The image​​​‌ shows a collection of​ medical procedure photos, likely​‌ from eye surgeries. Each​​ image is labeled with​​​‌ various surgical tools such​ as speculum, cannula, forceps,​‌ hook, phacoprobe, and keratome.​​ The tools are highlighted​​​‌ with colored boxes and​ labels, indicating their position​‌ and type. The images​​ are arranged in a​​​‌ grid format, displaying different​ stages of the procedure.​‌ The eye is opened​​ with a speculum, and​​​‌ various tools are used​ for precise surgical actions.​‌ The photos include confidence​​ levels for the identified​​​‌ tools. (Description generated at​ January 15th, 2026 by​‌ Albert AI with the​​ model Mistral-Small-3.2-24B)

8.22​ TBDM: Temporal Boundary Distillation​‌ Module for Surgical Gesture​​ Segmentation

This work was​​​‌ funded by 3IA Côte​ d'Azur.

Participants: Ezem Sura​‌ Ekmekci, Snehashis Majhi​​, Khodor Hamadi,​​​‌ Francois Bremond.

In​ 2025, in collaboration with​‌ CHU Nice and Caranx​​ Medical, a novel framework​​​‌ for surgical gesture segmentation​ was developed that addresses​‌ the challenging problem of​​ precise temporal localization during​​​‌ surgical action transitions. This​ work introduces the Temporal​‌ Boundary Distillation Module (TBDM),​​ an innovative approach that​​​‌ explicitly models temporal boundaries​ between surgical gestures using​‌ RGB-only video data (see​​ Figure  25). The​​​‌ framework employs knowledge distillation​ to learn boundary-aware features​‌ during training through cross-attention​​ mechanisms, while requiring no​​​‌ additional computational overhead at​ inference. TBDM was validated​‌ on two major surgical​​ datasets (CholecT50 and RARP-45),​​​‌ demonstrating consistent improvements across​ multiple baseline architectures, with​‌ up to +8.5 edit​​ score improvement on CholecT50.​​​‌ Notably, the approach achieved​ state-of-the-art performance on RARP-45​‌ (81.4 edit score, 77.9​​ F1@50), establishing TBDM as​​ a generalizable, plug-and-play solution​​​‌ for fine-grained surgical workflow‌ analysis. This work has‌​‌ been submitted to IPCAI​​ 2026.

Additionally, a comprehensive​​​‌ evaluation of YOLOv8 for‌ real-time surgical instrument recognition‌​‌ in robot-assisted and laparoscopic​​ surgeries was conducted 33​​​‌. Using a diverse‌ multi-source dataset of over‌​‌ 7,400 frames and 17,175​​ annotations, the model achieved​​​‌ a mean average precision‌ of 0.77 for binary‌​‌ detection and 0.72 for​​ multi-instrument classification across seven​​​‌ instrument types. The segmentation‌ performance demonstrated excellent accuracy‌​‌ with a mean Dice​​ score of 0.91 and​​​‌ mean intersection over union‌ of 0.86. With an‌​‌ inference speed of 1.12​​ milliseconds per frame, the​​​‌ model shows strong potential‌ for real-time clinical applications‌​‌ in surgical workflow analysis​​ and instrument tracking.

The​​​‌ image depicts a machine‌ learning framework for gesture‌​‌ recognition. The process starts​​ with a sequence of​​​‌ video frames input into‌ a pre-trained VideoMAE-v2 model‌​‌ with frozen parameters. The​​ extracted features are then​​​‌ passed to a projection‌ layer and a temporal‌​‌ model for prediction. Additionally,​​ there is a temporal​​​‌ boundary distillation module that‌ only operates during training.‌​‌ This module uses cross-attention​​ mechanisms and class presence​​​‌ maps to aggregate gesture‌ class information. This module‌​‌ helps in refining the​​ model's decision boundaries through​​​‌ a distillation loss calculated‌ between the projection features‌​‌ and the distilled boundary​​ features. The framework aims​​​‌ to improve gesture recognition‌ accuracy by leveraging temporal‌​‌ information and class distinctions.​​ (Description generated at January​​​‌ 19th, 2026 by Albert‌ AI with the model‌​‌ Mistral-Small-3.2-24B)

8.23 Effective Video‌​‌ Feature Extraction for Training​​ and Comprehension: Human-Centered Multimodal​​​‌ Video

Participants: Tanay Agrawal‌, Antitza Dantcheva,‌​‌ Francois Bremond.

Understanding​​ actions in videos is​​​‌ a crucial element of‌ computer vision, with significant‌​‌ implications in many fields.​​ Given our increasing reliance​​​‌ on visual data, understanding‌ and interpreting human actions‌​‌ in videos are becoming​​ essential for developing technologies​​​‌ in surveillance, healthcare, autonomous‌ systems, and human-computer interaction.‌​‌ Accurate interpretation of actions​​ in videos is fundamental​​​‌ to creating intelligent systems‌ capable of navigating and‌​‌ responding effectively to the​​ complexities of the real​​​‌ world. In this context,‌ advances in action understanding‌​‌ are pushing the boundaries​​ of computer vision and​​​‌ playing a crucial role‌ in the development of‌​‌ cutting-edge applications that impact​​ our daily lives.

Computer​​​‌ vision has seen significant‌ progress thanks to the‌​‌ rise of deep learning​​ methods such as convolutional​​​‌ neural networks (CNNs) and‌ transformers, pushing the boundaries‌​‌ of computer vision and​​ enabling the computer vision​​​‌ community to advance in‌ many areas, including image‌​‌ segmentation, object detection, scene​​ understanding, and more. However,​​​‌ video processing remains limited‌ compared to static images.‌​‌ In this thesis, we​​ focus on video understanding,​​​‌ dividing it into two‌ main parts: video classification‌​‌ and action detection, and​​​‌ their application in affective​ computing, particularly in interaction-based​‌ scenarios. In this thesis,​​ we explore efficient learning​​​‌ approaches for video feature​ extraction in various video​‌ classification and interaction understanding​​ tasks. Our contributions 51​​​‌ cover the computation of​ intermediate-level features for faster​‌ convergence, plugin adaptation for​​ handling diverse datasets and​​​‌ modalities, and evolutionary temporal​ modeling for understanding long​‌ videos. We begin by​​ improving personality and behavior​​​‌ recognition through geometry-based behavioral​ coding and segmentation-driven attention​‌ mechanisms. We then address​​ the challenges of modality​​​‌ availability and data diversity​ using knowledge distillation and​‌ a novel adapter-based cross-learning​​ framework that generalizes to​​​‌ all tasks. Finally, we​ tackle the analysis of​‌ long videos for temporal​​ action detection using temporal​​​‌ adapters with image models,​ as well as modular​‌ adapters and a two-stage​​ spatiotemporal learning strategy with​​​‌ a video basis. Together,​ this work contributes to​‌ building generalizable and efficient​​ learning systems for a​​​‌ wide range of video​ understanding applications.

8.24 Rotation-Induced​‌ Centroid Shift in Latent​​ Space

Participants: Benoit Lagadec​​​‌, Matthieu Saumard,​ Francois Bremond.

Convolutional​‌ neural networks are not​​ rotation-equivariant in practice: discrete​​​‌ image rotation requires interpolation​ and zero-padding, making the​‌ rotation operator non-invertible and​​ causing convolution and rotation​​​‌ to not commute. We​ show that this leads​‌ to a systematic and​​ measurable shift of the​​​‌ feature-space centroid when images​ are rotated, even when​‌ the model is trained​​ with standard rotation augmentation.​​​‌ We formalize this centroid​ drift analytically and verify​‌ it empirically. To mitigate​​ this effect, we introduce​​​‌ a set of angle-specialized​ Exponential Moving Average (EMA)​‌ teachers that provide stable​​ feature anchors at different​​​‌ rotation angles, optionally enhanced​ with low-rank angle adapters.​‌ This approach directly suppresses​​ rotation-induced centroid shift and​​​‌ significantly improves feature consistency​ and classification accuracy under​‌ rotation, outperforming both classical​​ augmentation and mean-teacher baselines​​​‌ while requiring minimal additional​ computation. We formalize discrete​‌ in-plane rotation on pixel​​ grids as a degraded​​​‌ permutation and show why​ convolution and rotation do​‌ not commute. In this​​ work, we empirically confirm​​​‌ that the centroid of​ feature representations shifts under​‌ rotation. Many studies are​​ dedicated to find invariant​​​‌ in detection. An illustration​ is detailed in Figure​‌ 26.

This image​​ illustrates a series of​​​‌ transformations applied to a​ grid of pixels. 1.​‌ **Original Grid**: A small​​ grid with green highlighted​​​‌ pixels. 2. **Add Padding**:​ The grid is expanded​‌ with black padding around​​ it. 3. **Rotation 45​​​‌ Degrees**: The grid is​ rotated by 45 degrees​‌ within the padded area.​​ 4. **Real/Numerical Transformation**: The​​​‌ rotated grid is transformed​ into a new format​‌ while retaining its pixel​​ structure. 5. **Flatten Operation​​​‌ (Vectorization)**: Both the original​ and transformed grids are​‌ flattened into a linear​​ vector format. 6. **Re-ordering**:​​​‌ Pixels are re-ordered, with​ some pixels not affected​‌ by the transformation. 7.​​ **Limited Permutation**: The vectors​​​‌ are permuted to demonstrate​ how a rotation translates​‌ to a simple permutation​​ of pixels. 8. **Final​​​‌ Sub-Space**: After keeping only​ specific pixels, a sub-space​‌ of the image is​​ shown, illustrating that rotation​​ can be represented by​​​‌ a simple permutation of‌ pixels. (Description generated at‌​‌ January 15th, 2026 by​​ Albert AI with the​​​‌ model Mistral-Small-3.2-24B)

This image‌ illustrates a series of‌​‌ transformations applied to a​​ grid of pixels. 1.​​​‌ **Original Grid**: A small‌ grid with green highlighted‌​‌ pixels. 2. **Add Padding**:​​ The grid is expanded​​​‌ with black padding around‌ it. 3. **Rotation 45‌​‌ Degrees**: The grid is​​ rotated by 45 degrees​​​‌ within the padded area.‌ 4. **Real/Numerical Transformation**: The‌​‌ rotated grid is transformed​​ into a new format​​​‌ while retaining its pixel‌ structure. 5. **Flatten Operation‌​‌ (Vectorization)**: Both the original​​ and transformed grids are​​​‌ flattened into a linear‌ vector format. 6. **Re-ordering**:‌​‌ Pixels are re-ordered, with​​ some pixels not affected​​​‌ by the transformation. 7.‌ **Limited Permutation**: The vectors‌​‌ are permuted to demonstrate​​ how a rotation translates​​​‌ to a simple permutation‌ of pixels. 8. **Final‌​‌ Sub-Space**: After keeping only​​ specific pixels, a sub-space​​​‌ of the image is‌ shown, illustrating that rotation‌​‌ can be represented by​​ a simple permutation of​​​‌ pixels. (Description generated at‌ January 15th, 2026 by‌​‌ Albert AI with the​​ model Mistral-Small-3.2-24B)

Convolutional neural networks​​ are often assumed to​​​‌ be robust to rotations‌ when trained with rotation‌​‌ augmentations. However, this assumption​​ overlooks a key property​​​‌ of real image rotations:‌ discrete rotation on a‌​‌ pixel grid is implemented​​ using interpolation and padding,​​​‌ making the operation non-invertible‌ and causing it to‌​‌ not commute with convolution.​​ As a result, rotating​​​‌ an input image and‌ then extracting features is‌​‌ not equivalent to extracting​​ features and then rotating​​​‌ them. We show that‌ this mismatch induces a‌​‌ systematic and predictable shift​​ in the feature-space centroid​​​‌ across rotation angles, even‌ when the network is‌​‌ trained with extensive rotation​​ augmentation.

This observation reframes​​​‌ rotation robustness as a‌ problem of representation geometry‌​‌, rather than data​​ diversity alone. If rotation​​​‌ induces angle-dependent sub-clusters in‌ feature space, enforcing global‌​‌ consistency (e.g., with a​​ single Mean Teacher model)​​​‌ can suppress meaningful structure‌ and lead to underfitting.‌​‌ We therefore propose a​​ simple alternative: a set​​​‌ of angle-specialized EMA teachers‌ that provide stable feature‌​‌ targets at different rotation​​ angles, coupled with a​​​‌ feature-space centroid alignment loss‌ that prevents rotation-induced drift‌​‌ without collapsing intra-class variability.​​ Our approach is architecture-agnostic,​​​‌ computationally lightweight, and complementary‌ to standard training pipelines.‌​‌ It improves rotation robustness​​ in both classification and​​​‌ detection settings without requiring‌ group-equivariant architectures or spatial‌​‌ transformer modules. The core​​ contribution of this work​​​‌ is to characterize the‌ geometric effect of discrete‌​‌ rotation in CNN feature​​ space and to introduce​​​‌ a training strategy that‌ explicitly stabilizes this geometry.‌​‌

Applied to detection (see​​ Figure  26), our​​​‌ method ensures that rotation-induced‌ feature sub-clusters remain compact‌​‌ and aligned. This contrasts​​ with our former work,​​​‌ which uses a related‌ mechanism in person re-identification‌​‌ to enlarge inter-cluster separation,​​​‌ whereas our objective is​ to preserve sub-cluster coherence.​‌

8.25 Dual Volume Skeleton-Guided​​ 3D Face Reconstruction from​​​‌ Sparse Views

Participants: Benoit​ Lagadec, Seongro Yoon​‌, Francois Bremond.​​

Reconstructing high-fidelity 3D face​​​‌ meshes from sparse 2D​ inputs is challenging due​‌ to limited depth cues​​ and structural ambiguity. We​​​‌ present a skeleton-guided, dual-volume​ diffusion framework for reconstructing​‌ editable, high-fidelity 3D face​​ meshes from only two​​​‌ sparse views, see Figure​ 27. By integrating​‌ part-level latent diffusion with​​ skeleton-based conditioning and symmetry-aware​​​‌ dual-volume packing, our approach​ preserves pose-consistent geometry, enables​‌ part-aware editing, and maintains​​ bilateral alignment. A teacher–student​​​‌ strategy with multi-view consistency​ further improves stability and​‌ fidelity, yielding significant gains​​ over state-of-the-art baselines. Our​​​‌ contributions:

• A skeleton-conditioned​ diffusion pipeline that injects​‌ explicit structural priors to​​ improve pose-consistent geometry under​​​‌ sparse views.

• A​ dual-volume latent representation, inspired​‌ by bipartite packing, enabling​​ part-aware decoding and preventing​​​‌ fusion of contacting parts.​ It allows to complete​‌ a partial view in​​ final face generation.

•​​​‌ A symmetry-aware objective coupling​ reconstruction accuracy and bilateral​‌ regularization for realistic midline​​ geometry.

• A self​​​‌ supervised teacher–student strategy enhances​ multi-view consistency.

The image​‌ depicts a process for​​ generating a 3D mesh​​​‌ from a 2D input​ image of a person's​‌ face. The process begins​​ with facial landmark detection.​​​‌ These key points are​ used to construct a​‌ 3D skeleton, which is​​ encoded into a clip.​​​‌ Using a dual U-Net-based​ diffusion model with skeleton​‌ conditioning, two 3D single​​ view generations are created.​​​‌ These views are decoded​ into volumes (left and​‌ right) and combined using​​ marching cubes with part​​​‌ mesh to produce the​ final 3D mesh. Loss​‌ functions like mutual contrastive​​ loss and symmetry losses​​​‌ are applied during the​ process to ensure accuracy​‌ and symmetry. (Description generated​​ at January 15th, 2026​​​‌ by Albert AI with​ the model Mistral-Small-3.2-24B)

The​‌ image depicts a process​​ for generating a 3D​​​‌ mesh from a 2D​ input image of a​‌ person's face. The process​​ begins with facial landmark​​​‌ detection. These key points​ are used to construct​‌ a 3D skeleton, which​​ is encoded into a​​​‌ clip. Using a dual​ U-Net-based diffusion model with​‌ skeleton conditioning, two 3D​​ single view generations are​​​‌ created. These views are​ decoded into volumes (left​‌ and right) and combined​​ using marching cubes with​​​‌ part mesh to produce​ the final 3D mesh.​‌ Loss functions like mutual​​ contrastive loss and symmetry​​​‌ losses are applied during​ the process to ensure​‌ accuracy and symmetry. (Description​​ generated at January 15th,​​​‌ 2026 by Albert AI​ with the model Mistral-Small-3.2-24B)​‌

Given​​ two input images (frontal​​​‌ and profile), we detect​ 2D landmarks to form​‌ a facial skeleton. Here​​ landmarks are replaced by​​​‌ facial skeleton to produce​ more realistic generation in​‌ diffusion. A skeleton encoder​​ produces a latent embedding​​​‌ that conditions a dual-UNet​ diffusion backbone via adaptive​‌ normalization. The denoiser outputs​​ two latent volumes (left/right),​​ which are decoded by​​​‌ a 3D VAE into‌ SDF (i.e., static and‌​‌ dynamic factorization) /occupancy grids.​​ Marching Cubes extracts meshes​​​‌ per side; parts remain‌ disjoint via dual-volume packing,‌​‌ see Figure 28.​​ A symmetry loss regularizes​​​‌ left/right consistency. A complete‌ view of architecture is‌​‌ defined in Figure 27​​.

The image displays​​​‌ three 3D models of‌ a human head and‌​‌ shoulders. The first model​​ is entirely red with​​​‌ a hat. The second‌ model is entirely blue‌​‌ with a hat. The​​ third model has the​​​‌ head in red and‌ the shoulders and upper‌​‌ body in blue. All​​ models are set against​​​‌ a grey grid background.‌ (Description generated at January‌​‌ 15th, 2026 by Albert​​ AI with the model​​​‌ Mistral-Small-3.2-24B)

8.26 Turbo Learning:​​ 3D Face Reconstruction with​​​‌ Mesh Re-Projection and Re-Identification‌ Consistency

Participants: Benoit Lagadec‌​‌, Francois Bremond.​​

We introduce Turbo Learning​​​‌, a two-stage iterative‌ refinement framework for 3D‌​‌ face reconstruction inspired by​​ the positive-feedback dynamics of​​​‌ turbocharged engines. Traditional pipelines‌ rely on sparse supervisory‌​‌ cues such as 2D​​ landmarks, limiting their ability​​​‌ to recover accurate geometry.‌ Our approach instead uses‌​‌ self-generated 3D meshes as​​ progressively stronger priors: Stage​​​‌ 1 predicts a coarse‌ mesh guided by MediaPipe‌​‌ landmarks, while Stage 2​​ uses this mesh as​​​‌ dense geometric supervision.

To‌ further enhance identity preservation,‌​‌ we introduce a Mesh​​ Re-Projection and Re-Identification Consistency​​​‌ Loss. By re-projecting‌ meshes from both stages‌​‌ into image space and​​ applying an InfoNCE contrastive​​​‌ Re-ID objective, we enforce‌ identity stability across refinement‌​‌ steps. The combination of​​ a geometric turbo loop​​​‌ and an identity turbo‌ loop produces reconstructions that‌​‌ are more stable, more​​ detailed, and more identity-faithful.​​​‌

We compare Turbo Learning‌ to classical iterative strategies‌​‌ such as EM, diffusion-based​​ refinement, boosting, and teacher–student​​​‌ systems, and show that‌ it occupies a distinctive‌​‌ position among them, see​​ Fig. 29.

The​​​‌ image depicts a two-step‌ process for a machine‌​‌ learning framework aimed at​​ improving 3D mesh reconstruction.​​​‌ In Step 1, the‌ input includes face and‌​‌ profile images, generating depth,​​ mask, and normal outputs.​​​‌ These are processed through‌ a dual uncertainty block‌​‌ and re-projected to calculate​​ re-projection loss and re-ID​​​‌ contrastive loss. Step 2‌ builds on this by‌​‌ refining the 3D mesh​​ guided by the initial​​​‌ output and additional ground‌ truth mesh, further minimizing‌​‌ re-projection loss. The dual​​ uncertainty block plays a​​​‌ central role in both‌ steps, ensuring accurate depth‌​‌ and geometric information. (Description​​ generated at January 15th,​​​‌ 2026 by Albert AI‌ with the model Mistral-Small-3.2-24B)‌​‌

8.27 THEval: Evaluation​​​‌ Framework for Talking Head‌ Video Generation

Participants: Nabyl‌​‌ Quignon, Baptiste Chopin​​, Yaohui Wang,​​​‌ Antitza Dantcheva.

Generative‌ models for talking head‌​‌ videos have witnessed remarkable​​ progress, achieving high-resolution and​​​‌ realistic results. However, evaluating‌ these models remains a‌​‌ significant challenge, as the​​​‌ rapid advancement in generation​ has outpaced the development​‌ of adequate metrics. Current​​ evaluations primarily rely on​​​‌ general image quality metrics​ or lip-synchronization scores, which​‌ often fail to capture​​ essential aspects of realism​​​‌ such as motion quality,​ temporal coherence, and naturalness.​‌ Furthermore, these existing metrics​​ have been shown to​​​‌ correlate poorly with human​ preferences, necessitating a more​‌ robust and perceptually aligned​​ evaluation approach.

Overview of​​​‌ the THEval scheme.

We introduce THEVAL​​​‌ 56, a novel​ evaluation framework designed to​‌ address these limitations by​​ aligning closely with human​​​‌ perception, a visual summary​ of the framework is​‌ available on Figure 30​​. We support this​​​‌ framework with a new,​ challenging evaluation dataset comprising​‌ over 5,000 videos sourced​​ from diverse YouTube channels,​​​‌ ensuring the content was​ unseen during model training​‌ to test generalization. The​​ dataset features a wide​​​‌ range of languages, head​ poses, and expressions. To​‌ assess performance comprehensively, we​​ decompose evaluation into three​​​‌ core dimensions: quality, naturalness,​ and synchronization, utilizing eight​‌ fine-grained metrics to analyze​​ dynamics such as lip​​​‌ and head motion alongside​ global aesthetics.

To validate​‌ our framework, we conduct​​ an extensive benchmark of​​​‌ 17 state-of-the-art audio- and​ video-driven models, generating and​‌ analyzing over 85,000 videos.​​ We leverage a user​​​‌ study to demonstrate that​ our final composite score​‌ achieves a strong Spearman​​ correlation of 0.870 with​​​‌ human ratings, significantly outperforming​ traditional metrics like FID​‌ and Syncnet. By applying​​ this pipeline, we identify​​​‌ that while many current​ algorithms excel in lip​‌ synchronization, they continue to​​ face challenges in generating​​​‌ expressive facial behavior and​ artifact-free details, establishing THEVAL​‌ as a vital tool​​ for fostering future progress​​​‌ in the field.

8.28​ Beyond Real versus Fake​‌ Towards Intent-Aware Video Analysis​​

Participants: Saurabh Atreya,​​​‌ Nabyl Quignon, Baptiste​ Chopin, Abhijit Das​‌, Antitza Dantcheva.​​

The rapid advancement of​​​‌ generative models has led​ to increasingly realistic deepfake​‌ videos, posing significant societal​​ and security risks. While​​​‌ existing detection methods focus​ primarily on distinguishing real​‌ from fake videos, such​​ approaches fail to address​​​‌ a fundamental question regarding​ the intent behind manipulated​‌ content. With the proliferation​​ of AI-generated media, the​​​‌ binary distinction of authenticity​ is becoming less relevant​‌ than understanding whether content​​ is malicious or benign.​​​‌ This shift necessitates a​ new paradigm in video​‌ analysis that moves beyond​​ artifact detection to the​​​‌ contextual understanding of underlying​ motivations.

Three-Way Contrastive Alignment​‌ Pipeline

We introduce​​​‌ IntentHQ 53, a‌ new benchmark for human-centered‌​‌ intent analysis designed to​​ formalize the task of​​​‌ intent recognition. We curate‌ a comprehensive dataset of‌​‌ 5,168 videos, meticulously annotated​​ with 23 fine-grained intent​​​‌ categories such as "Financial‌ fraud", "Political propaganda", and‌​‌ "Comedy", organized under five​​ broader dimensions including Deception​​​‌ and Persuasion. To effectively‌ analyze these videos, we‌​‌ propose a novel self-supervised​​ learning framework (see Figure​​​‌ 31) that leverages‌ a three-way contrastive alignment‌​‌ strategy. This method jointly​​ aligns video, audio, and​​​‌ textual modalities, utilizing data‌ augmentation techniques like semantic‌​‌ paraphrasing and text-to-speech synthesis​​ to learn robust representations​​​‌ without relying on manual‌ labels during pretraining.

To‌​‌ validate our approach, we​​ benchmark intent recognition using​​​‌ various state-of-the-art multimodal architectures.‌ Our proposed model, which‌​‌ integrates spatio-temporal video features​​ with audio and text​​​‌ analysis, achieves a classification‌ accuracy of 52.5%, establishing‌​‌ a new state-of-the-art by​​ significantly outperforming standard video​​​‌ classification baselines such as‌ VideoMAE and TimeSFormer. Ablation‌​‌ studies further reveal that,​​ while video remains the​​​‌ most predictive modality, the‌ fusion of text and‌​‌ audio is essential for​​ distinguishing complex, socially embedded​​​‌ intents. By releasing the‌ IntentHQ dataset and code,‌​‌ we aim to foster​​ further research in intent-aware​​​‌ media analysis, shifting the‌ focus towards a more‌​‌ nuanced understanding of digital​​ content.

8.29 AI killed​​​‌ the video star. Audio-driven‌ diffusion model for expressive‌​‌ talking head generation

Participants:​​ Baptiste Chopin, Antitza​​​‌ Dantcheva.

We proposed‌ Dimitra++ 55, a‌​‌ novel framework for audio-driven​​ talking head generation, streamlined​​​‌ to learn lip motion,‌ facial expression, as well‌​‌ as head pose motion.​​ Specifically, we proposed a​​​‌ conditional Motion Diffusion Transformer‌ (cMDT) to model facial‌​‌ motion sequences, employing a​​ 3D representation. The cMDT​​​‌ is conditioned on two‌ inputs: a reference facial‌​‌ image, which determines appearance,​​ as well as an​​​‌ audio sequence, which drives‌ the motion. Quantitative and‌​‌ qualitative experiments, as well​​ as a user study​​​‌ on two widely employed‌ datasets, i.e., VoxCeleb2 and‌​‌ CelebV-HQ, suggested that Dimitra++​​ is able to outperform​​​‌ existing approaches in generating‌ realistic talking heads imparting‌​‌ lip motion, facial expression,​​ and head pose. Code​​​‌ and qualitative results are‌ provided on our project‌​‌ page: Project Page.​​

8.30 LIA-X: Interpretable Latent​​​‌ Portrait Animator

Participants: Antitza‌ Dantcheva, François Brémond‌​‌.

We introduce LIA-X​​ 57, a novel​​​‌ interpretable portrait animator designed‌ to transfer facial dynamics‌​‌ from a driving video​​ to a source portrait​​​‌ with fine-grained control. LIA-X‌ is an autoencoder that‌​‌ models motion transfer as​​ a linear navigation of​​​‌ motion codes in latent‌ space. Crucially, it incorporates‌​‌ a novel Sparse Motion​​ Dictionary that enables the​​​‌ model to disentangle facial‌ dynamics into interpretable factors.‌​‌ Deviating from previous 'warp-render'​​ approaches, the interpretability of​​​‌ the Sparse Motion Dictionary‌ allows LIA-X to support‌​‌ a highly controllable 'edit-warp-render'​​ strategy, enabling precise manipulation​​​‌ of fine-grained facial semantics‌ in the source portrait.‌​‌ This helps to narrow​​​‌ initial differences with the​ driving video in terms​‌ of pose and expression.​​ Moreover, we demonstrate the​​​‌ scalability of LIA-X by​ successfully training a large-scale​‌ model with approximately 1​​ billion parameters on extensive​​​‌ datasets. Experimental results show​ that our proposed method​‌ outperforms previous approaches in​​ both self-reenactment and cross-reenactment​​​‌ tasks across several benchmarks.​ Additionally, the interpretable and​‌ controllable nature of LIA-X​​ supports practical applications such​​​‌ as fine-grained, user-guided image​ and video editing, as​‌ well as 3D-aware portrait​​ video manipulation. Project Page​​​‌

8.31 Simplicity-Bias-Aware Adaptation of​ Foundation Models for Deepfake​‌ Detection

Participants: Charbel Yahchouchi​​, Noemi Roggero,​​​‌ Laurent Saroul, Antitza​ Dantcheva.

Given the​‌ rapid advancement of deep​​ learning and generative models,​​​‌ the synthesis of realistic​ and plausible images and​‌ videos has reached unprecedented​​ levels. However, this accessibility​​​‌ also raises serious concerns,​ as such content can​‌ be misused for malicious​​ purposes such as identity​​​‌ impersonation, misinformation, and social​ manipulation. Consequently, deepfake detection​‌ has emerged as a​​ crucial area of research,​​​‌ aiming to develop robust​ and generalizable detectors capable​‌ of reliably identifying manipulated​​ media. Despite impressive progress,​​​‌ most current detectors struggle​ to generalize to unseen​‌ manipulation, limiting their real-world​​ reliability.

The image depicts​​​‌ a deep learning model​ designed for detecting fake​‌ or real images. The​​ process starts by converting​​​‌ an input image into​ patch embeddings. These embeddings​‌ go through a series​​ of Transformer and Adapter​​​‌ layers within the CLIP​ ViT backbone. The output​‌ is then split into​​ two paths: one leading​​​‌ to an auxiliary classification​ head (SiFeR Head) and​‌ the other to the​​ main classification head. Both​​​‌ heads use linear classifiers​ to determine if the​‌ image is fake or​​ real. The auxiliary head​​​‌ calculates a specific loss​ function that includes both​‌ auxiliary and forget loss,​​ while the main head​​​‌ calculates the main loss.​ The model aims to​‌ improve the detection of​​ manipulated images by using​​​‌ these dual classification heads.​ (Description generated at January​‌ 14th, 2026 by Albert​​ AI with the model​​​‌ Mistral-Small-3.2-24B)

In​​ this work, we study​​​‌ the limitations of foundation​ model adaptation for deepfake​‌ detection under distribution shifts​​ and address the impact​​​‌ of shortcut learning induced​ by parameter-efficient fine-tuning for​‌ deepfakes. We introduce a​​ simplicity-bias-aware adaptation framework, see​​​‌ Fig. 32, that​ augments a frozen CLIP​‌ visual encoder with lightweight​​ adapter modules and integrates​​​‌ the SIFER feature-sieving mechanism​ to identify and suppress​‌ simple but non-generalizable cues​​ during training. To validate​​​‌ our framework, we conduct​ an extensive evaluation on​‌ recent state-of-the-art deepfake detection​​ datasets, focusing on cross-dataset​​​‌ and cross-manipulation generalization under​ distribution shifts. Experimental results​‌ show consistent improvements in​​ video-level Area Under the​​​‌ Curve (AUC) compared to​ CLIP-based baselines and other​‌ parameter-efficient adaptation strategies, with​​ particularly strong gains on​​ subtle and localized manipulations.​​​‌

8.32 Now You See‌ Me, Now You Don't:‌​‌ A Unified Framework for​​ Expression Consistent Anonymization in​​​‌ Talking Head Videos

Participants:‌ Anil Egin, Antitza‌​‌ Dantcheva.

Face video​​ anonymization is aimed at​​​‌ privacy preservation while allowing‌ for the analysis of‌​‌ videos in a number​​ of computer vision downstream​​​‌ tasks such as expression‌ recognition, people tracking, and‌​‌ action recognition. We propose​​ here a novel unified​​​‌ framework 39 referred to‌ as Anon-NET, streamlined to‌​‌ de-identify facial videos, while​​ preserving age, gender, race,​​​‌ pose, and expression of‌ the original video. Specifically,‌​‌ we inpaint faces by​​ a diffusion-based generative model​​​‌ guided by high-level attribute‌ recognition and motion-aware expression‌​‌ transfer. We then animate​​ deidentified faces by video-driven​​​‌ animation, which accepts the‌ de-identified face and the‌​‌ original video as input.​​ Extensive experiments on the​​​‌ datasets VoxCeleb2, CelebV-HQ, and‌ HDTF, which include diverse‌​‌ facial dynamics, demonstrate the​​ effectiveness of AnonNET in​​​‌ obfuscating identity while retaining‌ visual realism and temporal‌​‌ consistency. Project Page

8.33​​ Beyond the visible: A​​​‌ survey on cross-spectral face‌ recognition

Participants: Antitza Dantcheva‌​‌.

Cross-spectral face recognition​​ (CFR) refers to recognizing​​​‌ individuals using face images‌ stemming from different spectral‌​‌ bands, such as infrared​​ versus visible. While CFR​​​‌ is inherently more challenging‌ than classical face recognition‌​‌ due to significant variation​​ in facial appearance caused​​​‌ by the modality gap,‌ it is useful in‌​‌ many scenarios including night-vision​​ biometrics and detecting presentation​​​‌ attacks. Recent advances in‌ deep neural networks (DNNs)‌​‌ have resulted in significant​​ improvement in the performance​​​‌ of CFR systems. Given‌ these developments, the contributions‌​‌ of this survey are​​ three-fold. First, we provide​​​‌ an overview of CFR,‌ by formalizing the CFR‌​‌ problem and presenting related​​ applications. Secondly, we discuss​​​‌ the appropriate spectral bands‌ for face recognition and‌​‌ discuss recent CFR methods,​​ placing emphasis on deep​​​‌ neural networks. In particular,‌ we describe techniques that‌​‌ have been proposed to​​ extract and compare heterogeneous​​​‌ features emerging from different‌ spectral bands. We also‌​‌ discuss the datasets that​​ have been used for​​​‌ evaluating CFR methods. Finally,‌ we discuss the challenges‌​‌ and future lines of​​ research on this topic.​​​‌

This work has been‌ published in Neurocomputing 31‌​‌.

9.1 Bilateral contracts​​ with industry

10.1 International research​‌ visitors

10.2 European initiatives​​​‌

10.3​ National initiatives

10.4 Regional initiatives

Since​‌ 2011, we have initiated​​ a strategic partnership (called​​​‌ CobTek) with Nice hospital​ (CHU Nice, Prof F.​‌ Askenazy) to start ambitious​​ research activities dedicated to​​​‌ healthcare monitoring and assistive​ technologies. These new studies​‌ address the analysis of​​ more complex spatiotemporal activities​​​‌ (e.g. complex interactions, long​ term activities).

11.1 Promoting scientific activities​​

11.2 Teaching - Supervision‌ - Juries - Educational‌​‌ and pedagogical outreach

Participant:​​​‌ François Brémond.

Francois​ Bremond held AI courses​‌ on Computer Vision $\&$ Deep Learning for the​​​‌ Data Science and AI​ - MSc program at​‌ Université Côte d'Azur: Teaching​​ Website. Academic year​​​‌ 2025: 24 hours.

11.3 Popularization

12.1 Major‌​‌ publications

• 1 inproceedingsD.​​Dhruv Agarwal, T.​​​‌Tanay Agrawal, L.‌Laura Ferrari and F.‌​‌ F.Francois F Bremond​​. From Multimodal to​​​‌ Unimodal Attention in Transformers‌ using Knowledge Distillation.‌​‌AVSS 2021 - 17th​​ IEEE International Conference on​​​‌ Advanced Video and Signal-based‌ SurveillanceVirtual, United States‌​‌November 2021HALDOI​​back to textback​​​‌ to text
• 2 inproceedings‌T.Tanay Agrawal,‌​‌ D.Dhruv Agarwal,​​ M.Michal Balazia,​​​‌ N.Neelabh Sinha and‌ F. F.Francois F‌​‌ Bremond. Multimodal Personality​​ Recognition using Cross-Attention Transformer​​​‌ and Behaviour Encoding.‌VISAPP '22: International Conference‌​‌ on Computer Vision Theory​​ and Applicationsvirtual, United​​​‌ StatesIEEE; SCITEPRESS -‌ Science and Technology Publications‌​‌February 2022, 501-508​​HALDOIback to​​​‌ textback to text‌
• 3 inproceedingsT.Tanay‌​‌ Agrawal, M.Michal​​ Balazia, P.Philipp​​​‌ Müller and F. F.‌Francois F Bremond.‌​‌ Multimodal Vision Transformers with​​ Forced Attention for Behavior​​​‌ Analysis.WACV '23:‌ IEEE International Winter Conference‌​‌ on Applications in Computer​​ VisionWaikoloa, United States​​​‌January 2023HALDOI‌back to text
• 4‌​‌ inproceedingsA.Abid Ali​​, F. F.Farhood​​​‌ F Negin, F.‌ F.Francois F Bremond‌​‌ and S.Susanne Thümmler​​. Video-based Behavior Understanding​​​‌ of Children for Objective‌ Diagnosis of Autism.‌​‌VISAPP 2022 - 17th​​ International Conference on Computer​​​‌ Vision Theory and Applications‌Online, FranceFebruary 2022‌​‌HALback to text​​back to text
• 5​​​‌ inproceedings M.Mahmoud Ali‌, D.Di Yang‌​‌, A.Arkaprava Sinha​​, D.Dominick Reilly​​​‌, S.Srijan Das‌, G.Gianpiero Francesca‌​‌ and F.Francois Bremond​​. Quo Vadis, Video​​​‌ Understanding with Vision-Language Foundation‌ Models? NeurIPS Proceedings Neurips‌​‌ 2024 - 38th Annual​​ Conference on Neural Information​​​‌ Processing Systems Vancouver (Canada),‌ Canada December 2024 HAL‌​‌
• 6 inproceedingsM.Michal​​ Balazia, P.Philipp​​​‌ Müller, Á. L.‌Ákos Levente Tánczos,‌​‌ A. V.August Von​​ Liechtenstein and F.François​​​‌ Brémond. Bodily Behaviors‌ in Social Interaction: Novel‌​‌ Annotations and State-of-the-Art Evaluation​​.MM'22: The 30th​​​‌ ACM International Conference on‌ MultimediaLisbon, PortugalACM;‌​‌ ACMOctober 2022,​​ 70-79HALDOIback​​​‌ to text
• 7 inproceedings‌H.Hava Chaptoukaev,‌​‌ V.Valeriya Strizhkova,​​​‌ M.Michele Panariello,​ B.Bianca D’alpaos,​‌ A.Aglind Reka,​​ V.Valeria Manera,​​​‌ S.Susanne Thümmler,​ E.Esma Ismailova,​‌ N.Nicholas Evans,​​ F. F.Francois F​​​‌ Bremond, M.Massimiliano​ Todisco, M. A.​‌Maria A. Zuluaga and​​ L. M.Laura M​​​‌ Ferrari. StressID: a​ Multimodal Dataset for Stress​‌ Identification.NeurIPS 2023​​ - 37th Conference on​​​‌ Neural Information Processing Systems​New Orleans, United States​‌December 2023HALback​​ to text
• 8 article​​​‌S.Siyuan Chen,​ Y.Youngjun Cho,​‌ K.Kun Yu,​​ L.Laura Ferrari and​​​‌ F. F.Francois F​ Bremond. Editorial: Recognizing​‌ the state of emotion,​​ cognition and action from​​​‌ physiological and behavioral signals​.Frontiers in Computer​‌ Science4August 2022​​HALDOIback to​​​‌ text
• 9 inproceedingsH.​Hao Chen, Y.​‌Yaohui Wang, B.​​Benoit Lagadec, A.​​​‌Antitza Dantcheva and F.​ F.Francois F Bremond​‌. Joint Generative and​​ Contrastive Learning for Unsupervised​​​‌ Person Re-identification.CVPR​ 2021 - IEEE Conference​‌ on Computer Vision and​​ Pattern RecognitionVirtual, United​​​‌ StatesJune 2021HAL​
• 10 articleH.Hao​‌ Chen, Y.Yaohui​​ Wang, B.Benoit​​​‌ Lagadec, A.Antitza​ Dantcheva and F.Francois​‌ Bremond. Learning Invariance​​ from Generated Variance for​​​‌ Unsupervised Person Re-identification.​IEEE Transactions on Pattern​‌ Analysis and Machine Intelligence​​December 2022, 1-15​​​‌In press. HALDOI​
• 11 articleC. F.​‌Carlos F Crispim-Junior,​​ V.Vincent Buso,​​​‌ K.Konstantinos Avgerinakis,​ G.Georgios Meditskos,​‌ A.Alexia Briassouli,​​ J.Jenny Benois-Pineau,​​​‌ Y.Yiannis Kompatsiaris and​ F.Francois Bremond.​‌ Semantic Event Fusion of​​ Different Visual Modality Concepts​​​‌ for Activity Recognition.​IEEE Transactions on Pattern​‌ Analysis and Machine Intelligence​​382016, 1598​​​‌ - 1611HALDOI​
• 12 inproceedingsR.Rui​‌ Dai, S.Srijan​​ Das and F. F.​​​‌Francois F Bremond.​ CTRN: Class Temporal Relational​‌ Network For Action Detection​​.BMVC 2021 -​​​‌ The British Machine Vision​ ConferenceVirtual, United Kingdom​‌November 2021HAL
• 13​​ inproceedingsR.Rui Dai​​​‌, S.Srijan Das​ and F. F.Francois​‌ F Bremond. Learning​​ an Augmented RGB Representation​​​‌ with Cross-Modal Knowledge Distillation​ for Action Detection.​‌ICCV 2021 - IEEE/CVF​​ International Conference on Computer​​​‌ VisionMontreal, CanadaOctober​ 2021HALback to​‌ textback to text​​
• 14 inproceedingsR.Rui​​​‌ Dai, S.Srijan​ Das, K.Kumara​‌ Kahatapitiya, M.Michael​​ Ryoo and F. F.​​​‌Francois F Bremond.​ MS-TCT: Multi-Scale Temporal ConvTransformer​‌ for Action Detection.​​CVPR - Conference on​​​‌ Computer Vision and Pattern​ RecognitionNew Orleans, United​‌ StatesJune 2022HAL​​
• 15 inproceedingsR.Rui​​​‌ Dai, S.Srijan​ Das, L.Luca​‌ Minciullo, L.Lorenzo​​ Garattoni, G.Gianpiero​​​‌ Francesca and F. F.​Francois F Bremond.​‌ PDAN: Pyramid Dilated Attention​​ Network for Action Detection​​​‌.WACV 2021 -​ Winter Conference on Applications​‌ of Computer Vision 2021​​Waikoloa / Virtual, United​​ StatesJanuary 2021HAL​​​‌back to text
• 16‌ inproceedingsR.Rui Dai‌​‌, S.Srijan Das​​, M.Michael Ryoo​​​‌ and F.Francois Bremond‌. AAN : Attributes-Aware‌​‌ Network for Temporal Action​​ Detection.BMVC 2023​​​‌ - The 34th British‌ Machine Vision ConferenceAberdeen,‌​‌ United KingdomNovember 2023​​HAL
• 17 articleA.​​​‌Antitza Dantcheva and F.‌François Brémond. Gender‌​‌ estimation based on smile-dynamics​​.IEEE Transactions on​​​‌ Information Forensics and Security‌2016, 11HAL‌​‌DOI
• 18 inproceedingsS.​​Srijan Das, R.​​​‌Rui Dai, M.‌Michal Koperski, L.‌​‌Luca Minciullo, L.​​Lorenzo Garattoni, F.​​​‌François Bremond and G.‌Gianpiero Francesca. Toyota‌​‌ Smarthome: Real-World Activities of​​ Daily Living.ICCV​​​‌ 2019 -17th International Conference‌ on Computer VisionSeoul,‌​‌ South KoreaOctober 2019​​HALback to text​​​‌
• 19 articleS.Srijan‌ Das, R.Rui‌​‌ Dai, D.Di​​ Yang and F. F.​​​‌Francois F Bremond.‌ VPN++: Rethinking Video-Pose embeddings‌​‌ for understanding Activities of​​ Daily Living.IEEE​​​‌ Transactions on Pattern Analysis‌ and Machine IntelligenceDecember‌​‌ 2021HALDOIback​​ to textback to​​​‌ text
• 20 inproceedingsS.‌Srijan Das, S.‌​‌Saurav Sharma, R.​​Rui Dai, F.​​​‌ F.Francois F Bremond‌ and M.Monique Thonnat‌​‌. VPN: Learning Video-Pose​​ Embedding for Activities of​​​‌ Daily Living.ECCV‌ 2020 - 16th European‌​‌ Conference on Computer Vision​​Glasgow (Virtual), United Kingdom​​​‌August 2020HAL
• 21‌ inproceedingsL. M.Laura‌​‌ M Ferrari, G.​​Guy Abi Hanna,​​​‌ P.Paolo Volpe,‌ E.Esma Ismailova,‌​‌ F. F.Francois F​​ Bremond and M. A.​​​‌Maria A. Zuluaga.‌ One-class autoencoder approach for‌​‌ optimal electrode set-up identification​​ in wearable EEG event​​​‌ monitoring.EMBC 2021‌ - 43rd Annual International‌​‌ Conference of the IEEE​​ Engineering in Medicine and​​​‌ Biology SocietyVirtuel, France‌October 2021HALDOI‌​‌back to text
• 22​​ inproceedingsJ.-C.Jen-Cheng Hou​​​‌, A.Aileen Mcgonigal‌, F.Fabrice Bartolomei‌​‌ and M.Monique Thonnat​​. A Self-Supervised Pre-Training​​​‌ Framework for Vision-Based Seizure‌ Classification.2022 IEEE‌​‌ International Conference on Acoustics,​​ Speech, and Signal Processing​​​‌ proceedingsIEEE ICASSP 2022‌ : IEEE International Conference‌​‌ on Acoustics, Speech and​​ Signal ProcessingSingapore, Singapore​​​‌May 2022HALDOI‌
• 23 articleI.Indu‌​‌ Joshi, M.Marcel​​ Grimmer, C.Christian​​​‌ Rathgeb, C.Christoph‌ Busch, F.Francois‌​‌ Bremond and A.Antitza​​ Dantcheva. Synthetic Data​​​‌ in Human Analysis: A‌ Survey.IEEE Transactions‌​‌ on Pattern Analysis and​​ Machine Intelligence2024,​​​‌ 1-20HALDOIback‌ to text
• 24 article‌​‌A.Alexandra König,​​ E.Elisa Mallick,​​​‌ J.Johannes Tröger,‌ N.Nicklas Linz,‌​‌ R.Radia Zeghari,​​ V.Valeria Manera and​​​‌ P.Philippe Robert.‌ Measuring neuropsychiatric symptoms in‌​‌ patients with early cognitive​​ decline using speech analysis​​​‌.European Psychiatry64‌12021, e64‌​‌HALDOIback to​​ text
• 25 articleA.​​​‌Alexandra König, P.‌Philipp Müller, J.‌​‌Johannes Tröger, H.​​​‌Hali Lindsay, J.​Jan Alexandersson, J.​‌Jonas Hinze, M.​​Matthias Riemenschneider, D.​​​‌Danilo Postin, E.​Eric Ettore, A.​‌Amandine Lecomte, M.​​Michel Musiol, M.​​​‌Maxime Amblard, F.​François Bremond, M.​‌Michal Balazia and R.​​Rene Hurlemann. Multimodal​​​‌ phenotyping of psychiatric disorders​ from social interaction: Protocol​‌ of a clinical multicenter​​ prospective study.Personalized​​​‌ Medicine in Psychiatry33-34​May 2022, 100094​‌HALDOIback to​​ textback to text​​​‌back to textback​ to text
• 26 inproceedings​‌N.Neelabh Sinha,​​ M.Michal Balazia and​​​‌ F. F.Francois F​ Bremond. FLAME: Facial​‌ Landmark Heatmap Activated Multimodal​​ Gaze Estimation.AVSS​​​‌ 2021 - 17th IEEE​ International Conference on Advanced​‌ Video and Signal-based Surveillance​​Virtual, United StatesNovember​​​‌ 2021HALDOIback​ to text
• 27 inproceedings​‌V.Valeriya Strizhkova,​​ Y.Yaohui Wang,​​​‌ D.David Anghelone,​ D.Di Yang,​‌ A.Antitza Dantcheva and​​ F.François Brémond.​​​‌ Emotion Editing in Head​ Reenactment Videos using Latent​‌ Space Manipulation.2021​​ 16th IEEE International Conference​​​‌ on Automatic Face and​ Gesture Recognition (FG 2021)​‌FG 2021 - IEEE​​ International Conference on Automatic​​​‌ Face and Gesture Recognition​Jodhpur, IndiaDecember 2021​‌HALDOIback to​​ text
• 28 inproceedingsY.​​​‌Yaohui Wang, P.​Piotr Bilinski, F.​‌ F.Francois F Bremond​​ and A.Antitza Dantcheva​​​‌. G3AN: Disentangling Appearance​ and Motion for Video​‌ Generation.CVPR 2020​​ - IEEE Conference on​​​‌ Computer Vision and Pattern​ RecognitionSeattle / Virtual,​‌ United StatesJune 2020​​HAL
• 29 inproceedingsD.​​​‌Di Yang, Y.​Yaohui Wang, Q.​‌Quan Kong, A.​​Antitza Dantcheva, L.​​​‌Lorenzo Garattoni, G.​Gianpiero Francesca and F.​‌ F.Francois F Bremond​​. Self-Supervised Video Representation​​​‌ Learning via Latent Time​ Navigation.Technical Tracks​‌ 3AAAI 2023 -​​ AAAI Conference on Artificial​​​‌ Intelligence37Proceedings of​ the 37th AAAI Conference​‌ on Artificial Intelligence3​​Washigton, D.C., United States​​​‌June 2023HALDOI​
• 30 articleR.Radia​‌ Zeghari, R.Rachid​​ Guerchouche, M.Minh​​​‌ Tran-Duc, F.François​ Bremond, K.Kai​‌ Langel, I.Inez​​ Ramakers, N.Nathalie​​​‌ Amiel, M. P.​Maria Pascale Lemoine,​‌ V.Vincent Bultingaire,​​ V.Valeria Manera,​​​‌ P.Philippe Robert and​ A.Alexandra König.​‌ Feasibility Study of an​​ Internet-Based Platform for Tele-Neuropsychological​​​‌ Assessment of Elderly in​ Remote Areas.Diagnostics​‌12April 2022HAL​​DOIback to text​​​‌

12.2 Publications of the​ year