3.1.1 Axis 1.1 — Integrating domain knowledge

State-of-the-art methods for speech and audio processing are based on discriminative neural networks trained for the targeted task. This paradigm faces major limitations: lack of interpretability, large data requirements and inability to generalize to unseen classes or tasks. Our approach is to combine the representation power of deep learning with our acoustic expertise to obtain smaller generative models describing the probability distribution of speech and audio signals. Particular attention will be paid to designing physically-motivated input layers, output layers, and unsupervised representations that capture complex-valued, multi-scale spectro-temporal dependencies. Given these models, we derive computationally efficient inference algorithms that address the above limitations. We also explore the integration of deep learning with symbolic reasoning and common-sense knowledge to increase the generalization ability of deep models.

3.1.2 Axis 1.2 — Learning from little/no labeled data

While supervised learning from fully labeled data is economically costly, unlabeled data are inexpensive but provide intrinsically less information. Our goal is to learn representations that disentangle the attributes of speech by equipping the unsupervised representation learning methods above with supervised branches exploiting the available labels and supervisory signals, and with multiple adversarial branches overcoming the usual limitations of adversarial.

3.1.3 Axis 1.3 — Preserving privacy

To preserve privacy, speech must be transformed to hide the users' identity and other privacy-sensitive attributes (e.g., accent, health status) while leaving intact those attributes which are required for the task (e.g., phonetic content for automatic speech recognition) and preserving the data variability for training purposes. We develop strong attacks to evaluate privacy. We also seek to hide personal identifiers and privacy-sensitive attributes in the linguistic content, focusing on their robust extraction and replacement from speech signals.

3.1.4 Axis 1.4 — Reducing computational footprint

This axis includes proposing reliable methods to quantify fine-grained energy consumption, computational footprint (in terms of operations), and memory footprint, so as to identify potential bottlenecks in the network at training and test time before applying compression methods.

3.2.1 Axis 2.1 — Linguistic speech content

Speech recognition is the main means to extract linguistic information from speech. Although it is a mature research area, performance drops in real-world environments pursue the development of speech enhancement and source separation methods to effectively improve robustness in such real-world scenarios. Semantic content analysis is required to interpret the spoken message. The challenges include learning from little real data, quickly adapting to new topics, and robustness to speech recognition errors. The detection and classification of hate speech in social media videos will also be considered as a benchmark, thereby extending the work on text-only detection. Finally, we also consider extracting phonetic and prosodic information to study the categorization of speech sounds and certain aspects of prosody by learners of a foreign language.

3.2.2 Axis 2.2 — Speaker identity and states

Speaker identity is required for the personalization of human-computer interaction. Speaker recognition and diarization are still challenging in real-world conditions. The speaker states that we aim to recognize include emotion and stress, which can be used to adapt the interaction in real time.

3.2.3 Axis 2.3 — Speech environment information

We develop audio event detection methods that exploit both strongly/weakly labeled and unlabeled data, operate in real-world conditions, can discover new events, and provide a semantic interpretation. Modeling the temporal, spatial and logical structure of ambient sound scenes over a long duration is also considered.

3.3.1 Axis 3.1 - Multimodality modeling and analysis

The study of multimodality concerns the interaction between modalities, their fusion, coordination and synchronization for a single speaker, as well as their synchronization across the speakers in a conversation. We focus on audiovisual speech enhancement to improve the intelligibility and quality of noisy speech by considering the speaker’s lip movements. We also consider the semi/weakly/self-supervised learning methods for multimodal data to obtain interpretable representations that disentangle in each modality the attributes related to linguistic and semantic content, emotion, reaction, etc. We also study the contribution of each modality to the intelligibility of spoken communication.

3.3.2 Axis 3.2 - Multimodal speech generation

Multimodal speech generation refers to articulatory, acoustic, and audiovisual speech synthesis techniques which output one or more facets. Articulatory speech synthesis relies on 2D and 3D modeling of the dynamics of the vocal tract from real-time MRI (rtMRI) data. We consider the generation of the full vocal tract, from the vocal folds to the lips, first in 2D then in 3D. This comprises the generation of the face and the prediction of the glottis opening. We also consider audiovisual speech synthesis. Both the animation of the lower part of the face related to speech and of the upper part related to the facial expressions are considered, and development continues towards a multilingual talking head. We investigate further the modeling of expressivity for both audio-only and audiovisual speech synthesis, for a better control of expressivity, where we consider several disentangled attributes at the same time.

3.3.3 Axis 3.3 — Interaction

Interaction is a new field of research for our project-team that we will approach gradually. We start by studying the multimodal components (prosody, facial expressions, gestures) used during interaction, both by the speaker and by the listener, where the goal is to simultaneously generate speech and gestures by the speaker, and generating regulatory gestures for the listener. We will introduce different dialog bricks progressively: Spoken language understanding, Dialog management, and Natural language generation. Dialog will be considered in a multimodal context (gestures, emotional states of the interlocutor, etc.) and we will break the classical dialog management scheme to dynamically account for the interlocutor's evolution during the speaker's response.

6.1.1 VAC

• Name:
  Virtual Assistant Creator
• Keywords:
  Artificial intelligence, Audio signal processing, Speech processing
• Functional Description:
  VAC is a framework for creating interconnectable components for speech and natural language processing. It also offers a set of standard components for creating virtual assistants such as noise reduction, speech recognition and synthesis, and natural language understanding.
• Release Contributions:
  Middleware for communication between components - Speech processing components - Access to microphone and speakers (via python-sounddevice and libportaudio2) - Denoising (ConvTasNet model) - Activity detection (via webrtcvad) - Speech recognition (K2/Sherpa zipformer model, NeMo ConformerTransducer model) - Speech synthesis (BalacoonTTS, FastPitch and HifiGAN from NeMo) - Natural language processing components - Text completion (via llama-cpp) - Chat (via llama-cpp) - Video processing components - Face detection (via dlib) - Facial landmark detection (via dlib) - Body detection (via opencv2) - Body pose estimation (via opencv2)
• URL:
  https://­gitlab.­inria.­fr/­multispeech/­vac
• Contact:
  Theo Biasutto-Lervat

6.1.2 MRI

• Name:
  Magnetic Resonance Imaging
• Keywords:
  Health, Medical imaging
• Functional Description:
  Magnetic Resonance Imaging (MRI) takes an increasing place in the investigation of speech production because it provides a complete geometrical information of the vocal tract. We thus initiated a cooperation with the IADI laboratory (Imagerie Adaptive Diagnostique et Interventionnelle) at Nancy Hospital, which studies in particular magnetic resonance imaging. This year, we acquired static MRI data for two speakers (approximately 90 blocked articulations corresponding to vowels and consonants followed by a vowel) and we carried out preliminary experiments intended to acquire dynamic data.
• Contact:
  Yves Laprie

6.2.1 Virtual Assistant Creator

Voice assistants and voice interfaces have become a key technology, simplifying the user experience and increasing the accessibility of many applications, and their use will intensify in the coming years. However, this technology is mainly driven by large technology companies (mainly American), raising questions about European digital sovereignty.

To address this problem, we are currently developing an open-source platform for the creation of virtual assistants, aiming to ease the integration of such technologies in any project. This platform will provide the main speech processing and natural language processing bricks that are necessary to build a voice interface, such as denoising, recognition or speech synthesis, and dialog management.

This year, a complete redesign of the platform has occured, due to the recent predominence of LLMs in human-machine interaction, natural language processing, and ultimetaly in voice-assitant technologies. These models and their high computing requirements make a fully embedded vision near impossible in the current state of technologies, hence a full rewrite of VAC as a client/server platform.

6.2.2 Record ECOS

We have developed a simple low-cost platform to record two persons during a face-to-face conversation.

On the hardware side, deploying one acquisition platform requires two webcams with tripods, a microphones stereo set with stands, an audio interface and a low-end laptop on Linux.

On the software side, we have delopped an engine based on ffmpeg to synchronously record from both cameras and the microphone stereo set, while associating the audio canals to their respective videos. The engine has a simple GUI to easily start and stop recording, as well as encrypt the data on the fly. We also developped several post-processing scripts to remove the speech signal residues of speaker A from the recording of speaker B, to generate the transcription for a speaker, and to merge the two individual transcriptions into a dialog transcription

7.1.1 Axis 1.1 — Integrating domain knowledge

7.1.2 Axis 1.2 - Learning from little/no labeled data

7.1.3 Axis 1.3 - Preserving privacy

Speech signals convey a lot of private information. To protect speakers, we pursued our investigation of x-vector based voice anonymization, which relies on splitting the speech signal into the speaker (x-vector), phonetic and pitch features and resynthesizing the signal with a different target x-vector. In particular, we measured and reduced the amount of speaker information carried by phoneme durations 28.

We published a detailed analysis of the results of the 2nd Voice Privacy Challenge which we co-organized in 2022 9. We organized the 3rd Voice Privacy Challenge 41, which allows the use of external datasets and pretrained models. The challenge evaluates utility using ASR and emotion classification models trained on original (unprocessed) data to ensure that linguistic and emotional content remains undistorted. Additionally, it significantly simplifies the evaluation protocol and reduces the running time of the evaluation scripts, which are now primarily in python.

We organized the 1st VoicePrivacy Attacker Challenge 42, which focuses on developing speaker re-identification attacks against three baseline anonymization systems and four anonymization systems developed by the Voice Privacy 2024 Challenge participants. The best attacker systems reduced the equal error rate (EER) by 25–44% relative w.r.t. the semi-informed attack used in the VoicePrivacy 2024 Challenge.

7.1.4 Axis 1.4 — Reducing computational footprint

In order to be able to compare the energy footprint of a system trained over a different site, we have been studying the relationship between the energy consumption of different model architectures on several GPU models based on the model's complexity and number of floating point operations 33. Some of the outcome of this study has been used within the DCASE challenge in order to normalize the energy consumption of the systems submitted by the participants. This allowed to study the evolution of the energy consumption of state-of-the-art sound event systems over the past years and its relation to the performance 26, 38. During her PhD thesis work, Zahra Benslimane studied in detail the computational load in terms of floating point operations of three different multichannel speech enhancement systems. She then proposed an analysis of the relationship between algorithm latency, computational footprint and performance in terms of speech enhancement. The end goal is to identify the bottleneck of the current system in order to convert them to low latency and low complexity speech enhancement algorithms targeting implementation in hearing aids.

7.2.1 Axis 2.1 — Linguistic speech content

7.2.2 Axis 2.2 — Speaker identity and states

7.2.3 Axis 2.3 — Speech in its environment

7.3.1 Axis 3.1 — Multimodality modeling and analysis

7.3.2 Axis 3.2 — Multimodal speech generation

7.3.3 Axis 3.3 — Interaction

8.1.1 Vivoka

• Company: Vivoka (France)
• Duration: Oct 2021 – Apr 2024
• Participants: Can Cui, Mostafa Sadeghi, Emmanuel Vincent
• Abstract: This contract funds the PhD of Can Cui on joint and embedded automatic speech separation, diarization and recognition for the generation of meeting minutes.

8.1.2 Meta AI

• Company: Meta AI (France)
• Duration: May 2022 – Apr 2025
• Participants: Robin San Roman, Romain Serizel
• Abstract: This CIFRE grant funds the PhD of Robin San Roman on self-supervised disentangled representation learning of audio data for compression and generation.

8.1.3 Orange Labs

• Company: Orange Labs (France)
• Duration: March 2023 – Feb 2026
• Participants: Taous Iatariene, Romain Serizel
• Abstract: This CIFRE grant funds the PhD of Taous Iatariene on sound source tracking.

9.1.1 Participation in other International Programs

9.2.1 Visits of international scientists

9.3.1 H2020 projects

ANR ROBOVOX

• Title:
  Robust Vocal Identification for Mobile Security Robots
• Duration:
  Mar 2019 – Apr 2024
• Coordinator:
  Laboratoire d'informatique d'Avignon (LIA)
• Partners:
  Inria (Nancy), LIA (Avignon), A.I. Mergence (Paris)
• Participants:
  Antoine Deleforge, Sandipana Dowerah, Denis Jouvet, Romain Serizel
• Abstract:
  The aim of ROBOVOX project is to improve speaker recognition robustness for a security robot in real environment. Several aspects will be particularly considered such as ambient noise, reverberation and short speech utterances.

ANR BENEPHIDIRE

• Title:
  Stuttering: Neurology, Phonetics, Computer Science for Diagnosis and Rehabilitation
• Duration:
  Mar 2019 - Dec 2024
• Coordinator:
  Praxiling (Montpellier)
• Partners:
  Praxiling (Montpellier), LORIA (Nancy), INM (Montpellier), LiLPa (Strasbourg).
• Participants:
  Yves Laprie, Slim Ouni, Shakeel Ahmad Sheikh
• Abstract:
  The BENEPHIDIRE project brings together neurologists, speech-language pathologists, phoneticians, and computer scientists specializing in speech processing to investigate stuttering as a speech impairment and to develop techniques for diagnosis and rehabilitation.

ANR JCJC DENISE

• Title:
  Tackling hard problems in audio using Data-Efficient Non-linear InverSe mEthods
• Duration:
  Oct 2020 – Sep 2024
• Coordinator:
  Antoine Deleforge
• Participants:
  Antoine Deleforge, Paul Magron, Tom Sprunck
• Collaborators:
  UMR AE, Institut de Recherche Mathématiques Avancées de Strasbourg, Institut de Mathématiques de Bordeaux
• Abstract:
  DENISE aims to explore the applicability of recent breakthroughs in the field of nonlinear inverse problems to audio signal reparation and to room acoustics, and to combine them with compact machine learning models to yield data-efficient techniques.

ANR Full3DTalkingHead

• Title:
  Synthèse articulatoire phonétique
• Duration:
  Apr 2021 - Mar 2025
• Coordinator:
  Yves Laprie
• Partners:
  Gipsa-Lab (Grenoble), IADI (Nancy), LPP (Paris)
• Participants:
  Slim Ouni, Vinicius Ribeiro, Yves Laprie
• Abstract:
  The objective is to realize a complete three-dimensional digital talking head including the vocal tract from the vocal folds to the lips and the face, and integrating the digital simulation of the aero-acoustic phenomena.

ANR Lorraine Artificicial Intelligence – LOR-AI LOR-AI

• Title:
  Lorraine Artificicial Intelligence Cofinancement de thèses en IA
• Duration:
  Sep 2020- Dec 2025
• Coordinator:
  Yves Laprie
• Partners:
  CNRS, Inria, Regional University Hospital Centre (CHRU)
• Participants:
  Doctoral school of Université de Lorraine
• Abstract:
  This project about Artificial Intelligence, led by the Université de Lorraine (UL), has a double objective by providing 12 co-fundings for doctoral theses: on the one hand, to strengthen UL areas of excellence in AI and domains tightly connected to IA, i.e. particularly Health, and on the other hand, to open other research areas to AI with the objective of leading to scientific breakthroughs.

ANR REFINED

• Title:
  Real-Time Artificial Intelligence for Hearing Aids
• Duration:
  Mar 2022 - Mar 2026
• Coordinator:
  CEA List (Saclay)
• Partners:
  CEA List (Saclay), Institut de l'audition (Paris), LORIA (Nancy)
• Participants:
  Paul Magron, Nasser-Eddine Monir, Romain Serizel
• Abstract:
  The Refined project brings together audiologists, computer scientists and specialists about hardware implementation to design new speech enhancement algorithms that both fit the needs of patients suffering of hearing losses and the computational constraints of hearing aid devices.

ANR ReNAR

• Title:
  Reducing Noise with Augmented Reality
• Duration:
  Feb 2024 - Jan 2028
• Coordinator:
  CEA List (Saclay)
• Partners:
  Ircam (Paris), Laboratoire des Sciences du Numérique de Nantes (Nantes), LORIA (Nancy)
• Participants:
  Romain Serizel, Aine Drelingyte
• Abstract:
  The aim of the ReNAR project is to design a solution that can attenaute the impact of noise in office working scenarios (in particular in open spaces). We will target two aspects: generating noise maskers that results in sound scenes that are pleasent to hear for workers and generating signals that can obfuscate surrounding speech.

ANR LLM4all

• Title:
  Large Language Models for All
• Duration:
  Oct 2023 - Mars 2027
• Coordinator:
  Synalp Loria (Nancy)
• Partners:
  LORIA-Synalp, LORIA-Multispeech, LIX, Linagora, Ap-HP, HuggingFace
• Participants:
  Irina Illina, Emmanuel Vincent
• Abstract:
  Large Language Models (LLM) of sufficient size exhibit outstanding emergent abilities, such as learning from their input context and decomposing a complex problem into a chain of simpler steps. The LLM4all project will thus focus on such large models, or on models at the same level of generic performances, and will propose methods to solve two related fundamental issues: how to update these LLMs automatically, and how to reduce their computing requirements in order to facilitate their deployment.

PEPR Cybersécurité, projet iPOP

• Title:
  Protection des données personnelles
• Duration:
  Oct 2022 – Sep 2028
• Coordinator:
  Vincent Roca (Inria PRIVATICS)
• Partners:
  Inria PRIVATICS (Lyon), COMETE, PETRUS (Saclay), MAGNET, SPIRALS (Lille), IRISA (Rennes), LIFO (Bourges), DCS (Nantes), CESICE (Grenoble), EDHEC (Lille), CNIL (Paris)
• Participant:
  Emmanuel Vincent
• Summary:
  The objectives of iPOP are to study the threats on privacy introduced by new digital technologies, and to design privacy-preserving solutions compatible with French and European regulations. Within this scope, Multispeech focuses on speech data.

DGA DEEP MAUVES

• Title:
  Deep automatic aircraft speech recognition for non native speakers
• Duration:
  Dec 2022 – Dec 2026
• Coordinator:
  Irina Illina
• Participant:
  Irina Illina, Raphaël Bagat, Emmanuel Vincent
• Summary:
  This project proposes methods and tools that increase the usability of ASR systems for non-native speakers in noisy conditions in the aeronautical domain.

Défi Inria COLaF

• Title:
  Corpus et Outils pour les Langues de France
• Duration:
  Aug 2023 – Jul 2027
• Coordinator:
  Slim Ouni and Benoît Sagot (Inria ALMANACH)
• Partners:
  Inria ALMANACH (Paris)
• Participant:
  Slim Ouni, Sam Bigeard, Vincent Colotte, Emmanuel Vincent
• Summary:
  This project aims to increase the inclusiveness of speech technologies by releasing open data, models and software for accented French and for regional, overseas and non-territorial languages of France.

LANGU:IA

• Title:
  Pôle de référence transversal et pluridisciplinaire du Traitement automatique du Français et des Langues de France
• Duration:
  Jan 2024 - Sep 2024
• Coordinator:
  Communauté de communes de Retz-en-Valois
• Partners:
  Communauté de communes de Retz-en-Valois, APIL, ATALA, CMN – Cité internationale de la langue française, Ministère de la Culture – Délégation générale à la langue française et aux langues de France, ELDA, LISN (Saclay), Le VoiceLab
• Participants:
  Emmanuel Vincent
• Abstract:
  LANGU:IA aims to create a national cluster for language technologies that will bring together all stakeholders (industrial, academic, institutional) in order to develop technologies and innovation for the automatic processing of French and the languages of France.

ANR SPEECHPRIVACY

• Title:
  Multiple-attribute disentanglement and semantic privacy
• Duration:
  Feb 2024 - Jan 2028
• Coordinator:
  Vincent Colotte
• Partners:
  LORIA (Nancy), EURECOM (Sophia Antipolis), LIA (Avignon)
• Participants:
  Vincent Colotte, Emmanuel Vincent
• Abstract:
  SpeechPrivacy will deliver a flexible solution to privacy preservation based on isolated/disentangled representations and the selective obfuscation/modification of individual attributes beyond the usual voice identity/sex and sensitive keywords.

ANSES IPIAMA

• Title:
  Reducing Noise with Augmented Reality
• Duration:
  Dec 2023 - Dec 2026
• Coordinator:
  Jean-Pierre Arz, INRS (Nancy)
• Partners:
  INRS (Nancy), Laboratoire Énergies et Mécanique Théorique et Appliquée (Nancy), LORIA (Nancy)
• Participants:
  Romain Serizel
• Abstract:
  The IPIAMA project aims to propose binaural speech intelligibility measurements (with both ears) for people equipped with hearing aids. The project will rely jointly on classic listening tests (reliable but expensive) and models based on data collected in realistic conditions.

10.1.1 Scientific events: organisation

10.1.2 Scientific events: selection

10.1.3 Invited talks

• Keynote "Observer l'humain pour générer un avatar multimodal", WACAI 2024, Bordeaux, Jun 2024 (S. Ouni)
• Invited speaker at Computational Analysis and Simulation of the Human Voice Dagstuhl Seminar, Jun 2024 (Y. Laprie)

10.1.4 Leadership within the scientific community

• Vice-president of AFCP - Association Francophone de la Communication Parlée (S. Ouni)
• Secretary/Treasurer, executive member of AVISA (Auditory-VIsual Speech Association), an ISCA Special Interest Group (S. Ouni)
• Member of the Steering Committee of ISCA's Special Interest Group on Security and Privacy in Speech Communication (E. Vincent)
• Board member of Le VoiceLab, the association of French voice tech players (E. Vincent)
• Chair of the Steering Committe of DCASE (R. Serizel)

10.1.5 Scientific expertise

• Expert for the Data Governance Working Group of the Global Partnership on AI (GPAI) (E. Vincent)
• Expertise for the Czech Science Foundation (I. Illina)
• Expertise for the CIFRE (I. Illina)
• Expertise for the « Collaborative Research in Computational Neuroscience » (CRCNS), NSF-ANR (R. Serizel)

10.1.6 Research administration

• Coordinator of the AI Cluster ENACT (European Centre for Artificial Intelligence through Innovation, E. Vincent)
• Head of Science of the Inria Research Center at Université de Lorraine (E. Vincent, until Feb)
• Head of pole scientifique Automatique, Mathématiques, Informatique et leurs interactions (AM2I) (Y. Laprie)
• Member of the selection committee for the position of assistant professor at Université de Strasbourg (S. Ouni)
• Member of the selection committee for the position of assistant professor at Université de Paris-Saclay (S. Ouni)
• Member of the selection committee for the position of assistant professor at IUT Lannion (I. Illina)
• Member of the evaluation committee of Haut Conseil de l’évaluation de la recherche et de l’enseignement supérieur (HCERES) for STIH (I. Illina)
• Member of the RIPEC jury, UL (I. Illina)
• Inria representative on the Lorraine Steering Committee for Open Science (E. Vincent)
• Vice-chair of the hiring committee for Junior Research Scientists, Inria Research Center at Université de Lorraine (E. Vincent)
• Member of Comité Espace Transfert, Inria Research Center at Université de Lorraine (E. Vincent)
• Member of Inria's Evaluation Committee (E. Vincent, until Feb)
• Member of Conseil de Laboratoie du LORIA (V. Colotte)
• Member of the bureau du pole scientifique Automatique, Mathématiques, Informatique et leurs interactions (AM2I) (I. Illina)
• Member of the Comité du pole scientifique Automatique, Mathématiques, Informatique et leurs interactions (AM2I) (I. Illina)
• Member of the board, Université de Lorraine (Y. Laprie)
• Member of the Comité Utilisateurs des Moyens de Calculs, Inria Research Center at Université de Lorraine (T. Biasutto–Lervat)
• Referent Plateformes-Outils, Inria Research Center at Université de Lorraine (T. Biasutto–Lervat)

10.2.1 Teaching

• BUT: I. Illina, Java programming (100 hours), Linux programming (58 hours), and Advanced Java programming (40 hours), Data Structures (50h), L1, Université de Lorraine, France
• BUT: I. Illina, Supervision of student projects and internships (50 hours), L2, Université de Lorraine, France
• BUT: R. Serizel, Introduction to office tools (108 hours), Multimedia and web (20 hours), Documents and databases (20 hours), L1, Université de Lorraine, France
• BUT: R. Serizel, Multimedia content and indexing (14 hours), Content indexing and retrieval software (20 hours), L2, Université de Lorraine, France
• BUT: S. Ouni, Programming in Java (24 hours), Web Programming (24 hours), Graphical User Interface (96 hours), L1, Université de Lorraine, France
• BUT: S. Ouni, Advanced Algorithms (24 hours), L2, Université de Lorraine, France
• Licence: Y. Laprie, Phonetics (16 hours), L2, École d’audioprothèse, Université de Lorraine, France
• Licence: V. Colotte, Digital literacy and tools (hybrid courses, 50 hours), L1, Université de Lorraine, France
• Licence: V. Colotte, System (80 hours), L2-L3, Université de Lorraine, France
• Licence: V. Colotte, Introduction to speech processing (20 hours), L3, Université de Lorraine, France
• Master: V. Colotte, Integration project: multimodal interaction with Pepper Robot (17 hours), M2, Université de Lorraine, France
• Master: V. Colotte, Multimodal oral communication (24 hours), M2, Université de Lorraine, France
• Master: V. Colotte, AI introduction (6 hours), M2 - intellectual property rights, Université de Lorraine, France
• Master: V. Colotte, Introduction to speech processing (24 hours), M1, Université de Lorraine, France
• Master: Y. Laprie, Speech corpora (30 hours), M1, Université de Lorraine, France
• Master: E. Vincent, P. Magron, and N. Tomashenko, Speech recognition and synthesis (20 hours), M2, Université de Lorraine
• Master: T. Biasutto-Lervat, Deep learning (labs, 16 hours), M2, Université de Lorraine
• Master: E. Vincent and P. Magron, Neural networks (32 hours + 11 hours), M2, Université de Lorraine
• Other: V. Colotte, Co-Responsible for NUMOC (Digital literacy by hybrid courses) for Université de Lorraine, France (for 7,000 students)
• Other: S. Ouni, Co-Responsible of parcours Ingénierie Logiciel, BUT, Université de Lorraine, France

10.2.2 Supervision

• PhD: Can Cui, “Joint speech separation, diarization and recognition for automatic meeting transcription”, Oct 2024, M. Sadeghi and E. Vincent 36.
• PhD: Sewade Olaolu Ogun, “Generating diverse synthetic data for ASR training data augmentation”, Oct 2024, V. Colotte and E. Vincent 37.
• PhD in progress: Louis Abel, "Expressive audio-visual speech synthesis in an interaction context", Oct 2021, V. Colotte and S. Ouni.
• PhD in progress: Mickaëlla Grondin, "Modeling gestures and speech in interactions", Nov 2021, S. Ouni and F. Hirsch (Praxiling).
• PhD in progress: Raphaël Bagat, “Automatic speech recognition for non-native speakers in a noisy environment”, Oct 2023, I. Illina and E. Vincent.
• PhD in progress: Yaya Sy "Efficient Continued pre-training of Large Language Models", Oct 2023, I. Illina and C. Cerisara.
• PhD in progress: Jean-Eudes Ayilo "Audio-visual Speech Enhancement: Bridging the Gap between Supervised and Unsupervised Approaches", Oct 2023, M. Sadeghi and R. Serizel.
• PhD in progress: Mohamed Imed Eddine Ghebriout, “LLM adaptation and exploitation for medical emergency call triage”, Apr 2024, G. Guibon (LIPN) and E. Vincent.
• PhD in progress: Orane Dufour, “Towards a comprehensive speech anonymization framework”, Oct 2024, M. Rouvier (LIA) and E. Vincent.
• PhD in progress: Guilhem Faure, "End-to-end speech-to-sign language generation", Oct 2024, M. Sadeghi and S. Ouni.

10.2.3 Juries

• Participation in the HDR jury of Lina Rojas (Université de Lorraine, Jun 2024), S. Ouni, examiner
• Participation in the PhD jury of Théo Mariotte (University of Le Mans, Jan 2024), E. Vincent, reviewer
• Participation in the PhD jury of Samir Sadok (CentraleSupélec Rennes, Mar 2024), S. Ouni, reviewer
• Participation in the PhD jury of Salah Zaiem (Télécom ParisTech, Mar 2024), E. Vincent, chair
• Participation in the PhD jury of Francisco Teixeira (University of Lisbon, Jul 2024), E. Vincent, reviewer
• Participation in the PhD jury of Sanjana Sankar (Université Grenoble-Alpes, Sep 2024), S. Ouni, reviewer
• Participation in the PhD jury of Hamza Bayd (CentraleSupélec Rennes, Dec 2024), S. Ouni, chair
• Participation in the PhD jury of Thomas Lebrun (INSA de Lyon, Dec 2024), E. Vincent, reviewer
• Participation in the PhD jury of Léa-Marie Lam-Yee-Mui (Université Paris Saclay, Nov 2024), I. Illina, reviewer
• Participation in the PhD jury of Etienne Labbé (Université Paul Sabatier, Feb 2024), R. Serizel, reviewer
• Participation in the PhD jury of Philippe Gonzales (Technical University of Denmark, Apr 2024), R. Serizel, reviewer
• Participation in the PhD jury of Kevin Wilkinghoff (Universität Bonn, May 2024), R. Serizel, reviewer

10.3.1 Participation in Live events

• Panel discussion at Viva Technology in Paris, Jun 2024 (S. Ouni)
• Panel discussion at the VoiceLab event on Ethics of AI systems for speech and language processing and generation, Jul 2024 (E. Vincent)
• Talk on automatic lipsync technology at Rencontres Animation Développement Innovation (RADI) in Angoulême, Nov 2024 (S. Ouni)

International journals

• 6 articleF.François Effa, J.-P.Jean-Pierre Arz, R.Romain Serizel and N.Nicolas Grimault. Evaluating and predicting the audibility of acoustic alarms in the workplace using experimental methods and deep learning.Applied Acoustics219March 2024, 109955HALDOI
• 7 articleS.Simon Leglaive, M.Matthieu Fraticelli, H.Hend ElGhazaly, L.Léonie Borne, M.Mostafa Sadeghi, S.Scott Wisdom, M.Manuel Pariente, J. R.John R. Hershey, D.Daniel Pressnitzer and J. P.Jon P. Barker. Objective and subjective evaluation of speech enhancement methods in the UDASE task of the 7th CHiME challenge.Computer Speech and Language89January 2025HALDOI
• 8 articleN.-E. E.Nasser-Eddine Eddine Monir, P.Paul Magron and R.Romain Serizel. A Phoneme-Scale Assessment of Multichannel Speech Enhancement Algorithms.Trends in Hearing28December 2024HALDOIback to text
• 9 articleM.Michele Panariello, N.Natalia Tomashenko, X.Xin Wang, X.Xiaoxiao Miao, P.Pierre Champion, H.Hubert Nourtel, M.Massimiliano Todisco, N.Nicholas Evans, E.Emmanuel Vincent and J.Junichi Yamagishi. The VoicePrivacy 2022 Challenge: Progress and perspectives in voice anonymisation.IEEE/ACM Transactions on Audio, Speech and Language Processing2024. In press. HALDOIback to text
• 10 articleA.Agnès Piquard-Kipffer, K.Karen Martinelli, L.Léa Dussere, A.Anne Sancier, J.Jérémy Zytnicki, C.Caroline Barbot-Bouzit and S.Slim Ouni. AVI-Corse : méthodologie et enjeux d'un projet participatif. Des avatars numériques au service du langage et de la communication.: AVI-Corse: methodology and challenges of a participatory project. Digital avatars, new tools for language and communication needs..La Nouvelle revue – Éducation et société inclusives982024, 237-250HALback to text
• 11 articleV.Vinicius Ribeiro, K.Karyna Isaieva, J.Justine Leclere, J.Jacques Felblinger, P.-A.Pierre-André Vuissoz and Y.Yves Laprie. Automatic segmentation of vocal tract articulators in real-time magnetic resonance imaging.Computer Methods and Programs in Biomedicine2432January 2024, 107907In press. HALDOI
• 12 articleM.Mostafa Sadeghi, X.Xavier Alameda-Pineda and R.Radu Horaud. Unsupervised Performance Analysis of 3D Face Alignment with a Statistically Robust Confidence Test.Neurocomputing564January 2024, 1-16HALDOIback to text
• 13 articleI. A.Imran Ahamad Sheikh, E.Emmanuel Vincent and I.Irina Illina. Training RNN Language Models on Uncertain ASR Hypotheses in Limited Data Scenarios.Computer Speech and Language83January 2024, 101555HALDOIback to text

International peer-reviewed conferences

• 14 inproceedingsL.Louis Abel, V.Vincent Colotte and S.Slim Ouni. Towards interpretable co-speech gestures synthesis using STARGATE.International Conference on Multimodal Interaction (ICMI Companion ’24: GENEA Workshop)San José, Costa RicaNovember 2024HALDOIback to text
• 15 inproceedingsL.Louis Abel, V.Vincent Colotte and S.Slim Ouni. Towards realtime co-speech gestures synthesis using STARGATE.25th Interspeech Conference (INTERSPEECH 2024)Kos Island, GreeceSeptember 2024HALback to text
• 16 inproceedingsJ.-E.Jean-Eudes Ayilo, M.Mostafa Sadeghi and R.Romain Serizel. Diffusion-based speech enhancement with a weighted generative-supervised learning loss.International Conference on Acoustics Speech and Signal Processing (ICASSP)Seoul (Korea), South KoreaApril 2024HALDOIback to text
• 17 inproceedingsC.Can Cui, I. A.Imran Ahamad Sheikh, M.Mostafa Sadeghi and E.Emmanuel Vincent. Improving Speaker Assignment in Speaker-Attributed ASR for Real Meeting Applications.The Speaker and Language Recognition Workshop Odyssey 2024Quebec, CanadaJune 2024HALback to text
• 18 inproceedingsA.Ali Golmakani, M.Mostafa Sadeghi, X.Xavier Alameda-Pineda and R.Romain Serizel. A weighted-variance variational autoencoder model for speech enhancement.ICASSP 2024 - International Conference on Acoustics Speech and Signal ProcessingSeoul (Korea), South Korea2024, 1-5HAL
• 19 inproceedingsM.Mickaëlla Grondin-Verdon, D.Domitille Caillat and S.Slim Ouni. Qualitative study of gesture annotation corpus : Challenges and perspectives.ICMI Companion '24: Companion Proceedings of the 26th International Conference on Multimodal InteractionSan Jose, Costa RicaACM2024, 147-155HALDOIback to text
• 20 inproceedingsL.Ladislav Mošner, R.Romain Serizel, L.Lukáš Burget, O.Oldřich Plchot, E.Emmanuel Vincent, J.Junyi Peng and J.Jan Černocký. Multi-channel extension of pre-trained models for speaker verification.InterspeechKos, GreeceSeptember 2024HALback to text
• 21 inproceedingsI.Ilyass Moummad, N.Nicolas Farrugia, R.Romain Serizel, J.Jeremy Froidevaux and V.Vincent Lostanlen. Mixture of Mixups for Multi-label Classification of Rare Anuran Sounds.EUSIPCO 2024Lyon, FranceAugust 2024HAL
• 22 inproceedingsB.Berné Nortier, M.Mostafa Sadeghi and R.Romain Serizel. Unsupervised speech enhancement with diffusion-based generative models.International Conference on Acoustics Speech and Signal Processing (ICASSP)Seoul (Korea), South KoreaApril 2024HALDOIback to text
• 23 inproceedingsS.Sewade Ogun, A. T.Abraham T. Owodunni, T.Tobi Olatunji, E.Eniola Alese, B.Babatunde Oladimeji, T.Tejumade Afonja, K.Kayode Olaleye, N. A.Naome A. Etori and T.Tosin Adewumi. 1000 African Voices: Advancing inclusive multi-speaker multi-accent speech synthesis.Interspeech 2024Kos Island, GreeceSeptember 2024HALback to text
• 24 inproceedingsC.Cennet Oguz, P.Pascal Denis, S.Simon Ostermann, N.Natalia Skachkova, E.Emmanuel Vincent and J.Josef van Genabith. MMAR: Multilingual and multimodal anaphora resolution in instructional videos.Findings of the 2024 Conference on Empirical Methods in Natural Language ProcessingMiami, United StatesNovember 2024HALback to text
• 25 inproceedingsR. S.Robin San Roman, P.Pierre Fernandez, H.Hady Elsahar, A.Alexandre Défossez, T.Teddy Furon and T.Tuan Tran. Proactive Detection of Voice Cloning with Localized Watermarking.Proceedings of the 41st International Conference on Machine LearningICML 2024 - 41st International Conference on Machine Learning235Vienna, AustriaJuly 2024, 1-17HAL
• 26 inproceedingsF.Francesca Ronchini and R.Romain Serizel. Performance and Energy Balance: A Comprehensive Study of State-of-the-Art Sound Event Detection Systems.ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)Seoul, South KoreaIEEEApril 2024, 1096-1100HALDOIback to textback to text
• 27 inproceedingsM.Mostafa Sadeghi and R.Romain Serizel. Posterior sampling algorithms for unsupervised speech enhancement with recurrent variational autoencoder.International Conference on Acoustics Speech and Signal Processing (ICASSP)Seoul (Korea), South KoreaApril 2024HALDOIback to text
• 28 inproceedingsN.Natalia Tomashenko, E.Emmanuel Vincent and M.Marc Tommasi. Analysis of Speech Temporal Dynamics in the Context of Speaker Verification and Voice Anonymization.2025 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2025)Hyderabad, IndiaApril 2025HALback to text

National peer-reviewed Conferences

• 29 inproceedingsL.Louis Abel, V.Vincent Colotte and S.Slim Ouni. Synthèse de gestes communicatifs via STARGATE.35èmes Journées d'Études sur la Parole (JEP 2024)35èmes Journées d'Études sur la Parole (JEP 2024) 31ème Conférence sur le Traitement Automatique des Langues Naturelles (TALN 2024) 26ème Rencontre des Étudiants Chercheurs en Informatique pour le Traitement Automatique des Langues (RECITAL 2024)Toulouse, FranceATALA & AFPC2024, 181-190HALback to text

Conferences without proceedings

• 30 inproceedingsS.Sam Bigeard, P.Panagiotis Tsolakis, E.Emmanuel Vincent, V.Vincent Colotte, P.Pascale Erhart and S.Slim Ouni. Retour d'expérience : Whisper pour les langues régionales.LIFT 2: Journées scientifiques du GdR Linguistique Informatique, Formelle et de TerrainOrléans, FranceNovember 2024HALback to text
• 31 inproceedings A.Alexis Dehais-Underdown, L.Lise Crevier-Buchman, D.Didier Demolin, P.-A.Pierre-André Vuissoz, M.Marc Fauvel, J.Jacques Felblinger and Y.Yves Laprie. Are glottalic mechanisms in Human Beatboxing really glottalic ? 13th International Seminar of Speech Production Autrans, France May 2024 HAL back to text
• 32 inproceedingsC.Constance Douwes and R.Romain Serizel. From Computation to Consumption: Exploring the Compute-Energy Link for Training and Testing Neural Networks for SED Systems.Detection and Classification of Acoustic Scenes and Events 2024Tokyo, JapanOctober 2024HALback to text
• 33 inproceedingsC.Constance Douwes and R.Romain Serizel. Normalizing Energy Consumption for Hardware-Independent Evaluation.2024 IEEE International Workshop on Machine Learning for Signal ProcessingLondon, United KingdomSeptember 2024HALback to textback to text
• 34 inproceedingsM.Mohammad Mohammadamini, D.Driss Matrouf, M.Michael Rouvier, J.-F.Jean-Francois Bonastre, R.Romain Serizel and T.Theophile Gonos. RoboVox: A Single/Multi-channel Far-field Speaker Recognition Benchmark for a Mobile Robot.LREC_COLINGTurino, ItalyApril 2024HALback to text
• 35 inproceedingsA.Agnès Piquard-Kipffer, A.Ana Krilanovic, J.Jérémy Zytnicki, K.Karen Martinelli, L.Léa Dussere, A.Anne Sancier and S.Slim Ouni. Assessment of avatar lip-reading technology (AVI-Corse project). Perspectives of young people with and without hearing loss.36th WCA 2024, World Congress of AudiologyParis (CNIT - La Défense), FranceSeptember 2024HALback to text

Doctoral dissertations and habilitation theses

• 36 thesisC.Can Cui. Joint speech separation, diarization, and recognition for automatic meeting transcription.Université de LorraineOctober 2024HALback to textback to text
• 37 thesisS.Sewade Ogun. Generating diverse synthetic data for ASR training data augmentation.Université de LorraineOctober 2024HALback to textback to text

Reports & preprints

• 38 miscC.Constance Douwes and R.Romain Serizel. Energy Consumption Trends in Sound Event Detection Systems.September 2024HALback to textback to text
• 39 miscI.Ilyass Moummad, R.Romain Serizel, E.Emmanouil Benetos and N.Nicolas Farrugia. Domain-Invariant Representation Learning of Bird Sounds.September 2024HAL
• 40 miscR. S.Robin San Roman, P.Pierre Fernandez, A.Antoine Deleforge, Y.Yossi Adi and R.Romain Serizel. Latent Watermarking of Audio Generative Models.2024HALDOI
• 41 reportN.Natalia Tomashenko, X.Xiaoxiao Miao, P.Pierre Champion, S.Sarina Meyer, X.Xin Wang, E.Emmanuel Vincent, M.Michele Panariello, N.Nicholas Evans, J.Junichi Yamagishi and M.Massimiliano Todisco. The Voice Privacy 2024 Challenge Evaluation Plan.Inria; Eurecom; NIIMarch 2024HALback to text
• 42 miscN.Natalia Tomashenko, X.Xiaoxiao Miao, E.Emmanuel Vincent and J.Junichi Yamagishi. The First VoicePrivacy Attacker Challenge Evaluation Plan.October 2024HALback to text

Scientific popularization

• 43 inbookG.Guillaume Coiffier, S.Sewade Ogun, L.Leo Valque and P.Priyansh Trivedi. STATE OF THE ART.THINK BEFORE LOADING2024HAL