The goal of the project is the modeling of speech for facilitating oral-based communication. The name MULTISPEECH comes from the following aspects that are particularly considered:
Multisource aspects - which means dealing with speech signals originating from several sources, such as speaker plus noise, or overlapping speech signals resulting from multiple speakers; sounds captured from several microphones are also considered.
Multilingual aspects - which means dealing with speech in a multilingual context, as for example for computer assisted language learning, where the pronunciations of words in a foreign language (i.e., non-native speech) is strongly influenced by the mother tongue.
Multimodal aspects - which means considering simultaneously the various modalities of speech signals, acoustic and visual, in particular for the expressive synthesis of audio-visual speech.
The project is organized along the three following scientific challenges:
The explicit modeling of speech. Speech signals result from the movements of articulators. A good knowledge of their position with respect to sounds is essential to improve, on the one hand, articulatory speech synthesis, and on the other hand, the relevance of the diagnosis and of the associated feedback in computer assisted language learning. Production and perception processes are interrelated, so a better understanding of how humans perceive speech will lead to more relevant diagnoses in language learning as well as pointing out critical parameters for expressive speech synthesis. Also, as the expressivity translates into both visual and acoustic effects that must be considered simultaneously, the multimodal components of expressivity, which are both on the voice and on the face, are addressed to produce expressive multimodal speech.
The statistical modeling of speech. Statistical approaches are common for processing speech and they achieve performance that makes possible their use in actual applications. However, speech recognition systems still have limited capabilities (for example, even if large, the vocabulary is limited) and their performance drops significantly when dealing with degraded speech, such as noisy or reverberated signals, distant microphone recording and spontaneous speech. Source separation and speech enhancement approaches are investigated as a way of making speech recognition systems more robust. Handling new proper names is an example of critical aspect that is tackled, along with the use of statistical models for speech-text automatic alignment and for speech production.
The estimation and the exploitation of uncertainty in speech processing. Speech signals are highly variable and often disturbed with noise or other spurious signals (such as music or undesired extra speech). In addition, the output of speech enhancement and of source separation techniques is not exactly the accurate “clean” original signal, and estimation errors have to be taken into account in further processing. Hence, one goal consists to compute and handle the uncertainty of the reconstructed signal provided by source separation approaches. Finally, MULTISPEECH also aims to estimate the reliability of phonetic segment boundaries and prosodic parameters for which no such information is currently available.
Although being interdependent, each of these three scientific challenges constitutes a founding research direction for the MULTISPEECH project. Consequently, the research program is organized along three research directions, each one matching a scientific challenge. A large part of the research is conducted on French speech data; English and German languages are also considered in speech recognition experiments and language learning. Adaptation to other languages of the machine learning based approaches is possible, depending on the availability of corresponding speech corpora. Most of our research on signal processing, speech recognition, speech synthesis, etc., relies on deep learning approaches.
Speech signals are the consequence of the deformation of the vocal tract under the effect of the movements of the articulators (jaw, lips, tongue, ...) to modulate the excitation signal produced by the vocal cords or air turbulence. These deformations are visible on the face (lips, cheeks, jaw) through the coordination of different orofacial muscles and skin deformation induced by the latter. These deformations may also express different emotions. We should note that human speech expresses more than just phonetic content, to be able to communicate effectively. In this project, we address the different aspects related to speech production from the modeling of the vocal tract up to the production of expressive audiovisual speech. Phonetic contrasts used by the phonological system of any language result from constraints imposed by the nature of the human speech production apparatus. For a given language these contrasts are organized so as to guarantee that human listeners can identify (categorize) sounds robustly. The study of the categorization of sounds and prosody thus provides a complementary view on speech signals by focusing on the discrimination of sounds by humans, particularly in the context of language learning.
Modeling speech production is a major issue in speech sciences. Acoustic simulation makes the link between articulatory and acoustic domains. Unfortunately this link cannot be fully exploited because there is almost always an acoustic mismatch between natural and synthetic speech generated with an articulatory model approximating the vocal tract. However, the respective effects of the geometric approximation, of the fact of neglecting some cavities in the simulation, of the imprecision of some physical constants and of the dimensionality of the acoustic simulation are still unknown. Hence, the first objective is to investigate the origin of the acoustic mismatch by designing more precise articulatory models, developing new methods to acquire tridimensional Magnetic Resonance Imaging (MRI) data of the entire vocal tract together with denoised speech signals, and evaluating several approaches of acoustic simulation. Up to now, acoustic-to-articulatory inversion has been addressed as an instantaneous problem, articulatory gestures being recovered by concatenating local solutions. The second objective is thus to investigate how more elaborated strategies (a syllabus of primitive gestures, articulatory targets…) can be incorporated in the acoustic-to-articulatory inversion algorithms to take into account dynamic aspects.
Speech is considered as a bimodal communication means; the first modality is audio, provided by acoustic speech signals and the second one is visual, provided by the face of the speaker. In our approach, the Acoustic-Visual Text-To-Speech synthesis (AV-TTS) is performed simultaneously with respect to its acoustic and visible components. A first AV-TTS system has been developed resulting in a talking head; the system relied on 3D-visual data and on an extension of our acoustic-unit concatenation text-to-speech synthesis system (SOJA). An important goal is to provide an audiovisual synthesis that is intelligible, both acoustically and visually. Thus, we continue working on adding visible components of the head through a tongue model and a lip model. Another challenging research goal is to add expressivity in the AV-TTS. The expressivity comes through the acoustic signal (prosody aspects) and also through head and eyebrow movements. One objective is to add a prosodic component in the TTS engine in order to take into account some prosodic entities such as emphasis (to highlight some important key words).
Discriminating speech sounds and prosodic patterns is the keystone of language learning whether in the mother tongue or in a second language. This issue is associated with the emergence of phonetic categories, i.e., classes of sounds related to phonemes and prosodic patterns. The study of categorization is concerned not only with acoustic modeling but also with speech perception and phonology. Foreign language learning raises the issue of categorizing phonemes of the second language given the phonetic categories of the mother tongue. Thus, studies on the emergence of new categories, whether in the mother tongue (for people with language deficiencies) or in a second language, must rely upon studies on native and non-native acoustic realizations of speech sounds and prosody, and on perceptual experiments. Concerning prosody, studies are focused on native and non-native realizations of modalities (e.g., question, affirmation, command, ...), as well as non-native realizations of lexical accents and focus (emphasis). For language learning, the aim is to provide automatic feedback to language learners with respect to acquisition of prosody as well as acquisition of a correct pronunciation of the sounds of the foreign language. Concerning the mother tongue we are interested in the monitoring of the process of sound categorization in the long term (mainly at primary school) and its relation with the learning of reading and writing skills, especially for children with language deficiencies.
Whereas the first research direction deals with the physical aspects of speech and its explicit modeling, this second research direction investigates machine learning-based approaches for handling speech data.
Acoustic models are used to represent the pronunciation of the sounds or other acoustic events such as noise. Whether they are used for source separation, for speech recognition, for speech transcription, or for speech synthesis, the achieved performance strongly depends on the accuracy of these models.
At the linguistic level, MULTISPEECH investigates models for handling the context (thus going beyond the few preceding words of the
Acoustic modeling is a key issue for automatic speech recognition. Despite the progress made for many years, speech recognition performance depends on the quality of the input speech signals, and performance degrades quickly with noisy or reverberated signals. Accurate signal enhancement techniques are therefore essential to increase the robustness of both automatic speech recognition and speech-text alignment systems to noise and non-speech events. In MULTISPEECH, focus is set on source separation techniques using multiple microphones and/or models of non-speech events. Some of the challenges include getting the most of the new modeling frameworks based on alpha-stable distributions and on deep neural networks, combining them with established spatial filtering approaches, modeling more complex properties of speech and audio sources (phase, inter-frame and inter-frequency properties), and exploiting large data sets of speech, noise, and acoustic impulse responses to automatically discover new models. Beyond the definition of such models, one difficulty is to design scalable estimation algorithms robust to overfitting, to integrate them into the recently developed FASST and KAM software frameworks if relevant, and to develop new software frameworks otherwise.
We are constantly surrounded by a complex audio stream carrying information about our environment. Hearing is a privileged way to detect and identify events that may require quick action (ambulance siren, baby cries...). Indeed, audition offers several advantages compared to vision: it allows for omnidirectional detection, up to a few tens of meters and independently of the lighting conditions. For these reasons, automatic audio analysis has become increasingly popular over the past few years. Yet, machines are still limited to detecting and classifying a few tens of sound event classes while human can generally recognize a few thousand. Besides, current algorithms rely heavily on the availability of annotated data that are extremely costly to obtain. In MULTISPEECH we focus on developing new methods, independent of applications, that would enable the detection of thousands of audio events from a little amount of annotated data while being robust to “out-of-the lab” conditions.
MULTISPEECH investigates lexical and language models in speech recognition with a focus on improving the processing of proper names and of spontaneous speech. Proper names are relevant keys in information indexing, but are a real problem in transcribing many diachronic spoken documents which refer to data, especially proper names, that evolve over time. This leads to the challenge of dynamically adjusting lexicons and language models through the use of the context of the documents or of some relevant external information. We also investigate language models defined on a continuous space (through neural network based approaches) in order to achieve a better generalization on unseen data, and to model long-term dependencies. We also want to introduce into these models additional relevant information such as linguistic features, semantic relation, topic or user-dependent information. Other topics are spontaneous speech, for which utterances are often ill-formed and frequently contain disfluencies (hesitations, repetitions, ...) that degrade speech recognition performance, and pronunciation lexicons which are critical especially when dealing with non-native speech and foreign names.
Speaker identification is the task that consists in identifying a person based on a voice recording. It has recently been deployed in several real-world application including secured access to bank services via telephone or internet. However, identification based solely on voice remains a modality with limited reliability under real conditions including several acoustic perturbations (noise, reverberation…) when the speaker might not be cooperative (a limited amount of data is available). In MULTISPEECH we focus on exploring new approaches exploiting multichannel speech enhancement techniques and uncertainty propagation to improve the performance of speaker identification systems in real conditions and with short speech utterances.
Over the last few years parametric speech synthesis has emerged as an alternative to corpus-based speech synthesis. The announced advantages of the parametric speech synthesis are the possibility to deal with small amounts of speech resources and the flexibility for adapting models (for new emotions or new speakers). MULTISPEECH investigates parametric approaches (currently based on deep learning) to produce expressive audio-visual speech. Also, in the context of acoustic feedback in foreign language learning, voice modification approaches are studied to modify the learner’s (or teacher’s) voice in order to emphasize the difference between the learner’s acoustic realization and the expected realization.
This axis focuses on the uncertainty associated with some processing steps. Uncertainty stems from the high variability of speech signals and from imperfect models. For example, enhanced speech signals resulting from source separation are not exactly the clean original speech signals. Words or phonemes resulting from automatic speech recognition contain errors, and the phone boundaries resulting from an automatic speech-text alignment are not always correct, especially in acoustically degraded conditions. Hence the goal of estimating the reliability and/or the uncertainty on the results.
Because small distortions in the separated source signals can translate into large distortions in the cepstral features used for speech recognition, this limits the recognition performance on noisy data. One way to address this issue is to estimate the uncertainty of the separated sources in the form of their posterior distribution and to propagate this distribution, instead of a point estimate, through the subsequent feature extraction and speech decoding stages. MULTISPEECH seeks to provide more accurate estimates of the posterior distribution of the separated source signals accounting for, e.g., posterior correlations over time and frequency which have not been considered so far. Mappings learned on training data and fusion of multiple uncertainty estimators are also explored. The estimated uncertainties are then exploited for acoustic modeling in speech recognition and, in the future, also for speech-text alignment. This approach may later be extended to the estimation of the resulting uncertainty of the acoustic model parameters and of the acoustic scores themselves.
The accuracy of the phonetic segmentation is important in several cases, as for example for the computation of prosodic features, for avoiding incorrect feedback to the learner in computer assisted foreign language learning, or for the post-synchronization of speech with face/lip images. Currently the phonetic boundaries obtained are quite correct on good quality speech, but the precision degrades significantly on noisy and non-native speech. Phonetic segmentation aspects need to be investigated, both in speech recognition (i.e., spoken text unknown) and in forced alignment (i.e., when the spoken text is known). In the same way that combining several speech recognition outputs leads to improved speech recognition performance, MULTISPEECH plans to investigate the combination of several speech-text alignments as a way of improving the quality of speech-text alignment and of getting information on the reliability of the phonetic boundaries. When segmenting speech corpora, knowing the reliability of the boundaries will help deciding which parts of the corpora need to be manually checked and corrected, thus avoiding an exhaustive checking of the whole corpus.
Prosody information is also investigated as a means for structuring speech data (determining sentence boundaries, punctuation…) possibly in addition to syntactic dependencies. Structuring automatic transcription output is important for further exploitation of the transcription results such as easier reading after the addition of punctuation, or exploitation of full sentences in automatic translation. Prosody information is also necessary for determining the modality of the utterance (question or not), as well as determining accented words. Prosody information comes from the fundamental frequency, the duration of the sounds and their energy. Any error in estimating these parameters may lead to a wrong decision. MULTISPEECH aims at estimating the uncertainty of the duration of the phones (see uncertainty of phonetic boundaries above) and of the fundamental frequency estimates, as well as how this uncertainty shall be propagated in the detection of prosodic phenomena such as accented words, utterance modality, or determination of the structure of the utterance.
Approaches and models developed in the MULTISPEECH project are intended to be used for facilitating oral communication in various situations through enhancements of the communication channels, either directly via automatic speech recognition or speech production technologies, or indirectly, thanks to computer assisted language learning. Applications also include the usage of speech technologies for helping people in handicapped situations or for improving their autonomy. Foreseen application domains are related to computer assisted learning, health and autonomy (more precisely aided communication and monitoring), annotation and processing of spoken documents, and multimodal computer interaction.
Although speaking seems quite natural, learning foreign languages, or learning the mother tongue for people with language deficiencies, represents critical cognitive stages. Hence, many scientific activities have been devoted to these issues either from a production or a perception point of view. The general guiding principle with respect to computer assisted mother or foreign language learning is to combine modalities or to augment speech to make learning easier. Based upon a comparison of the learner’s production to a reference, automatic diagnoses of the learner’s production can be considered, as well as perceptual feedback relying on an automatic transformation of the learner’s voice. The diagnosis step strongly relies on the studies on categorization of sounds and prosody in the mother tongue and in the second language. Furthermore, making a reliable diagnosis on each individual utterance is still a challenge, which requires a temporally accurate segmentation of the speech utterance into phones; this explains why accurate segmentation of speech is an important topic in the field of acoustic speech modeling.
A foreseen application aims at improving the autonomy of elderly or disabled people, and fits with smartroom applications. In a first step, source separation techniques should help for locating and monitoring people through the detection of sound events inside apartments. In a longer perspective, adapting speech recognition technologies to the voice of elderly people should also be useful for such applications, but this requires the recording of adequate databases. Sound monitoring in other application fields (security, environmental monitoring) can also be envisaged.
A first type of annotation consists in transcribing a spoken document in order to get the corresponding sequences of words, with possibly some complementary information, such as the structure (punctuation) or the modality (affirmation/question) of the utterances to make the reading and understanding easier. Typical applications of the automatic transcription of radio or TV shows, or of any other spoken document, include making possible their access by deaf people, as well as by text-based indexing tools.
A second type of annotation is related to speech-text alignment, which aims at determining the starting and ending times of the words, and possibly of the sounds (phonemes). This is of interest in several cases such as for annotating speech corpora for linguistic studies, and for synchronizing lip movements with speech sounds (for example, for avatar-based communications). Although good results are currently achieved on clean data, automatic speech-text alignment needs to be improved for properly processing noisy spontaneous speech data and needs to be extended to handle overlapping speech.
Large audio archives are important for some communities of users, e.g., linguists, ethnologists or researchers in digital humanities in general. In France, a notorious example is the “Archives du CNRS — Musée de l’homme”, gathering about 50,000 recordings dating back to the early 1900s. When dealing with very old recordings, the practitioner is often faced with the problem of noise. This stems from the fact that a lot of interesting material from a scientific point of view is very old or has been recorded in very adverse noisy conditions, so that the resulting audio is poor. The work on source separation can lead to the design of semi-automatic denoising and enhancement features, that would allow these researchers to significantly enhance their investigation capabilities, even without expert knowledge in sound engineering.
Finally, there is also a need for speech signal processing techniques in the field of multimedia content creation and rendering. Relevant techniques include speech and music separation, speech equalization, prosody modification, and speaker conversion.
Speech synthesis has tremendous applications in facilitating communication in a human-machine interaction context to make machines more accessible. For example, it started to be widely common to use acoustic speech synthesis in smartphones to make possible the uttering of all the information. This is valuable in particular in the case of handicap, as for blind people. Audiovisual speech synthesis, when used in an application such as a talking head, i.e., virtual 3D animated face synchronized with acoustic speech, is beneficial in particular for hard-of-hearing individuals. This requires an audiovisual synthesis that is intelligible, both acoustically and visually. A talking head could be an intermediate between two persons communicating remotely when their video information is not available, and can also be used in language learning applications as vocabulary tutoring or pronunciation training tool. Expressive acoustic synthesis is of interest for the reading of a story, such as audiobook, to facilitate the access to literature (for instance for blind people or for illiterate people).
E. Vincent has co-edited a 500-page book on audio source separation and speech enhancement, which provides a unifying view of various established and recent methods .
2018 ISCA Award for the best paper published in Computer Speech and Language (2013–2017)
Best paper award of MISSI 2018 (11th International Conference on Multimedia and Network Information Systems)
Multichannel audio source separation with deep neural networks
Keywords: Audio - Source Separation - Deep learning
Scientific Description: dnnsep is the only source separation software relying on multichannel Wiener filtering based on deep learning. Deep neural networks are used to initialize and reestimate the power spectrum of the sources at every iteration of an expectation-maximization (EM) algorithm. This results in state-of-the-art separation quality for both speech and music.
Functional Description: Combines deep neural networks and multichannel signal processing for speech enhancement and separation of musical recordings.
Release Functional Description: This version derives from version 1.0 (not 1.9). Differences concerns the use of a bidirectional long short-term memory (BLSTM) neural network, smoothing of the multichannel Wiener filter (MWF) over time and frequency, usage of the principal component of the MWF filter, adding a new generalized eigenvector beamformer with blind analytical normalization (GEVB) filter, and normalizing the training and test signals.
Participants: Aditya Nugraha, Emmanuel Vincent and Antoine Liutkus
Contact: Emmanuel Vincent
High realistic lip synchronization for 3d animated characters
Keywords: 3D animation - Graphics - Speech Synthesis
Functional Description: Dynalips provides a solution to synchronize precisely and automatically the movements of the lips of a 3D character with speech (we address 3D animation movies and video games). We have developed a demonstrator that illustrates the whole process: from audio + text to the generation of the animation trajectory, and controlling the animation of a 3D model (e.g. an avatar). The demonstrator is composed mainly by the player developed in Unity 3D (but can be used with any other system) and plays the animation synchronously with speech in realtime. It is possible to generate an animation for Autodesk Maya 3D.
News Of The Year: The whole lip-sync demonstrator is fully operational. From text and recorded speech, the system allows animating two different 3D models. The player is running on Unity 3D.
Partners: Université de Lorraine - Sayens (SATT Grand Est)
Contact: Slim Ouni
Kaldi-based Automatic Transcription System
Keyword: Speech recognition
Functional Description: KATS is a multipass system for transcribing audio data, and in particular radio or TV shows in French, English or Arabic. It is based on the Kaldi speech recognition tools. It relies on Deep Neural Network (DNN) modeling for speech detection and acoustic modeling of the phones (speech sounds). Higher order statistical language models and recurrent neural network language models can be used for improving performance through rescoring of multiple hypotheses.
News Of The Year: New models have been trained for British English and evaluated on MGB data
Contact: Dominique Fohr
Visualization of multimodal speech data
Keywords: Data visualization - 3D movement - Speech processing - Videos
Scientific Description: VisArtico is a multimodal data visualization software acquired by several systems: articulograph, motion capture, depth camera. This software makes it possible to visualize the positions of real or virtual sensors and to animate them simultaneously with acoustics. Regarding the articulatory data, the user has the possibility to visualise the contour of the tongue and the lips. It also makes it possible to find the midsagittal plane of the speaker, and to deduce the position of the palate, if this information is absent during the acquisition. The software makes it possible to display the segmentation at the level of sentences, words or phonemes. The goal is to provide an effective multimodal data visualization tool that can be useful to anyone studying speech production, audio-visual synthesis, or animation in a more general way.
Functional Description: VisArtico is a user-friendly software which allows visualizing multimodal data acquired by several systems : an articulograph (AG500, AG501 or NDI Wave), motion capture system, depth camera. This visualization software has been designed so that it can directly use the data provided by the different systems to display the spatial and temporal positions of the sensors (real and virtual), synchronized with the corresponding acoustic recordings. Moreover, for articulatory data, VisArtico not only allows viewing the sensors but also enriches the visual information by indicating clearly and graphically the data for the tongue, lips and jaw. Finally, it is possible to generate a movie for any articulatory-acoustic sequence. This software can be useful for researchers in speech production, audiovisual speech synthesis or articulatory speech analysis.
Release Functional Description: The main improvement in this version is the ability to view a video that was recorded along with the articulatory or motion capture data. The software also allows for automatic speech segmentation.
News Of The Year: This year, we have added the possibility to visualize a video simultaneously with the multimodal data and the acoustic data. Several bugs have been fixed.
Participants: Ilef Ben Farhat, Loïc Mangeonjean and Slim Ouni
Partners: CNRS - Université de Lorraine
Contact: Slim Ouni
Publication: VisArtico: a visualization tool for articulatory data
Keywords: Medical imaging - Natural language processing
Functional Description: The Xarticulators software is intended to delineate contours of speech articulators in X-ray and MR images, construct articulatory models and synthesize speech from X-ray films. This software provides tools to track contours automatically, semi-automatically or by hand, to make the visibility of contours easier, to add anatomical landmarks to speech articulators and to synchronize images with the sound. In addition we also added the possibility of processing digitized manual delineation results made on sheets of papers when no software is available. Xarticulators also enables the construction of adaptable linear articulatory models from the X-ray or MR images and incorporates acoustic simulation tools to synthesize speech signals from the vocal tract shape. Recent work was on the possibility of synthesizing speech from 2D-MRI films, and on the construction of better articulatory models for the velum, lips and epiglottis.
Release Functional Description: The new version allows MRI films to be processed and, above all, it offers a better transition from the shape of the vocal tract to the area function, which corresponds to an approximation of the vocal tract using a series of elementary tubes from the glottis to the lips.
News Of The Year: Improvements to the articulatory model which now takes into account "small" filiform articulators such as the epiglottis and velum using models using the central line. Improvement of the transition from the medio-sagittal 2D form of the vocal tract to the function of area.
Contact: Yves Laprie
Publication: Articulatory model of the epiglottis
Since articulatory modeling, i.e. representing the geometry of the vocal tract with a small number of parameters, is a key issue in articulatory synthesis the improvement of the articulatory models remains an important objective. This year we put emphasis on thin articulators as the epiglottis and velum. Indeed, the delineation of those contours often leads to erroneous transverse dimensions (too thin or too thick contours) which generates some artificial swelling deformations. Before the determination of the deformation modes, the central lines of the velum and epiglottis are extracted in the images use to build the model. The deformation modes thus only concern the central line, which prevents artificial swelling factors to emerge from the factor analysis. A reconstruction algorithm has been developed to obtain the contour from the central line.
One of the issues in articulatory synthesis is to assess the impact of the geometric simplifications that are made on the vocal tract so as to enable faster acoustic simulation and to decrease the number of parameters required to approximate the vocal tract shape. The other issue concerns the impact of the plane wave assumption. The idea consists of comparing the signal or spectrum synthesized via numerical acoustic simulation against the one measured on a real human subject. However, this requires that both geometric and corresponding acoustic data are available at the same time. This can be achieved with MRI data when the acquisition duration is sufficiently short to allow the speaker to phonate the sound during the whole acquisition. The MRI acquisition protocol has thus been optimized on the new Siemens Prisma MRI machine of Nancy hospital so as to reduce the acquisition time to 7 seconds, which makes it possible for the subject to produce a sound throughout the acquisition. The acoustic simulation was achieved by using the Matlab K-wave package, either from the entire 3D volume extracted from the MRI data, or from the 2D shape extracted from the mid-sagittal plane. Several simplifications have been carried out (with or without the epiglottis, with or without the velum…) so as to assess their acoustic impacts. These simulations only concern vowels because these sounds can be sustained by subjects and the MRI machine noise does not change the position of formant frequencies dramatically. This work has been carried out in cooperation with IADI laboratory.
The size of the dynamic database (recorded last year in the Max Planck Institute for Biophysical Chemistry in Göttingen), in the form of MRI films of the mid-sagittal plane acquired at 55 Hz, is about 200.000 images. Even if the long term objective is to exploit the whole database, efforts were dedicated to manual delineation of contours in some films with the idea of using those data to train a machine learning technique. Several students were trained, and in total more than 1000 images have been delineated. The corresponding films have been exploited to achieve articulatory copy synthesis by improved acoustic simulations developed last year.
Deriving articulatory dynamics from the acoustic speech signal is a recurrent topic in our team. This year, we have investigated whether it is possible to predict articulatory dynamics from phonetic information without having the acoustic speech signal. The input data may be considered as not sufficiently rich acoustically, as there is probably no explicit coarticulation information, but we expect that the phonetic sequence provides compact yet rich knowledge. We have experimented a recurrent neural network architecture, where we have trained the model with an electromagnetic articulography (EMA) corpus, and have obtained good performances similar to the state-of-the-art articulatory inversion from line spectral frequencies (LSF) features .
A comparison between emotional speech and neutral speech has been carried on using a small corpus of acted speech. The analysis was focused on the way pronunciations and prosodic parameters are modified in emotional speech, compared to neutral style .
Experiments with deep learning-based approaches for expressive speech synthesis are described in .
This year, we have acquired audiovisual 3D corpus (using the optitrack system, using 8 cameras) for a set of emotions acted by a professional actress. We recorded 6 basic emotions: joy, fear, disgust, sadness, anger, surprise; in addition to neutral speech. The corpus contains 5000 utterances (2000 utterances for the neutral speech and 500 utterances per emotion). The visual and acoustic data have been processed, segmented and labeled spatially and temporally. An important aspect of the work was to study the evaluation of the quality of the animation of a 3D talking head where the animation is generated from the acquired 3D data. For this purpose, we studied the relevance of root mean square error (RMSE) measure which is classically used to evaluate the error of the prediction. Our preliminary results confirmed that RMSE can be irrelevant in our field, as we may not reach critical articulatory target, and we still obtain very low RMSE. Thus the audiovisual intelligibility of the system would be low. To improve the results, we have worked on improving the 3D model controls using better key-shapes and reduced redundant and confusing blendshapes.
The processed neutral-speech data have been used to train a deep neural network to predict from speech and linguistic information the trajectories of the animation controls of the talking head, which is the core of the lipsync system. We have also used this expressive-speech data to train a DNN-based TTS to synthesize expressive audiovisual speech from text. Currently, we are performing extensive testing and validation of the results.
We continue our research focused on the importance of multimodal speech combining oral and visual clues. We investigated identification and production of morpho-syntactic skills in ten deaf children (severe with cochlear implant using French cued-speech LPC - Langue française Parlée Complétée) and ten age-matched children with typical development. Our goal was to examine the production of morpho-syntactic structures in auditory channel versus audiovisual speech. Five conditions were observed: audiovisual conditions with a 3D avatar speaking or coding oral language with LPC versus a human speaker with or without LPC and auditory channel. We used the 3D avatar coding set up in the ADT Handicom project. Statistical analysis and interpretation of results is ongoing.
We set-up standardized norms on the development of reading and related skills in French: EVALEC Primaire software (in collaboration with the LPC - Laboratoire de Psycholologie Cognitive, UMR 7290, Aix-Marseille Université). This year, LPC collected new data at the end of grade 5 (about 100 children) and added them to those previously collected at the end of grades 1–4, about 100 children for each level . EVALEC primaire software includes five tests focused on written word processing, recording both accuracy scores and processing time (time latency and vocal response duration for the reading aloud tests). EVALEC primaire software also includes tests of phonemic and syllabic awareness, phonological short-term memory, and rapid naming. These data would allow researchers and speech therapists to assess the reading and reading-related skills of dyslexic children as compared to average readers.
We have examined the effects of L1/L2 interferences at the segmental level, and of the lack of fluency at the sentence level, on the realizations of French final fricatives by German learners. Due to L1/L2 interference, German speakers tend to devoice French final fricatives. A well-known effect of the lack of L2 mastering is the decrease of the speech articulation rate, which lengthens the average duration of segments. In order to better apprehend the impact of categorization and fluency, we selected four series of consonants from the IFCASL corpus, i.e. voiced and unvoiced fricatives uttered by French native and German non-native speakers. The realizations of French unvoiced consonants uttered by German speakers are essentially dependent on fluency, whereas the realizations of voiced consonants by the same speakers are dependent on both fluency and categorization. We evaluated a set of acoustic cues related to the voicing distinction -including consonant duration and periodicity-, and submitted the data to a hierarchical clustering analysis. Results, discussed as a function of speaker's level and prosodic boundaries, confirmed the mutual importance of fluency and segmental categorization on non-native realizations .
Within the METAL project, work is on-going for integrating speech processing technology in an application to help learning foreign language and for experimenting it with middle and high school students learning German. This includes tutoring aspects using a talking head to show proper articulation of words and sentences; as well as using automatic tools derived from speech recognition technology, for analyzing student pronunciations. Preliminary experiments have shown the poor quality of speech signals recorded from groups of students in classrooms.
Emmanuel Vincent has co-edited a 500-page book on audio source separation and speech enhancement, which provides a unifying view of array processing, matrix factorization, deep learning and other methods, with application to speech and music . We also contributed to five chapters in that book , , , , and three chapters in another book , , .
In multichannel scenarios, source localization and source separation are tightly related tasks. We introduced the real and imaginary parts of the acoustic intensity vector in each time-frequency bin as suitable input features for deep learning based speaker localization . We analyzed the inner working of the neural network using a methodology called layerwise relevance propagation, which points the time-frequency bins on which the network relies to output a given location . We defined a new task called text-informed speaker localization, which consists of localizing the speaker uttering a known word or sentence such as the wake-up word of a hands-free voice command system in a situation when other speakers are overlapping. We proposed a method to address this task, where a phonetic alignment is obtained, converted into an estimated time-frequency mask, and fed to a convolutional neural network together with interchannel phase difference features in order to localize the desired speaker . We published a new dataset using a microphone array embedded in an unmanned aerial vehicle in , organized an international sound source localization challenge associated to this dataset and participated to the 2018 LOCATA sound source localization challenge. We published a book chapter on audio-motor integration, showing an application to sound source localization with robots .
In a given room, each possible position of the microphones and the sources corresponds to different room transfer functions. The goal of room acoustic modeling is to model the manifold formed by these transfer functions. Past studies have focused on learning a supervised mapping between the relative transfer function and the source location for localization purposes. We introduced the reverse task consisting of learning a mapping between the source location and the corresponding relative transfer function, which may be used as a prior on the relative transfer function for source separation purposes. We proposed a semi-supervised algorithm to learn this mapping in a situation when the location of each relative transfer function measurement is not precisely known . We also started investigating the estimation and modeling of early acoustic echoes. In we showed how their knowledge could improve performance of sound source separation algorithms. In we proposed a new method to estimate them blindly from multichannel recordings with much higher precision than conventional blind channel identification methods.
We pursued our research on the use of deep learning for multichannel source separation . We introduced a method that exploits knowledge of the source locations in order to estimate multichannel Wiener filters for two or more sources . We explored several variants of the multichannel Wiener filter, which turned out to result in better speech recognition performance on the CHiME-3 dataset . We also used deep neural networks for reducing the residual nonlinear echo after linear acoustic echo cancellation and started extending this approach to joint reverberation, echo, and noise reduction. Finally, we recently started exploring the case where the microphones composing a multichannel array are not distributed according to a predefined geometry and do not have a common sampling clock.
This year, our work on heavy tails distribution has witnessed a significant advance with the development of a multichannel model that is able to account for the inter-channel delays and time difference of arrivals in an alpha-stable framework, hence benefiting from the inherent robustness of such distributions. This work has been submitted to the IEEE transactions on Signal Processing by Mathieu Fontaine and is still under review. Its main applications are: i/ the separation of multichannel sources, for which we have demonstrated a superiority with respect to the multichannel Wiener filter in the oracle setting, and ii/ localizations of heavy tailed sources, where we worked on the theoretical foundations
The team has investigated a number of alternative probabilistic models to the symmetric local complex Gaussian (LCG) model for audio source separation. An important limit of LCG is that most signals of interest such as speech or music do not exhibit Gaussian distributions but heavier-tailed ones due to their important dynamic. In we proposed a new sound source separation algorithm using heavy-tailed alpha stable priors for source signals. Experiments showed that it outperformed baseline Gaussian-based methods on under-determined speech or music mixtures. Another limitation of LCG is that it implies a zero-mean complex prior on source signals. This induces a bias towards low signal energies, in particular in under-determined settings. With the development of accurate magnitude spectrogram models for audio signals using deep neural networks, it becomes desirable to use probabilistic models enforcing stronger magnitude priors and better accounting for phases. In , we presented the BEADS (Bayesian Expansion Approximating the Donut Shape) model. The prior considered is a mixture of isotropic Gaussians regularly placed on a zero-centered complex circle. We showed it outperformed LCG on an informed source separation task.
Our work on interference reduction focused this year in scaling our previous work to full-length recording. This has been achieved thanks to a new method we proposed, which estimates the interference reduction parameters based on random projections of the full length recordings . This technique scales linearly with the duration of the recording, making it usable in real-world use-cases.
The book chapter we published on audio-motor integration, shows an application to ego-noise reduction for robots . In the context of robotics, ego-noise refers to the acoustic noise produced in a robot's microphones by its own movement.
Achieving robust speech recognition in reverberant, noisy, multi-source conditions requires both speech enhancement and separation and robust acoustic modeling. In order to motivate further work by the community, we created the series of CHiME Speech Separation and Recognition Challenges in 2011 . We oversaw the collection of a new dataset sponsored by Google, which considers a `dinner party' scenario. Twenty parties of four people, who know each other well, were recorded in their own homes using 2 binaural in-ear microphones per participant and 6 distant Kinects, for a total duration of about 50 h. We organized the CHiME-5 Challenge based on these data . We also participated in the collection of two French datasets for ambient assisted living applications as part of the voiceHome and VOCADOM projects.
We are constantly surrounded by sounds and we rely heavily on these sounds to obtain important information about what is happening around us. Our team has been involved in the community on ambient sound recognition for the past few years. In collaboration with Johannes Kepler University (Austria) and Carnegie Mellon University (USA), we co-organized a task on large-scale sound event detection as part of the Detection and Classification of Acoustic Scenes and Events (DCASE) 2018 Challenge . It focused on the problem of learning from audio segments that are either weakly labeled or not labeled, targeting domestic applications. In this context, we work on semi-supervised sampling strategies to create triplets (a triplet is composed of the current sample, a so-called positive sample from the same class as the current sample and a negative sample from a different class) and studied their application to train triplet networks for audio tagging.
Automatic Speech Recognition (ASR) of multimedia content such as videos or multi-genre broadcasting requires a correct extraction of speech segments. We explored the efficiency of deep neural models for speech/non-speech segmentation. We used a bidirectional LSTM model to obtain speech/non-speech probabilities and a decision module (4-state automaton with safety margins). Compared to a Gaussian Mixture Model (GMM) based speech/non-speech segmenter, the results achieved on the MGB British Challenge data, show a reduction of the ASR word error rate (23.7% versus 29.4%). We have also trained models for the Arabic and French languages.
Within the AMIS project, speech recognition systems have been developed for the transcription of videos in French, English and Arabic. They have been integrated with other components (such as translation and summarization) to allow for the summarization of videos in a target language , , .
Speaker recognition is the task of recognizing a person from its voice. The performances of speaker recognition systems severely degrade due to several practical challenges such as the limited amount of speech data, real-world noises and spoofing. We explored the efficiency of DNN-based distance metric learning methods for speaker recognition in short duration conditions. Currently, we are developing a neural network architecture that gives phone-invariant speaker embeddings for robust speaker recognition. We also participated in the NIST speaker recognition evaluation 2018 as a part of the I4U consortium. The speaker recognition technology is vulnerable to spoofing attacks where mimicked voice, synthetic speech, or playback voice is used to get illegitimate access. We are investigating whether technology-assisted speaker selection can help in improving mimicry attack . In , we proposed an enhanced baseline system for replay spoofing detection with ASVspoof 2017 dataset. In , we demonstrated that playback speech enhanced with DNN-based speech enhancement method can severely degrade the speaker recognition and countermeasure performance as compared to the conventional replay attacks with voice samples from covert recording. We also proposed a common feature and back-end fusion scheme for the integration of spoofing countermeasures and speaker recognition . Currently, we are co-organizing the third edition of automatic speaker verification spoofing challenge (ASVspoof 2019) where our newly developed cost function will be adopted for the performance assessment of integrated systems. In the context of multimodal authentication with the voice as a modality, we investigated the optimization of speech features for audio-visual synchrony detection .
With respect to language identification, the current research activity focuses on lightly supervised or unsupervised domain adaptation. The goal is to adapt a language identification system optimized for a given transmission channel to a new transmission channel.
Despite recent progress in developing Large Vocabulary Continuous Speech Recognition Systems (LVCSR), these systems suffer from Out-Of-Vocabulary words (OOV). In many cases, the OOV words are Proper Nouns (PNs). The correct recognition of PNs is essential for broadcast news, audio indexing, etc. We addressed the problem of OOV PN retrieval in the context of broadcast news LVCSR. We focused on dynamic (document dependent) extension of LVCSR lexicon. To retrieve relevant OOV PNs, we proposed to use a very large multipurpose text corpus: Wikipedia. This corpus contains a huge number of PNs. These PNs are grouped in semantically similar classes using word embedding. We used a two-step approach: first, we selected OOV PN pertinent classes with a multi-class Deep Neural Network (DNN). Secondly, we ranked the OOVs of the selected classes. The experiments on French broadcast news show that a bi-directional Gated Recurrent Unit model outperforms other studied models. Speech recognition experiments demonstrate the effectiveness of the proposed methodology .
Within the AMIS project, the update of speech recognition vocabularies has been investigated using web data collected over a time period similar to that of the collected videos, for three languages: French, English and Arabic. Results have been analyzed globally, and also with respect to names only. This analysis has shown the poor coverage of the names by the baseline lexicons, and has also demonstrated the benefits of the updated lexicons, both in term of WER reduction and OOV rate reduction .
Similarly to speech, music involves several levels of information, from the acoustic signal up to cognitive quantities such as composer style or key, through mid-level quantities such as a musical score or a sequence of chords. The dependencies between mid-level and lower- or higher-level information can be represented through acoustic models and language models, respectively. Ken Déguernel defended his PhD on automatic music improvisation and he proposed a polyphonic music improvisation approach that takes the structure of the musical piece at multiple time scales into account . We also explored the ability of a conventional recurrent neural network with moving history to account for long-term dependencies in music melodies, and compared it with two new architectures with growing or parallel history .
Nowadays, Twitter, LinkedIn, Facebook and YouTube are very popular for communicating ideas, beliefs, feelings or any other form of information. At the same time, the dark side of these new technologies has led to an increase in hate speech or racism. Our work seeks to study hate speech in user-generated contents in France, which thus requires French resources. We plan to design a hate speech corpus and a lexicon in French; whereas such hate speech lexicons exist for other languages, no such tool can be found in French. We began, on English data, to develop a new methodology to automatically detect hate speech, based on machine learning and Neural Networks. Human detection of this material is unfeasible since the contents to be analyzed are huge. Current machine learning methods use only certain task specific features to model hate speech. We propose to develop an innovative approach to combine these pieces of information into a multi-feature approach so that the weaknesses of the individual features are compensated by the strengths of other features. We began a collaboration with the CREM laboratory in Metz and Saarland University.
Work on Arabic speech synthesis was carried out within a CMCU PHC project with ENIT (École Nationale d’Ingénieurs de Tunis, Tunisia), using HMM and NN based approaches applied to Modern Standard Arabic language. Speech synthesis systems rely on a description of speech segments corresponding to phonemes, with a large set of features that represent phonetic, phonologic, linguistic and contextual aspects. When applied to Modern Standard Arabic, two specific phenomena have to be taken in account: vowel quantity and gemination. This year, we studied thoroughly the modeling of these phenomena. Results of objective and subjective evaluations showed that the use of a deep neural architecture in speech synthesis (more specifically in predicting the speech parameters) enhanced the accuracy of acoustic modelling so that the quality of generated speech is better than that of HMM-based speech synthesis , .
Deep neural network (DNN) approaches have been further investigated for the modeling of phoneme duration. According to the specific phenomena of the Arabic language, we proposed a class-specific modeling of the phoneme durations. An objective evaluation showed that the proposed approach leads to a more accurate modeling of the phoneme duration (compared to HMM-based or MERLIN DNN-based approaches) .
Expressive speech synthesis using parametric approaches is constrained by the style of the speech corpus used. We carried out a preliminary study on developing expressive speech synthesis for a new speaker voice without requiring a specific recording of expressive speech by this new speaker. For that, we focused on deep neural network based layer adaptation for investigating the transfer the expressive characteristics to a new speaker for which only neutral speech data is available. Such transfer learning mechanism should accelerate the efforts towards exploiting existing expressive speech corpora. However, there is a trade-off between the knowledge transfer of expressivity characteristics and the retaining of the speaker’s identity in the synthesized speech.
In many real-world conditions, the target speech signal overlaps with noise and some distortion remains after speech enhancement. The framework of uncertainty decoding assumes that this distortion has a Gaussian distribution and seeks to estimate its covariance matrix and propagate it through the acoustic model for robust ASR . We introduced new Gaussian mixture model-derived (GMMD) uncertainty features for robust DNN-based acoustic model training and decoding, which are computed as the difference between the closed-form GMM log-likelihoods obtained with vs. without uncertainty. We concatenated the GMMD features with conventional acoustic features and showed that they improve ASR performance on both the CHiME-2 and CHiME-3 datasets .
Besides the above application, we finalized our exploration of uncertainty modeling for wind turbine control. Baldwin Dumortier defended his PhD thesis on this topic .
In the METAL project, experiments are planned to investigate further the use of speech technologies for foreign language learning in middle and high schools. Besides adapting acoustic models to teenager voices, current work investigates the reliability of speech technologies for analyzing student pronunciations, and for detecting miss-pronunciations. Also, besides making the pronunciation diagnostics more reliable, the aim is to elaborate robust strategies that will make it possible to handle sets of unreliable individual results, and still be able to provide a relevant feedback on recurrent miss-pronunciations.
The analysis of prosodic correlates of discourse particles has continued. Some additional data has been annotated. The automatic word and phonetic segmentation of the discourse particles has been manually checked and corrected when necessary. Once more, this has shown that automatic segmentation is not perfect, especially on spontaneous speech recording in real conditions. For each discourse particle, prosodic characteristics of occurrences of each pragmatic function (conclusive, introductive, etc.) were automatically extracted. For each discourse particle and each pragmatic function, the most frequent F0 patterns were retained as the representative forms. Results show that a pragmatic function, common to several discourse particles, gives rise to a uniform prosodic marking .
Company: Dolby (Spain)
Duration: Sep – Dec 2018
Participants: Antoine Liutkus (Inria Zenith), Emmanuel Vincent
Abstract: This contract aims to evaluate the feasibility of state-of-the-art source separation technology and related technologies for four use cases, and to identify those which could be commercially exploited, possibly after a follow-up R&D phase.
Company: Honda Research Intitute Japan (Japan)
Duration: Feb – Mar 2018
Participants: Aditya Nugraha, Romain Serizel, Emmanuel Vincent
Abstract: This contract targets collaborative research on multichannel speech and audio processing and eventual software licensing in order to enable voice-based communication in challenging noisy and reverberant conditions in which current hands-free voice-based interfaces perform poorly.
Company: Honda Research Intitute Japan (Japan)
Duration: Aug 2018 – Mar 2019
Participants: Nancy Bertin (CNRS - IRISA), Antoine Deleforge, Diego Di Carlo
Abstract: This is a follow-up contract, which also targets collaborative research on multichannel speech and audio processing and eventual software licensing in order to enable voice-based communication in challenging noisy and reverberant conditions in which current hands-free voice-based interfaces perform poorly.
Company: Studio Maia SARL (France)
Other partners: Imaging Factory
Duration: Jul 2017 – March 2019
Participants: Yassine Boudi, Vincent Colotte, Mathieu Hu, Emmanuel Vincent
Abstract: This Inria Innovation Lab aims to develop a software suite for voice processing in the multimedia creation chain. The software is aimed at sound engineers and it will rely on the team's expertise in speech enhancement, robust speech and speaker recognition, and speech synthesis.
Company: Orange SA (France)
Duration: Nov 2016 – Nov 2019
Participants: Laureline Perotin, Romain Serizel, Emmanuel Vincent
Abstract: This CIFRE contract funds the PhD thesis of Laureline Perotin with Orange Labs. Our goal is to develop deep learning based speaker localization and speech enhancement algorithms for robust hands-free voice command. We are especially targeting difficult scenarios involving several simultaneous speakers.
Company: Invoxia SAS (France)
Duration: Mar 2017 – Mar 2020
Participants: Guillaume Carbajal, Romain Serizel, Emmanuel Vincent
Abstract: This CIFRE contract funds the PhD thesis of Guillaume Carbajal. Our goal is to design a unified end-to-end deep learning based speech enhancement system that integrates all steps in the current speech enhancement chain (acoustic echo cancellation and suppression, dereverberation, and denoising) for improved hands-free voice communication.
Company: Ministère des Armées (France)
Duration: Sep 2018 – Aug 2021
Participants: Raphaël Duroselle, Denis Jouvet, Irina Illina
Abstract: This contract corresponds to the PhD thesis of Raphaël Duroselle on the application of deep learning techniques for domain adaptation in speech processing.
Company: Facebook AI Research (France)
Duration: Nov 2018 – Nov 2021
Participants: Adrien Dufraux, Emmanuel Vincent
Abstract: This CIFRE contract funds the PhD thesis of Adrien Dufraux. Our goal is to explore cost-effective weakly supervised learning approaches, as an alternative to fully supervised or fully unsupervised learning for automatic speech recognition.
Project acronym: CPER LCHN
Project title: CPER “Langues, Connaissances et Humanités Numériques”
Duration: 2015-2020
Coordinator: Bruno Guillaume (LORIA) & Alain Polguère (ATILF)
Participants: Dominique Fohr, Denis Jouvet, Odile Mella, Yves Laprie
Abstract: The main goal of the project is related to experimental platforms for supporting research activities in the domain of languages, knowledge and numeric humanities engineering.
MULTISPEECH contributes to automatic speech recognition, speech-text alignment and prosody aspects.
Project acronym: CPER IT2MP
Project title: CPER “Innovation Technologique Modélisation et Médecine Personalisée”
Duration: 2015-2020
Coordinator: Faiez Zannad (Inserm-CHU-UL)
Participants: Romain Serizel, Emmanuel Vincent
Abstract: The goal of the project is to develop innovative technologies for health, and tools and strategies for personalized medicine.
MULTISPEECH will investigate acoustic monitoring using an array of microphones.
Project title: Control of the movements of the lips in the context of facial animation for an intelligible lipsync.
Duration: February 2017 - August 2018
Coordinator: Slim Ouni
Participants: Valerian Girard, Slim Ouni
Funding: SATT
Abstract: We proposed in this project the development of tools of lipsync which, from recorded speech, provide realistic mechanisms of animating the lips. These tools are meant to be integrated into existing 3D animation software and existing game engines. One objective was that these lipsync tools fit easily into the production pipeline in the field of 3D animation and video games. The goal of this maturation was to propose a product ready to be exploited in the industry whether by the creation of a start-up or by the distribution of licenses.
A first prototype of the lipsync system has been developed for French. From audio and text, the system allows animating a 3D model of the face (an avatar) realistically. This work has been presented at Annecy International Animation Film Festival.
Project acronym: DYCI2 (http://
Project title: Creative Dynamics of Improvised Interaction
Duration: March 2015 - February 2018
Coordinator: Ircam (Paris)
Other partners: Inria (Nancy), University of La Rochelle
Participants: Ken Déguernel, Nathan Libermann, Emmanuel Vincent
Abstract: The goal of this project was to design a music improvisation system able to listen to the other musicians, to improvise in their style, and to modify its improvisation according to their feedback in real time.
MULTISPEECH was responsible for designing a system able to improvise on multiple musical dimensions (melody, harmony) across multiple time scales.
Project acronym: ArtSpeech
Project title: Synthèse articulatoire phonétique
Duration: October 2015 - March 2019
Coordinator: Yves Laprie
Other partners: Gipsa-Lab (Grenoble), IADI (Nancy), LPP (Paris)
Participants: Ioannis Douros, Yves Laprie, Anastasiia Tsukanova
Abstract: The objective is to synthesize speech from text via the numerical simulation of the human speech production processes, i.e. the articulatory, aerodynamic and acoustic aspects. Corpus based approaches have taken a hegemonic place in text to speech synthesis. They exploit very good acoustic quality speech databases while covering a high number of expressions and of phonetic contexts. This is sufficient to produce intelligible speech. However, these approaches face almost insurmountable obstacles as soon as parameters intimately related to the physical process of speech production have to be modified. On the contrary, an approach which rests on the simulation of the physical speech production process makes explicitly use of source parameters, anatomy and geometry of the vocal tract, and of a temporal supervision strategy. It thus offers direct control on the nature of the synthetic speech.
Static MRI acquisition of vowels (images plus acoustic signal) have been carried out this year and their exploitation started to explore the impact of the articulatory modeling and the plane wave assumption. Manual delineations of approximately 1000 images have been done and used to generate speech signals with articulatory copy synthesis.
Project acronym: KAMoulox
Project title: Kernel additive modelling for the unmixing of large audio archives
Duration: January 2016 - September 2019
Coordinator: Antoine Liutkus (Inria Zenith)
Participants: Mathieu Fontaine, Antoine Liutkus
Abstract: The objective is to develop the theoretical and applied tools required to embed audio denoising and separation tools in web-based audio archives. The applicative scenario is to deal with large audio archives, and more precisely with the notorious “Archives du CNRS — Musée de l'homme”, gathering about 50,000 recordings dating back to the early 1900s.
Project acronym: ISITE LUE
Project title: Lorraine Université d’Excellence
Duration: starting in 2016
Coordinator: Univ. Lorraine
Participants: Ioannis Douros, Yves Laprie
Abstract: The initiative aims at developing and densifying the initial perimeter of excellence, within the scope of the social and economic challenges, so as to build an original model for a leading global engineering university, with a strong emphasis on technological research and education through research. For this, we have designed LUE as an “engine” for the development of excellence, by stimulating an original dialogue between knowledge fields.
MULTISPEECH is mainly concerned with challenge number 6: “Knowledge engineering”, i.e., engineering applied to the field of knowledge and language, which represent our immaterial wealth while being a critical factor for the consistency of future choices. This project funds the PhD thesis of Ioannis Douros.
Project acronym: E-FRAN METAL
Project title: Modèles Et Traces au service de l’Apprentissage des Langues
Duration: October 2016 - September 2020
Coordinator: Anne Boyer (LORIA)
Other partners: Interpsy, LISEC, ESPE de Lorraine, D@NTE (Univ. Versailles Saint Quentin), Sailendra SAS, ITOP Education, Rectorat.
Participants: Theo Biasutto-Lervat, Anne Bonneau, Vincent Colotte, Dominique Fohr, Denis Jouvet, Odile Mella, Slim Ouni, Anne-Laure Piat-Marchand, Elodie Gauthier, Thomas Girod
Abstract: METAL aims at improving the learning of languages (both written and oral components) through the development of new tools and the analysis of numeric traces associated with students' learning, in order to adapt to the needs and rythm of each learner.
MULTISPEECH is concerned by oral language learning aspects.
Project acronym: VOCADOM (http://
Project title: Robust voice command adapted to the user and to the context for ambient assisted living
Duration: January 2017 - December 2020
Coordinator: CNRS - LIG (Grenoble)
Other partners: Inria (Nancy), Univ. Lyon 2 - GREPS, THEORIS (Paris)
Participants: Dominique Fohr, Md Sahidullah, Sunit Sivasankaran, Emmanuel Vincent
Abstract: The goal of this project is to design a robust voice control system for smart home applications.
MULTISPEECH is responsible for wake-up word detection, overlapping speech separation, and speaker recognition.
Project acronym: DiSCogs
Project title: Distant speech communication with heterogeneous unconstrained microphone arrays
Duration: September 2018 – March 2022
Coordinator: Romain Serizel
Participants: Nicolas Furnon, Irina Illina, Romain Serizel, Emmanuel Vincent
Collaborators: Télécom ParisTech, 7sensing
Abstract: The objective is to solve fundamental sound processing issues in order to exploit the many devices equipped with microphones that populate our everyday life. The solution proposed is to apply machine learning methods based on deep learning to recast the problem of synchronizing devices at the signal level as a multi-view learning problem aiming at extracting complementary information from the devices at hand.
Program: H2020 ICT-29-2018 (RIA)
Project acronym: COMPRISE
Project title: Cost-effective, Multilingual, Privacy-driven voice-enabled Services
Duration: Dec 2018- Nov 2021
Coordinator: Emmanuel Vincent
Other partners: Inria Magnet, Ascora GmbH, Netfective Technology SA, Rooter Analysis SL, Tilde SIA, University of Saarland
Participants: Irina Illina, Denis Jouvet, Emmanuel Vincent
Abstract: COMPRISE will define a fully private-by-design methodology and tools that will reduce the cost and increase the inclusiveness of voice interaction technologies.
Program: CHIST-ERA
Project acronym: AMIS
Project title: Access Multilingual Information opinionS
Duration: Dec 2015- Nov 2018
Coordinator: Kamel Smaïli (LORIA)
Other partners: University of Avignon, University of Science and Technology Krakow, University of DEUSTO (Bilbao)
Participants: Dominique Fohr, Denis Jouvet, Odile Mella
Abstract: The idea of the project is to develop a multilingual help system of understanding without any human being intervention. This should help people understanding broadcasting news, presented in a foreign language and to compare it to a corresponding one available in the mother tongue of the user.
MULTISPEECH contributions concern mainly the speech recognition in French, English and Arabic videos.
Jon Barker: University of Sheffield (UK)
Tomi Kinnunen: University of Eastern Finland (Finland)
Speaker verification and spoofing countermeasures for voice biometrics , , , , ,
Nicholas Evans: EURECOM (France)
Hamid Eghbal-Zadeh: Johannes Kepler University (Austria)
Shinji Watanabe, Johns Hopkins University (USA)
Junichi Yamagishi, National Institute of Informatics (Japan)
Elected chair, Steering Committee of the Latent Variable Analysis and Signal Separation (LVA/ICA) conference series (E. Vincent)
General co-chair, 5th CHiME Speech Separation and Recognition Challenge, September 2018 (E. Vincent)
General co-chair, 5th International Workshop on Speech Processing in Everyday Environments, Hyderabad, India, September 2018 (E. Vincent)
Area chair, Latent Variable Analysis and Signal Separation (LVA/ICA), July 2018 (A. Deleforge)
Area Chair, INTERSPEECH, September 2018 (D. Jouvet)
Co-chair, special session “Geometry-Aware Auditory Scene Analysis” at ICASSP, April 2018 (A. Deleforge)
Co-chair, special session “Advances in Phase Retrieval and Application” at LVA/ICA, July 2018 (A. Deleforge)
Co-chair, special session “Distant speech processing for smart speakers” at EUSIPCO, September 2018 (R. Serizel)
Chair, IEEE signal processing cup (SPCup), 2019 (A. Deleforge)
Steering Committee of the Detection and Classification of Acoustic Scenes and Events (DCASE) challenge series (E. Vincent)
Organizing Committe of Automatic Speaker Verification Spoofing and Countermeasures Challenge 2019 (ASVspoof 2019) (M. Sahidullah)
Organizing Committee of the DCASE challenge series, 2018 (R. Serizel)
Review chair, IEEE Technical Committee on Audio and Acoustic Signal Processing, responsible for organizing the review of the 443 papers submitted to the 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) in the general AASP domain (E. Vincent)
Plate-Forme Intelligence Artificielle (PFIA), July 2018 (A. Bonneau, S. Ouni, R. Serizel)
International Conference on Multimedia and Network Information Systems (MISSI), September, 2018 (D. Jouvet)
International Conference on Speech and Computer (SPECOM), September 2018 (D. Jouvet)
International Conference on Text, Speech, and Dialogue (TSD), September 2018 (D. Jouvet)
ACLing 2018 - Arabic Computational Linguistics series (D. Jouvet)
APSIPA ASC 2018 - Asia-Pacific Signal and Information Processing Association - Annual Summit and Conference (M. Sahidullah)
CHiME 2018 - International Workshop on Speech Processing in Everyday Environments (E. Vincent)
EUSIPCO 2018 - European Signal Processing Conference (D. Jouvet, M. Sahidullah, R. Serizel)
ICASSP 2018 - IEEE International Conference on Acoustics, Speech and Signal Processing (A. Deleforge, I. Illina, D. Jouvet, S. Ouni, R. Serizel, E. Vincent)
ICPR 2018 - International Conference on Pattern Recognition (M. Sahidullah)
INTERSPEECH 2018 (A. Bonneau, I. Illina, S. Ouni, M. Sahidullah, E. Vincent)
IROS 2018 - International Conference on Intelligent Robots (A. Deleforge)
IWAENC 2018 - International Workshop on Acoustic Signal Enhancement (E. Vincent)
JEP 2018 - Journées d'Etudes sur la Parole (A. Bonneau, Y. Laprie, S. Ouni)
LVA/ICA 2018 - International Conference on Latent Variable Analysis and Signal Separation (E. Vincent)
MISSI 2018 - International Conference on Multimedia and Network Information Systems (D. Jouvet)
Speaker Odyssey 2018 - The Speaker and Language Recognition Workshop (M. Sahidullah)
SPECOM 2018 - International Conference on Speech and Computer (D. Jouvet)
SPEECH PROSODY 2018 (A. Bonneau, A. Deleforge, D. Jouvet)
SLT 2018 - IEEE Spoken Language Technology Workshop (D. Jouvet, M. Sahidullah, I. Illina)
Speech Communication (D. Jouvet)
Speech Communication, special issue on Realism in Robust Speech and Language Processing (E. Vincent)
EURASIP Journal on Audio, Speech, and Music Processing (Y. Laprie)
Computers in Biology and Medicine (M. Sahidullah)
Computer Speech & Language (M. Sahidullah)
Digital Signal Processing (M. Sahidullah)
Expert Systems With Applications (M. Sahidullah)
IEEE Access (M. Sahidullah)
IEEE/ACM Transactions on Audio, Speech, and Language Processing (A. Deleforge, M. Sahidullah, R. Serizel)
IEEE Journal of Selected Topics in Signal Processing (A. Deleforge, R. Serizel)
IEEE Signal Processing Letters (A. Deleforge, M. Sahidullah)
IEEE Transactions on Emerging Topics in Computational Intelligence (R. Serizel)
IEEE Transactions on Multimedia (S. Ouni)
Jasa Express Letters (Y. Laprie)
Journal of the Acoustical Society of America (Y. Laprie, S. Ouni)
Language Resources and Evaluation (S. Ouni)
Signal Processing Letters (D. Jouvet)
Speech Communication (A. Deleforge, M. Sahidullah)
Deep learning for distant-microphone speech enhancement and recognition, Facebook AI Research, Paris, January 2018 (E. Vincent)
Multidimensional and multi-level learning of music structure for machine improvisation in the DYCI2 project, University Pompeu Fabra, Spain, February 2018 (E. Vincent)
Des difficultés aux troubles de l’apprentissage de la lecture – Impact sur l’estime de soi. Canopé, Nancy, February 2018 (A. Piquard-Kipffer)
A brief introduction to deep learning and its application to multichannel speech enhancement, Laboratoire des systèmes perceptifs, Paris, March 2018 (R. Serizel)
Speaker embeddings: from i-vector to x-vector and beyond in summer school on Machine Learning Applied to Speech Technology and Autonomous Agents, University of Eastern Finland, Finland, August 2018 (M. Sahidullah)
Elected chair, ISCA Special Interest Group on Robust Speech Processing, until September 2018 (E. Vincent)
Secretary/Treasurer, executive member of AVISA (Auditory-VIsual Speech Association), an ISCA Special Interest Group (S. Ouni)
Expertise of ANR project proposals (Y. Laprie)
Expertise of an ASTRID proposal (R. Serizel)
Expertise of a Netherlands Orgaadminisnisation for Scientific Research (NWO) proposal (R. Serizel)
Member of an expertise Committee for specific language disabilities (MDPH 54) (A. Piquard-Kipffer)
Head of the AM2I Scientific Pole of Université de Lorraine (Y. Laprie)
Vice Scientific Deputy of Inria Nancy - Grand Est (E. Vincent)
Member of Management board of Université de Lorraine (Y. Laprie)
Member of the Comité Espace Transfert of Inria Nancy - Grand Est (E. Vincent)
Member of the HCERES committee for the Laboratoire Psychologie de la Perception (E. Vincent)
Member of the HCERES committee for the LIMSI laboratory (D. Jouvet)
Vice-chair of the recruitement jury for an Associate Professor, Polytech Nancy (E. Vincent)
Member of the national recruitment jury for Inria Junior Research Scientists (E. Vincent)
Member of the Commission du personnel scientifique of Inria Nancy - Grand Est (R. Serizel)
Member of the Commission développement technologique of Inria Nancy - Grand Est (R. Serizel)
Member of “Commission paritaire” of Université de Lorraine (Y. Laprie)
DUT: I. Illina, Programming in Java, 150 hours, L1, University of Lorraine, France
DUT: I. Illina, Linux System, 65 hours, L1, University of Lorraine, France
DUT: I. Illina, Supervision of student projects and stages, 50 hours, L2, University of Lorraine, France
DUT: S. Ouni, Programming in Java, 24 hours, L1, University of Lorraine, France
DUT: S. Ouni, Web Programming, 24 hours, L1, University of Lorraine, France
DUT: S. Ouni, Graphical User Interface, 96 hours, L1, University of Lorraine, France
DUT: S. Ouni, Advanced Algorihms, 24 hours, L2, University of Lorraine, France
DUT: R. Serizel, Computer science basics, 90 hours, L1, University of Lorraine, France
DUT: R. Serizel, Introduction to office software applications, 18 hours, L2, University of Lorraine, France
DUT: R. Serizel, Multimedia and web applications, 20 hours, L1, University of Lorraine, France
DUT: R. Serizel, Multimedia content indexing, 20 hours, L2, University of Lorraine, France
Licence: A. Bonneau, Phonetics, 17 hours, L2, Ecole d'audioprothèse, University of Lorraine, France
Licence: A. Bonneau, Speech manipulations, 2 hours, L1, Ecole d'orthophonie, University of Lorraine, France
Licence: V. Colotte, C2i - Certificat Informatique et Internet, 50 hours, L1, University of Lorraine, France
Licence: V. Colotte, System, 45 hours, L3, University of Lorraine, France
Licence: O. Mella, Introduction to Web Programming, 30 hours, L1, University of Lorraine, France
Licence: O. Mella, Computer Networking, 98 hours, L2-L3, University of Lorraine, France
Licence: A. Piquard-Kipffer, Education Science, 32 hours, L1, Département d'Orthophonie, University of Lorraine, France
Licence: A. Piquard-Kipffer, Learning to Read, 34 hours, L2, Département d'Orthophonie, University of Lorraine, France
Licence: A. Piquard-Kipffer, Dyslexia, Dysorthographia, 12 hours, L3, Département d'Orthophonie, University of Lorraine, France
Licence: A. Piquard-Kipffer, Mathematics Didactics, 9 hours, L3, Département d'Orthophonie, University of Lorraine, France
Master: V. Colotte, Introduction to Speech Analysis and Recognition, 18 hours, M1, University of Lorraine, France
Master: V. Colotte, Integration project: multimodal interaction with Pepper, 10 hours, M2, University of Lorraine, France
Master: D. Jouvet, Speech recognition, 12 hours, M2 University of Lorraine, France
Master: D. Jouvet and S. Ouni, Multimodal oral comunication, 24 hours, M2, University of Lorraine, France
Master: Y. Laprie, Analysis, perception and automatic recognition of speech, 30 hours, M1, University of Lorraine, France
Master: O. Mella, Computer Networking, 76 hours, M1, University of Lorraine, France
Master: S. Ouni, Multimedia in Distributed Information Systems, 31 hours, M2, University of Lorraine, France
Master: E. Vincent and T. Biasutto-Lervat, Neural networks, 38 hours, M2, University of Lorraine
Master: A. Piquard-Kipffer, Dyslexia, Dysorthographia diagnosis, 6 hours, M1, Département d'Orthophonie, University of Lorraine, France
Master: A. Piquard-Kipffer, Deaf people and reading, 21 hours, Département d'Orthophonie, University of Lorraine, France
Master: A. Piquard-Kipffer, Psycholinguistics, 20 hours, Département d'Orthophonie, University Pierre et Marie Curie-Paris, France
Master: A. Piquard-Kipffer, French Language Didactics, 53 hours, ESPE, University of Lorraine, France
Master: A. Piquard-Kipffer, Psychology, 6 hours, M2, Department of Psychology, University of Lorraine, France
Continuous training: O. Mella, Computer science courses for secondary school teachers (ISN courses), 21 hours, ESPE, University of Lorraine, France
Continuous training: A. Piquard-Kipffer, Special Educational Needs, 53 hours, ESPE, University of Lorraine, France
Doctorat: A. Piquard-Kipffer, Language Pathology, 20 hours, EHESP, University of Sorbonne-Paris cité, France
Other: V. Colotte, Responsible for “Certificat Informatique et Internet” for the University of Lorraine, France (50000 students, 30 departments)
PhD: Ken Déguernel, “Apprentissage de structures musicales en contexte d’improvisation”, University of Lorraine, March 6, 2018, Emmanuel Vincent and Gérard Assayag (IRCAM) .
PhD: Baldwin Dumortier, “Contrôle acoustique d’un parc éolien”, University of Lorraine, September 25, 2018, Emmanuel Vincent and Madalina Deaconu (Inria Tosca) .
PhD in progress: Amal Houidhek, "Synthèse paramétrique de parole arabe", December 2015, cotutelle, Vincent Colotte, Denis Jouvet and Zied Mnasri (ENIT, Tunisia).
PhD in progress: Imène Zangar, “Amélioration de la qualité de synthèse vocale par HMM pour la parole arabe”, December 2015, codirection, Vincent Colotte, Denis Jouvet and Zied Mnasri (ENIT, Tunisia).
PhD in progress: Amine Menacer, “Traduction automatique de vidéos”, May 2016, Kamel Smaïli and Denis Jouvet.
PhD in progress: Mathieu Fontaine, “Processus alpha-stable pour le traitement du signal”, May 2016, Antoine Liutkus and Roland Badeau (Télécom ParisTech).
PhD in progress: Anastasiia Tsukanova, “Coarticulation modeling in articulatory synthesis”, May 2016, Yves Laprie.
PhD in progress: Nathan Libermann, “Deep learning for musical structure analysis and generation”, October 2016, Frédéric Bimbot (IRISA) and Emmanuel Vincent.
PhD in progress: Lauréline Perotin, “Séparation aveugle de sources sonores en milieu réverbérant”, November 2016, Romain Serizel, Emmanuel Vincent, and Alexandre Guérin (Orange).
PhD in progress: Théo Biasutto, "Multimodal coarticulation modeling: Towards the animation of an intelligible speaking head", December 2016, Slim Ouni.
PhD in progress: Sara Dahmani, "Modeling facial expressions to animate a realistic 3D virtual talking head", January 2017, Slim Ouni and Vincent Colotte.
PhD in progress: Guillaume Carbajal, “Apprentissage profond bout-en-bout pour le rehaussement de la parole”, March 2017, Romain Serizel, Emmanuel Vincent, and Éric Humbert (Invoxia).
PhD in progress: Sunit Sivasankaran, “Exploiting contextual information in the speech processing chain”, July 2017, Dominique Fohr and Emmanuel Vincent.
PhD in progress: Ioannis Douros, “Combining cineMRI and static MRI to analyze speech production”, July 2017, Pierre-André Vuissoz (IADI) and Yves Laprie.
PhD in progress: Diego Di Carlo, “Estimating the Geometry of Audio Scenes Using Virtually-Supervised Learning”, October 2017, Antoine Deleforge and Nancy Bertin.
PhD in progress: Lou Lee, “Du lexique au discours: les particules discursives en français”, October 2017, Yvon Keromnes and Mathilde Dargnat (ATILF) and Denis Jouvet.
PhD in progress: Nicolas Turpault, “Deep learning for sound scene analysis in real environments”, January 2018, Romain Serizel and Emmanuel Vincent.
PhD in progress: Raphaël Duroselle, “Adaptation de domaine par réseaux de neurones appliquée au traitement de la parole”, September 2018, Denis Jouvet and Irina Illina.
PhD in progress: Nicolas Furnon, “Deep-learning based speech enhancement with ad-hoc microphone arrays”, October 2018, Romain Serizel, Irina Illina and Slim Essid (Télécom ParisTech).
PhD in progress: Ajinkya Kulkarni, "Synthèse de parole expressive par apprentissage profond", October 2018, Vincent Colotte and Denis Jouvet.
PhD in progress: Manuel Pariente, “Deep learning-based phase-aware audio signal modeling and estimation”, October 2018, Antoine Deleforge and Emmanuel Vincent.
PhD in progress: Adrien Dufraux, “Leveraging noisy, incomplete, or implicit labels for automatic speech recognition”, November 2018, Emmanuel Vincent, Armelle Brun (LORIA), and Matthijs Douze (Facebook AI Research).
Participation in the Habilitation (HDR) Jury for Anthony Larcher (University of Le Mans, December 2018), D. Jouvet
Participation in the PhD jury of Anastasios Alexandridis (University of Crete, January 2018), E. Vincent.
Participation in the PhD jury of Marius Miron (University Pompeu Fabra, Spain, February 2018), E. Vincent, reviewer.
Participation in the PhD jury of Victor Bisot (Télécom ParisTech, March 2018), E. Vincent, reviewer.
Participation in the PhD jury of Boaz Schwarz (Bar-Ilan university, Israel, March 2018), A. Deleforge, reviewer.
Participation in the PhD jury of Jean-Rémy Gloaguen (École Centrale de Nantes, October 2018), E. Vincent, reviewer.
Participation in the PhD jury of Mathieu Andreux (Université Paris Sciences et Lettres, November 2018), E. Vincent, reviewer.
Participation in the PhD jury of Ming Xiu (Université de Strasbourg, December 2018), Y. Laprie.
Participation in CAFIPEMPF Jury - Master Learning Facilitator, Académie de Nancy-Metz & Université de Lorraine, April, May 2018, A. Piquard-Kipffer
Participation in the Competitive Entrance Examination into Speech-Language Pathology Departement, University of Lorraine, June 2018, A. Piquard-Kipffer.
Participation to the "Ada Lovelace Day" organized by Inria Nancy - Grand Est, October 2018 (A. Bonneau).
Interview for “GAFA : la voix est libre”, le Magazine de la Rédaction, France Culture, January 5, 2018 (E. Vincent)
Interview for “Les enceintes connectées vont révolutionner vos vies et vous ne pourrez plus vous en passer”, 18h39.fr, February 22, 2018 (E. Vincent)
Interview for “Annecy 2018 : Dynalips, solution de lip-sync automatisée.”, 3DVF.com, June 18, 2018 (S. Ouni)
Demonstration at Journée des métiers, Collège Péguy, le Chesnay, April 2018 (A. Piquard-Kipffer)
Exhibition of an automatic lip-sync system at The Annecy International Animation Film Festival, Annecy, 12-15 June 2018 (S. Dahmani, V. Girard, T. Biasutto-Lervat, S. Ouni)
Demonstration at Fête de la Science, University of Lorraine, October 12, 2018 (N. Turpault, R. Serizel, E. Vincent)
Demonstration and participation in a panel discussion at Forum des Sciences Cognitives, Artem, Nancy, November 7, 2018 (N. Turpault, E. Vincent)
Invitation to Serbia Science Festival 2018 (Belgrade) by the French institute of Serbia to give four popularization lectures and present a demonstration on "teaching robots to hear us", November 2018 (A. Deleforge).