MULTISPEECH is a joint project between Inria, CNRS and University of Lorraine, hosted in the LORIA laboratory (UMR 7503). The goal of the project is the modeling of speech for facilitating oral-based communication. The name MULTISPEECH comes from the following aspects that are particularly considered:
Multisource aspects - which means dealing with speech signals originating from several sources, such as speaker plus noise, or overlapping speech signals resulting from multiple speakers; sounds captured from several microphones are also considered.
Multilingual aspects - which means dealing with speech in a multilingual context, as for example for computer assisted language learning, where the pronunciations of words in a foreign language (i.e., non-native speech) is strongly influenced by the mother tongue.
Multimodal aspects - which means considering simultaneously the various modalities of speech signals, acoustic and visual, in particular for the expressive synthesis of audio-visual speech.
The project is organized along the three following scientific challenges:
The explicit modeling of speech. - Speech signals result from the movements of articulators. A good knowledge of their position with respect to sounds is essential to improve, on the one hand, articulatory speech synthesis, and on the other hand, the relevance of the diagnosis and of the associated feedback in computer assisted language learning. Production and perception processes are interrelated, so a better understanding of how humans perceive speech will lead to more relevant diagnoses in language learning as well as pointing out critical parameters for expressive speech synthesis. Also, as the expressivity translates into both visual and acoustic effects that must be considered simultaneously, the multimodal components of expressivity, which are both on the voice and on the face, will be addressed to produce expressive multimodal speech.
The statistical modeling of speech. - Statistical approaches are common for processing speech and they achieve performance that makes possible their use in actual applications. However, speech recognition systems still have limited capabilities (for example, even if large, the vocabulary is limited) and their performance drops significantly when dealing with degraded speech, such as noisy signals, distant microphone recording and spontaneous speech. Source separation based approaches are investigated as a way of making speech recognition systems more robust to noise. Handling new proper names is an example of critical aspect that is tackled, along with the use of statistical models for speech-text automatic alignment and for speech production.
The estimation and the exploitation of uncertainty in speech processing. - Speech signals are highly variable and often disturbed with noise or other spurious signals (such as music or undesired extra speech). In addition, the output of speech enhancement and of source separation techniques is not exactly the accurate "clean" original signal, and estimation errors have to be taken into account in further processing. This is the goal of computing and handling the uncertainty of the reconstructed signal provided by source separation approaches. Finally, MULTISPEECH also aims at estimating the reliability of phonetic segment boundaries and prosodic parameters for which no such information is yet available.
Although being interdependent, each of these three scientific challenges constitutes a founding research direction for the MULTISPEECH project. Consequently, the research program is organized along three research directions, each one matching a scientific challenge. A large part of the research is conducted on French speech data; English and German languages are also considered in speech recognition experiments and language learning. Adaptation to other languages of the machine learning based approaches is possible, depending on the availability of corresponding speech corpora.
Speech signals are the consequence of the deformation of the vocal tract under the effect of the movements of the articulators (jaw, lips, tongue, ...) to modulate the excitation signal produced by the vocal cords or air turbulence. These deformations are visible on the face (lips, cheeks, jaw) through the coordination of different orofacial muscles and skin deformation induced by the latter. These deformations may also express different emotions. We should note that human speech expresses more than just phonetic content, to be able to communicate effectively. In this project, we address the different aspects related to speech production from the modeling of the vocal tract up to the production of expressive audiovisual speech. Phonetic contrasts used by the phonological system of any language result from constraints imposed by the nature of the human speech production apparatus. For a given language these contrasts are organized so as to guarantee that human listeners can identify (categorize) sounds robustly. The study of the categorization of sounds and prosody thus provides a complementary view on speech signals by focusing on the discrimination of sounds by humans, particularly in the context of language learning.
Modeling speech production is a major issue in speech sciences. Acoustic simulation makes the link between articulatory and acoustic domains. Unfortunately this link cannot be fully exploited because there is almost always an acoustic mismatch between natural and synthetic speech generated with an articulatory model approximating the vocal tract. However, the respective effects of the geometric approximation, of the fact of neglecting some cavities in the simulation, of the imprecision of some physical constants and of the dimensionality of the acoustic simulation are still unknown. Hence, the first objective is to investigate the origin of the acoustic mismatch by designing more precise articulatory models, developing new methods to acquire tridimensional Magnetic Resonance Imaging (MRI) data of the entire vocal tract together with denoised speech signals, and evaluating several approaches of acoustic simulation. The articulatory data acquisition relies on a head-neck antenna at Nancy Hospital to acquire MRI of the vocal tract, and on the articulograph Carstens AG501 available in the laboratory.
Up to now, acoustic-to-articulatory inversion has been addressed as an instantaneous problem, articulatory gestures being recovered by concatenating local solutions. The second objective is thus to investigate how more elaborated strategies (a syllabus of primitive gestures, articulatory targets…) can be incorporated in the acoustic-to-articulatory inversion algorithms to take into account dynamic aspects.
Speech is considered as a bimodal communication means; the first modality is audio, provided by acoustic speech signals and the second one is visual, provided by the face of the speaker. In our approach, the Acoustic-Visual Text-To-Speech synthesis (AV-TTS) is performed simultaneously with respect to its acoustic and visible components, by considering a bimodal signal comprising both acoustic and visual channels. A first AV-TTS system has been developed resulting in a talking head; the system relied on 3D-visual data and on an extension of our acoustic-unit concatenation text-to-speech synthesis system (SoJA). An important goal is to provide an audiovisual synthesis that is intelligible, both acoustically and visually. Thus, we continue working on adding visible components of the head through a tongue model and a lip model. We will also improve the TTS engine to increase the accuracy of the unit selection simultaneously into the acoustic and visual domains. To acquire the facial data, we consider using a marker-less motion capture system using a kinect-like system with a face tracking software, which constitutes a relatively low-cost alternative to the Vicon system.
Another challenging research goal is to add expressivity in the AV-TTS. The expressivity comes through the acoustic signal (prosody aspects) and also through head and eyebrow movements. One objective is to add a prosodic component in the TTS engine in order to take into account some prosodic entities such as emphasis (to highlight some important key words). One intended approach will be to explore an expressivity measure at sound, syllable and/or sentence levels that describes the degree of perception or realization of an expression/emotion (audio and 3D domain). Such measures will be used as criteria in the selection process of the synthesis system. To tackle the expressivity issue we will also investigate Hidden Markov Model (HMM) based synthesis which allows for easy adaptation of the system to available data and to various conditions.
Discriminating speech sounds and prosodic patterns is the keystone of language learning whether in the mother tongue or in a second language. This issue is associated with the emergence of phonetic categories, i.e., classes of sounds related to phonemes and prosodic patterns. The study of categorization is concerned not only with acoustic modeling but also with speech perception and phonology. Foreign language learning raises the issue of categorizing phonemes of the second language given the phonetic categories of the mother tongue. Thus, studies on the emergence of new categories, whether in the mother tongue (for people with language deficiencies) or in a second language, must rely upon studies on native and non-native acoustic realizations of speech sounds and prosody, and on perceptual experiments. Concerning prosody, studies are focused on native and non-native realizations of modalities (e.g., question, affirmation, command, ...), as well as non-native realizations of lexical accents and focus (emphasis).
For language learning, the analysis of the prosody and of the acoustic realization of the sounds aims at providing automatic feedback to language learners with respect to acquisition of prosody as well as acquisition of a correct pronunciation of the sounds of the foreign language. Concerning the mother tongue we are interested in the monitoring of the process of sound categorization in the long term (mainly at primary school) and its relation with the learning of reading and writing skills , especially for children with language deficiencies.
Whereas the first research direction deals with the physical aspects of speech and its explicit modeling, this second research direction investigates statistical models for speech data.
Acoustic models are used to represent the pronunciation of the sounds or other acoustic events such as noise. Whether they are used for source separation, for speech recognition, for speech transcription, or for speech synthesis, the achieved performance strongly depends on the accuracy of these models. At the linguistic level, MULTISPEECH investigates models for handling the context (beyond the few preceding words currently handled by the
Acoustic modeling is a key issue for automatic speech recognition. Despite the progress made for many years, current speech recognition applications rely on strong constraints (close-talk microphone, limited vocabulary, or restricted syntax) to achieve acceptable performance. The quality of the input speech signals is particularly important and performance degrades quickly with noisy signals. Accurate signal enhancement techniques are therefore essential to increase the robustness of both automatic speech recognition and speech-text alignment systems to noise and non-speech events.
In MULTISPEECH, focus is set on source separation techniques using multiple microphones and/or models of non-speech events. Some of the challenges include getting the most of the new modeling frameworks based on alpha-stable distributions and deep neural networks, combining them with established spatial filtering approaches, modeling more complex properties of speech and audio sources (phase, inter-frame and inter-frequency properties), and exploiting large data sets of speech, noise, and acoustic impulse responses to automatically discover new models. Beyond the definition of such models, the difficulty will be to design scalable estimation algorithms robust to overfitting, integrate them into the recently developed FASST and KAM software frameworks if relevant, and develop new software frameworks otherwise.
MULTISPEECH investigates lexical and language models in speech recognition with a focus on improving the processing of proper names and of spontaneous speech. Proper names are relevant keys in information indexing, but are a real problem in transcribing many diachronic spoken documents which refer to data, especially proper names, that evolve over time. This leads to the challenge of dynamically adjusting lexicons and language models through the use of the context of the documents or of some relevant external information. We also investigate language models defined on a continuous space (through neural network based approaches) in order to achieve a better generalization on unseen data, and to model long-term dependencies. We also want to introduce into these models additional relevant information such as linguistic features, semantic relation, topic or user-dependent information.
Other topics are spontaneous speech and pronunciation lexicons. Spontaneous speech utterances are often ill-formed and frequently contain disfluencies (hesitations, repetitions, ...) that degrade speech recognition performance. Hence the objective of improving the modeling of disfluencies and of spontaneous speech pronunciation variants. Attention will also be set on pronunciation lexicons with respect to non-native speech and foreign names. Non-native pronunciation variants have to take into account frequent mis-pronunciations due to differences between mother tongue and target language phoneme inventories. Proper name pronunciation variants are a similar problem where difficulties are mainly observed for names of foreign origin that can be pronounced either in a French way or kept close to foreign origin native pronunciation.
Over the last few years statistical speech synthesis has emerged as an alternative to corpus-based speech synthesis. The announced advantages of the statistical speech synthesis are the possibility to deal with small amounts of speech resources and the flexibility for adapting models (for new emotions or new speakers), however, the quality is not as good as that of the concatenation-based speech synthesis. MULTISPEECH will focus on a hybrid approach, combining corpus-based synthesis, for its high-quality speech signal output, and HMM-based speech synthesis for its flexibility to drive selection, and the main challenge will be on its application to producing expressive audio-visual speech.
Moreover, in the context of acoustic feedback in foreign language learning, voice modification approaches are investigated to modify the learner’s (or teacher’s) voice in order to emphasize the difference between the learner’s acoustic realization and the expected realization.
This axis focuses on the uncertainty associated with some processing steps. Uncertainty stems from the high variability of speech signals and from imperfect models. For example, enhanced speech signals resulting from source separation are not exactly the clean original speech signals. Words or phonemes resulting from automatic speech recognition contain errors, and the phone boundaries resulting from an automatic speech-text alignment are not always correct, especially in acoustically degraded conditions. Hence it is important to know the reliability of the results and/or to estimate the uncertainty of the results.
Because small distortions in the separated source signals can translate into large distortions in the cepstral features used for speech recognition, this limits the recognition performance on noisy data. One way to address this issue is to estimate the uncertainty of the separated sources in the form of their posterior distribution and to propagate this distribution, instead of a point estimate, through the subsequent feature extraction and speech decoding stages. Although major improvements have been demonstrated in proof-of-concept experiments using knowledge of the true uncertainty, accurate uncertainty estimation and propagation remains an open issue.
MULTISPEECH seeks to provide more accurate estimates of the posterior distribution of the separated source signals accounting for, e.g., posterior correlations over time and frequency which have not been considered so far. The framework of variational Bayesian (VB) inference appears to be a promising direction. Mappings learned on training data and fusion of multiple uncertainty estimators are also explored. The estimated uncertainties are then exploited for acoustic modeling in speech recognition and, in the future, also for speech-text alignment. This approach may later be extended to the estimation of the resulting uncertainty of the acoustic model parameters and of the acoustic scores themselves.
The accuracy of the phonetic segmentation is important in several cases, as for example for the computation of prosodic features, for avoiding incorrect feedback to the learner in computer assisted foreign language learning, or for the post-synchronization of speech with face/lip images. Currently the phonetic boundaries obtained are quite correct on good quality speech, but the precision degrades significantly on noisy and non-native speech. Phonetic segmentation aspects will be investigated, both in speech recognition (i.e., spoken text unknown) and in forced alignment (i.e., when the spoken text is known).
In the same way that combining several speech recognition outputs leads to improved speech recognition performance, MULTISPEECH will investigate the combination of several speech-text alignments as a way of improving the quality of speech-text alignment and of determining which phonetic boundaries are reliable and which ones are not, and also for estimating the uncertainty of the boundaries. Knowing the reliability of the boundaries will also be useful when segmenting speech corpora; this will help deciding which parts of the corpora need to be manually checked and corrected without an exhaustive checking of the whole corpus.
Prosody information is also investigated as a means for structuring speech data (determining sentence boundaries, punctuation…) possibly in addition to syntactic dependencies. Structuring automatic transcription output is important for further exploitation of the transcription results such as easier reading after the addition of punctuation, or exploitation of full sentences in automatic translation. Prosody information is also necessary for determining the modality of the utterance (question or not), as well as determining accented words.
Prosody information comes from the fundamental frequency, the duration of the sounds and their energy. Any error in estimating these parameters may lead to a wrong decision. MULTISPEECH will investigate estimating the uncertainty of the duration of the phones (see uncertainty of phonetic boundaries above) and on the fundamental frequency, as well as how this uncertainty shall be propagated in the detection of prosodic phenomena such as accented words, utterance modality, or determination of the structure of the utterance.
Approaches and models developed in the MULTISPEECH project are intended to be used for facilitating oral communication in various situations through enhancements of the communication channels, either directly via automatic speech recognition or speech production technologies, or indirectly, thanks to computer assisted language learning. Applications also include the usage of speech technologies for helping people in handicapped situations or for improving their autonomy. Foreseen application domains are related to computer assisted learning, health and autonomy (more precisely aided communication and monitoring), annotation and processing of spoken documents, and multimodal computer interaction.
Although speaking seems quite natural, learning foreign languages, or learning the mother tongue for people with language deficiencies, represents critical cognitive stages. Hence, many scientific activities have been devoted to these issues either from a production or a perception point of view. The general guiding principle with respect to computer assisted mother or foreign language learning is to combine modalities or to augment speech to make learning easier. Based upon a comparison of the learner’s production to a reference, automatic diagnoses of the learner’s production can be considered, as well as perceptual feedback relying on an automatic transformation of the learner’s voice. The diagnosis step strongly relies on the studies on categorization of sounds and prosody in the mother tongue and in the second language. Furthermore, reliable diagnosis on each individual utterance is still a challenge, and elaboration of advanced automatic feedback requires a temporally accurate segmentation of speech utterances into phones and this explains why accurate segmentation of native and non-native speech is an important topic in the field of acoustic speech modeling.
A foreseen application aims at improving the autonomy of elderly or disabled people, and fit with smartroom applications. In a first step, source separation techniques could be tuned and should help for locating and monitoring people through the detection of sound events inside apartments. In a longer perspective, adapting speech recognition technologies to the voice of elderly people should also be useful for such applications, but this requires the recording of adequate databases. Sound monitoring in other application fields (security, environmental monitoring) could also be envisaged.
A first type of annotation consists in transcribing a spoken document in order to get the corresponding sequences of words, with possibly some complementary information, such as the structure (punctuation) or the modality (affirmation/question) of the utterances to make the reading and understanding easier. Typical applications of the automatic transcription of radio or TV shows, or of any other spoken document, include making possible their access by deaf people, as well as by text-based indexing tools.
A second type of annotation is related to speech-text alignment, which aims at determining the starting and ending times of the words, and possibly of the sounds (phonemes). This is of interest in several cases as for example, for annotating speech corpora for linguistic studies, and for synchronizing lip movements with speech sounds, for example for avatar-based communications. Although good results are currently achieved on clean data, automatic speech-text alignment needs to be improved for properly processing noisy spontaneous speech data and needs to be extended to handle overlapping speech.
Large audio archives are important for some communities of users, e.g., linguists, ethnologists or researchers in digital humanities in general. In France, a notorious example is the "Archives du CNRS — Musée de l’homme", gathering about 50,000 recordings dating back to the early 1900s. When dealing with very old recordings, the practitioner is often faced with the problem of noise. This stems from the fact that a lot of interesting material from a scientific point of view is very old or has been recorded in very adverse noisy conditions, so that the resulting audio is poor. The work on source separation can lead to the design of semi-automatic denoising and enhancement features, that would allow these researchers to significantly enhance their investigation capabilities, even without expert knowledge in sound engineering.
Finally, there is also a need for speech signal processing techniques in the field of multimedia content creation and rendering. Relevant techniques include speech and music separation, speech equalization, prosody modification, and speaker conversion.
Speech synthesis has tremendous applications in facilitating communication in a human-machine interaction context to make machines more accessible. For example, it started to be widely common to use acoustic speech synthesis in smartphones to make possible the uttering of all the information. This is valuable in particular in the case of handicap, as for blind people. Audiovisual speech synthesis, when used in an application such as a talking head, i.e., virtual 3D animated face synchronized with acoustic speech, is beneficial in particular for hard-of-hearing individuals. This requires an audiovisual synthesis that is intelligible, both acoustically and visually. A talking head could be an intermediate between two persons communicating remotely when their video information is not available, and can also be used in language learning applications as vocabulary tutoring or pronunciation training tool. Expressive acoustic synthesis is of interest for the reading of a story, such as audiobook, to facilitate the access to literature (for instance for blind people or illiterate people).
We ranked 1st ex aqueo for the ”Professionally produced music recordings” task of the 2016 Signal Separation Evaluation Campaign (SiSEC) .
Automatic Speech-Text Alignment Software
Keyword: Speech-text alignment
Functional Description
ASTALI is a software for aligning a speech signal with its corresponding orthographic transcription (given in simple text file for short audio signals or in .trs files as generated by transcriber for longer speech signals). Using a phonetic lexicon and automatic grapheme-to-phoneme converters, all the possible sequences of phones corresponding to the text are generated. Then, using acoustic models, the tool finds the best phone sequence and provides the boundaries at the phone and at the word levels. ASTALI is available through a web application, which makes the service easy to use, without requiring any software downloading. This year, the integration of the web application on the ORTOLANG platform has been finalized.
Participants: Dominique Fohr, Odile Mella, Antoine Chemardin, Valérian Girard and Denis Jouvet
Contact: Dominique Fohr
URLs: https://
Multichannel audio source separation with deep neural networks
Keywords: Audio - Source Separation - Deep learning
Scientific Description
dnnsep is the only source separation software relying on multichannel Wiener filtering based on deep learning. Deep neural networks are used to initialize and reestimate the power spectrum of the sources at every iteration of an expectation-maximization (EM) algorithm. This results in state-of-the-art separation quality for both speech and music.
Functional Description
dnnsep is a new software that combines deep neural networks and multichannel signal processing for speech enhancement and separation of musical recordings.
Participants: Aditya Nugraha, Antoine Liutkus and Emmanuel Vincent
Contact: Emmanuel Vincent
Functional Description
JSnoori is written in Java and uses signal processing algorithms developed within the WinSnoori software with the double objective of being a platform independent signal visualization and manipulation tool, and also for designing exercises for learning the prosody of a foreign language. JSnoori can be used directly or via scripts written in Jython. This year, several approaches for computing the fundamental frequency have been added; and, JSnoori is now available through the ORTOLANG platform.
Participants: Yves Laprie, Slim Ouni, Aghilas Sini and Ilef Ben Farhat
Contact: Yves Laprie
Kaldi-based Automatic Transcription System
Keyword: Speech recognition
Functional Description
KATS is a multipass system for transcribing audio data, and in particular radio or TV shows. The audio stream is first split into homogeneous segments that are decoded using the most adequate acoustic model with a large vocabulary continuous speech recognition engine. In this new software, the recognition engine is based on the Kaldi toolkit, and uses Deep Neural Network - DNN - based acoustic models. An extra processing pass is run in order to rescore the
Participants: Odile Mella, Dominique Fohr and Denis Jouvet
Contact: Dominique Fohr
URL: Available online on the A||go platform: https://
Sofware for audio-visual and multimodal data acquisition and processing
Functional Description
Within the ADT PLAVIS (cf. ), we have developed a software for 3D audiovisual data acquisition and synthesis. The system incorporates an animation module of the talking head to reconstruct the animated face along with audio. The acquisition software handles one or several acquisition systems: motion-capture (Kinect-like), Vicon or EMA systems. The various acquisition channels are synchronized. The animation technique can exploit multimodal data to define blendshapes that controls the face; the advantage of using blendshapes is to be able to transfer the animation from one 3D human model to another. A semi-automatic acoustic boundary correction process is integrated in the corpus building process. The text-to-speech processing is driven by the Soja software.
Participants: Vincent Colotte, Slim Ouni, Sara Dahmani
Contact: Vincent Colotte
Speech Synthesis platform in JAva
Functional Description
SOJA is a software for Text-To-Speech synthesis (TTS) which relies on a non uniform unit selection algorithm. It performs all steps from text input to speech signal output. A set of associated tools is available for elaborating a corpus for a TTS system (transcription, alignment. . . ). Currently, the corpus contains about 3 hours of speech recorded by a female speaker. Most of the modules are in Java, some are in C. The SOJA software runs under Windows and Linux. It can be launched with a graphical user interface or directly integrated in a Java code or by following the client-server paradigm. During 2016, the part of code in C was reduced to go to a full-Java software in the future. The natural language processing can now be restarted from any step. This functionality is useful for instance during corpus processing when using semi-automatic boundaries correction.
Participants: Vincent Colotte and Alexandre Lafosse
Contact: Vincent Colotte
Visualization of EMA Articulatory data
Functional Description
VisArtico is a user-friendly software which allows visualizing EMA data acquired by an articulograph (AG500, AG501 or NDI Wave). This visualization software has been designed so that it can directly use the data provided by the articulograph to display the articulatory coil trajectories, synchronized with the corresponding acoustic recordings. Moreover, VisArtico not only allows viewing the coil trajectories but also enriches the visual information by indicating clearly and graphically the data for the tongue, lips and jaw. In addition, it is possible to insert images (MRI or X-Ray, for instance) to compare the EMA data with data obtained through other acquisition techniques. It is possible to generate a movie for any articulatory-acoustic sequence. During 2016, we have made a new version of VisArtico where the 3D view is now based on OpenGL. This allows a better quality rendering. It is possible to make measurement between sensors to compute the distance. Finally, we added the possibility to display the fundamental frequency on the spectrogram.
Participants: Slim Ouni, Loic Mangeonjean, Ilef Ben Farhat and Bertrand Muller
Contact: Slim Ouni
Keyword: Medical imaging
Functional Description
The Xarticulators software is intended to delineate contours of speech articulators in X-ray images, construct articulatory models and synthesize speech from X-ray films. This software provides tools to track contours automatically, semi-automatically or by hand, to make the visibility of contours easier, to add anatomical landmarks to speech articulators and to synchronize images with the sound. In addition we also added the possibility of processing digitized manual delineation results made on sheets of papers when no software is available. Xarticulators also enables the construction of adaptable linear articulatory models from the X-ray images and incorporates acoustic simulation tools to synthesize speech signals from the vocal tract shape. Recent work was on the possibility of synthesizing speech from X-ray or 2D-MRI films.
During 2016, we developed a new version of the articulatory model which incorporates a more realistic model of the epiglottis and lips.
Contact: Yves Laprie
Functional Description
Within a LORIA exploratory project (cf. ), we have set up an acquisition hardware platform to acquire multimodal data in speech communication context. The system is composed of the articulograph Carstens AG501 (which was acquired as part of the EQUIPEX ORTOLANG - cf. ), 4 Vicon cameras (a motion capture system), an Intel RealSense which is a depth camera (acquired as part of the project CORExp - cf. ), a video camera and a microphone. With such heterogeneous hardware the synchronization is essential; this is achieved through a trigger device. All the data processing is performed with the PLAVIS software. This year, the system has been used to acquire multimodal data for the MCC project (cf. ) and a first exploratory expressive multimodal corpus .
Participants: Slim Ouni, Vincent Colotte, Valerian Girard, Sara Dahmani
Contact: Slim Ouni
The acoustic simulations play a central role in articulatory synthesis and should enable the production of all classes of sounds in a realistic manner. The production of voiced fricatives relies on a partial closure of the glottis which simultaneously creates an airflow which generates turbulence downwards from the constriction and the vibration of the vocal folds. Our acoustic simulation framework has been extended to incorporate a glottal chink in a self-oscillating vocal fold model. The glottis is then made up of two main separated components: a self-oscillating part and a constantly open chink. This feature allows the simulation of voiced fricatives, thanks to a self-oscillating model of the vocal folds to generate the voiced source, and the glottal opening that is necessary to generate the frication noise.
The acoustic propagation paradigm is appropriately chosen so that it can deal with complex geometries and a time-varying length of the vocal tract. Temporal scenarios for the dynamic shapes of the vocal tract and the glottal configurations were derived from the simultaneous acquisition of X-ray or MRI images and audio recording. Copy synthesis of a few French sentences , , shows the accuracy of the simulation framework to reproduce acoustic cues of phrase-level utterances containing most of French phone (sound) classes while considering the real geometric shape of the speaker. For this purpose the articulatory model has been extended to offer a better precision of the epiglottis and of lips.
The acquisition of dynamic data is a key objective since speech production gestures involve the anticipation of the articulatory targets of the coming sounds. Cine-MRI represents an invaluable tool since it can image the whole vocal tract. However, speech requires a sampling frequency above 30 Hz to capture interesting information. Compressive sampling relies on partially collecting data in the Fourier space of the images acquired via MRI. The combination of compressed sensing technique, along with homodyne reconstruction, enables the missing data to be recovered . The good reconstruction is guaranteed by an appropriate design of the sampling pattern. It is based on a pseudo-random Cartesian scheme, where each line is partially acquired for use of the homodyne reconstruction, and where the lines are pseudo-randomly sampled: central lines are constantly acquired and the sampling density decreases as the lines are far from the center.
With the spread of depth cameras (kinect-like systems), many researchers consider using these systems to track the movement of some speech articulators as lips and jaw. We are considering using this kind of system if it is suitable for speech production studies. For this reason, we have assessed the precision of markerless acquisition techniques when used to acquire articulatory data for speech production studies . Two different markerless systems have been evaluated and compared to a marker-based one. The main finding is that both markerless systems provide reasonable results during normal speech and the quality is uneven during fast articulated speech. The quality of the data is dependent on the temporal resolution of the markerless system.
A comparison between emotional and neutral speech was conducted using a small database containing utterances recorded in six emotional types (anger, fear, sadness, disgust, surprise and joy) as well as in a neutral pronunciation. The prosodic analysis focused on the main prosodic parameters such as vowel duration, energy and fundamental frequency (F0) level, and pause occurrences. The values of prosodic parameters were compared among the various emotional styles, as well as between emotional style and neutral style utterances. Moreover, the structuration of the sentences, in the various emotional styles, was particularly studied through a detailed analysis of pause occurrences and their length, and of the length of prosodic groups .
Concerning expressive audiovisual speech synthesis, a case study of a semi-professional actor who uttered a set of sentences for 6 different emotions in addition to neutral speech was conducted. Our purpose is to identify the main characteristics of audiovisual expressions that need to be integrated during synthesis to provide believable emotions to the virtual 3D talking head. We have recorded concurrently audio and motion capture data. The acoustic and the visual data have been analyzed. The main finding is that although some expressions are not well identified, some expressions were well characterized and tied in both acoustic and visual space . The acquisition of the corpus was done with the platform software PLAVIS (cf. ).
We examined the schooling experiences of 166 young people with disabilities, aged from 6 to 20 years old. These children and teenagers had specific language impairment : SLI (severe language impairment), dyslexia, dysorthographia. The phonemic discrimination, phonological and phonemic analysis difficulties faced in their childhoods had raised reading difficulties which constituted a major obstacle, which the pupils did not overcome. Consequently, this led them to repeat one or more grades. This rate is 18 times higher than the French average. The importance of this cycle of learning can be better understood through this data, which could also enable, if not overcoming the handicap, to at least improving their learning possibilities .
Three digital albums for language impaired children were designed within the Handicom (ADT funded by Inria). These three prototypes focus on the importance of multimodal speech combining written words and visual clues: a 3D avatar telling the stories and coding oral language in LPC (french cued speech) for hearing impaired children. Eight speech and language therapists used one of these albums (the digital prototype Nina fête son anniversaire !) with 8 children who are aged 5 years: 4 hearing impaired children, 2 children with SLI and 2 children with autism. The training they experienced with these children showed that the use of the digital book can foster some capacities involved in language learning .
The IFCASL corpus is a French-German bilingual phonetic learner corpus designed, recorded and annotated in the IFCASL project (cf. ). It incorporates data for a language pair in both directions, i.e. in our case French learners of German, and German learners of French. In addition, the corpus is complemented by two sub-corpora of native speech by the same speakers. The corpus has been finalized, and provides spoken data by about 100 speakers with comparable productions, annotated and segmented at the word and phone levels, with more than 50% of manually checked and corrected data .
We investigated the correct placement of lexical (German) or post-lexical (French) accents . French and German differ with respect to the representation and implementation of prominence. French can be assumed to have no prominence represented in the mental lexicon and accents are regularly assigned post-lexically on the last full vowel of an accentual group. In German, prominence is considered to be represented lexically. This difference may give rise to interferences when German speakers learn French and French speakers learn German.
Results of a judgment task (conducted with 3 trained phoneticians) of native and nonnative productions of French learners of German and German learners of French, all of them beginners, show that both groups have not completely acquired the correct suprasegmental structures in the respective L2
In view of implementing acoustic feedback in foreign language learning we analyzed acoustic cues which could explain that final fricatives are perceived as voiced or unvoiced. The ratio of unvoiced frames in the consonantal segment and also the ratio between consonantal duration and vowel duration were measured. As expected, we found that beginners face more difficulties to produce voiced fricatives than advanced learners. Also, the production becomes easier for the learners, especially for beginners, if they practice repetition after a native speaker. We use these findings to design and develop feedback via speech analysis/synthesis technique TD-PSOLA using the learner’s own voice and voiced fricatives uttered by French speakers . We selected fully voiced exemplars and evaluated whether the presence of an additional schwa fosters the perception of voicing by native French speakers.
We pursued our research on the use of deep learning for multichannel source separation . Our technique exploits both the spatial properties of the sources as modeled by their spatial covariance matrices and their spectral properties as modeled by a deep neural network. The model parameters are alternately estimated in an expectation-maximization (EM) fashion. We used this technique for music separation in the context of the 2016 Signal Separation Evaluation Campaign (SiSEC) . We also used deep learning to address the fusion of multiple source separation techniques and found it to perform much better than the variational Bayesian model averaging techniques previously investigated .
We wrote an article about music source separation for the general public .
The alpha-harmonizable model has recently been proposed by A. Liutkus et al. as the only available probabilistic framework to account for signal processing methods manipulating fractional spectrograms instead of more traditional power spectrograms. Indeed, they generalize the classical Gaussian formulation and permit to handle large uncertainties or signal dynamics, which are both common in audio.
Our work on this topic this year has notably focused on its extension to the multichannel setting, which is important for music processing and source localization. Since inference in multivariate alpha-stable distribution is a very intricate issue, the approach undertaken has focused on analysing the multichannel signals through the joint analysis of multiple scalar projections on the real line. This results in an original algorithm called PROJET that combines computational tractability with the inherent robustness of alpha-stable models , .
In many real-world conditions, the target speech signal is reverberated and noisy. In order to motivate further work by the community, we created an international evaluation campaign on that topic in 2011: the CHiME Speech Separation and Recognition Challenge. After three successful editions , , we organized the fourth edition in 2016. We also summarized the speech distortion conditions in real scenarios for speech processing applications and collected a French corpus for distant-microphone speech processing in real homes .
Speech enhancement and automatic speech recognition (ASR) are most often evaluated in matched (or multi-condition) settings where the acoustic conditions of the training data match (or cover) those of the test data. We conducted a systematic assessment of the impact of acoustic mismatches (noise environment, microphone response, data simulation) between training and test data on the performance of recent DNN-based speech enhancement and ASR techniques . The results show that most algorithms perform consistently on real and simulated data and are barely affected by training on different noise environments. This suggests that DNNs generalize more easily than previously thought.
We explored acoustic modeling for the classification of environmental sound events and sound scenes and submitted our system to the DCASE 2016 Challenge .
The diachronic nature of broadcast news causes frequent variations in the linguistic content and vocabulary, leading to the problem of Out-Of-Vocabulary (OOV) words in automatic speech recognition. Most of the OOV words are found to be proper names whereas proper names are important for automatic indexing of audio-video content as well as for obtaining reliable automatic transcriptions. New proper names missed by the speech recognition system can be recovered by a dynamic vocabulary multi-pass recognition approach in which new proper names are added to the speech recognition vocabulary based on the context of the spoken content . The goal of this work is to model the semantic and topical context of new proper names in order to retrieve OOV words which are relevant to the spoken content in the audio document. Probabilistic topic models and word embeddings from neural network models are explored for the task of retrieval of relevant proper names. Neural network context models trained with an objective to maximise the retrieval performance are proposed. A Neural Bag-of-Words (NBOW) model trained to learn context vector representations at a document level is shown to outperform the generic representations. The proposed Neural Bag-of-Weighted-Words (NBOW2) model learns to assign a degree of importance to input words and has the ability to capture task specific key-words . Experiments on automatic speech recognition of French broadcast news videos demonstrate the effectiveness of the proposed models. Further evaluation of the NBOW2 model on standard text classification tasks, including movie review sentiment classification and newsgroup topic classification, shows that it learns interesting information about the task and gives the best classification accuracies among the bag-of-words models.
Out-of-vocabulary (OOV) words can pose a particular problem for automatic speech recognition of broadcast news. The language models (LMs) of ASR systems are typically trained on static corpora, whereas new words (particularly new proper nouns) are continually introduced in the media. Additionally, such OOVs are often content-rich proper nouns that are vital to understanding the topic. We explore methods for dynamically adding OOVs to language models by adapting the n-gram language model used in our ASR system. We propose two strategies: the first one relies on finding in-vocabulary (IV) words similar to the OOVs, where word embeddings are used to define similarity. Our second strategy leverages a small contemporary corpus to estimate OOV probabilities. The models we propose yield improvements in perplexity over the baseline; in addition, the corpus-based approach leads to a significant decrease in proper noun error rate over the baseline in recognition experiments .
Similarly to speech, music involves several levels of information, from the acoustic signal up to cognitive quantities such as composer style or key, through mid-level quantities such as a musical score or a sequence of chords. The dependencies between mid-level and lower- or higher-level information can be represented through acoustic models and language models, respectively. We published two articles that summarize our work on the System & Contrast model for the characterization of the mid-term and long-term structure of music and on the structural segmentation of popular music pieces using a regularity constraint that naturally stems from this model , . We also proposed a new model for automatic music improvisation that combines a multi-dimensional probabilistic model encoding the musical experience of the system and a factor oracle encoding the local context of the improvisation .
Work on HMM-based Arabic speech synthesis was carried out within a CMCU PHC project with ENIT (Engineer school at Tunis-Tunisia; cf. ). A first version of the system, based on the HTS toolkit (HMM-based Speech Synthesis System), is now working; and the study of the impact of some parameters is ongoing.
In parallel, the HTS system is also applied to the French language.
In many real-world conditions, the target speech signal overlaps with noise and some distortion remains after speech enhancement. The framework of uncertainty decoding assumes that this distortion has a Gaussian distribution and seeks to estimate its covariance matrix in order to exploit it for subsequent feature extraction and decoding. A number of uncertainty estimators have been proposed in the literature, which are typically based on fixed mathematical approximations or heuristics. We finalized our work on a principled variational Bayesian approach to uncertainty estimation and showed its benefit w.r.t. other estimators for speech and speaker recognition . We also pursued our work on the propagation of uncertainty in deep neural network acoustic models.
Besides the above applications, we pursued our exploration of uncertainty modeling for robot audition and wind turbine control. In the first context, uncertainty arises about the location of acoustic sources and the robot is controlled to locate the sources as quickly as possible . In the second context, uncertainty arises about the noise intensity of each wind turbine and the turbines are controlled to maximize electrical production under a maximum noise threshold .
We have continued our work on determining more accurate phonetic boundaries with two new approaches based on DNN. The first approach proposes to find phonetic boundaries directly from the parameterized speech signal using an LSTM (Long Short-Term Memory) neural network. The aim of the second approach is twofold: provide confidence measures for evaluating speech-text alignment outputs and refine these outputs. One of these studies was done with the Synalp team of LORIA in the framework of the project ORFEO (cf. ). The achieved confidence measure outperforms a confidence score (based on acoustic posterior probability) derived from a state-of-the-art text-to-speech aligner .
Within the IFCASL project (cf. ), we have also developed a speech-text alignment system for German which will be integrated into the ASTALI software.
The study of discourse particles that was initiated last year, has continued in the framework of the CPER LCHN (cf. ). A larger set of words and expressions that can be used either as normal lexical words or as discourse particles (as for example quoi (what), voilà (there it is), ...) has been considered. For each of these words/expressions and for each speech corpus that was aligned in the ORFEO project (cf. ), a subset of about one hundred occurrences were selected. Thanks to the CPER LCHN support, a part of these occurrences have been annotated as "discourse particle" or "non discourse particle". Detailed analysis is in progress, with respect to the function (discourse particle or not), the type of speech corpus, and the associated prosodic features.
The fundamental frequency is one of the prosodic features. Numerous approaches exist for the computation of F0. Most of them lead to good performance on good quality speech. The performance degradation with respect to noise level has been studied on reference databases, for several (about ten) F0 detection approaches. It was observed that for each algorithm, a large part of the errors are due to incorrect voiced/unvoiced decision. Studies have also been initiated for computing a confidence measure on the estimated F0 values through the use of neural network approaches.
Company: Venathec SAS
Other partners: ACOEM Group, GE Intelligent Platforms (contracted directly with Venathec)
Duration: June 2014 - August 2017
Supported by: Bpifrance
Abstract: The project aims to design a real-time control system for wind farms that will maximize energy production while limiting sound nuisance. This will leverage our know-how on audio source separation and uncertainty modeling and propagation.
Project acronym: CORExp
Project title: Acquisition, Processing and Analysis of a Corpus for the Synthesis of Expressive Audiovisual Speech
Duration: December 2014 - December 2016
Coordinator: Slim Ouni
Cofunded by Inria and Région Lorraine
Abstract: The main objective of this project was the acquisition of a bimodal corpus of a considerable size (several thousand sentences) to study the expressiveness and emotions during speech (for example, how to decode facial expressions that are merged with speech signals). The main purpose was to acquire, process and analyze the corpus and to study the expressiveness; the results will be used for the expressive audiovisual speech synthesis system.
Project acronym: CPER LCHN
Project title: CPER "Langues, Connaissances et Humanités Numériques"
Duration: 2015-2020
Coordinator: Bruno Guillaume (LORIA) & Alain Polguère (ATILF)
Abstract: The main goal of the project is related to experimental platforms for supporting research activities in the domain of languages, knowledge and numeric humanities engineering.
MULTISPEECH contributes to automatic speech recognition, speech-text alignment and prosody aspects.
Project acronym: CPER IT2MP
Project title: CPER "Innovation Technologique Modélisation et Médecine Personalisée"
Duration: 2015-2020
Coordinator: Faiez Zannad (Inserm-CHU-UL)
Abstract: The goal of the project is to develop innovative technologies for health, and tools and strategies for personalized medicine.
MULTISPEECH will investigate acoustic monitoring using an array of microphones.
Project title: Control of the movements of the lips in the context of facial animation for an intelligible lipsync.
Duration: May 2016 - December 2017
Coordinator: Slim Ouni
Abstract: We propose in this project the development of tools of lipsync which from recorded speech will provide realistic mechanisms of animating the lips. These tools will be available to be integrated into existing 3D animation software and existing game engines. One objective is that these lipsync tools fit easily into the production pipeline in the field of 3D animation and video games. The goal of this maturation is to propose a product ready to be exploited in the industry whether by the creation of a start-up or by the distribution of licenses.
Project acronym: ORTOLANG
Project title: Open Resources and TOols for LANGuage
Duration: September 2012 - December 2016 (phase I)
Coordinator: Jean-Marie Pierrel, ATILF (Nancy)
Other partners: LPL (Aix en Provence), LORIA (Nancy), Modyco (Paris), LLL (Orléans), INIST (Nancy)
Abstract: The aim of ORTOLANG was to propose a network infrastructure offering a repository of language data (corpora, lexicons, dictionaries, etc.) and tools and their treatment that are readily available and well-documented. This will enable a real mutualization of analysis research, of modeling and automatic treatment of the French language. This will also facilitate the use and transfer of resources and tools set up within public laboratories towards industrial partners, in particular towards SME which often cannot develop such resources and tools for language treatment due to the costs of their realization. Moreover, this will promote the French language and local languages of France by sharing knowledge which has been acquired by public laboratories.
Several teams of the LORIA laboratory contribute to this Equipex, mainly with respect to providing tools for speech and language processing. MULTISPEECH contributes with text-speech alignment and speech visualization tools.
Project acronym: E-FRAN METAL
Project title: Modèles Et Traces au service de l’Apprentissage des Langues
Duration: October 2016 - September 2020
Coordinator: Anne Boyer (LORIA)
Other partners: Interpsy, LISEC, ESPE de Lorraine, D@NTE (Univ. Versailles Saint Quentin), Sailendra SAS, ITOP Education, Rectorat.
Abstract: METAL aims at improving the learning of languages (both written and oral components) through the development of new tools and the analysis of numeric traces associated with students' learning, in order to adapt to the needs and rythm of each learner.
Multispeech is concerned by oral language learning aspects.
Project acronym: ISITE LUE
Project title: Lorraine Université d’Excellence
Duration: starting in 2016
Coordinator: Univ. Lorraine
Abstract: The initiative aims at developing and densifying the initial perimeter of excellence, within the scope of the social and economic challenges, so as to build an original model for a leading global engineering university, with a strong emphasis on technological research and education through research. For this, we have designed LUE as an “engine” for the development of excellence, by stimulating an original dialogue between knowledge fields.
MULTISPEECH is mainly concerned with challenge number 6: "Knowledge engineering", i.e., engineering applied to the field of knowledge and language, which represent our immaterial wealth while being a critical factor for the consistency of future choices. In 2016, this project has funded a new PhD thesis.
Project acronym: ContNomina
Project title: Exploitation of context for proper names recognition in diachronic audio documents
Duration: February 2013 - March 2017
Coordinator: Irina Illina
Other partners: LIA, Synalp
Abstract: The ContNomina project focuses on the problem of proper names in automatic audio processing systems by exploiting in the most efficient way the context of the processed documents. To do this, the project addresses the statistical modeling of contexts and of relationships between contexts and proper names; the contextualization of the recognition module (through the dynamic adjustment of the lexicon and of the language model in order to make them more accurate and certainly more relevant in terms of lexical coverage, particularly with respect to proper names); and the detection of proper names (on the one hand, in text documents for building lists of proper names, and on the other hand, in the output of the recognition system to identify spoken proper names in the audio/video data).
Project acronym: ORFEO
Project title: Outils et Ressources pour le Français Écrit et Oral
Duration: February 2013 - February 2016
Coordinator: Jeanne-Marie DEBAISIEUX (Université Paris 3)
Other partners: ATILF, CLLE-ERSS, ICAR, LIF, LORIA, LATTICE, MoDyCo
Abstract: The main objective of the ORFEO project is the constitution of a corpus for the study of contemporary French.
In this project, we were concerned by the automatic speech-text alignment at the word and phoneme levels for audio files from several corpora gathered by the project. These corpora orthographically transcribed with Transcriber contain mainly spontaneous speech, recorded under various conditions with a large SNR range and a lot of overlapping speech and anonymised speech segments. For the forced speech-text alignment phase, we applied our 2-step methodology (the first step uses a detailed acoustic model for finding the pronunciation variants; then, in the second step a more compact model is used to provide more temporally accurate boundaries).
Project acronym: IFCASL
Project title: Individualized feedback in computer-assisted spoken language learning
Duration: March 2013 - December 2016
Coordinator: Jürgen Trouvain (Saarland University)
Other partners: Saarland University (COLI department)
Abstract: The main objective of IFCASL is to investigate learning of oral French by German speakers, and oral German by French speakers at the phonetic level.
A French-German learner corpus was designed and recorded. French speakers were recorded in Nancy, whereas German speakers were recorded in Saarbrücken. An automatic speech-text alignment process was applied on all the data. Then, the French speech data (native and non-native) were manually checked and annotated in France, and the German speech data (native and non-native) were manually checked and annotated in Germany. The corpora are currently used for analyzing non-native pronunciations, and studying feedback procedures.
Project acronym: DYCI2
Project title: Creative Dynamics of Improvised Interaction
Duration: March 2015 - February 2018
Coordinator: Ircam (Paris)
Other partners: Inria (Nancy), University of La Rochelle
Abstract: The goal of this project is to design a music improvisation system which will be able to listen to the other musicians, improvise in their style, and modify its improvisation according to their feedback in real time.
Project acronym: KAMoulox
Project title: Kernel additive modelling for the unmixing of large audio archives
Duration: January 2016 - January 2019
Coordinator: Antoine Liutkus
Abstract: Develop the theoretical and applied tools required to embed audio denoising and separation tools in web-based audio archives. The applicative scenario is to deal with large audio archives, and more precisely with the notorious "Archives du CNRS — Musée de l'homme", gathering about 50,000 recordings dating back to the early 1900s.
Project acronym: ArtSpeech
Project title: Synthèse articulatoire phonétique
Duration: October 2015 - March 2019
Coordinator: Yves Laprie
Other partners: Gipsa-Lab (Grenoble), IADI (Nancy), LPP (Paris)
Abstract: The objective is to synthesize speech from text via the numerical simulation of the human speech production processes, i.e. the articulatory, aerodynamic and acoustic aspects. Corpus based approaches have taken a hegemonic place in text to speech synthesis. They exploit very good acoustic quality speech databases while covering a high number of expressions and of phonetic contexts. This is sufficient to produce intelligible speech. However, these approaches face almost insurmountable obstacles as soon as parameters intimately related to the physical process of speech production have to be modified. On the contrary, an approach which rests on the simulation of the physical speech production process makes explicitly use of source parameters, anatomy and geometry of the vocal tract, and of a temporal supervision strategy. It thus offers direct control on the nature of the synthetic speech.
Measurements of glottis opening during the production of fricatives via EPGG (ElectroPhotoGlottoGraphy), the design of acoustic experiments with a replica of the vocal tract and the design of dynamic acquisition with MRI were the main activities of this first year.
Project acronym: RAPSODIE
Project title: Automatic Speech Recognition for Hard of Hearing or Handicapped People
Duration: March 2012 - February 2016
Coordinator: eRocca (Mieussy, Haute-Savoie)
Other partners: CEA (Grenoble), Inria (Nancy), CASTORAMA (France)
Abstract: The goal of the project was to realize a portable device to help a hard-of-hearing person to communicate with other people. To achieve this goal the portable device needs to access a speech recognition system, adapted to this task. Another application of the device is environment vocal control for handicapped persons.
In this project, MULTISPEECH was involved in optimizing the speech recognition models for the envisaged task, and in finding the best way of presenting the speech recognition results in order to maximize the communication efficiency between the hard-of-hearing person and the speaking person.
Project acronym: VoiceHome
Duration: February 2015 - July 2017
Coordinator: onMobile
Other partners: Orange, Delta Dore, Technicolor Connected Home, eSoftThings, Inria (Nancy), IRISA, LOUSTIC
Abstract: The goal of this project is to design a robust voice control system for smart home and multimedia applications. We are responsible for the robust automatic speech recognition brick.
Project acronym: Plavis
Project title: Platform for acquisition and audiovisual speech synthesis
Duration: January 2015 - December 2016
Coordinator: Vincent Colotte
Abstract: The objective of this project was to develop a platform acquisition and audiovisual synthesis system (3D animation of the face synchronously with audio). The main purpose was to build a comprehensive platform for acquisition and processing of audiovisual corpus (selection, acquisition and acoustic processing, 3D visual processing and linguistic processing). The acquisition was performed using a motion-capture system (Kinect-like), a Vicon system, and an electromagnetic articulography (EMA) system.
Project title: Acquisition and processing of multimodal corpus in the context of interactive human communication
Duration: June 2015 - May 2016
Coordinator: Slim Ouni
Abstract: The aim of this project was the study of the various mechanisms involved in multimodal human communication that can be oral, visual, gestural and tactile. This project focused on the identification and acquisition of a very large corpus of multimodal data from multiple information sources and acquired in the context of interaction and communication between two people or more.
Project acronym: SYNABE
Project title: Articulatory data synchronization for studying stuttering
Duration: January 2016 - December 2016
Coordinator: Fabrice Hirsch (Praxiling, UMR 5267, Montpellier)
Other partners: S. Ouni
Funding: CNRS DEFI Instrumentation aux limites
Abstract: The objective of this project is to use simultaneously three hardware allowing having information on the subglottic (respiratory belt), glottic (electroglottograph) and supraglottic (articulograph) levels during the production of the speech in order to know the timing of the gestures during speech. This system will be used to study the motor coordination between the three levels mentioned in the stuttering and normo-fluent words. We will propose a new typology of normal and pathological disfluencies.
Our main contribution concerned the articulatory data acquisition using the articulograph AG501.
Jon Barker: University of Sheffield (UK)
Jonathan Le Roux, Shinji Watanabe, John R. Hershey: Mitsubishi Electric Research Labs (MERL, Boston, USA)
Dayana Ribas Gonzalez, Ramón J. Calvo: CENATAV (Habana, Cuba)
STIC-AmSud: MCC - Multimodal Communication Corpus. A collaboration: Argentina, Chile and France (01/2015-12/2016)
Project acronym: MCC
Project title: Multimodal Communication Corpus
Duration: January 2015 - December 2016
International Coordinator: S. Ouni
National Coordinators: Nancy Hitschfeld (Depto. de Ciencias de la Computación (DCC), Universidad de Chile) - Chile; and, Juan Carlos Gomez (Centro Internacional Franco Argentino de Ciencias de la Información y de Sistemas (CIFASIS), UNR, CONICET) - Argentina
Abstract: The project aims to collect a multimodal speech corpus containing synchronized audio-visual data recorded from talking individuals. The corpus will incorporate several communication modes which appear in the communication among humans, such as the acoustic signal, facial movements and body gestures during speech. During 2016, a complete corpus of 8 speakers (4 French and 4 Spanish) has been acquired and processed. The corpus will be distributed using the Ortolang platform.
PHC UTIQUE - HMM-based Arabic speech synthesis, with ENIT (Engineer school at Tunis-Tunisia)
Duration: 2015 - 2018.
Coordinators: Vincent Colotte (France) and Noureddine Ellouze (Tunisia).
Abstract: Development of an HMM-based speech synthesis system for the Arabic language. This includes the development of an Arabic speech corpus, the selection of linguistic features relevant to Arabic HMM-based speech synthesis, as well as improving the quality of the speech signal generated by the system.
Sebastian Gonzales Mora
Date: Jan 2016
Faculty de Cs. Físicas y Matemáticas, University of Chile
Benjamin Martinez Elizalde
Date: May 2016 - Aug 2016
Institution: Carnegie Mellon University (USA)
Dayana Ribas Gonzalez
Date: Sep 2016 - Dec 2016
Institution: CENATAV (Cuba)
Ziteng Wang
Date: Sep 2016 - Sep 2017
Institution: Institute of Acoustics, Chinese Academy of Sciences (China)
General co-chair, 4th CHiME Speech Separation and Recognition Challenge (E. Vincent)
General co-chair, 4th International Workshop on Speech Processing in Everyday Environments, San Francisco, USA, September 2016 (E. Vincent)
General co-chair, 14th International Conference on Auditory-Visual Speech Processing, Stockholm, Sweden August 2017 (S. Ouni)
Chair, SiSEC 2016, Signal Separation Evaluation Challenge (A. Liutkus)
Elected chair, Steering Committee of the Latent Variable Analysis and Signal Separation (LVA/ICA) conference series (E. Vincent)
Chair, Challenges Subcommittee, IEEE Technical Committee on Audio and Acoustic Signal Processing (E. Vincent)
Member of the organizing committee, 2017 IEEE Automatic Speech Recognition and Understanding Workshop, Okinawa, Japan, December 2017 (E. Vincent)
Member of the steering committee, Detection and Classification of Acoustic Scenes and Events (DCASE) challenge series (E. Vincent)
Area chair for Analysis of Speech and Audio Signal, INTERSPEECH'2016 (D. Jouvet)
Area chair for Audio and Speech Source Separation, IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (E. Vincent)
CHiME 2016 - Speech Separation and Recognition Challenge (E. Vincent)
EUSIPCO 2016 - European Signal Processing Conference (D. Jouvet)
ICECS 2016 - Conference on Environmental and Computer Science (R. Serizel)
INTERSPEECH 2016 (A. Bonneau, S.Ouni, E. Vincent, I. Illina, Y. Laprie)
IROS 2016 - International Conference on Intelligent Robots and Systems (E. Vincent)
IVA 2016 - International Conference on Intelligent Virtual Agents (S. Ouni)
JEP 2016 - Journées d'Etudes sur la Parole (D. Jouvet, A. Bonneau, E. Vincent, D. Fohr, I. Illina, O. Mella, Y. Laprie)
SLT 2016 - IEEE Spoken Language Technology Workshop (E. Vincent)
Computer Speech and Language, special issue on Multi-Microphone Speech Recognition in Everyday Environments (E. Vincent)
Speech Communication (D. Jouvet)
Traitement du signal (E. Vincent)
International Journal of Learner Corpus Research, special issue on "Investigating segmental, prosodic and fluency features in spoken learner corpora" (A. Bonneau, Guest Editor)
Speech Communication, special issue on Realism in Robust Speech and Language Processing (E. Vincent)
Computer Speech and Language (D. Jouvet, S. Ouni, E. Vincent)
Digital Signal Processing (E. Vincent)
IEEE Transactions on Audio, Speech and Language Processing (A. Liutkus, S. Ouni, R. Serizel)
IEEE Transactions on Signal Processing (A. Liutkus)
IEEE Signal Processing Letters (A. Liutkus, R. Serizel)
Journal of the Acoustical Society of America (Y. Laprie)
JASA Express Letters (Y. Laprie)
Speech Communication (E. Vincent, Y. Laprie)
Séparation de sources: quand l'acoustique rencontre le machine learning, keynote talk, 13e Congrès Français d'Acoustique (E. Vincent)
Speech recognition, Ecole Nationale d’Ingénieurs de Tunis, May 2016 (D. Jouvet)
Les corpus acoustiques et langagiers pour la reconnaissance de la parole, Seminar about Big Data, LIA-LINOS (Laboratoire International Associé), May 2016 (O. Mella)
Parole audiovisuelle : pour faciliter la communication parlée, Praxiling, Université de Montpellier 3, October 2016 (S. Ouni)
Elected chair, ISCA Special Interest Group on Robust Speech Processing (E. Vincent)
Secretary/Treasurer, executive member of AVISA (Auditory-VIsual Speech Association), an ISCA Special Interest Group (S. Ouni)
Member IEEE Technical Committee on Audio and Acoustic Signal Processing (A. Liutkus)
Expertise of an ANR project proposal (D. Jouvet, Y. Laprie, E. Vincent)
Expertise of a project for the Research Foundation Flanders – FWO (S. Ouni)
Elected Member of the board of the AM2I Scientific Pole - Université de Lorraine (Y. Laprie)
Member of the HCERES visiting committee for LIUM (D. Jouvet)
Chairman of selection committee for the position of Assistant Professor (ESSTIN 0762, May 2016), Y. Laprie.
Member of a selection committee (Université d'Avignon, May 2016), E. Vincent
Member of a selection committee (Université du Maine, May 2016), D. Jouvet
Member of a selection committee (Télécom ParisTech, June 2016), E. Vincent
Member of the "Commission de développement technologique" (A. Bonneau)
DUT: I. Illina, Programming in Java, 150 hours, L1, University of Lorraine, France
DUT: I. Illina, Linux System, 65 hours, L1, University of Lorraine, France
DUT: I. Illina, Supervision of student projects and stages, 50 hours, L2, University of Lorraine, France
DUT: S. Ouni, Programming in Java, 24 hours, L1, University of Lorraine, France
DUT: S. Ouni, Web Programming, 24 hours, L1, University of Lorraine, France
DUT: S. Ouni, Graphical User Interface, 96 hours, L1, University of Lorraine, France
DUT: S. Ouni, Advanced Algorihms, 24 hours, L2, University of Lorraine, France
DUT: R. Serizel, Introduction to computer tools, 108h, L1, University of Lorraine – IUT Nancy Charlemagne, France
Licence: V. Colotte, C2i - Certificat Informatique et Internet, 50h, L1, University of Lorraine, France
Licence: V. Colotte, System, 115h, L3, University of Lorraine, France
Licence: O. Mella, C2i - Certificat Informatique et Internet, 28h, L1, University of Lorraine, France
Licence: O. Mella, Introduction to Web Programming, 30h, L1, University of Lorraine, France
Licence: O. Mella, Computer Networking, 80h, L2-L3, University of Lorraine, France
Licence: A. Piquard-Kipffer, Education Sciences, 36h, L1, France
Licence: A. Piquard-Kipffer, Reading and Writing, 27h, L2, Département Orthophonie, University of Lorraine, France
Licence: A. Piquard-Kipffer, Psycholinguistics, 6 hours, L2 Département Orthophonie, University Pierre et Marie Curie-Paris, France
Licence: A. Piquard-Kipffer, Reading and Writing assessment, 10h, L3, Département Orthophonie, University of Lorraine, France
Master: V. Colotte, Introduction to Speech Analysis and Recognition, 18h, M1, University of Lorraine, France
Master: Y. Laprie, Analyse, perception et reconnaissance de la parole, 32 hours, M1, University of Lorraine, France
Master: O. Mella, Computer Networking, 74h, M1, University of Lorraine, France
Master: O. Mella, Introduction to Speech Analysis and Recognition, 12h, M1, University of Lorraine, France
Master: S. Ouni, Multimedia in Distributed Information Systems, 31 hours, M2, University of Lorraine, France
Master: A. Piquard-Kipffer, Dyslexia, 25 hours, M1, Département Orthophonie, University of Lorraine, France
Master: A. Piquard-Kipffer, Reading and writing, 6 hours, M1, Département Orthophonie, University Pierre et Marie Curie-Paris, France
Master: A. Piquard-Kipffer, Deaf People and Reading, 15 hours, M2 Département Orthophonie, University of Lorraine, France
Master: A. Piquard-Kipffer, Psychology, 40 hours, M2, ESPE, University of Lorraine, France
Master: A. Piquard-Kipffer, French Language Didactics, 80 hours, M2, ESPE, University of Lorraine, France
Engineer school: V. Colotte, Conception and developpement in XML, 20h, Bac+3, Telecom Nancy, France
Ecole d'audioprothèse : A. Bonneau, Phonetics, 16 h, University of Lorraine
Doctorat: A. Piquard-Kipffer, Language Pathology - speech and language screening, 15 hours, EHESP, University of Sorbonne- Paris Cité, France
Adults: O. Mella, Computer science courses for seconday school teachers (Informatique et Sciences du Numérique courses) (21h), ESPE of Academy Nancy-Metz, University of Lorraine, France
Other: V. Colotte, Responsible for "Certificat Informatique et Internet" for the University of Lorraine, France (50000 students, 30 departments)
Other: S. Ouni, Responsible of Année Spéciale DUT, University of Lorraine, France
PhD: Imran Sheikh, "Exploiting Semantic and Topic Context to Improve Recognition of Proper Names in Diachronic Audio Documents", November 2016, Irina Illina, Dominique Fohr and Georges Linares.
PhD in progress: Baldwin Dumortier, "Contrôle acoustique d'un parc éolien", September 2014, Emmanuel Vincent and Madalina Deaconu.
PhD in progress: Quan Nguyen, "Mapping of a sound environment by a mobile robot", November 2014, Francis Colas and Emmanuel Vincent.
PhD in progress: Aditya Nugraha, "Deep neural networks for source separation and noise-robust speech recognition", January 2015, Antoine Liutkus and Emmanuel Vincent.
PhD in progress: Ken Deguernel, "Apprentissage de structures musicales en situation d’improvisation", March 2015, Emmanuel Vincent and Gérard Assayag.
PhD in progress: Amal Houidhek, "Élaboration et analyse d’une base de parole arabe pour la synthèse vocale", December 2015, Denis Jouvet and Vincent Colotte (France) and Zied Mnasri (Tunisia).
PhD in progress: Imène Zangar, "Amélioration de la qualité de synthèse vocale par HMM pour la parole arabe", December 2015, Denis Jouvet and Vincent Colotte (France) and Zied Mnasri (Tunisia).
PhD in progress: Mathieu Fontaine, "Processus alpha-stable pour le traitement du signal", May 2016, Antoine Liutkus and Roland Badeau (Télécom ParisTech).
PhD in progress: Amine Menacer, "Traduction automatique de vidéos", May 2016, Kamel Smaïli and Denis Jouvet.
PhD in progress: Anastasiia Tsukanova, "Coarticulation modeling in articulatory synthesis", May 2016, Yves Laprie.
PhD in progress: Nathan Libermann, "Deep learning for musical structure analysis and generation", October 2016, Frédéric Bimbot and Emmanuel Vincent.
PhD in progress: Yang Liu, "Merging acquisition and processing of cineMRI of the vocal tract", October 2016, Pierre-André Vuissoz and Yves Laprie.
Participation in PhD thesis Jury for David Guennec (Université Rennes 1, September 2016), Y. Laprie.
Participation in PhD thesis Jury for Ugo Marchand (Université Paris 6, November 2016), E. Vincent, reviewer.
Participation in PhD thesis Jury for Joachim Flocon-Cholet (Université Rennes 1, June 2016), E. Vincent, reviewer.
Participation in PhD thesis Jury for Aly Magassouba (Université Rennes 1, December 2016), E. Vincent, reviewer.
Participation in PhD thesis Jury for Diandra Fabre (Université Grenoble Alpes, December 2016), S. Ouni, reviewer.
Participation in PhD thesis Jury for Ivana Didirková (Université Montpellier 3, December 2016), Y. Laprie, reviewer.
Chairman of Scientific « Baccalauréat », specialty Earth Sciences (Académie de Nancy-Metz and Université de Lorraine, July 2016), A. Piquard-Kipffer.
Participation in the Competitive Entrance Examination into Speech-Language Pathology Departement (University of Lorraine, June 2016), A. Piquard-Kipffer.
Demonstration at Village Sciences LORIA, March 2016 (K. Deguernel, E. Vincent, S. Ouni).
Demonstration at Forum des métiers, Collège Peguy, Le Chesnay, March 2016 (A. Piquard-Kipffer).
Demonstration at EHESP-University of Sorbonne- Paris Cité, March 2016 (A. Piquard-Kipffer).
Demonstration at LORIA's 40th Anniversary, June 2016 (K. Deguernel, E. Vincent).
"Démixer la musique", Interstices, January 2016 (A. Liutkus and E. Vincent).
Intervention lors d'une action pour la Maison pour la Science en Lorraine au service des professeurs (A. Bonneau)
Démonstration lors de la journée Rencontre Inria Industrie sur le thème « Ed-Techs au service de e-Education », December 2016 (D. Jouvet)