Node infrastructure:

We have access to three types of GPU node on Jean Zay:12

1. Nodes comprising 4 Nvidia Tesla V100 SXM2 32GB GPUs, 192GB of RAM, and two Intel Cascade Lake 6248 processors. One Nvidia Tesla V100 card is rated at around 300W,13 while the Intel Cascade Lake processor is rated at 150W.14 For the DRAM we can use the work of 129 to estimate the total power draw of 192GB of RAM at approximately 20W. The total power draw of one Jean Zay node at peak use therefore adds up to around 1520W.
2. Nodes comprising 8 Nvidia A100 SXM4 80GB GPUs, 512GB of RAM, and two AMD Milan EPYC 7543 processors. One Nvidia A100 card is rated at around 400W.15 while the AMD Milan processor is rated at 225W.16 Following 129, we estimate the total power draw of 512GB of RAM at approximately 50W. The total power draw of one A100 node at peak use therefore adds up to around 3700W.
3. Nodes comprising 4 Nvidia H100 SXM5 80GB GPUs, 512GB of RAM and two Intel Xeon Platinum 8468 processors. One Nvidia H100 card is rated around 700W 17, while the Intel Xeon Platinum 8468 processor is rated at 300W.18 The total power draw of one H100 node at peak use therefore adds up to around 3450W.

We have recently gained access to the Adastra supercomputer at CINES,19 which has nodes comprising 4 AMD MI250X's and 1 AMD EPYC Trento processor. Adastra uses the same hardware as the LUMI cluster,20 for which the total power draw of one node at peak use has been reported as being approximately 2500W.21

With this information, we use the formula proposed by 156 and compute the total power required for each setting:

Where $c$ and $g$ are the number of CPUs and GPUs respectively, $p_c$ is the average power draw (in W) from all CPU sockets, $p_r$ the average power draw from all DRAM sockets, and $p_g$ the average power draw of a single GPU. We estimate the total power consumption by adding GPU, CPU and DRAM consumption, and then multiplying by the Power Usage Effectiveness (PUE), which accounts for the additional energy required to support the compute infrastructure. We use a PUE coefficient of 1.20, which is the value reported by IDRIS for the Jean Zay supercomputer. For the real time $t$ we have to divide the reported time for each Jean Zay project by 4 for V100 and H100 nodes and 8 for A100 nodes, as Jean Zay reports the computing time of each project in GPU hours and not in per-node hours. In Table 1 we report the training times in hours, as well as the total power draw (in kWh) of each Jean Zay and Adastra project associated with the ALMAnaCH team during 2024. We use this information to compute the total power consumption (multiplying by the PUE) of each project, also reported in Table 1.

We can further estimate the CO${}_2$ emissions in kilograms of each single project by multiplying the total power consumption by the average CO${}_2$ emissions per kWh in France, which were around 27g/kWh on average for 2024.22 The total CO${}_2$ emissions in kg for one single model can therefore be computed as:

All emissions are also reported in Table 1. The total emission estimate for the team is 3,782.1kg of CO${}_2$. The carbon footprint of a single passenger on a round trip from Paris to Mexico City (Boeing 787), flying economy, amounts to around 3,900kg of CO${}_2$.23 This means that our computing emissions from Jean Zay for 2024 amount to just below one Paris-Mexico City trip, and therefore the largest source of emissions from the team are from flights to conferences (e.g. NAACL 2024 in Mexico City), which are a necessary part of communicating about our research to the international community.

7.1.1 HTR-United

• Keywords:
  HTR, OCR
• Functional Description:

  HTR-United is a Github organization without any other form of legal personality. It aims at gathering HTR/OCR transcriptions of all periods and styles of writing, mostly but not exclusively in French. It was born from the mere necessity for projects- to possess potentiel ground truth to rapidly train models on smaller corpora.

  Datasets shared or referenced with HTR-United must, at minimum, take the form of: (i) an ensemble of ALTO XML and/or PAGE XML files containing either only information on the segmentation, either the segmentation and the corresponding transcription, (ii) an ensemble of corresponding images. They can be shared in the form of a simple permalink to ressources hosted somewhere else, or can be the contact information necessary to request access to the images. It must be possible to recompose the link between the XML files and the image without any intermediary process, (iii) a documentation on the transcription practices followed for the segmentation and the transcription. In the cases of a Github repository, this documentation must be summarized in the README.

  A corpus can be sub-diveded into smaller ensembles if it seems necessary.

• Release Contributions:
  First version.
• URL:
  https://­htr-united.­github.­io/
• Contact:
  Alix Chague

7.1.2 VGAMT

• Name:
  Visually Guided and Adapted Machine Translation system
• Keyword:
  Machine translation
• Functional Description:
  Machine translation model that lets the user adding an image as input in addition to the sentence to be translated.
• Publication:
  hal-03977982
• Contact:
  Matthieu Futeral-Peter

7.1.3 CamemBERTa

• Name:
  a DeBERTa v3-based French language model
• Keywords:
  Language model, French
• Functional Description:
  CamemBERTa was initially evaluated on five distinct downstream tasks for French: part-of-speech (POS) tagging, dependency parsing, named entity recognition (NER), FLUE (French Language Understanding Evaluation), including natural language inference (NLI). It improves the state of the art for most tasks compared to previous monolingual and multilingual approaches, which again confirms the effectiveness of large pretrained language models for French. CamemBERTa is particularly effective for high-level tasks.
• URL:
  https://­gitlab.­inria.­fr/­almanach/­CamemBERTa
• Contact:
  Wissam Antoun

7.1.4 CamemBERT-bio

• Keywords:
  Language model, Deep learning, NLP, Transformer
• Functional Description:
  CamemBERT-bio is a state-of-the-art french biomedical language model built using continual-pretraining from camembert-base. It was trained on a french public biomedical corpus of 413M words containing scientific documments, drug leaflets and clinical cases extrated from theses and articles. It shows significant improvement on multiple biomedical named entity recognition tasks compared to camembert-base.
• URL:
  http://­camembert-bio-model.­fr/
• Contact:
  Rian Touchent

7.1.5 RoCS-MT

• Name:
  Robust Challenge Set for Machine Translation
• Keywords:
  Machine translation, NLP, Evaluation, Robustness, User-generated content
• Functional Description:
  RoCS-MT, a Robust Challenge Set for Machine Translation (MT), is designed to test MT systems’ ability to translate user-generated content (UGC) that displays non-standard characteristics, such as spelling errors, devowelling, acronymisation, etc. RoCS-MT is composed of English comments from Reddit, selected for their non-standard nature, which have been manually normalised and professionally translated into five languages: French, German, Czech, Ukrainian and Russian. The challenge set was included as a test suite at the WMT 2023 conference. This repository therefore also includes automatic translations from the submissions to the general MT task.
• URL:
  https://­github.­com/­rbawden/­RoCS-MT
• Publication:
  hal-04300824
• Contact:
  Rachel Bawden

7.1.6 3MT_French Dataset

• Name:
  3 Minutes Thesis Corpus
• Keywords:
  Multimodal Corpus, Video annotation
• Functional Description:
  This new resource will be useful to computer science and social science researchers working on public speaking assessment and training. It will help refine the analysis of speaking from a fresh perspective based on social-cognitive theories rarely studied in this context, such as first impressions and theories of primacy and recency.
• URL:
  https://­zenodo.­org/­records/­7603511#.­Y904l3CZOUk
• Publication:
  https://hal.science/hal-04366763
• Contact:
  Chloe Clavel

7.1.7 CATMuS Medieval (Model)

• Name:
  Consistent Approach to Transcribing ManuScripts - Medieval model
• Keyword:
  Handwritten Text Recognition
• Functional Description:
  CATMuS (Consistent Approach to Transcribing ManuScript) Medieval is a model for automatically transcribing medieval manuscripts using Latin scripts, in particular Old and Middle French, Latin, Spanish (and other languages of Spain), and Italian. The model was trained on the largest and most diverse dataset known for medieval manuscripts in Latin scripts, with more than 110 000 lines of training data.
• Contact:
  Thibault Clerice
• Partners:
  University of Toronto, Ecole nationale des chartes, CIHAM UMR 5648, VeDPH - Ca' Foscari, Université de Genève, ENS Lyon

7.1.8 HTRomance

• Keyword:
  Handwritten Text Recognition
• Functional Description:
  The ground truth produced as part of the HTRomance project aims to provide diverse data, from the 12th century to the 19th century, for training handwritten text recognition models. It covers the following languages: Latin, various states of French, Spanish, Occitan and Italian.
• URL:
  https://­htromance-project.­github.­io/
• Contact:
  Thibault Clerice
• Partners:
  VeDPH - Ca' Foscari, Ecole nationale des chartes, CIHAM UMR 5648, ENS Lyon

7.1.9 OSCAR

• Name:
  Open Super-large Crawled ALMAnaCH coRpus
• Keywords:
  Raw corpus, Multilingual corpus
• Functional Description:

  OSCAR is a huge multilingual corpus obtained by language classification and filtering of the Common Crawl corpus using the goclassy architecture.

  OSCAR is currently shuffled at line level and no metadata is provided. Thus it is mainly intended to be used in the training of unsupervised language models for natural language processing.

  Data is distributed by language in both original and deduplicated form. There are currently 166 different languages available.

• Release Contributions:
  Version 21.09 was generated using Ungoliant version v1, a new generation tool, faster and better documented/tested than the previous one, goclassy, used for OSCAR 1.0 (aka OSCAR 2019). As per OSCAR Schema v1.1, each document/record now has associated metadata. New languages with respect to version 2019: Manx, Rusyn, Scots and West Flemish. Their size and quality still has to be assessed. Removed languages with respect to version 2019: Central Bikol and Cantonese. Cantonsese was of a very low quality. Central Bikol corpus is still available on OSCAR 2019.
• URL:
  https://­oscar-corpus.­com/
• Publications:
  hal-02148693, hal-03301590, hal-03536361, hal-03177623
• Contact:
  Pedro Ortiz Suarez
• Participants:
  Pedro Ortiz Suarez, Benoit Sagot, Julien Abadji

7.1.10 Expresso

• Name:
  Expresso - A Benchmark and Analysis of Discrete Expressive Speech Resynthesis
• Functional Description:
  Expresso est un corpus de parole expressive de haute qualité (48 kHz) qui comprend à la fois de la parole lue de manière expressive (8 styles, au format wav mono) et des dialogues improvisés (26 styles, au format wav stéréo). L'ensemble de données inclut 4 locuteurs (2 hommes, 2 femmes) et totalise 40 heures (11h de lecture, 30h d'improvisation). Les transcriptions de la parole lue sont également fournies. La tâche du Benchmark Expresso est de resynthétiser l'audio entrant en utilisant un code discret à faible débit qui a été obtenu sans supervision à partir du texte.
• Contact:
  Tu Nguyen

7.1.11 SONAR

• Keywords:
  Sentence embeddings, Natural language processing, Multimodality, Speech, Text, Machine translation, Zero-shot
• Functional Description:
  SONAR (for Sentence-level multimOdal and laNguage-Agnostic Representations) is a multilingual and multimodal fixed-size sentence embedding space, with a full suite of speech and text encoders and decoders. It substantially outperforms existing sentence embeddings such as LASER3 and LabSE on the xsim and xsim++ multilingual similarity search tasks. Speech segments can be embedded in the same SONAR embedding space using language-specific speech encoders trained in a teacher-student setting on speech transcription data. We also provide a single text decoder, which allows us to perform text-to-text and speech-to-text machine translation, including for zero-shot language and modality combinations.
• URL:
  https://­github.­com/­facebookresearch/­SONAR
• Publication:
  hal-04264028
• Contact:
  Paul-Ambroise Duquenne

7.1.12 eScriptorium Documentation

• Name:
  Open documentation for eScriptorium
• Functional Description:
  Collaborative and open documentation redacted using the functionalities offered by Github et deployed thanks to Readthedocs. It offers an illustrated description of all the features of the eScriptorium application, which does not offer any other form of complete documentation, as well as tutorials.
• URL:
  https://­escriptorium.­readthedocs.­io/
• Contact:
  Alix Chague

7.1.13 Narabizi Treebank

• Name:
  A multi-layered treebank for the Arabic dialect spoken in North Africa and written in Latin Script
• Keywords:
  Treebank, Named entities, Machine translation, Evaluation, Low resources languages
• Functional Description:
  We introduce the first treebank for a romanized user-generated content variety of Algerian, a North-African Arabic dialect known for its frequent usage of code-switching. Made of 1300 sentences, after deduplication, fully annotated in morpho-syntax and Universal Dependency syntax, with full translation at both the word and the sentence levels, this treebank is made freely available. It is supplemented with 50k unlabeled sentences collected from Common Crawl and web-crawled data using intensive data-mining techniques. Preliminary experiments demonstrate its usefulness for POS tagging and dependency parsing. We believe that what we present in this paper is useful beyond the low-resource language community. This is the first time that enough unlabeled and annotated data is provided for an emerging user-generated content dialectal language with rich morphology and code switching, making it an challenging test-bed for most recent NLP approaches. The current version of the dataset adds two novel annotation layers (named entity recognition and offensive language detection) and a re-annotation of the tokenization, morpho-syntactic and syntactic layers that ensure annotation consistency.
• URL:
  https://­gitlab.­inria.­fr/­ariabi/­release-narabizi-treebank
• Contact:
  Djame Seddah

7.1.14 CATMuS Medieval (Dataset)

• Name:
  Consistent Approaches to Transcribing ManuScripts - Medieval Dataset
• Keywords:
  Handwritten Text Recognition, HTR, OCR
• Functional Description:
  Developed through collaboration among various institutions and projects, CATMuS provides an inter-compatible handwritten text recognition dataset spanning more than 240 manuscripts and incunabula in 10 different languages, comprising over 170,000 lines of text and 5 million characters spanning from the 8th century to the 16th.
• Release Contributions:
  - 40 new mansucripts
• Publication:
  https://inria.hal.science/hal-04453952
• Contact:
  Thibault Clerice
• Partners:
  CIHAM UMR 5648, Ecole nationale des chartes, University of Toronto, Antwerp University

7.1.15 Counter dataset

• Name:
  Counter Radicalization Dataset
• Keywords:
  NLP, Multilingual corpus, Data protection, Pseudonymization, Online radicalization
• Functional Description:
  This dataset includes multilingual content from forums, Telegram, social media, and other sources in English, French, and Arabic. It covers various radical ideologies and is pseudonymized to protect privacy while maintaining data utility. It includes annotations for call to action, radicalization level, and named entity recognition.
• URL:
  https://­gitlab.­inria.­fr/­ariabi/­counter-dataset-public
• Contact:
  Djame Seddah

7.1.16 CUBANSPVARIETY

• Keywords:
  Variety identification, NLP
• Functional Description:
  A Cuban Spanish variety identification dataset consisting of 1,762 manually annotated tweets by three native speakers, with labels assigned based on agreement and covering Cuban and Cuban varieties and common examples.
• Contact:
  Djame Seddah

7.1.17 CamemBERTav2

• Keywords:
  Language model, French
• Functional Description:

  It is the second version of the CamemBERTa model, which is based on the DebertaV2 architecture with the Replaced Token Detection (RTD) objective.

  The new update includes: 1) Much larger pretraining dataset: 275B unique tokens (previously 2B). 2) A newly built tokenizer based on WordPiece with 32,768 tokens, addition of the newline and tab characters, support emojis, and better handling of numbers (numbers are split into two digits tokens). 3) Extended context window of 1024 tokens

• Contact:
  Benoit Sagot

7.1.18 CamemBERTv2

• Keywords:
  Language model, French
• Functional Description:
  CamemBERTv2 is a French language model pretrained on a large up-to-date corpus of 275B tokens of French text. The model still based on the BERT architecture, demonstrates improved performance across a variety of NLP tasks, over the previous version
• URL:
  https://­huggingface.­co/­almanach/­camembertv2-base
• Contact:
  Benoit Sagot

7.1.19 HaSCoSVa

• Name:
  Hate Speech Corpus for Spanish
• Keywords:
  Hate Speech Detection, NLP
• Functional Description:
  A new corpus of tweets related to hate speech towards immigrants written in Spanish. This corpus contains information regarding the language variant. The dataset is subdivided into two subsets according to the language variant: (1) Latin American and (2) European.
• URL:
  https://­gitlab.­inria.­fr/­counter/­HaSCoSVa
• Contact:
  Djame Seddah

7.1.20 LADaS

• Name:
  Layout Analysis Dataset with Segmonto
• Keyword:
  Layout Analysis
• Functional Description:
  LADaS is a dataset for training layout analysis on documents from the 16th to the 21st century. It helps reproduce documents in XML TEI.
• Publication:
  hal-04513725
• Contact:
  Thibault Clerice

7.1.21 CamemBERT-bio-gliner

• Keywords:
  Language model, Deep learning, NLP, Transformer
• Functional Description:
  CamemBERT-bio-gliner is a Named Entity Recognition (NER) model capable of identifying any french biomedical entity type using a BERT-like encoder. It provides a practical alternative to traditional NER models, which are limited to predefined entities, and Large Language Models (LLMs) that, despite their flexibility, are costly and large for resource-constrained scenarios. CamemBERT-bio is used as a backbone.
• Contact:
  Rian Touchent

7.1.22 CoMMuTE

• Name:
  Contrastive multilingual and multimodal translation evaluation
• Keywords:
  Machine translation, Evaluation, Image analysis
• Functional Description:
  CoMMuTE is a contrastive evaluation dataset designed to assess the ability of multimodal machine translation models to exploit images in order to disambiguate the sentence to be translated. In other words, given a sentence containing a word that can be translated in several ways, the additional image determines the meaning of the word to be translated. The model must then take the image into account to propose a correct translation. CoMMuTE is available from English into French, German and Czech.
• URL:
  https://­github.­com/­MatthieuFP/­CoMMuTE
• Contact:
  Matthieu Futeral-Peter

7.1.23 mOSCAR

• Keywords:
  Raw corpus, Multilingual corpus, Multimodal Corpus, Multimodality, Text-image processing, Web crawling
• Functional Description:
  mOSCAR is a the first large-scale multilingual and multimodal web-crawled document corpus. It covers 163 languages, 315M documents, 214B tokens and 1.2B images. We carefully conduct a set of filtering and evaluation steps to make sure mOSCAR is sufficiently safe, diverse and of good quality.
• URL:
  https://­oscar-project.­github.­io/­documentation/­versions/­mOSCAR/
• Publication:
  hal-04629451
• Contact:
  Matthieu Futeral-Peter
• Participants:
  Matthieu Futeral-Peter, Benoit Sagot, Rachel Bawden, Julien Abadji, Armel Zebaze Dongmo, Cordelia Schmid, Remi Lacroix

8.1.1 OSCAR and mOSCAR: Collecting Corpora from the Web

Since the introduction of large language models (LLMs) in Natural Language Processing (NLP), large raw corpora have played a crucial role. However, most of these large raw corpora are either available only for English or not available to the general public due to copyright issues. There are some examples of freely available multilingual corpora for the training of deep learning NLP models, such as the Paracrawl corpus 125 and our own large-scale multilingual corpus OSCAR 150, 121, 120.24 However, they have quality issues, especially for low-resource languages, an issue investigated in a large-scale study we were involved in and whose initial publication in 2021 139 was followed by a publication in the Transactions of the Association for Computational Linguistics140. The latest OSCAR version is OSCAR 23.01.

In 2024, we worked on the development of a new, multimodal version of OSCAR, mOSCAR (for multimodal OSCAR). This works was carried out as part of Matthieu Futeral-Peter 's PhD thesis, a joint effort between the ALMAnaCH and WILLOW teams (Cordelia Schmid, from WILLOW, being one of Matthieu's supervisors). It is also the result of a collaboration with engineers working at GENCI on the Jean Zay supercomputer. The goal of the project, initiated in 2023, was to address the lack of publicly available interleaved text-image data for languages beyond English. We automatically collected a large number of images from the internet associated with snippets of text (some captions but also other types of associated texts related to the image from accompanying documents). Unlike existing datasets, which are either English-only, composed of caption-like data, or fully private, mOSCAR covers 163 languages and consists of 303 million documents, 200 billion tokens, and 1.15 billion images. We conducted extensive filtering and evaluation to ensure that the dataset is diverse, safe, and of high quality 105. The dataset is already publicly accessible under the Creative Commons CC BY 4.0 license, promoting further research in multilingual multimodal learning.25

8.1.2 COLaF: Collecting high-quality text corpora for French and other languages of France

We finalised the creation of COLaF (“Corpus et Outils pour les Langues de France” — corpora and tools for the languages of France), an Inria “DEFI” (Inria-internal multi-team project) jointly led by Benoît Sagot and Slim Ouni. COLaF is carried out in close interaction with the (informal) OSCAR project and in collaboration with our colleagues from the MULTISPEECH team in the Nancy Inria Centre (Emmanuel Vincent, Slim Ouni) and the former ALMAnaCH member Laurent Romary, as well as with the Inria headquarters' support (in particular Jean-Frédéric Gerbeau). Its goal is to contribute to the development of free corpora and tools for French and French-based Creole, as spoken in France and abroad, as well as of other languages of France, in close collaboration with academic and institutional partners.

In 2024 we worked on multiple research directions in the context of COLaF:

• We strengthened and initiated interactions with a number of relevant external partners, including:

  • language-specific institutions, including Académie Régionale de la Langue Picarde (Picard), Lo Congrès (Occitan), and, since 2024, The Historic New Orleans Collection (Louisiana French, Louisiana Creole) and the Institut du Monde Réunionnais (Réunion Créole).
  • research laboratories such as LILPa (Université de Strasbourg and CNRS, Alsacian and other languages), LISA (Université de Corse and CNRS, Corsican), MoDyCo (Université Paris-Ouest and CNRS, Breton and other languages), and, since 2024, Tulane University (Louisiana).

  In May we organised a kick-off meeting attended by over 50 people representing most of these partners and others.

• We finalised our work on text corpus representation based on the TEI guidelines, and published a first version of our COLaF TEI metadata scheme and the associated documentation (Web site, GitHub repository).
• We resumed our work on language identification, at the intersection between the OSCAR and COLaF projects, in the context of Rasul Dent's newly started PhD (funded by COLaF) and with a focus on French-based Creole languages. Our first result, the open corpus Molyé 47, combines stereotypical representations of three kinds of language variation in Europe with early attestations of French-based Creole languages across a period of 400 years. It is intended to facilitate future research on the continuity between contact situations in Europe and Creolophone (former) colonies. New results in this direction will be published in 2025. This work is being carried out in close collaboration with Pedro Ortiz, a former ALMAnaCH PhD student 147 now working for Common Crawl.
• We have been working on collecting Occitan corpora and improving the state of the art in Occitan dialect identification and Occitan part of speech tagging, as part of Oriane Nédey 's engineer contract and subsequent PhD thesis.
• In the context of the ANR TraLaLaM project, and more precisely as a side subproject of Malik Marmonier 's engineering work, we carried out preliminary work on the identification of resources for Franco-Provençal (also known as Arpitan, not to be confused with Provençal, a variety of Occitan).
• We developed a first version of our database of language resources for French, other languages of France and French-based Creoles, as well as a bibliography on COLaF-related topics.

8.2.1 Training language models

Pretrained language models are now ubiquitous in NLP. Despite their success, many early models were either trained on English data or on the concatenation of data in multiple languages 130, 141. One of the most visible achievements of the ALMAnaCH team was the training and release of CamemBERT in 2019, a BERT-like 130 (and more specifically a RoBERTa-like) neural language model for French trained on the French section of our large-scale web-based OSCAR corpus 150 (see Section 8.1.1) and still downloaded several million times a month. In previous years we also developed CamemBERT variants 143, ELMo models trained on OSCAR corpora for other languages, including French 148, 149, as well as (more recently) CamemBERTa 123, an alternative of CamemBERT based on the DeBERTaV3 architecture 138 that further improved the state of the art for French NLP.

However, models published several years ago, like CamemBERT, face challenges due to temporal concept drift, where outdated training data leads to a decline in performance, especially when encountering new topics and terminology. This issue emphasizes the need for updated models that reflect current linguistic trends. In 2024 we introduce two new versions of the CamemBERT and CamemBERTa base models, named CamemBERTav2 and CamemBERTv2, designed to address these challenges 95. CamemBERTav2, like CamemBERTa, is based on the DeBERTaV3 architecture. It makes use of the Replaced Token Detection (RTD) objective for better contextual understanding. CamemBERTv2 is built on RoBERTa, like CamemBERT, which uses the Masked Language Modeling (MLM) objective. We trained both models on a significantly larger and more recent dataset with longer context length and an updated tokeniser that enhances tokenisation performance for French. We demonstrated the versatility of these models and their effectiveness across a range of use cases. Our results show that these updated models vastly outperform their predecessors, making them valuable tools for modern NLP systems. All our new models, as well as intermediate checkpoints, are made openly available on Huggingface.

8.2.2 Understanding language models

Another crucial research direction is gaining a better understanding of how language models actually work and the properties of the vector representations they produce. This is the focus of Nathan Godey 's PhD work as well as one of the research topics of Aina Garí Soler . In this context, we worked on two different directions.

• Language models have long been shown to embed geographical information in their hidden representations. This line of work has recently been revisited in the context of LLMs. In a study published this year 53, we proposed to fill the gap between well-established and recent literature by observing how geographical knowledge evolves when scaling language models. We showed that geographical knowledge is observable even for tiny models and that it scales consistently as we increase the model size. Notably, we observed that larger language models cannot mitigate the geographical bias that is inherent to the training data.
• It has been observed that smaller language models can suffer from saturation, characterised as a drop in performance at some advanced point in training followed by a plateau. In NLP, it takes the form of anisotropy, a singular property of hidden representations that makes them unexpectedly close to each other in terms of angular distance (cosine-similarity). Some recent works tend to show that anisotropy is a consequence of optimising the cross-entropy loss on long-tailed distributions of tokens. Following our 2023 work on model anisotropy 137, we showed that anisotropy can also be observed empirically in language models with specific objectives that should not suffer directly from the same consequences 52. We also showed that the anisotropy problem extends to Transformers trained on other modalities. Our observations tend to demonstrate that anisotropy might actually be inherent to Transformers-based models. Investigating further, we found that such saturation can be explained by a mismatch between the hidden dimension of smaller models and the high rank of the target contextual probability distribution 106. This mismatch affects the performance of the linear prediction head used in such models through the well-known softmax bottleneck phenomenon. We measured the effect of the softmax bottleneck in various settings and find that models based on fewer than 1000 hidden dimensions tend to adopt degenerate latent representations in late pretraining, which leads to reduced evaluation performance.
• As part of Aina Garí Soler 's postdoctoral research, under the supervision of Chloé Clavel and Matthieu Labeau (Telecom-Paris), we investigated how subword-based tokenisation impacts word representations. When deriving contextualised word representations from language models, a decision needs to be made on how to obtain representations for out-of-vocabulary (OOV) words that are segmented into subwords. What is the best way to represent these words with a single vector, and are these representations of worse quality than those of in-vocabulary words? We carried out an intrinsic evaluation of embeddings from different models on semantic similarity tasks involving OOV words. We compared the contextualised representations of words that are segmented into subwords with those of words that have a dedicated embedding in BERT and other models through an intrinsic evaluation relying on similarity estimation 30. Our findings are relevant for any NLP practitioner working with contextualised word representations, and particularly for applications relying on word similarity. We showed that, out of the tested strategies for split-word representation, averaging subword embeddings is the best one, with few exceptions, the quality of split-word representations is often worse than that of full-words, although this depends on the kind of words considered, similarity values obtained for split-word pairs are generally higher than similarity estimations involving full-words, the best layers to use differ across split-types, a higher number of tokens does not necessarily, as intuitively thought, decrease representation quality and, in the within-word setting, word form has a negative impact on results when words are split. Our results also point to specific aspects to which future research and efforts of improvement should be directed. We make our split-sim dataset available to facilitate research on split-word representation.

Understanding how LLMs can make use of potentially complex and/or structured prompts is another research direction aimed at finding ways to improve their performance. For complex reasoning tasks that require step-by-step thinking, Chain-of-Thought (CoT) prompting has given impressive results, especially when combined with an approach called self-consistency. Nonetheless, some tasks remain particularly difficult for LLMs to solve. Tree of Thoughts (ToT) 160 and Graph of Thoughts (GoT) 127 emerged as alternatives, dividing complex problems into paths of subproblems. In 2024, we proposed a new type of in-context learning, which we called Tree of Problems (ToP), a simpler version of ToT, which improves performance on complex tasks that can be divided into identical subtasks 72. Our empirical results show that our approach outperforms ToT and GoT, and in addition performs better than CoT on complex reasoning tasks.

8.3.1 Shared Tasks, Benchmarking and Evaluation

As in previous years, we continued our participation in the organisation of the shared tasks at the main conference in MT (WMT). Rachel Bawden was again member of the organising committees of the general MT task 59, 107 and the biomedical MT task 64, whose purpose is to evaluate models on multiple domains and on the biomedical domain respectively. With the continuing use of LLMs for MT, both tasks studied some of the new challenges linked to the use of LLMs for translation.

In terms of benchmarking, there were a few other contributions other than the shared tasks. Rachel Bawden participated in an initiative led by Jesujoba Alabi, a previous engineer in the team and now a PhD student at Saarland University, on the creation of a docment-level corpus for African languages, AfriDoc, which was then used to evaluate LLM performance at the document level 94. She also continued her contribution to the benchmarking of the BLOOM language model through an informal initiative “Bloume” for the evaluation of the LLM specifically for French. Several tasks, including MMT were included in the evaluation 41.

8.3.2 Low-resource MT

Improving low-resource MT by understanding how LLMs can take advantage of analogies is the topic of Armel Zebaze's PhD, supervised by Rachel Bawden and Benoît Sagot. He has been studying the use of different language models for few-shot example selection, particularly to help the translation of low-resource language pairs. The ability of generative LLMs to perform in-context learning has given rise to a large body of research into how best to prompt models for various NLP tasks. For low-resource MT, no systematic studies had been published on how best to select examples, and mixed results had been reported on the usefulness of similarity-based selection over random selection. We published a study covering multiple LLMs and multiple in-context example retrieval strategies, comparing multilingual sentence embeddings 114. We covered several language directions, representing different levels of language resourcedness (English into French, German, Swahili and Wolof). Contrarily to previously published results, we found that sentence embedding similarity can improve MT, especially for low-resource language directions, and studied the balance between selection pool diversity and quality. We also highlighted potential problems with the evaluation of LLM-based MT (dealing with copying of the source text, empty translations and translating in the wrong language) and suggested a more appropriate evaluation protocol, adapting the COMET metric to the evaluation of LLMs.

Capitalising on our work on Tree of Problems (cf. Section 8.2) we developed a new LLM-based translation paradigm, named compositional translation, which relies on the notion of analogy in a different way 113. Compositional translation uses an LLM to decompose a sentence into simpler phrases (or segments), and then to translate each phrase with the help of retrieved demonstrations. Finally, the LLM is prompted to translate the initial sentence with the help of the self-generated phrase-translation pairs, thereby relying on another way to characterise analogy in translation. Our intuition was that this approach should improve translation because these shorter, simpler phrases should be intrinsically easier to translate and easier to match with relevant examples. This proves especially beneficial in low-resource scenarios, and more generally whenever the selection pool is small or out of domain. We showed that compositional translation boosts LLM translation performance on a wide range of popular MT benchmarks, including FLORES 200, NTREX 128 and TICO-19.

Another approach to LLM-based low-resource MT is to explore whether LLMs can benefit from an explicit description of the low-resource language (such as a grammar, a monolingual lexicon, or a bilingual lexicon) in order to facilitate translation to or from that language. In the context of the TraLaLaM project, we investigated the capacity of LLMs to engage in explicit learning of a new, unseen language, i.e. assimilating metalinguistic explanations, to perform language tasks 109. Using constructed languages derived from French using cryptographic methods as controlled test environments, we designed experiments to evaluate how well an LLM can learn and apply grammatical rules. Our findings indicate that while LLMs demonstrate a measurable ability for explicit learning, contrarily to what previous authors published 124, this capability diminishes as the complexity of the linguistic phenomena increases. Supervised fine-tuning on chains of thought significantly enhances LLM performance but struggles to generalise to typologically novel or more complex linguistic features.

8.3.3 MT applied to specific domains

One of the major challenges for the development of high quality MT models is adapting them to specific domains (including scientific domains), or even ideally to multiple domains at once. We have made several contributions to MT for specific domains this year.

In the context of the DadaNMT project led by Rachel Bawden , which ended at the end of 2023, in we finalised and published several works in 2024 investigating how to improve MT for domain adaptation and cross-domain transfer. Firstly, we studied how incorporating bilingual lexicons could help MT of specific domains by using a technique that is commonly used in domains with a controlled language (i.e. with no ambiguity): inline injection of candidate translations within the source sentence 132. We sought to see whether this technique could be successfully used in the general MT setting (where there is ambiguity and the lexicon is not specific to the specific language of the domain) to help better translate specific domains. Working with a strong encoder-decoder model (mT5 finet-uned on parallel data) and using the test case of German-to-English, we show that the technique does not result in the same improvements as previously shown. However, in our analysis using distractor annotations, we show that the model can learn to select appropriate translation candidates and ignore irrelevant ones, thereby exhibiting more than a systematic copying behaviour, particularly in a low- or mid-resource setting 35. In a follow-up, we moved to LLMs and how to specialise them to specific domains through in-context example selection and topic models. We tested the relevance of topic models by using them to select informative examples even for out-of-domain inputs, experimenting on 7 diverse domains and 11 language pairs of differing resourcedness and comparing performance to the use of domain labels and keywords. Our results showed that few-shot examples and related keywords consistently improved translation quality, that example diversity must be balanced with source similarity, and that our pipeline was overly restrictive for example selection when a targeted development set was available. However, this work provided a starting point for later work carried out on analogy for low-resource MT int he context of Armel Zebaze Dongmo 's PhD (see Section 8.3.2).

A large part of our work in domain-specific MT has been on the translation of scientific documents in the context of the MaTOS (Machine Translation for Open Science) ANR project, led by François Yvon (CNRS). There is a specific focus on English–French and French–English translation, since the high quality of MT for these directions allows us to work on more high-level challenges such as term translations, consistency and document coherence. Two PhD students are currently co-supervised by Rachel Bawden jointly with François Yvon in the projet: Ziqian Peng (recruited by the CNRS) on document-level MT for scientific documents and Nicolas Dahan on the evaluation of MT. As part of the project, we published two surveys on each of these topics 110, 102 and trained MT models for the translation of documents in the NLP and Earth and Planetary Science domains 112. One of the major challenges for the translation of scientific documents is the translation of documents in one go (rather than sentence by sentence) to ensure consistency in the translation of terms and to maintain document coherence. MT models tend to struggle to translate longer segments, a challenge we studied in the context of Ziqian Peng 's PhD, looking at how we can evaluate the effect of document length on translation performance and what effect it has. Through carefully controlled experiments, we showed that translation performance decreases with the length of the input text and that translation quality depends on the position within the document, i.e. sentences at the beginning are better translated than those at the end 75, 111.Having identified this challenge, we studied an approach to improve the translation of longer segments and of later positions within source texts by manipulating the distribution of positions within documents during training so that all positions up to a maximum length are seen approximately the same number of times (i.e. exposing the model to later positions as much as earlier ones). Our results showed that the approach only marginally mitigates such problems and that document-level MT does not yet match the performance of sentence-based MT 111. In the context of Nicolas Dahan 's PhD, we continued to look at ways of evaluating document-level consistency and coherence when translating. In research inspired by the use of contrastive test examples, consisting of correct and incorrect translations, we automatically generate contrastive examples designed to stress test different metrics on potential errors of document MT systems (e.g. truncating translations, shufflinf sentences), document-level effects to which metrics should be robust (splitting sentences, use of synonyms) and other challenges linked to consistency and coherences. Testing on English–French, we evaluated several MT metrics, using multiple segmentation strategies and span lengths. Our experiments showed that some metrics highly sensitive to local changes under-estimate long-range disruptions, and vice versa and that certain perturbations (e.g., related to coherence and repetition) reveal shortcomings in metrics that would appear robust under sentence-level evaluations. Finally, we also published our study on postediting automatic translations, analysing the performance of the different models and studied the differences between postedits produced by professional translators and those produced by authors in the NLP community 42, and we extended our study to the Earth and Planetary Sciences domain.

8.3.4 MT Applied to Non-standard Texts

User-generated content (UGC) such as texts found on social media are characterised by various phenomena not typically present in standard edited texts and which present challenges for MT (e.g. spelling mistakes, acronyms, truncations, and contractions). It is important to develop MT models that are robust, which means they are able to translate these kinds of text just as well as if the texts had not displayed non-standard variation.

In the context of Lydia Nishimwe 's PhD on robust MT, supervised by Rachel Bawden and Benoît Sagot , we have been working on building robust sentence embeddings, using the test case of LASER, and, inspired by T-modules (research carried out in the context of Paul-Ambroise Duquenne's PhD), using distillation to reduce the distance between non-standard sentences and their normalised versions. We introduced RoLASER, a robust English encoder trained using a teacher-student approach to reduce the distances between the representations of UGC sentences and the LASER representations of their standard counterparts 67. We showed that with training only on standard and synthetic UGC-like data, RoLASER significantly improves LASER's robustness to both natural and artificial UGC data by achieving up to 2x and 11x better scores. Evaluation on downstream tasks showed that RoLASER performs comparably to or better than LASER on standard data, while consistently outperforming it on UGC data. Since then, we have extended the approach to SONAR 133, the follow-up to LASER, thereby creating RoSONAR, and we have carried out experiments to see whether RoSONAR can be used to create robust and modular MT architectures.

8.3.5 Multimodal MT

In the context of Matthieu Futeral-Peter 's PhD thesis, co-supervised by Rachel Bawden , Benoît Sagot , and Cordelia Schmid (WILLOW project-team), we have continued work on multimodal (image-text) MT. Our focus has remained on exploiting visual context to address ambiguity in translation, which is a fundamental challenge.

In 2024, our research resulted in two major publications. The first paper involves mOSCAR, our new large-scale multilingual and multimodal document corpus (see Section 8.1.1). We demonstrated that training multilingual models on mOSCAR significantly improves few-shot learning capabilities across various multilingual image-text tasks, reinforcing the importance of diverse multimodal data for multimodal LLMs 105.

Our second major contribution in 2024, ZeroMMT 104, introduced a method to train multimodal MT systems without requiring fully supervised multimodal parallel data (i.e., sentence pairs with corresponding images). Instead, we adapted a strong text-only MT model using a joint training objective: visually conditioned masked language modeling and a Kullback-Leibler divergence loss that encourages the MMT model's outputs to remain consistent with the original MT model while integrating visual cues. This approach allows for effective learning from multimodal English data alone, making it applicable to language pairs with no existing multimodal training corpora. ZeroMMT demonstrated disambiguation performance close to state-of-the-art multimodal MT models trained on fully supervised data, as evaluated on standard MMT benchmarks and our contrastive evaluation set CoMMuTE 135. To further validate its generalisation capabilities, we extended CoMMuTE to three new languages: Arabic, Russian, and Chinese. Additionally, we introduced an inference-time trade-off mechanism using classifier-free guidance, allowing for a controlled balance between translation fidelity and disambiguation strength without requiring additional data.

8.5.1 Identifying Common Examples in Spanish Dialects in Hate-Speech Contexts

Our previous research on multilingual zero-shot hate speech detection 145 highlighted the cultural gap in cross-lingual settings. For instance, the Spanish word puta `prostitute', is often used as a non-offensive intensifier but was flagged as offensive by models trained on an English dataset that focuses on misogyny. While such cultural differences across languages are increasingly studied, their impact on dialects of the same language remains unclear. After our work on hate speech contrasting different varieties of Spanish spoken either in South America or in Spain 128, we extended our approach, this time focusing on the Cuban Spanish variant and the identification of cross-dialect common examples.

As mentioned above, variations in languages across geographic regions or cultures are crucial to addressing biases in NLP systems designed for culturally sensitive tasks, such as hate speech detection or dialogue with conversational agents. In languages such as Spanish, where varieties can significantly overlap, many examples can be valid across varieties, which we refer to as common examples. Ignoring these examples may cause misclassifications, reducing model accuracy and fairness. Accounting for these common examples is therefore essential to improving the robustness and representativeness of NLP systems trained on such data. We used training dynamics to automatically detect common examples or errors in existing Spanish datasets. We demonstrated the effectiveness of using predicted label confidence for our Datamaps 157 implementation for the identification of hard-to-classify examples, especially common examples, enhancing model performance in a variety of identification tasks. Besides the original use of the Datamaps for this task, an interesting outcome lies in the introduction of a Cuban Spanish Variety Identification dataset with common example annotations developed to facilitate more accurate detection of Cuban and Caribbean Spanish varieties. To our knowledge, this is the first dataset focused on identifying the Cuban, or any other Caribbean, Spanish variety. After a first preprint in Autumn of 2024, this work was published in the VARDIAL specialised workshop, colocated with COLING 36.

8.5.2 The CounteR Dataset

The first half of the CounteR project faced an almost fatal challenge, namely the lack of availability of a multilingual radical content dataset covering the project's target ideology spectrum (from white-supremacist to Jihadism). This is why we had to first develop our multitask learning architecture on adjacent domains such as hate speech before being able to focus on real radical content suitable for the law enforcement agencies that were part of the project and designed to be our end-users. Such data was made available to us by an EU-external third-party contractor, validated by the European Commission project officer, over an 18-month period during the second half of the project. The final dataset covers 12 languages, some of which were actually provided by the LEAs themselves. ALMAnaCH was responsible for the whole NLP work package (“Data Analytics for Detecting Radical Content”), which spanned the entire duration of the project. Due to the UE classified nature of our core deliverables (D4.3 “NLP Basic Features” and D4.4 “transfer learning for NLP”) we cannot present our results on cross-lingual transfer and out-of-domain scenarios on radical content detection here. As per the consortium agreement, the only data that could be released would have to be fully GDPR-compliant, entailing a perfect pseudo-anonymisation process. Therefore, given the extent of the work needed for this task, we focused on a subset of languages comprising French, Arabic and English. The next section, Section 8.5.2, presents this work, the analysis we carried out, as well as the challenges of using such dataset in classification tasks, are presented in Section 8.5.2.

8.5.3 Miscellaneous

We adapted the Narabi Treebank NER annotation scheme so it could be part of the Universal NER framework. This work led to a publication at the NAACL 2024 conference 60.

8.6.1 Biomedical NLP

In the context of the BPI-funded project ONCOLAB, we explored the application of NLP techniques to the medical domain. The medical domain is a very specialised one, and clinical documents (EHR – Electronic Health Records) are very sensitive data and not readily available, especially for French. During his Master's thesis, Rian Touchent tested several approaches to collect a dataset of French biomedical documents (biomed-fr) and use it to continue the pretraining of our CamemBERT language model, leading to CamemBERT-bio 159, 71, one of the first French language models specialised for the biomedical domain. Continual pre-training ensured a very low environmental impact (compared to a pre-training from scratch) and CamemBERT-bio exhibits state-of-the-art performance on several French biomedical benchmarks.

The lack of biomedical data and especially annotated biomedical data also led Simon Meoni to explore the use of LLMs (in particular Instruct-GPT) to automaticaly annotate medical entities in the E3C corpus (without any prior fine-tuning) in the context of his CIFRE PhD (with the company Arkhn), supervised by Éric Villemonte de La Clergerie for ALMAnaCH. In 2024, to overcome the issue of the necessary confidentiality of medical data, which leads to a lack of any freely available datasets, he explored the generation of synthetic documents using keywords extracted from real health records that do not contain confidential information 61. Furthermore, we introduced a reward mechanism that iteratively refines the quality of synthetic documents. This involves scoring synthetic candidates against real clinical reports using a semantic textual similarity score and performing an alignment step to align the model with its best-scored utterances.

8.6.2 NLP for Patents

As part of You Zuo 's CIFRE PhD, in collaboration with the start-up qatent acquired by Questel in 2024, supervised by Benoît Sagot and Éric Villemonte de La Clergerie for ALMAnaCH and Kim Gerdes for qatent (see Section 9), we have broadened our research into NLP for patents, with a focus on automatic patent generation and computer-aided patent writing. In particular, we published a comparative evaluation and analysis of language models for the task of patent generation, focusing on two different tasks 73: (i) the generation of abstract from claims and (ii) the generation of claims given previous ones. We developed a benchmark, PatentEval, and a comprehensive error typology for both tasks. We manually annotated various models, including those specific to the patent domain as well as general-purpose language models. In addition, we explored and evaluated several metrics to approximate human judgments in patent text evaluation, analysing the extent to which these metrics align with expert assessments. These approaches provide valuable insights into the capabilities and limitations of current language models in the specialized field of patent text generation.

8.10.1 Story Generation Evaluation

In the context of Cyril Chhun 's thesis, co-supervised by Chloé Clavel and Fabian Suchanek (Telecom-Paris), we examine automatic story evaluation. Storytelling is an integral part of human experience and plays a crucial role in social interactions. Automatic Story Evaluation (ASE) and Generation (ASG) could therefore benefit society in multiple ways, but they are challenging tasks that require high-level human abilities such as creativity, reasoning, and deep understanding. Meanwhile, LLMs now achieve state-of-the-art performance on many NLP tasks. In 2024, we studied whether LLMs can be used as substitutes for human annotators for ASE. We performed an extensive analysis of the correlations between LLM ratings, other automatic measures, and human annotations, and we explored the influence of prompting on the results and the explainability of LLM behaviour 25. Most notably, we found that LLMs outperform current automatic measures for system-level evaluation but still struggle at providing satisfactory explanations for their answers.

8.10.2 Linguistic Diversity

In the context of Yanzhu Guo 's PhD thesis, under the supervision of Chloé Clavel , we examine the linguistic diversity of LLM outputs. While LLM evaluation has gained significant attention in recent years, most research has focused on assessing the task-solving abilities of models. However, this focus often neglects whether machine-generated language matches the human level of diversity, in terms of vocabulary choice, syntactic construction, and expression of meaning, raising questions about whether the fundamentals of language generation have been properly addressed. Our study emphasises the importance of preserving the linguistic richness of human language, given the concerning surge in online content produced or aided by LLMs.

In the first part of our study, we proposed a comprehensive framework for evaluating LLMs from various linguistic diversity perspectives including lexical, syntactic, and semantic dimensions. Using this framework, we benchmarked six state-of-the-art LLMs across three diversity dimensions on five NLG tasks, and conducted an in-depth case study for syntactic diversity. We also examined how various development and deployment strategies for LLMs influence the linguistic diversity of their outputs. Our findings reveal a significant gap in replicating the linguistic richness of human language, particularly in more creative tasks, despite the impressive capabilities of LLMs in generating coherent and contextually appropriate text. Specifically, we observed that factors such as model scale, training data volume, and finetuning techniques critically influence diversity metrics.

In the second part of our study, we investigated the consequences of training LLMs on synthetic data generated by their predecessors. We conducted recursive finetuning experiments across three NLG tasks 55. Our findings reveal a consistent decrease in the diversity of the model outputs through successive iterations, especially remarkable for tasks demanding high levels of creativity. This trend underscores the potential risks of training language models on synthetic text, highlighting the need for careful consideration of the long-term effects of such training approaches on the linguistic capabilities of language models.

8.11.1 Repair modelling

In the scope of Ha Anh Ngo 's PhD thesis, under the supervision of Chloé Clavel , Catherine Pelachaud (Sorbonne Université, ISIR) and Nicolas Rollet (Telecom-Paris, on detachment in ALMAnaCH), we investigated the phenomenon of conversational repair mechanisms in human dialogue. Repair in conversations refers to the process by which interlocutors identify, address, and resolve issues, such as misunderstandings, mishearings, or ambiguous expressions, which help maintain mutual understanding between participants and ensure that the conversation proceeds smoothly.

In the first part of our study, in collaboration with Professor Dirk Heylen from Twente University (Netherlands), we have been exploring the multimodal patterns (linguistic and prosodic cues) involved when a speaker initiates and resolves a repair request. Using a task-oriented dialogue corpus in Dutch, we used an automatic coreference solver and analysed patterns of self-repetition and other-repetition to identify linguistic features that distinguish turns where the addressee initiates a repair request from those where they do not 66. Our findings revealed distinct linguistic patterns, including coreference chains, repetition, and grammatical structures, that differentiate repair initiation requests from regular dialogue turns, as well as among the three types of repair initiation. These results established the foundation for modelling the other-initiated repair sequence, which could significantly contribute to developing a computational model for recognising repair initiation and generating appropriate repair responses.

For future work, we are developing a computational model to detect other-initiated repair requests, incorporating multimodal cues (text, audio, and video) 65. The next step is to enable real-time online detection. Additionally, to address potential biases and limitations of the current corpus, we also plan to annotate a new corpus in French.

8.11.2 Word Meaning Negotiation

As part of Aina Garí Soler 's postdoctoral research, under the supervision of Chloé Clavel and Matthieu Labeau (Telecom-Paris), we have been exploring word usage dynamics in dialogue. In collaboration with researchers Jenny Myrendal and Staffan Larsson from Gothenburg University (Sweden), we have developed the NeWMe (Negotiating Word Meaning) corpus, a collection of annotated Word Meaning Negotiation (WMN) sequences. WMNs are parts of conversations where a speaker raises a clarification request or an objection about the usage of a specific word, prompting a metalinguistic discussion aimed at clarifying or refining the word's meaning. The corpus contains various kinds of interactions (written, oral, dyadic, and multi-party) and is annotated with WMN sequences, their components, and related phenomena. The effort, which included creating annotation guidelines and establishing inter-annotator agreement with recruited annotators, led to a submission to the LRE (Language Resources and Evaluation) journal in early 2025. This corpus offers valuable insights into conversational repair, disagreement, and semantic conflicts. The annotations can be used to identify potentially controversial or problematic terms and to investigate the mechanisms of conceptual or lexico-semantic alignment.

WMNs have three key components: the trigger, which is a problematic word usage; the indicator, an utterance signaling a problem with a word's meaning; and the negotiation – one or more conversational turns where participants address the misunderstanding or disagreement. To better understand the mechanisms by which a word can be misunderstood or disputed, we paid particular attention to the trigger component. We conducted an extensive literature review, identifying linguistic factors that contribute to potentially problematic word usages, as well as computational methods and data that facilitate their detection. This review connects works from various disciplines and highlights areas for future research.

Together, these two works will serve as foundations for our future research on lexico-semantic alignment, the detection of problematic word usages, and the expansion of WMN annotations to more languages and conversational contexts.

8.11.3 Socio-emotional Response Generation

In the context of Chenwei Wan 's research internship, co-supervised by Chloé Clavel and Matthieu Labeau (Telecom-Paris), we examine socio-emotional response generation in dialogue systems. Recently, with advancements in LLMs, end-to-end dialogue agents without explicit strategy prediction steps have become prevalent. However, implicit strategy planning lacks transparency, and recent studies show that LLMs' inherent preference bias towards certain socio-emotional strategies hinders the delivery of high-quality emotional support. To address this challenge, we proposed to decouple strategy prediction from language generation, and introduced a novel dialogue strategy prediction framework, EmoDynamiX, which models the discourse dynamics between user fine-grained emotions and system strategies using a heterogeneous graph for better performance and transparency.26 Experimental results showed that EmoDynamiX outperforms previous state-of-the-art methods by a significant margin (better proficiency and lower preference bias). Our approach also exhibits better transparency by allowing backtracing of decision making.

8.12.1 Hate Speech Detection in Conversations

As part of Célia Nouri 's research internship, under the supervision of Chloé Clavel and Jean-Philippe Cointet (Medialab, Sciences Po), we have been using graph neural networks to model enhance abusive language detection in social media conversations. In the context of this study we investigated the detection of abusive language in social media conversations using graph-based approaches. Traditional abusive language detection (ALD) methods often classify comments in isolation, neglecting crucial conversational context. While recent efforts have attempted to incorporate conversational context into NLP classifiers, they often employ a limited scope, primarily focusing on the immediately preceding comment and using flattened representations of conversations. To overcome these shortcomings, we propose a novel framework that models social media conversations as structured graphs, where nodes correspond to individual comments and edges represent reply relationships.

Our approach exploits Graph Neural Networks (GNNs), specifically Graph Attention Networks (GATs), to effectively capture conversational dependencies and enhance classification accuracy. We systematically compared different graph construction strategies to model Reddit conversations and introduce an affordance-based graph-trimming strategy aligned with Reddit's default rendering to enhance computational efficiency while preserving relevant context. Experiments on the Contextual Abuse Dataset (CAD) confirm that this approach yields superior performance in ALD. We also examined the impact of varying context depth by adjusting the number of GAT layers. Our results show that incorporating up to three-hop neighbours provides optimal performance, underscoring the importance of considering extended conversational structures beyond a single preceding comment. Additionally, our graph-based model outperforms both context-agnostic and traditional context-aware baselines, particularly in cases requiring contextual disambiguation. These findings establish GNNs as a robust framework for improving automated moderation of abusive content in online discussions.

Future research directions include extending the methodology to other social media platforms, integrating multimodal cues such as images and videos, and addressing annotation biases in conversational toxicity datasets. The proposed approach represents a significant advancement in context-aware hate speech detection and offers scalable solutions for moderating online discourse.

8.12.2 Fallacy Detection

In the context of Chadi Helwé 's thesis, supervised by Chloé Clavel and Fabian Suchanek (Telecom-Paris), we introduced MAFALDA, a benchmark for fallacy classification that merges and unites previous fallacy datasets 56. A fallacy is an erroneous or invalid way of reasoning. Consider, e.g., the argument “You must either support my presidential candidacy or be against America!”. This argument is a false dilemma fallacy: it wrongly assumes no other alternatives. The dataset integrates four pre-existing datasets into a cohesive whole, achieved through developing a new, comprehensive taxonomy. The proposed taxonomy aligns, refines, and unifies existing classifications of fallacies. We further provided a manual annotation of a part of the dataset together with manual explanations for each annotation. We proposed a new annotation scheme tailored for subjective NLP tasks, and a new evaluation method designed to handle subjectivity. We then evaluated several language models under a zero-shot learning setting and human performances on MAFALDA to assess their capability to detect and classify fallacies bias mitigation in language models. While a coarse grain classification (presence of fallacy) shows good results, a classification into fine-grained categories of fallacies is largely out of reach of LLMs in zero-shot settings. We hope that our benchmark will enable researchers to improve the results of this challenging task.

The PhD thesis recently started (November 2024) by Cecilia Graiff , supervised by Benoît Sagot , Chloé Clavel and Emiliano Grossmann, in collaboration with the “Action Exploratoire” SaLM, aims at extending this line of research by investigating argument structures in political discourse in a cross-lingual and cross-cultural perspective.

8.12.3 Disentangling Gender Information

In the context of Gabrielle Le Bellier 's research internship, supervised by Benoît Sagot and Chloé Clavel , we investigated bias mitigation in language models. Recent models have proved to be biased towards some demographics and display stereotypes. For instance, models identify nurses as women more than men in an excessive proportion compared to reality. This raises ethical issues about the ideas conveyed by the models and their consequences to societies. In this work, we focused on binary gender bias in occupation classification. Using a labelled dataset of biographies written in English (BiasBios), our aim was to build a model that is fair across genders. We used disentanglement to isolate gender information from occupation information and perform classification on top of the disentangled occupation vector. We explored several pre-processing and disentangled-based in-processing bias mitigation techniques. We evaluated disentanglement and fairness through metrics on embeddings, mutual information, and True Positive Rate on the downstream classification task. We found that a good disentanglement between occupation and gender does not necessarily involve a fairer occupation classification. Moreover, among the techniques explored, a multi-estimator setting seemed to enable different combinations to balance information loss and fairness, and estimators based on contrastive losses showed promising results.

8.13.1 Using Interbrain Synchrony and Rapport Building to Understand Productive Peer Collaboration

As a part of our large-scale approach to studying collaboration, our interdisciplinary team, composed of researchers, research engineers and Master's students within the neurosciences field, linguistics, education and computer sciences, designed an experiment that aims to examine the nature of collaboration between pairs of schoolchildren of the same age. We are particularly interested in the relationship between interpersonal rapport, as displayed by children's language and nonverbal behaviour, and the success of collaboration, as demonstrated by the learning gains of pairs of children collaborating on 4 scientific tasks, and by interbrain synchrony (IBS) - a similarity in the brain waves of two people while collaborating. Our team collected multimodal data from 22 dyads of adult participants (verbal, non verbal and neural data) in a longitudinal setting over 3 weeks using fNIRS hyperscanning. This study allows us to move towards our goals to better understand collaboration in children, as we are the first lab to carry out research on IBS between children in a peer learning setting, using the fNIRS (functional near-infrared spectroscopy), and to associate it to, on the one hand, success in collaboration, and on the other hand, to interpersonal rapport.

8.13.2 Conversational Grounding in Dialogue Systems

The team is dedicated to developing more socially capable conversational agents, with a focus on enhancing their ability to engage in natural and effective interactions. In this direction, Marius Le Chapelier is leading the Son of Sara project, which aims to create a new embodied conversational agent powered by an LLM. Throughout 2024, he has been implementing various modules to equip the agent with essential capabilities such as comprehension, reasoning, and response generation. These modules include Automatic Speech Recognition (ASR), Natural Language Generation (NLG), Text-to-Speech (TTS), and a Dialogue Manager, among others. In parallel, to further improve the agent's social capacities, the team is conducting in-depth research on LLMs' conversational abilities and social behaviors. Within this context, Biswesh Mohapatra 's PhD thesis, supervised by Justine Cassell and Laurent Romary , focuses on the understanding and evaluation of conversational grounding in dialogue systems, particularly in LLMs and LLM-based agents, developing new approaches into making LLMs that can engage in conversational grounding. Mutual understanding plays a critical role in effective communication, and we are working on addressing the notable gap in both theoretical frameworks and empirical evaluations of grounding capabilities in modern dialogue agents. In 2024, we annotated dialogue corpora using Grounding Acts, Grounding Units and a measure of their degree of grounding, together with a baseline classification model 62. We also conducted large-scale evaluations across various LLMs to assess their performance of LLMs in various aspects of conversational grounding, identifying correlations between model scale, pre-training data size, and grounding performance 63. Increasingly, our explorations have taken the form of multimodal representations, as conversational grounding–where interlocutors monitor and ensure that their conversational partners are following–can take the form of language, but also aspects of prosody, and visual phenomena.

8.13.3 Exploring Interpersonality: Multimodal Personality Cues in Embodied Conversational Agents

In 2024, in the context of our collaboration with KETI, we explored the most impactful cues to generate in embodied conversational agents (ECA) in order to convey particular personality traits to the people interacting with the ECA. These traits include speech style (speed of talk, and pitch excurses, for example) linguistic content (number of emotion words, for example), non-verbal behaviours (mutual eye gaze and head nods, for example). Breaking with previous work on personality in agents, the team developed the idea of “interpersonality”, which captures the fact that the manifestation of personality (that is, how it is displayed multimodally) is impacted by the personality of the interlocutor. A final report on interpersonality was subsequently accepted for publication in The Handbook of Personality Psychology, to appear in 2026.

Verbatim Analysis

Participants: Benoît Sagot.

• Partner type:
  Inria start-up
• Leader for ALMAnaCH:
  Benoît Sagot .
• Dates:
  1 Jul. 2009–31 Jan. 2024
• Description:
  Verbatim Analysis is an Inria start-up co-created in 2009 by Benoît Sagot . It uses some of ALMAnaCH's free NLP software (SxPipe) as well as a data mining solution co-developed by Benoît Sagot , VERA, for processing employee surveys with a focus on answers to open-ended questions. Its activities have been progressively taken over by opensquare (see below), and the company was discontinued at the end of January 2024.

opensquare

Participants: Benoît Sagot.

• Partner type:
  Inria start-up
• Leader for ALMAnaCH:
  Benoît Sagot .
• Dates:
  1 Dec 2016–present
• Description:
  Opensquare was co-created in December 2016 by Benoît Sagot with 2 senior specialists of human resources (HR) consulting. It is dedicated to designing, carrying out and analysing employee surveys as well as HR consulting based on these results. It uses a new employee survey analysis tool, enqi, which is still under development. This tool being co-owned by opensquare and Inria, both parties have signed a Software Licence Agreement in exchange for a yearly fee paid by opensquare to Inria based on its turnover. Benoît Sagot currently contributes to opensquare, under the “Concours scientifique” scheme.

META

Participants: Benoît Sagot, Tú Anh Nguyên, Paul-Ambroise Duquenne, Pierre Chambon.

• Partner type:
  Company
• Leader for ALMAnaCH:
  Benoît Sagot .
• Dates:
  1 Jan 2018–present
• Funding received:
  €331,260
• Description:

  Our collaboration with META AI is centered around the joint supervision of CIFRE PhD theses. A first collaboration (Louis Martin's PhD thesis), co-supervised by Benoît Sagot , Éric de La Clergerie and Antoine Bordes (META) was dedicated to text simplification (“français Facile À Lire et à Comprendre”, FALC), in collaboration with UNAPEI. This collaboration was part of a larger initiative called Cap'FALC involving (at least) these three partners as well as the relevant ministries. Louis defended his PhD in 2022. Two other joint PhD theses started in 2021 and were defended in 2024. Paul-Ambroise Duquenne's PhD, co-supervised by Benoît Sagot and Holger Schwenk (META), is dedicated to sentence embeddings for massively multilingual speech and text processing. Tú Anh Nguyen's PhD, co-supervised by Benoît Sagot and Emmanuel Dupoux (META), is dedicated to the unsupervised learning of linguistic representations from speech data, with a focus on textless dialogue modelling and speech generation. Finally, a new PhD started in PhD: Pierre Chambon's PhD is dedicated to code generation with language models.

  In addition, Benoît Sagot received a $50,000 gift grant from META AI in the context of the release of the LLAMA 3 series of models.

Qatent

Participants: You Zuo, Benoît Sagot, Éric de La Clergerie.

• Partner type:
  Inria start-up
• Leader for ALMAnaCH:
  Benoît Sagot .
• Dates:
  1 Jan 2021–present
• Description:
  Qatent is a startup supported by the Inria Startup Studio and ALMAnaCH that applies NLP to help write better patents faster. Its creation followed the 18-month secondment (“détachement”) at ALMAnaCH of Kim Gerdes, one of the three founders of the company, and benefitted from ALMAnaCH's scientific expertise and the Inria Startup Studio's counselling and financial support. It also led to You Zuo's CIFRE PhD thesis, co-supervised by Benoît Sagot and Kim Gerdes (now at qatent), which continues the collaboration. Note that in 2024, qatent was aquired by Questel, a global leader in intellectual property (IP) management and technology services, without a significant impact on You Zuo 's PhD.

10.1.1 Participation in other International Programs

10.2.1 Visits of international scientists

10.2.2 Visits to international teams

10.3.1 Horizon Europe

10.3.2 H2020 projects

ANR MaTOS

Participants: Rachel Bawden, Éric de La Clergerie, Nicolas Dahan, Ziqian Peng, Panagiotis Tsolakis.

• Duration:
  1 Jan 2023–31 Dec 2026.
• PI:
  François Yvon.
• Coordinator for ALMAnaCH:
  Rachel Bawden .
• Partners:
  • Sorbonne-Université
  • Université de Paris
  • CNRS
• Funding:
  €782,529 total, €280,520 for ALMAnaCH
• Summary:
  The MaTOS (Machine Translation for Open Science) project aims to develop new methods for the machine translation (MT) of complete scientific documents, as well as automatic metrics to evaluate the quality of these translations. Our main application target is the translation of scientific articles between French and English, where linguistic resources can be exploited to obtain more reliable translations, both for publication purposes and for gisting and text mining. However, efforts to improve MT of complete documents are hampered by the inability of existing automatic metrics to detect weaknesses in the systems and to identify the best ways to remedy them. The MaTOS project aims to address both of these issues.

ANR TraLaLaM

Participants: Rachel Bawden, Benoît Sagot, Malik Marmonier.

• Duration:
  1 Oct 2023–30 Sept 2026.
• PI:
  Josep Crego (Systran by ChapsVision).
• Coordinator for ALMAnaCH:
  Rachel Bawden .
• Partners:
  • Systran by ChapsVision
  • CNRS
• Funding:
  €595,348 total, €169,566 for ALMAnaCH
• Summary:
  The aim of TraLaLaM is to explore the use of large language models (LLMs) for machine translation, by asking two main questions: (i) in what scenarios can contextual information be effectively used via prompting? and (ii) for low-resource scenarios (with a focus on dialects and regional languages), can LLMs be effectively trained without any parallel data?

ANR PRME SINNet

Participants: Chloé Clavel.

• Duration:
  1 Mar 2024–1 Oct 2027.
• PI:
  Chloé Clavel .
• Coordinator for ALMAnaCH:
  Chloé Clavel .
• Funding:
  €474,255
• Summary:
  SINNet proposes a paradigm shift for rendering conversational systems and social robotics a more acceptable and trustworthy technology even when using deep learning approaches. It will focus on the verbal component of the interaction, will target the agent-user social relationship, and model the behaviors indexing the state of the social relationship between agent and user, thus going beyond the analysis of the user's positive and negative sentiments. It implies developing easy-to-adapt and easy-to-explain neural models able to analyse the user's behavior contributing to user-agent co-construction processes such as the ones characterising the rapport with the agent, or the trust and affiliation in the agent, as well as to generate the agent's answer fostering the user-agent social relationship. This SINNet project will establish interdisciplinarity as a core challenge by providing a shared formalism between complex (e.g. psychological or socio-linguistic) theories of social interactions and the underlying formalism in deep learning and language models.

ANR PRCE REVITALISE

Participants: Chloé Clavel.

• Duration:
  15 Feb 2022–15 Nov 2025.
• PI:
  Magalie Ochs (LIS).
• Coordinator for ALMAnaCH:
  Chloé Clavel .
• Partners:
  • LIS
  • Umanis
  • ISM
  • IMT Atlantique, Telecom-Paris
• Funding:
  €580,124 total, €123,800 for ALMAnaCH
• Summary:
  More than ever, with the increasing use of online video-conferencing solutions in daily professional interactions, public speaking skills are becoming crucial. The aim of this project is to obtain better insights into the best approaches allowing the practice of public speaking skills with technologically mediated tools. To this end, we will investigate different training environments (e.g. without a virtual/real audience) and different training approaches (e.g. modelling-based, feedback-based, simulation-based) to help users acquire, improve, and practise public speaking skills in full autonomy. For this purpose, different research challenges will be tackled to (i) automatically learn, from different corpora, the multimodal cues correlated to the quality of public speaking, (ii) provide pedagogical activities rooted in coaching practice, taking a user-centered approach and (iii) provide a global evaluation of the training session as well as the specific behavioural characteristics to improve.

10.4.1 Competitivity Clusters and Thematic Institutes

10.4.2 Other National Initiatives

10.4.3 Regional Initiatives

11.1.1 Scientific events: organisation

11.1.2 Scientific events: selection

11.1.3 Journal

• Éric de La Clergerie : Reviewer for ACM Computing Surveys and SN Computer Science.
• Rachel Bawden : Reviewer for Revue TAL (Special thematic issue on “Robustesse et limites des modèles de traitement automatique des langues”) and Revue TAL (Special thematic issue on “Explicabilité des modèles de TAL”).
• Djamé Seddah : Reviewer for Transaction on Affective Computing.

11.1.4 Invited talks

• Rachel Bawden :
  • Demi-heure de Science, Inria Paris (11 Jan 2024): “Machine Translation and Variation”.
  • Conference on Multilingualism, Aix-Marseille University, Aix-en-Provence (11 Jun 2024): “Machine Translation and the Challenge of Cross-Lingual Ambiguity”.
• Justine Cassell :
  • World Economic Forum, Davos (17 Jan 2024): “Communication at the Speed of AI”. Panel organised by Fast Company & Verizon
  • World Economic Forum, Davos (18 Jan 2024): “Future of XR”.
  • Shonan Village Center Meeting on The Future of Education with AI (5 Mar 2024): “The Future of Education with AI”. Shonan Village workshop, organised by Laurence deVillers
  • AI Research Center (AIRC), Japan (8 Mar 2024): “Interlocutor effects: What LLMs can't do and why”.
  • EACL, Malta (22 Mar 2024): “Marginalized Dialects and Language Technologies for Education: Can Culture-Sensitive EdTech Help Children Flourish”. Workshop EACL-EMNLP workshop “Towards Ethical and Inclusive Conversational AI: Language Attitudes, Linguistic Diversity, and Language Rights” (TEICAI)
  • Paris, France (27 Mar 2024): “Teaching and learning with generative AI”. Unesco Chair Conference on Teaching and Learning with generative AI.
  • Cambridge, MA, USA (5 Apr 2024): “Potential impacts of Social Bias and Machine Learning Bias on Precision Medicine”. “Harvard-Radcliffe workshop on “Sex and Gender at the Crossroads of 21st Century Precision Medicine”
  • Paris, France (22 May 2024): “The impact of on-Device LLMs on usability and security of cell phones”. Salon VivaTech
  • Paris, France (2 Jul 2024): “IA Génératives, chatbots, tuteurs et étudiants - What's next ?”. Seminaire AGIR (Paris Dauphine)
  • Paris, France (17 Sept 2024): “IA, neuroscience et l'avenir de l'éducation”. Evenement à Dauphine: Réunion Agir
  • Paris, France (27 Sept 2024): “Chatbots as interlocutors not tutors”. XYZParis (Station F)
  • French Embassy, Seoul, South Korea (22 Oct 2024): “IA and the future of education”. Joint meeting between France Korea, Taiwan and Japan
  • College de France (28 Oct 2024): “Conversations à Venir”. IA et Neuroscience
  • Conversations à Venir (6 Dec 2024): “Vivre avec l'Intelligence Artificielle: l'éducation”. Entretiens de Royaumont
  • Versailles, France (11 Dec 2024): “Giving AI a Voice and a Body”. Joint with Patrick Perez (Kyutai). Jolt Capital's 2024 Annual LPs & CEOs Day
• Floriane Chiffoleau :
  • Saarland University, Saarbrücken (22 Feb 2024): “Introduction into Publishing Workflows with TEI Publisher”. Master Class “Digital Scholarly Editing”' 2024, Saarbrücken
  • Charles University, Prague, Czech Republic (28 Mar 2024): “Leveraging EHRI Online Editions for training automated edition tools”. Joint with hscheith. Workshop “Natural Language Processing meets Holocaust Archives”
• Benoît Sagot :
  • PhD day of the Île-de-France AI4IDF institute (5 Mar 2024): “Image-enhanced machine translation: modelling and evaluation challenges”. Phd day of the Île-de-France AI4IDF institute
  • CORIA 2024, La Rochelle, France (3 Apr 2024): “Mieux comprendre les modèles de langue et les textes qu'ils produisent”. Keynote speech
  • École polytechnique, Palaiseau, France (9 Apr 2024): “Apprendre les langues aux machines: Introduction au traitement automatique des langues”. “Amphi 0” for computer science: invited scientific talk to encourage new École polytechique students to enroll in computer science courses
  • Maison Irène et Frédéric Joliot Curie, Bruxelles, Belgium (16 Apr 2024): “Language models and their training data: challenges and perspectives”. ERCIM Visionary Event on Generative AI: Introductory keynote
  • BnF, Paris, France (26 Apr 2024): “Training data for language models: challenges and perspectives”. Web Archiving Conference (WAC): Closing keynote
  • George Mason University, Fairfax, Virginia, USA (3 May 2024): “Why do small language models underperform?”. GMU Computer Science Seminar
  • University of Edinburgh, Edinburgh, UK (14 Jun 2024): “Doing more with sentence embeddings”. ILCC seminar
  • Paris, France (24 Jun 2024): “Modèles de langue et données d'entraînement : progrès récents et enjeux actuels”. Executive seminar “L'intelligence artificielle à la BnF” (AI at BnF)
  • Grenoble, France (29 Aug 2024): “Modèles génératifs pour le traitement automatique des langues”. Journées Scientifiques Inria
  • Montreal, Canada (1 Oct 2024): “Why do small language models underperform? Studying Language Model Saturation via the Softmax Bottleneck”. MILA Workshop on NLP in the era of generative AI, cognitive sciences, and societal transformation. Joint work with Nathan Godey and Éric de La Clergerie
  • Campus Cyber, Puteaux, France (11 Dec 2024): “Modèles de langue génératifs : contexte, état de l'art et enjeux”. Journées scientifiques du PEPR Cybersécurité
• Chloé Clavel :
  • Paris, France (23 May 2024): “AI4IDF: recherche en intelligence artificielle centrée sur l'humain”. Salon VivaTech
  • Bordeaux, France (24 May 2024): “Transparency in AI”. Symposium on Neuroscience & Artificial Intelligence Mechanisms, Perspectives & Consequences.
  • Glasgow, UK (19 Sept 2024): “Multimodal analysis of the socio-emotional layer in interactions — Or how to replicate a subjective perception process?”. Workshop of the IVA conference
  • Grenoble, France (15 Oct 2024): “Socio-conversational systems”. 2nd HRI Symposium organized by Naver Labs Europe
• Djamé Seddah :
  • Paris, France, (30 May 2024): “Should we be afraid of the big bad GPT? Demystifying language models and preventing their weaponization”. Forum Terratec 2024. Applications of AI in research and industry. Invited Talk. .html
  • Forum 30 ans de l'ORAP, BNF, Paris, France (15 Oct 2024): “Should we be afraid of the big bad GPT? Demystifying language models and preventing their weaponization”. Forum 30 ans de l'ORAP
  • CNCTR, Paris, France (29 Apr 2024): “Prévenir la weaponisation des LLM”. Commission nationale de controle des techniques de renseignement
  • ANSII, Paris, France (26 Sept 2024): “Prévenir la weaponisation des LLM”. Online à partir du Campus Santé
  • Terratech, Paris, France (30 May 2024): “Should we be afraid of the big bad GPT? Demystifying language models and preventing their weaponization”. Teratec 2024 Forum
  • IMS, Stuttgart, Germany (20 Nov 2024): “Demystifying language models and preventing their weaponization”.
  • Ambassade UK, workshop Inria-UK AI Institute (6 Nov 2024): “Preventing Language Models Weaponisation”. Joint with Arij Riabi and Wissam Antoun .
  • Ministère de l'intérieur, Levallois-Perret, France (11 Dec 2024): “Preventing Language Models Weaponisation: Insights from the Counter Dataset”. Joint with Arij Riabi and Wissam Antoun .
• Thibault Clérice :
  • Paris, France (13 Nov 2024): “CATMuS Medieval: The Importance of Medieval Manuscript Automatic Transcription for Computer Vision”.
• Alix Chagué :
  • Naples, Italie (16 Apr 2024): “FAIRer transcriptions: HTR-United and the possibility of a common for training data”. Horizons of digital philology. The Greek Anthology for rethinking [formats], [paradigms] and [collaboration]
  • Paris, France (13 Dec 2024): “Transcription of discursive and non-discursive documents: connecting project specificities and generic frameworks”. Workshop: texts in historical maps, diagrams and illustrations
• Éric de La Clergerie :
  • workshop e-VITA on Knowledge Graphs and LLMs (8 Mar 2024): “Coupling KG and LLM: a few directions”.

11.1.5 Scientific expertise

• Alix Chagué :
  • Member of the Comité de Perfectionnement du Master Technologies Numériques Appliquées à l'Histoire de l'Ecole nationale des chartes (Paris).
• Benoît Sagot :
  • Member of the scientific advisory board of ERIC CLARIN.
  • Member of the Working Group on IA of the Couperin consortium.
  • Member of the Scientific Council of ARTE France.
• Aina Garí Soler :
  • Reviewer of UTTER-FSTP (UTTER project: Unified Transcription and Translation for Extended Reality Financial Support for Third Parties).
• Chloé Clavel :
  • Reviewer for BPI.
• Éric de La Clergerie :
  • Reviewer of Expert BPI (8 project reviews on BPI call “Usages IA Generative”).
  • Reviewer of ANR (MRSEI call) (Project review).
• Rachel Bawden
  • Co-opted member of the executive committee of the EAMT (European Association for Machine Translation)
  • Reviewer of Thematic committee for Genci projects (Reviewing projects for the allocation of computational resources).

11.1.6 Research administration

• Laurent Romary :
  • President of the scientific board of ABES (Website).
  • Member of the scientific board of the Schloss Dagstuhl Scientific Advisory Board (Website).
  • Member of the international advisory board of the Research Infrastructure project LINDAT/CLARIAH-CZ.
• Benoît Sagot :
  • Member of the board of Inria Paris's Comité des Projets (Inria Paris research centre's Bureau du Comité des Projets).
  • Member of the scientific board of the Société de Linguistique de Paris (Administrateur).
• Rachel Bawden :
  • Member of the scientific board of the Société de Linguistique de Paris (Administratrice).

11.2.1 Teaching

• Rachel Bawden :
  • Master's course (M2), Master “Mathématiques, Vision Apprentissage”, ENS Paris-Saclay, France. CM: Speech and Language Processing, coorganised with Benoît Sagot , Chloé Clavel , Djamé Seddah and Guillaume Wisniewski. 3hrs.
• Justine Cassell :
  • Master's course (M2), Master “Biologie Intégrative et Psychologie”, speciality Neurosciences Parcours N°2 “Neurosciences cognitives et comportementales”, Sorbonne Université, Paris, France. CM: Metacognition dans l'Interaction Virtuelle. 4hrs.
  • Inria Paris. École Académique Formation Continue: IA et Education: Quels rôles pour l'IA? 4hrs.
  • Bénin Start-up Incubator. Continued training for start-ups creating education applications: Les Sciences de l'Education, coorganised with Mastercard Foundation. 12hrs.
  • Senegal Start-up Incubator. Continued training for start-ups creating education applications: Les Sciences de l'Education, coorganised with Mastercard Foundation. 12hrs.
• Alix Chagué :
  • Bachelor; Master; Doctorate's course (1er, 2e et 3e cycles), Faculté des Arts et des Sciences, Université de Montréal, Canada. CM: HNU-6059: Humanités Numériques: Langages de Programmation. 15hrs.
  • Bachelor's course (1er cycle), Faculté des Arts et des Sciences, Université de Montréal, Canada. CM: HNU-2000: Humanités Numériques: Pratique. 45hrs.
  • Università degli Studi di Napoli Federico II, Naples, Italie. Workshop on eScriptorium for HTR projects: eScriptorium. 3hrs.
• Floriane Chiffoleau :
  • Charles University, Prague, Czech Republic. Workshop “Natural Language Processing meets Holocaust Archives”: EHRI editions and TEI Publisher annotation tool. 2hrs.
• Chloé Clavel :
  • Master's course (M2), X Datascience, Polytechnique. CM: Introduction to NLP, Recent advances in NLP. 6hrs.
  • Master's course (M2), Master “Mathématiques, Vision Apprentissage”, ENS Paris-Saclay, France. CM: Speech and Language Processing, coorganised with Benoît Sagot , Chloé Clavel , Djamé Seddah and Guillaume Wisniewski. 6hrs.
  • Master's course (M2), Mastère spécialisé en Intelligence Artificielle, Telecom-Paris. CM: Conversational Systems. 3hrs.
  • Master's course (M2), Mastère spécialisé en Intelligence Artificielle, Telecom-Paris. CM: Introduction to Natural Language Processing. 3hrs.
  • Master's course (M2), Master Data IA, Polytechnique. CM: Introduction to Natural Language Processing. 3.5hrs.
• Thibault Clérice :
  • Doctorate's course (PhD & Higher), Invited professor course, Federico II, Napoli. CM: Introduction to Computational Humanities. 6hrs.
  • Doctorate's course (PhD & Higher), Invited professor course, Federico II, Napoli. CM: Natural Language Processing for Computational Humanities. 6hrs.
  • Doctorate's course (PhD & Higher), Invited professor course, Federico II, Napoli. CM: Computer Vision for Computational Humanities. 6hrs.
  • Doctorate's course (PhD & Higher), Invited professor course, Federico II, Napoli. TD/TP: Workshop for Handwritten Text Recognition & Layout Analysis. 18hrs.
  • ENS de Lyon, Lyon, France. Summer School: TranscriboQuest 2025, coorganised with CNRS (CIHAM). 21hrs.
• Aina Garí Soler :
  • Master's course (M2), M2 DS - Data Science, École Polytechnique. CM: CSC_5DS25_TP: Natural Language Processing and Sentiment Analysis, coorganised with Matthieu Labeau. 1.5hrs.
  • Master's course (M2), Mastère spécialisé - IA : Intelligence Artificielle, Télécom-Paris. Project supervision: IA717 : Natural Language Processing, coorganised with Matthieu Labeau, Maria Boritchev, Changhong Wang. 19.5hrs.
• Arij Riabi :
  • Master's course (M1), Cycle d'ingénieur, ENSAE. Project supervision: Applied statistics project. 20hrs.
• Benoît Sagot :
  • Course for the general public, Chaire annuelle “Informatique et sciences numériques”, Collège de France. CM: Apprendre les langues aux machines. 9hrs.
  • Master's course (M2), Master “Mathématiques, Vision Apprentissage”, ENS Paris-Saclay, France. CM: Speech and Language Processing, coorganised with Benoît Sagot , Chloé Clavel , Djamé Seddah and Guillaume Wisniewski. 7.5hrs.
  • PariSanté Campus. Certificate Data AI Product Owner Société des Ingénieurs de l'Automobile: Natural Language Processing. 14hrs.
  • PariSanté Campus. “IA for creative business” training by We Are_ school: Introduction au TAL et à l'IA générative. 5hrs $\times$ 3.
  • Institut national du service public. 3ème journée de formation des Directeurs d'Administrations Centrales au numérique, organisée par la DINUM: Introduction au traitement automatique des langues. 1hr.
• Hugo Scheithauer :
  • Master's course (M2), Master Humanités numériques, Université Rennes 2. TD/TP: L'édition numérique avec XML-TEI. 6hrs.
• Djamé Seddah :
  • Master's course (M2), Master “Mathématiques, Vision Apprentissage”, ENS Paris-Saclay, France. CM: Speech and Language Processing, coorganised with Benoît Sagot , Chloé Clavel , Djamé Seddah and Guillaume Wisniewski. 1.5hrs.

11.2.2 Supervision

11.2.3 Juries

11.3.1 Productions (articles, videos, podcasts, serious games, ...)

11.3.2 Participation in Live events

International journals

• 23 articleJ.Julie Bonnaire, G.Guillaume Dumas and J.Justine Cassell. Bringing together multimodal and multilevel approaches to study the emergence of social bonds between children and improve social AI.Frontiers in Neuroergonomics5May 2024HALDOI
• 24 articleL.Lucie Chenain, R.Rachid Riad, N.Nicolas Fraisse, C.Cécilia Jubin, G.Graça Morgado, K.Katia Youssov, M.Marine Lunven and A.-C.Anne-Catherine Bachoud-Levi. Graph methods to infer spatial disturbances: Application to Huntington's Disease's speech.Cortex176May 2024, 144 - 160HALDOI
• 25 articleC.Cyril Chhun, F. M.Fabian M. Suchanek and C.Chloé Clavel. Do Language Models Enjoy Their Own Stories? Prompting Large Language Models for Automatic Story Evaluation.Transactions of the Association for Computational Linguistics12September 2024, 1122–1142HALDOIback to text
• 26 articleL.Lith Lefranc, I.Ilja van Damme, T.Thibault Clérice and M.Mike Kestemont. ARletta. Open-Source Handwritten Text Recognition Models for Historic Dutch.Journal of Open Humanities Data1043July 2024, 1--7HALDOIback to text
• 27 articleC.Christian Licoppe, N.Nicolas Rollet and L.Luca Greco. ‘I know what it is’. An interactional study of sex discovery in prenatal ultrasound examinations.Discourse Studies265April 2024, 643-668HALDOI
• 28 articleT. A.Tu Anh Nguyen, B.Benjamin Muller, B.Bokai Yu, M. R.Marta R. Costa-Jussa, M.Maha Elbayad, S.Sravya Popuri, C.Christophe Ropers, P.-A.Paul-Ambroise Duquenne, R.Robin Algayres, R.Ruslan Mavlyutov, I.Itai Gat, M.Mary Williamson, G.Gabriel Synnaeve, J.Juan Pino, B.Benoît Sagot and E.Emmanuel Dupoux. SpiRit-LM: Interleaved Spoken and Written Language Model.Transactions of the Association for Computational Linguistics13January 2025, 30-52HALback to text
• 29 articleA.Ana Salgado, L.Laurent Romary, R.Rute Costa, T.Toma Tasovac, A. F.Anas Fahad Khan, M.Margarida Ramos, B.Bruno Almeida, S.Sara Carvalho, M.Mohamed Khemakhem, R.Raquel Silva and B.Boris Lehečka. The Morais Dictionary: Following Best Practices in a Retro-digitized Dictionary Project.International Journal of Humanities and Arts Computing181March 2024, 125 - 147HALDOI
• 30 articleA. G.Aina Garí Soler, M.Matthieu Labeau and C.Chloé Clavel. The Impact of Word Splitting on the Semantic Content of Contextualized Word Representations.Transactions of the Association for Computational Linguistics12April 2024, 299-320HALDOIback to text