2023Activity reportTeamFLOWERS

7.1.1 SocialAI

• Name:
  SocialAI: Benchmarking Socio-Cognitive Abilities in Deep Reinforcement Learning Agents
• Keywords:
  Artificial intelligence, Deep learning, Reinforcement learning, Large Language Models
• Functional Description:

  Source code for the paper https://arxiv.org/abs/2107.00956.

  A suite of environments for testing socio-cognitive abilities of artificial agents. Environments can be used in the multimodal setting (suitable for RL agents) and in the pure text setting (suitable for Large Language Model-based agents). Also contains RL and LLM baselines.

• URL:
  https://­gitlab.­inria.­fr/­gkovac/­act-and-speak
• Contact:
  Grgur Kovac

7.1.2 AutoDisc

• Keyword:
  Complex Systems
• Functional Description:
  AutoDisc is a software built for automated scientific discoveries in complex systems (e.g. self-organizing systems). It can be used as a tool to experiment automated discovery of various systems using exploration algorithms (e.g. curiosity-driven). Our software is fully Open Source and allows user to add their own systems, exploration algorithms or visualization methods.
• URL:
  https://­gitlab.­inria.­fr/­cromac/­AutomatedDiscoveryTool
• Contact:
  Clément Romac

7.1.3 Kids Ask

• Keywords:
  Human Computer Interaction, Cognitive sciences
• Functional Description:
  Kids Ask is a web-based educational platform that involves an interaction between a child and a conversational agent. The platform is designed to teach children how to generate curiosity-based questions and use them in their learning in order to gain new knowledge in an autonomous way.
• URL:
  https://­github.­com/­RaniaAbdelghani/­KidAsk
• Contact:
  Rania Abdelghani

7.1.4 ToGather

• Keywords:
  Education, Handicap, Environment perception
• Scientific Description:
  With participatory design methods, we have designed an interactive website application for educational purposes. This application aims to provide interactive services with continuously updated content for the stakeholders of school inclusion of children with specific educational needs.
• Functional Description:
  Website gathering information on middle school students with neurodevelopmental disorders. Authentication is required to access the site's content. Each user can only access the student file(s) of the young person(s) they are accompanying. A student file contains 6 tabs, in which each type of user can add, edit or delete information: 1. Profile: to quickly get to know the student 2. Skills: evaluation at a given moment and evolution over time 3. Compendium of tips: includes psycho-educational tips 4. Meetings: manager and reports 5. News: share information over time 6. Contacts: contact information for stakeholders The student only has the right to view information about him/her.
• Publication:
  https://hal.inria.fr/hal-03436355/
• Contact:
  Cécile Mazon
• Participants:
  Isabeau Saint-Supery, Cécile Mazon, Eric Meyer, Hélène Sauzéon

7.1.5 mc_training

• Name:
  Platform for metacognitive training
• Keywords:
  Human Computer Interaction, Education
• Functional Description:

  This is a web platform for children between 9 and 11 years old, designed to help children practice 4 metacognitive skills that are thought to be involved in curiosity-driven learning: - the ability to identify uncertainties - the ability to generate informed hypotheses - the ability to ask questions - the ability to evaluate the value of a preconceived inference.

  Children work on a reading-comprehension tasks and, for each of these skills, the platform offers help through a "conversation" with conversational agents that give instructions to perform the task, with respect to every skill, and can give suggestions if the child asks for it.

• Contact:
  Rania Abdelghani

7.1.6 Evolution of adaptation mechanisms in complex environments

• Name:
  Plasticity and evolvability under environmental variability: the joint role of fitness-based selection and niche-limited competition
• Keywords:
  Evolution, Ecology, Dynamic adaptation
• Functional Description:

  This is the code accompannying our paper Plasticity and evolvability under environmental variability: the joint role of fitness-based selection and niche-limited competition" which is to be presented at the Gecco 2022 conference.

  In this work we have studied the evolution of a population of agents in a world where the fitness landscape changes with generations based on climate function and a latitudinal model that divides the world in different niches. We have implemented different selection mechanisms (fitness-based selection and niche-limited competition).

  The world is divided into niches that correspond to different latitudes and whose state evolves based on a common climate function.

  We model the plasticity of an individual using tolerance curves originally developed in ecology. Plasticity curves have the form of a Gaussian the capture the benefits and costs of plasticity when comparing a specialist (left) with a generalist (right) agent.

  The repo contains the following main elements :

  folder source contains the main functionality for running a simulation scripts/run/reproduce_gecco.py can be used to rerun all simulations in the paper scripts/evaluate contains scripts for reproducing figures. reproduce_figures.py will produce all figures (provided you have already run scripts/run/reproduce_gecco.py to generate the data) folder projects contains data generated from running a simulation How to run To install all package dependencies you can create a conda environment as:

  conda env create -f environment.yml

  All script executions need to be run from folder source. Once there, you can use simulate.py, the main interface of the codebase to run a simulation, For example:

  python simulate.py –project test_stable –env_type stable –num_gens 300 –capacity 1000 –num_niches 10 –trials 10 –selection_type NF –climate_mean_init 2

  will run a simulation with an environment with a climate function whose state is constantly 2 consisting of 100 niches for 300 generations and 10 independent trials. The maximum population size will be 1000*2 and selection will be fitness-based (higher fitness means higher chances of reproduction) and niche limited (individuals reproduce independently in each niche and compete only within a niche),

  You can also take a look at scripts/run/reproduce_gecco.py to see which flags were used for the simulations presented in the paper.

  Running all simulations requires some days. You can instead download the data produced by running scripts/run/reproduce_gecco.py from this google folder and unzip them under the projects directory.

• URL:
  https://­github.­com/­eleninisioti/­ClimateAndLearning
• Contact:
  Eleni Nisioti

7.1.7 SAPIENS

• Name:
  SAPIENS: Structuring multi-Agent toPology for Innovation through ExperieNce Sharing
• Keywords:
  Reinforcement learning, Multi-agent
• Functional Description:

  SAPIENS is a reinforcement learning algorithm where multiple off-policy agents solve the same task in parallel and exchange experiences on the go. The group is characterized by its topology, a graph that determines who communicates with whom.

  All agents are DQNs and exchange experiences have the form of transitions from their replay buffers.

  Using SAPIENS we can define groups of agents that are connected with others based on a a) fully-connected topology b) small-world topology c) ring topology or d) dynamic topology.

  Install required packages You can install all required python packages by creating a new conda environment containing the packages in environment.yml:

  conda env create -f environment.yml

  And then activating the environment:

  conda activate sapiens

  Example usages Under notebooks there is a Jupyter notebook that will guide you through setting up simulations with a fully-connected and a dynamic social network structure for solving Wordcraft tasks. It also explains how you can access visualizations of the metrics produced during th$

  Reproducing the paper results Scripts under the scripts directory are useful for reproducing results and figures appearing in the paper.

  With scripts/reproduce_runs.py you can run all simulations presented in the paper from scratch.

  This file is useful for looking at how the experiments were configured but better avoid running it: simulations will run locally and sequentially and will take months to complete.

  Instead, you can access the data files output by simulations on this online repo.

  Download this zip file and uncompress it under the projects directory. This should create a projects/paper_done sub-directory.

  You can now reproduce all visualization presented in the paper. Run:

  python scripts/reproduce_visuals.py

  This will save some general plots under visuals, while project-specific plots are saved under the corresponding project in projects/paper_done

• URL:
  https://­github.­com/­eleninisioti/­SAPIENS
• Contact:
  Eleni Nisioti

7.1.8 architect-builder-abig

• Name:
  Architect-Builder Iterated Guiding
• Keyword:
  Artificial intelligence
• Functional Description:

  Codebase for the paper Learning to guide and to be guided in the Architect-Builder Problem

  ABIG stands for Architect-Builder Iterated Guiding and is an algorithmic solution to the Architect-Builder Problem. The algorithm leverages a learned model of the builder to guide it while the builder uses self-imitation learning to reinforce its guided behavior.

• URL:
  https://­github.­com/­flowersteam/­architect-builder-abig
• Contact:
  Tristan Karch

7.1.9 EAGER

• Name:
  Exploit question-Answering Grounding for effective Exploration in language-conditioned Reinforcement learning
• Keywords:
  Reinforcement learning, Language, Question Generation Question Answering, Reward shaping
• Functional Description:
  A novel QG/QA framework for RL called EAGER In EAGER, an agent reuses the initial language goal sentence to generate a set of questions (QG): each of these self-generated questions defines an auxiliary objective. Here, generating a question consists in masking a word of the initial language goal. Then the agent tries to answer these questions (guess the missing word) only by observing its trajectory so far. When it manages to answer a question correctly (QA) it obtains an intrinsic reward proportional to its confidence in the answer. The QA module is trained using a set of successful example trajectories. If the agent follows a path too different from correct ones at some point in its trajectory, the QA module will not answer the question correctly, resulting in zero intrinsic reward. The sum of all the intrinsic rewards measures the quality of a trajectory in relation to the given goal. In other words, maximizing this intrinsic reward incentivizes the agent to produce behaviour that unambiguously explains various aspects of the given goal.
• URL:
  https://­github.­com/­flowersteam/­EAGER
• Contact:
  Thomas Carta

7.1.10 IMGC-MARL

• Name:
  Intrinsically Motivated Goal-Conditioned Reinforcement Learning in Multi-Agent Environments
• Keywords:
  Reinforcement learning, Multi-agent, Curiosity
• Functional Description:

  This repo contains the code base of the paper Intrinsically Motivated Goal-Conditioned Reinforcement Learning in Multi-Agent Environments.

  In this work, we have studied the importance of the alignment of goals in the training of instrinsically motivated agents in the multi agent goal conditioned RL case. We also proposed a simple algorithm called goal coordination game which allows such agent to learn, in a completely decentralized/selfish way, to communicate in order to align their goal.

  The repository contains the code to reproduce the results of the paper. Which includes a custom RL environment ( using SimplePlayground "game engine"), model used (architecture + hyperparameters) and custom training (mostly based on RLlib ) to train both the model and the communication. We also provide the scripts for the training of every condition we test and notebook to study the results.

• URL:
  https://­github.­com/­Reytuag/­imgc-marl
• Contact:
  Gautier Hamon

7.1.11 Flow-Lenia

• Name:
  Flow Lenia: Mass conservation for the study of virtual creatures in continuous cellular automata
• Keywords:
  Cellular automaton, Self-organization
• Functional Description:

  This repo contains the code to run the Flow Lenia system which is a continuous parametrized cellular automaton with mass conservation. This work extends the classic Lenia system with mass conservation and allows to implement new feature like local parameter, environment components etc

  Several declination of the system (1 or several channels etc ) are available

  Please refer to the associated paper for the details of the system

  Implemented in JAX

• URL:
  https://­github.­com/­erwanplantec/­FlowLenia
• Contact:
  Gautier Hamon

7.1.12 Kidlearn: money game application

• Functional Description:
  The games is instantiated in a browser environment where students are proposed exercises in the form of money/token games (see Figure 1). For an exercise type, one object is presented with a given tagged price and the learner has to choose which combination of bank notes, coins or abstract tokens need to be taken from the wallet to buy the object, with various constraints depending on exercises parameters. The games have been developed using web technologies, HTML5, javascript and Django.

  

  

  

  

• URL:
  https://­flowers.­inria.­fr/­research/­kidlearn/
• Contact:
  Benjamin Clement

7.1.13 cognitive-testbattery

• Name:
  Cognitive test battery of human attention and memory
• Keywords:
  Open Access, Cognitive sciences
• Scientific Description:
  Cognitive test batteries are widely used in diverse research fields, such as cognitive training, cognitive disorder assessment, or brain mechanism understanding. Although they need flexibility according to the objectives of their usage, most of the test batteries are not be available as open-source software and not be tuned by researchers in detail. The present study introduces an open-source cognitive test battery to assess attention and memory, using a javascript library, p5.js. Because of the ubiquitous nature of dynamic attention in our daily lives, it is crucial to have tools for its assessment or training. For that purpose, our test battery includes seven cognitive tasks (multiple-objects tracking, enumeration, go/no-go, load-induced blindness, task-switching, working memory, and memorability), common in cognitive science literature. By using the test battery, we conducted an online experiment to collect the benchmark data. Results conducted on two separate days showed the high cross-day reliability. Specifically, the task performance did not largely change with the different days. Besides, our test battery captures diverse individual differences and can evaluate them based on the cognitive factors extracted from latent factor analysis. Since we share our source code as open-source software, users can expand and manipulate experimental conditions flexibly. Our test battery is also flexible in terms of the experimental environment, i.e., it is possible to experiment either online or in a laboratory environment.
• Functional Description:
  The evaluation battery consists of 6 cognitive activities (serious games: multi-object tracking, enumeration, go/no-go, Corsi, load-induced blindness, taskswitching, memorability). Easily deployable as a web application, it can be re-used and modified for new experiments. The tool is documented in order to facilitate the deployment and the analysis of results.
• URL:
  https://­github.­com/­flowersteam/­cognitive-testbattery
• Publication:
  hal-03723887
• Contact:
  Maxime Adolphe
• Participants:
  Pierre-yves Oudeyer, Hélène Sauzéon, Masataka Sawayama, Maxime Adolphe

7.1.14 LLM_stability

• Keywords:
  Artificial intelligence, Deep learning, Large Language Models
• Functional Description:

  Source code for the paper https://arxiv.org/abs/2307.07870

  Code enabling systematic evaluation of Large Language Models with various psychology questionnaires in different contexts, e.g. following conversations on different topics.

• URL:
  https://­gitlab.­inria.­fr/­gkovac/­llm_perspectives
• Contact:
  Grgur Kovac

7.1.15 Sensorimotor-lenia

• Keywords:
  Cellular automaton, Gradient descent, Curriculum Learning
• Functional Description:
  Source code for the search of sensorimotor agency in cellular automata associated to this blogpost https://developmentalsystems.org/sensorimotor-lenia/. The code allows to find rules in the cellular automata Lenia (through gradient descent, curriculum learning and diversity search) that lead to the self-organization of moving agents robust to perturbation by obstacles.
• URL:
  https://­github.­com/­flowersteam/­sensorimotor-lenia-search
• Contact:
  Gautier Hamon

7.1.16 MetaIPPO

• Keywords:
  Reinforcement learning, Exploration
• Functional Description:

  Code for the paper "Emergence of collective open-ended exploration from Decentralized Meta-Reinforcement learning" https://arxiv.org/pdf/2311.00651.pdf

  We train two decentralized agents together on an open ended tasks space to study the emergence of collective exploration behaviors. Our agents are able to generalize to novel objects and tasks, as well as an essentially open ended setting.

• URL:
  https://­github.­com/­RBorn02/­MetaIPPO
• Contact:
  Gautier Hamon

7.1.17 Lamorel

• Keywords:
  Large Language Models, Reinforcement learning, Distributed computing
• Functional Description:
  Lamorel is a Python library designed for people eager to use Large Language Models (LLMs) in interactive environments (e.g. RL setups).
• URL:
  https://­github.­com/­flowersteam/­lamorel
• Publication:
  hal-03970122
• Contact:
  Clément Romac

7.1.18 GLAM

• Name:
  Grounding LAnguage Models
• Keywords:
  Large Language Models, Reinforcement learning
• Scientific Description:
  Recent works successfully leveraged Large Language Models' (LLM) abilities to capture abstract knowledge about world's physics to solve decision-making problems. Yet, the alignment between LLMs' knowledge and the environment can be wrong and limit functional competence due to lack of grounding. In this paper, we study an approach (named GLAM) to achieve this alignment through functional grounding: we consider an agent using an LLM as a policy that is progressively updated as the agent interacts with the environment, leveraging online Reinforcement Learning to improve its performance to solve goals. Using an interactive textual environment designed to study higher-level forms of functional grounding, and a set of spatial and navigation tasks, we study several scientific questions: 1) Can LLMs boost sample efficiency for online learning of various RL tasks? 2) How can it boost different forms of generalization? 3) What is the impact of online learning? We study these questions by functionally grounding several variants (size, architecture) of FLAN-T5.
• Functional Description:
  GLAM is a new approach to achieve alignment between a Large Language Model (LLM) and a considered environment/world through functional grounding: we consider an agent using an LLM as a policy that is progressively updated as the agent interacts with the environment, leveraging online Reinforcement Learning to improve its performance to solve goals.
• URL:
  https://­github.­com/­flowersteam/­Grounding_LLMs_with_online_RL
• Publication:
  hal-03970122
• Contact:
  Clément Romac

7.1.19 ER-MRL

• Keywords:
  Reservoir Computing, Reinforcement learning
• Functional Description:
  Code for the Evolving-Reservoirs-for-Meta-Reinforcement-Learning (ER-MRL) paper. Our goal is to study the following question : How neural structures, optimized at an evolutionary scale, can enhance the capabilities of agents to learn complex tasks at a developmental scale? To achieve this, we adopt a computational framework based on meta reinforcement learning, modeling the interplay between evolution and development. At the evolutionary scale, we evolve reservoirs, a family of recurrent neural networks generated from hyperparameters. These evolved reservoirs are then utilized to facilitate the learning of a behavioral policy through reinforcement learning. This is done by encoding the environment state through the reservoir before presenting it to the agent.
• URL:
  https://­github.­com/­corentinlger/­ER-MRL
• Contact:
  Corentin Leger

7.1.20 ecoevojax_analysis

• Keywords:
  Evolutionary Algorithms, Evolution, Grid
• Functional Description:
  Code for the paper "Eco-evolutionary dynamics of non-episodic neuroevolution in large multi-agent environments." https://dl.acm.org/doi/10.1145/3583133.3590703. In this work we have proposed a grid-world foraging environment with reset-less neuroevolution. It is implemented in JAX and can support very large population and grid sizes. For example, evolving a population of 1000 agents in a world of 100 K pixels for 1 million time steps takes about 10 minutes on a GPU.
• URL:
  https://­github.­com/­eleninisioti/­ecoevojax_analysis
• Contact:
  Gautier Hamon

7.1.21 SBMLtoODEjax

• Keywords:
  SBML, JAX, Python, Numerical simulations, Numerical optimization, Automatic differentiation, Ordinary differential equations, Biomedical data
• Scientific Description:
  Advances in bioengineering and biomedicine demand a deep understanding of the dynamic behavior of biological systems, ranging from protein pathways to complex cellular processes. Biological networks like gene regulatory networks and protein pathways are key drivers of embryogenesis and physiological processes. Comprehending their diverse behaviors is essential for tackling diseases, including cancer, as well as for engineering novel biological constructs. Despite the availability of extensive mathematical models represented in Systems Biology Markup Language (SBML), researchers face significant challenges in exploring the full spectrum of behaviors and optimizing interventions to efficiently shape those behaviors. Existing tools designed for simulation of biological network models are not tailored to facilitate interventions on network dynamics nor to facilitate automated discovery. Leveraging recent developments in machine learning (ML), this paper introduces SBMLtoODEjax, a lightweight library designed to seamlessly integrate SBML models with ML-supported pipelines, powered by JAX. SBMLtoODEjax facilitates the reuse and customization of SBML-based models, harnessing JAX's capabilities for efficient parallel simulations and optimization, with the aim to accelerate research in biological network analysis.
• Functional Description:
  SBMLtoODEjax extends SBMLtoODEpy, a python library developed in 2019 for converting SBML files into python files written in Numpy/Scipy. The chosen conventions for the generated variables and modules are slightly different from the standard SBML conventions (used in the SBMLtoODEpy library) with the aim here to accommodate for more flexible manipulations while preserving JAX-like functional programming style.
• URL:
  https://­developmentalsystems.­org/­sbmltoodejax/
• Publication:
  hal-04317246
• Contact:
  Mayalen Etcheverry
• Partner:
  Tufts University

7.2.1 ToGather application

Participants: Cécile Mazon, Hélène Sauzéon, Eric Meyer, Isabeau Saint-Supery.

• Name:
  Application for Specialized education
• Keywords:
  Parent-professional relationships; user-centered design; school inclusion; autism spectrum disorder; ecosystemic approach
• Participants:
  Isabeau Saint-supery, Cécile Mazon, Hélène Sauzéon, Agilonaute
• Scientific Description:
  With participatory design methods, we have designed an interactive website application for educational purposes. This application aims to provide interactive services with continuously updated content for the stakeholders of school inclusion of children with specific educational needs. Especially, the services provide: 1) the student's profile with strengths and weaknesses; 2) an evaluation and monitoring over time of the student's repertoire of acquired, emerging or targeted skills; 3) a shared notebook of effective psycho-educational solutions for the student ; 4) a shared messaging system for exchanging "news" about the student and his/her family and, 5) a meeting manager allowing updates of evaluations (student progress). This application is currently assessed with a field study. Then, it will be transferred to the Academy of Nouvelle-Aquitaine-Bordeaux of the National Education Ministery.
• URL:
  The website is not online yet, but all informations such as tutorials are https://flowers.inria.fr/projet-tous-ensemble/ .
• Publication:
  hal-03436355

8.1.1 Language-Model-Augmented Autotelic Agents, Towards Open-Ended Skill Discovery

In this project, we take a step towards truly open-ended language-based autotelic agents by leveraging GPT3 99, a Large Language Model (LLM) demonstrating impressive language understanding capabilities. For an autotelic agent to be truly open-ended, it needs to be able to:

• Generate its own goals creatively (Goal generator)
• For an arbitrary goal, decide whether a given trajectory achieved the goal or not (Reward function)
• For a given trajectory, give a list of relevant achieved goals (Relabeller or Social Partner)

In this project, we also place ourselves in a textual environment. CookingWorld has been proposed as part of the First TextWorld Problems competition on text agents, and features a house with ingredients that can be cooked to achieve a recipe. We leverage the textual nature of the trajectories of agents in this environment by using GPT (ChatGPT-3.5 in our experiments), with different prompts, as our goal generator, reward function, and relabeller. More precisely (see Figure 2):

• Goal generator: we present GPT3 with a prompt composed of a context explanation ("We are in this video game environment"), a list of previously seen rooms, a list of previously seen objects, a list of previously achieved goals, possibly a past trajectory example, and ask it to generate new goals;
• Reward function: given an arbitrary language goal and a trajectory including actions and observations, labelled with the steps index at which those actions and observations were issued, we ask GPT to decide whether a given goal is achieved or not in the trajectory, and if it is, at which time step. In this way, we only need to call the GPT-based reward function once per episode (compared with once per step in a traditional RL setup), significantly speeding up our experiments. since the GPT calls are expensive.
• Relabeller: Once a trajectory is finished, we show GPT the completed trajectory and ask it to provide a list of goals achieved in this trajectory. We extract these goals and we add the relabelled trajectories to the replay buffer. The relabelled goals are given again to the reward function for it to double-check correctness.

Goal-conditioned policy implementation

Once a goal has been selected by the Goal Generator we need to be able to pursue this goal in the environment. In preliminary experiments we noticed that Deep RL based agents were too sample inefficient to be able to learn an open-ended repertoire of skills in a reasonable time. The solution we implemented is closer to an evolutionary algorithm: the policy maintains a dictionary mapping already experienced goal strings to the shortest experienced sequence of action achieving this goal (according to the relabeller or the reward function). When presented with a new goal, the agent embeds this goal and selects the sequence of actions whose goal key in the dictionary is the closest in embedding space to the one being executed (this match may be exact). The agent then executes this action sequence.

Goal Generator details

The LM-based Goal generator starts off with a warmup period of 4000 episodes, where goals are sampled based on the ones that have already been experienced. After this period the goal imagination kicks in, and the language model is shown in its prompt a sample of 60 goals that have been achieved in the past. The goal generator is asked to provide a high-level goal that is composed of the chaining of lower level subgoals among the list of goals in the prompt. The goal-conditioned policy then executes the subgoals one after the other, and there is a probability that the last subgoal be cutoff and random exploration be performed. In the random exploration text actions are given priority inversely proportional to the amount of time they have been seen (to encourage the agent to take rare actions when they become available). The overview of the algorithm, as well as an illustration of the information flow, are given in Figures 2 and 3. Figure 4 shows a confusion matrix quantifying how much the reward function agrees with human judgement.

Results

Our results show that LMA3, across methods, is able to discover a very large number of goals diverse goals, on the order of 1 goal per episode. The goals are distinct and also exhibit high diversity in the predicates and objects they cover. We see that removing human tips reduces the diversity of goals generated, because the LM has less diverse examples on which to base itself. Removing the LM goal generation has minimal influence beyond that, and removing chain-of-thought has the most drastic effect. The baseline oracle agent is limited to the 69 goals for which we have an exact reward function and relabeler. We additionally evaluate the agent on the oracle goals for which we have an exact reward function. This is represented in Figure 4. We see that most LMA3 variants achieve a high eval score on these held-out goals, despite never having been told what it would be evaluated on.

This work was presented at the Conference on Lifelong Learning Agents (August 2023) in Montreal.

8.1.2 Code-Generating Autotelic Agents

In this section we will present our recent work on code-generating autotelic agents. The idea of exploring this came from the realization that the open-endedness of our agents is upper-bounded by the complexity of the environment. The LMA3 agent was very powerful but could never learn complex skills because CookingWorld is extremely limited. On the other hand, programming is open-ended and an agent writing code has very few limits on the complexity of what it can accomplish.

We will present two works in a programming domain: Codeplay and ACES.

8.1.3 Codeplay: Autotelic Learning through Collaborative Self-Play in Programming Environments

Participants: Laetitia Teodorescu [correspondant], Cédric Colas, Matthew Bowers, Thomas Carta, Pierre-Yves Oudeyer.

In this work we propose an approach that is both a way to implement autotelic agents discovering a truly open-ended set of goals, and a way to allow language models to master novel coding skills in interaction with an interpreter. We ground LM-agents in a coding environment that provides straightforward binary rewards through code execution. We set ourselves in the Python Programming Puzzles domain 188, and we define an autotelic agent composed of 2 sub-agents: a Setter that generates puzzles, and a Solver whose objective is to solve the generated puzzles. Both agents are LM policies. They play a collaborative game where the Setter has to create puzzles that push the solver to learn, and the solver sees and tries to solve puzzles in its zone of proximal development (ZPD): hard but still solvable. First experiments presented here (with a fixed Solver) highlight the possibility to tradeoff difficulty of the generated puzzles and their novelty by tuning the reward experienced by the Setter, trained with deep RL (PPO:  186).

Related work

Closely related to this work are approaches to autotelic learning or automatic curriculum learning involving goal setter agents 123, 196, 101, 165, 158, as well as the PowerPlay framework of 184. Very closely related to this work as well are approaches for generating novel code puzzles for augmenting the capabilities of code-puzzle-solving language models 127, as well as recent attempts to cast program synthesis as a reinforcement learning problem 144, 208.

Prompts

We use a fixed few-shot prompt for both the solver and the setter. The few-shot prompt simply includes a series of puzzles and solutions from the tutorial set of the P3 training set (which are pretty short, so we can fit all of them). The puzzles and solutions are separated by assert(f(g())) statements. The prompt for the Setter finishes after an assertion, the prompt for the Solver (evaluated $N_a$ times to produce the difficulty estimation) additionally includes the puzzle to be solved at the end.

8.1.4 ACES: Generating diverse programming puzzles with autotelic language models and semantic descriptors

Participants: Julien Pourcel [correspondant], Cédric Colas, Pierre-Yves Oudeyer, Laetitia Teodorescu.

Results

We report results of our runs in Figure 10. Overall, the methods based on semantic archives, ACES and ELM-Semantic, achieve the highest diversity in the semantic space. We report diversity measures inside the embedding spaces of various smaller language models in Figure 11. In these figures we see that overall ACES outperforms other methods in this measure of diversity. We additionally perform tests of the suitability of generated puzzles as finetuning data for smaller LMs. For all methods, we finetune a smaller model (OpenLlama-3b) on the generated set and we test the pass@k metric for different values of k on the P3 test set; we report the scores in Figure 12. From that figure we see that we encounter a tradeoff between how diverse the data is and how useful it is to get a high score on the P3 test set. Further work is needed to get data that is both diverse and useful.a

[ ]

 [ ]

 [ ]

[ ]

          [ ]

[ ]

 [ ]

 [ ]

8.1.5 Grounding Large Language Models in Interactive Environments with Online Reinforcement Learning (GLAM)

Participants: Thomas Carta, Clément Romac, Pierre-Yves Oudeyer, Olivier Sigaud [ISIR Sorbonne Université, Paris, France], Sylvain Lamprier [Univ Angers, LERIA], Thomas Wolf [Hugging Face].

The recent rise of Transformer-based Large Language Models (LLMs) trained on massive text datasets has led to models exhibiting impressive capabilities (e.g. natural language generation, question answering, reasoning, translation...). Recently, LLMs were shown to also capture aspects of the physical rules in our world, e.g. about space, colors or even affordances between bodies and objects.

However, LLMs are known to suffer from a lack of grounding (i.e. connecting their inner representation of language to a world) which prevents them from properly dealing with the meaning of concepts and their direct use to solve tasks in interactive environments. Indeed, alignment between statistical structures in such LLMs and environments can be very limited, or even sometimes entirely wrong. This is partly due to 1) a training process (predicting next words) that is not directly incentivized to solve problems in an environment, 2) lack of abilities to intervene in the environment to identify causal structures; 3) lack in abilities to learn based on data collected as a result of interacting with the environment.

Focusing on such a functional competence, we propose to use an LLM as the policy interacting with a textual environment (i.e. a textual description of the scene is provided by the environment and possible actions are text commands) for decision-making problems. Using Reinforcement Learning (RL) to finetune the LLM to make it solve various tasks in this environment, our method named GLAM “functionally grounds” the LLM on the environment. That is, grounding the dynamics and physical rules of an environment to solve problems and obtain, in the end, an operational LLM able to use natural language to solve tasks in this interactive environment.

Schema of how GLAM metod.

Applying GLAM to Flan-T5 780M shows how such an LLM can be functionally grounded and be able to solve the tasks in an interactive textual environment. The resulting LLM also exhibits strong generalization when exposed to variations in the objects it must interact with. Finally, we show that incremental intervention using RL is key by comparing it to passive Imitation Learning. Our results highlight that our GLAM method leads to both better results and generalization even when Imitation Learning is performed using an optimal policy.

8.1.6 Automatic Curriculum Learning for Language Modeling

Participants: Clément Romac [correspondant], Rémy Portelas, Pierre-Yves Oudeyer.

We showed in recent works how Automatic Curriculum Learning (ACL) could help Deep Reinforcement Learning methods by tayloring a curriculum adapted to learner's capabilities 30, 27. Using ACL can lead to sample efficiency, asymptotic performance boost and help in solving hard tasks.

Parallel to this, recent works in Language Modeling using Transformers (e.g. GPT-2) have starting to get more interested in better understanding convergence and learning dynamics of these models. Trained in a supervised setup, these models are fed with hundred of millions of natural language sequences crawled from the web. The current standard way of training these models (i.e. constructing batches of randomly selected sequences) makes the assumption that all sequences have same interest for the model. However, recent works showed that this does not seem to be the case and that datasets can contain outliers harming training. Additionally, some works also showed that hand-designing a curriculum over sequences (e.g. ordered by their length) could speed up and stabilize training.

Building on this, we propose to experiment how ACL could help taylor such a curriculum in an automated way relying on Learning Progress. Our study has several contributions:

• Propose a standardized and more in-depth comparison of current curriculum learning methods used to train language models
• Introduce the first study of ACL in such a training
• Use ACL to propose deeper insights about training dynamics of Transformer models when doing Language Modeling by analysing generated curricula and Learning Progress estimations

We chose to train GPT-2 on the standard OSCAR dataset and use teacher algorithms to select samples that are shown to the model (see fig. 14).

Using ACL, we perform an in-depth analysis of prior methods changing the size of tokens' sequences observed during training following a hand-designed curriculum. Our experiments showed that a Random baseline outperforms these methods. We also provide, thanks to ACL methods based on Learning-Progress Multi-Armed Bandits, hints that while short sequences should not be used as training advances (as Large Language Models quickly learn them), there is no clear evidence that short sequences should be prioritized (and thus long sequences avoided) at the beginning of training.

Additionally, we performed several experiments using more advanced ACL methods on different task spaces and show that these lead to overfitting and underperform in comparison the no-curriculum strategy usually applied in Language Modeling. We hypothesize that, given how large models used in Language Modeling are, it is better to give a huge amount of very diverse and different samples (even though outliers or harmful samples exist) without any curriculum than using a curriculum that restrains the diversity of samples and introduces duplicates (leading to overfitting).

Schema of how ACL was integrated to Language Modeling.

8.2.1 The effect of social network structure on collective innovation

Participants: Eleni Nisioti [correspondant], Mateo Mahaut, Pierre-Yves Oudeyer, Ida Momennejad, Clément Moulin-Frier.

Innovations are a central component of open-ended skill acquisition: they denote the emergence of new solutions by the recombination of existing ones and their presence is necessary to ensure a continuous complexification of an agent's cultural repertoire. While we often tend to attribute discoveries to certain innovative individuals, if we shed a broad perspective at the history of our species we see that human innovation is primarily a collective process. Fields such as psychology and anthropology have been studying the ability of human groups to innovate for some time, with studies indicating that the social network structure has a significant impact: fully-connected structures are better suited for quick convergence in easy problems with clear global optima, while partially-connected structures perform best in difficult tasks where local optima may lure agents away from the globally optimal solution 119. At the same time a parallel story is unfolding in reinforcement learning (RL): distributed RL is a sub-field where multiple agents solve a task collectively 159. Compared to the single-agent paradigm, distributed RL algorithms converge quicker and often achieve superior performance. However, these algorithms have only considered full connectivity. In this inter-disciplinary project, we presented a novel learning framework that augments distributed RL with the notion of a social network structure and employed it to study the hypothesis from human studies that partial connectivity performs best in innovation tasks.

We implemented such innovation tasks using Wordcraft, a recently introduced RL playground inspired from the Little Alchemy 2 game (see left of figure 15 for an illustration of how this task works). We considered a wide diversity of social network structures: static structures that remain constant throughout learning (fully-connected, ring, small-world) and a dynamic structure where the group oscillates between phases of low and high connectivity (we illustrate this dynamic structure on the right of figure 15). Each agent in our implementation employs the DQN learning algorithm and exchanges experiences that have the form of sequences of state-action combinations with its neighbors.

A central conclusion of our empirical analysis was that the dynamic social network structure performs best. In addition to the performance groups achieve we measured behavioral and mnemonic metrics such as behavioral conformity and mnemonic diversity. Such metrics were inspired from human studies and helped us further analyze the behavior of groups. For example, one empirical observation was that sharing experiences did not help the group learn quicker in a very simple innovation task; instead the fully-connected group was the slowest. By looking at the diversity in the memories of the agents we observed that the fully-connected structure had the highest individual diversity (left of figure 16 ) and the lowest group diversity (right of figure 16): sharing experiences with others diversifies an individual's experiences but also homogenizes the group, which is bad for its performance.

We see the contribution of this project as two-fold. From the perspective of fields studying human intelligence, we have shown that using RL algorithms as computational tool is a promising direction towards increasing the verisimilitude of simulations and analyzing both behavior and memory. From the perspective of RL, we have shown that distributed RL algorithm should move beyond the fully-connected architecture and explore groups with dynamic topologies. This work is currently a preprint 162 and is about to be submitted in PNAS. We open-source the code at this link.

8.2.2 Cultural evolution in population with heterogeneous and variable preferences

Participants: Jérémy Perez [correspondant], Martí Sànchez Fibla, Clément Moulin-Frier.

Theoretical and empirical work in cultural evolution often assume populations in which individuals all agree on the quality of a given cultural trait (i.e. they have homogeneous preferences), and where those preferences are stable in time. Yet, these assumptions are not always met: for example, an uneven distribution of information in a population could lead to heterogeneous preferences; moreover, in some cultural domains (e.g. aesthetic culture), diverse preferences may be the norm rather than the exception. In this project in collaboration with Martí Sànchez Fibla from Universitat Pompeu Fabra, we designed an agent-based model in which we can control the heterogeneity of preferences, as well as the effect of cultural traits on the evolution of preferences. We find that assuming homogeneous or heterogeneous preferences leads to different predictions on several outcomes. First, populations with greater heterogeneity of preferences converge toward greater cultural diversity. Second, while we replicate the classical result that increasing opportunities to learn socially leads to less diversity in homogeneous populations, we find that this relationship is reversed in heterogeneous populations. We show that this happens because increasing social learning opportunities leads the distribution of cultural traits to converge toward the distribution of preferences. We also look at the consequences of allowing cultural traits to modify the preferences of individuals that possess them. This can for example capture self-reinforcing beliefs, or traits where the acquisition costs make individuals less likely to switch to another trait after possessing them for some time. We find that such “attractive” cultural traits naturally emerge in our model, and that they tend to decrease cultural diversity when preferences are not homogeneous. Overall, by showing that the effect of different parameters on cultural diversity are dependent on the assumed distribution of preferences, we highlight the importance of taking into account the possible heterogeneity of preferences when making predictions about cultural dynamics. An abstract for a poster was submitted to the conference of the European Human Behaviour and Evolution Association (EHBEA 2024).

8.2.3 Interactions between intrinsically motivated goal-exploration processes and cummulative cultural evolution

Participants: Jérémy Perez [correspondant], Maxime Derex, Pierre-Yves Oudeyer, Clément Moulin-Frier.

Cumulative culture describes the gradual accumulation and spread of innovations in a population, which results in the formation of a cultural repertoire that no individual could have independently invented on its own. As cumulative culture has been argued to underlie the ecological success of humans, understanding the mechanisms that underpin it has raised a lot of interest. However, computational models of cultural evolution have mostly been restricted agent-based models, which made it difficult to study the consequences of some cognitive capacities on cultural dynamics. In particular, these models often assume that cultural variation is generated in a random manner, which overlooks the sophisticated exploration strategies employed by humans and other animals, such as curiosity-driven exploration. In this project, we aim to fill this gap by modeling agents as reinforcement learners endowed with an intrinsic motivation to generate and pursue their own goals. This will allow to study how a group of curious agents may take cultural trajectories compared to non-curious agents, as well as to look at more sophisticated forms of cultural transmission such as goal transmission. This project is done in collaboration with Maxime Derex (IAST, Toulouse)

8.2.4 Learning and Self-organization of Cultural Conventions Between Artificial Agents

Participants: Tristan Karch [correspondant], Clément Moulin-frier, Pierre-Yves Oudeyer.

As introduced earlier, Vygotskian artificial agents internalize cultural conventions in order to transform linguistic production into cognitive tools that help them acquire new skills. A fundamental question is therefore to investig,ate how such cultural conventions can emerge between agents situated in social contexts.

Self-organizing cultural conventions in a new interactive AI paradigm: the Architect-Builder Problem

In this experiment, we are interested in interactive agents that learn to coordinate, namely, a builder – which performs actions but ignores the goal of the task, i.e. has no access to rewards – and an architect which guides the builder towards the goal of the task. We define and explore a formal setting where artificial agents are equipped with mechanisms that allow them to simultaneously learn a task while at the same time evolving a shared communication protocol. Ideally, such learning should only rely on high-level communication priors and be able to handle a large variety of tasks and meanings while deriving communication protocols that can be reused across tasks. We present the Architect-Builder Problem (ABP): an asymmetrical setting in which an architect must learn to guide a builder towards constructing a specific structure. The architect knows the target structure but cannot act in the environment and can only send arbitrary messages to the builder. The builder on the other hand can act in the environment, but receives no rewards nor has any knowledge about the task, and must learn to solve it relying only on the messages sent by the architect. Crucially, the meaning of messages is initially not defined nor shared between the agents but must be negotiated throughout learning. The Architect-Builder problem was initially introduced by Vollmer et. al 204 in an experiment named the CoCo game studying the formation of communication protocol between humans in such a context. Diagrams of interactions in the CoCo game and in our numerical adaptation are given in figure 17.

(a)	(b)

(a) Schematic view of the CoCo Game (the inspiration for ABP). The architect and the builder should collaborate in order to build the construction target while located in different rooms. The architecture has a picture of the target while the builder has access to the blocks. The architect monitors the builder workspace via a ca,mera (video stream) and can communicate with the builder only through the use of 10 symbols (button events). (b) Interaction diagram between the agents and the environment in our proposed ABP. The architect communicates messages ($m$) to the builder. Only the builder can act ($a$) in the environmen,t. The builder conditions its action on the message sent by the builder ($\left(a\right|s,m)$). The builder nev,er perceives any reward from the environment.

Under these constraints, we propose Architect-Builder Iterated Guiding (ABIG), a solution to ABP where the architect leverages a learned ,model of the builder to guide ,it while the builder uses self-imitation learning to reinforce its guided behavior. We analyze the key learning mechanisms of ABIG and te, i,t in 2D tasks involving grasping cubes, placing them at a given location, or building various shapes. ABIG results in a low-level, high-frequency, guiding communication protocol that not only enables an architect-builder pair to solve the task at hand, but that can also generalize to unseen tasks as illustrated in figure 18. These results were published at the International Conference on Representation Learning (ICLR 2022) 91.

ABIG transfer performances without retraining depending on the training goal. ABIG agents learn a communication protocol that transfers to new tasks. Highest performances reached when training on `place'.

8.2.5 Autotelic Reinforcement Learning in Multi-Agent Environments

Participants: Eleni Nisioti, Elias Masquil, Gautier Hamon [correspondant], Clément Moulin-Frier.

The intrinsically-motivated goal-conditioned learning paradigm is well-established in single-agent settings: by setting its own goals and pursuing them in an environment without external supervision, such an agent is able to acquire a wide diversity of skills 112. Such agents are called autotelic, from the Greek words auto (self) and telos (end). What happens when you transfer the autotelic paradigm to multi-agent environments, where some skills may require the cooperation of multiple agents (lifting a heavy box for example)? This is the question we aimed at addressing in this project. We believe that multi-agent applications will benefit from agents that can autonomously discover and learn cooperative skills, but entail additional challenges to the ones found in single-agent settings: agents that independently set their own goals will have a very low probability of simultaneously sampling the same cooperative goal, which will make solving these goals difficult.

To explore this question, we implemented what we call cooperative navigation tasks in the Simple Playgrounds environments. This 2-D environment, illustrated on the left of figure 19, consists of a room with 6 landmarks on its walls and two agents that receive continuous-valued observations about the distance and angle to all landmarks and other agents and perform discrete-valued actions that control their angular velocity and longitudinal force. A navigation task is a description of the landmarks that need to be reached: some tasks are individual (for example "at least one agent reaches the red landmark") and some are cooperative (for example "at least one agent reaches the red landmark and at least one agent reaches the blue landmark"). Each autotelic agent is learning using the RL algorithm PPO and, at each training episode, chooses which goal to pursue by random sampling within the goal distribution. In addition to a policy conditioned on its goals, the agent also needs a reward function that indicates whether a goal is achieved (see the schematic on the right of figure 19 for an illustration of the main algorithmic components of autotelic agents). In this project, we assume that the two agents already know this reward function and focus on examining the process of decentralized goal selection.

(a)	(b)

(left) The Cooperative landmarks environment consists of a room with two agents and six (right) landmarks. Landmarks are indicated as colored rectangles and navigation tasks are formulated as a set of landmarks the agent needs to navigate to, which might require coordination between the agents. (Right) Two autotelic agents in a multi-agent environment: the agents can exchange messages and condition their goal selection on them, which enables goal alignment.

Our empirical analysis of this set-up showed that goal alignment is important for solving the cooperative tasks we considered: agents that were independently sampling the goals failed to solve all tasks (see orange curve in figure 20) while agents whose goals were determined by a centralized process (blue curve) that guaranteed that the two agents are always pursuing the same goal performed optimally. We then wondered: can we achieve the same performance without requiring centralization? To achieve this we designed a communication-based algorithm that enables a group to align its goals while remaining decentralized: at the beginning of an episode and before determining their goals the two agents exchange messages and then use these messages to condition their goal-selection (see dashed arrows in the schematic on the right of figure 19). This communication is asymmetric: one randomly chosen agent, the leader, uses its goal generator to choose which goal to pursue and then decides what message to transmit to the follower, which conditions its goal selection on the received message. We observed that the agents learn a communication protocol that leads to the alignment of cooperative goals, even though they were not directly incentivised to do so. They were both independently learning a protocol that maximised their individual rewards but, as we show in our experiments corresponding to figure 20, goal alignment was able to emerge from such decentralized learning. We called this algorithm the Goal-coordination game, as it was inspired from another emergent communication algorithm called the Naming game 194.

To get a better understanding of how alignment helps solve this task we measured specialization, which is the tendency of agents to always go the same landmark when there are two options. For example, if for the goal "at least one agent reaches the red landmark and at least one agent reaches the blue landmark" the first agent always go to red and the second goes to blue, then specialization is maximum. We empirically observed that specialization correlates with alignment and is an optimal strategy in our tasks (see right plot of figure 20).

In 2023, as advised by reviewers, additional experiments on baselines to compare were conducted. We also conducted further experiments trying to understand better what are the causes of this inability to learn in the independant (0%align) case.

This work was accepted at the Conference on Lifelong Learning Agents (CoLLAs) 2023 and presented at the poster session. A preprint version is available on HAL 52. The source code for reproducing the experiments is available at this link.

8.2.6 The SocialAI School: Insights from Developmental Psychology Towards Artificial Socio-Cultural Agents

Participants: Grgur Kovač [correspondant], Remy Portelas, Peter Ford Dominey, Pierre-Yves Oudeyer.

Developmental psychologists have long-established socio-cognitive abilities as a core aspect of human intelligence and development 205100. Those abilities enable us to enter, participate in and benefit from human culture. Humans are then able to push our culture forward by improving on already existing cultural artifacts. This is referred to as the cumulative cultural evolution, and it has been argued that the most impressive of human achievements are the product of it 201. It seems clear that to construct artificial agents capable of interacting with and learning from humans one must equip them with socio-cognitive abilities enabling them to enter the human culture. This would enable artificial agents to benefit from our culture, and also push it forwards, i.e. to participate in the cumulative cultural evolution.

Current AI research mostly studies asocial settings or, in the case of Multi-Agent Reinforcement Learning, the emergence of culture (how culture emerges in the first place, rather than how one enters an already existing culture). Furthermore, this research is often done without a strong grounding in developmental psychology.

In this project, we follow the work of Michael Tomasello and Jerome Bruner who outlined a set of core socio-cognitive skills, e.g. social cognition (joint attention, theory of mind), referential and conventionalized communication, imitation, role-reversal, scaffolding, and many more 200, 100. Importantly, they also showed the importance the (cultural) environment for cognitive development.

To introduce some of those concepts to the AI community, we created a tool for procedural generation of environments - The SocialAI school. With The SocialAI school, experiments studying those socio-cognitive abilities can be easily conducted and cognitive-science experiments reconstructed. Furthermore, The SocialAI School enables us to generate both multimodal environments (suited for RL agents) and pure text version of those enviroments (suited for LLM-based agents). An example of a SocialAI school environment is shown in figure 21. In it the peer is pointing towards the red box. The agent (the red triangle) has to infer this to mean that the apple is hidden inside the red box.

We conducted many experiments, here we outline a few more important ones. We experimented with with multimodal RL agents. We tested generalization of inferring the meaning of referential communication (ex. the pointing gesture) to new scenarios/objects. We found that such a generalization is very hard for standard reinforcement learning agents. We show how a scaffolded environment helps with learning complex interaction sequences (formats). To show how cognitve science experiments can be recreated - we reconstruct a study of role reversal from 122. Furthermore, we conducted experiments regarding other aspects of social-cognition: joint attention, imitation, perspective taking, etc. We experimented with LLM-based interactive agents. We show that a simple LLM-based agent can achieve some perfomrance but still fails to generalize. This motivates future work for creating more complex LLM-based agents.

An example of an environment that can be created with the The SocialAI school. The red peer is pointing toward the red box. The task of the agent (the red triangle) is to infer this to mean that the red box contains an apple.

Most of this project was done in 2022. In this year, we extended the project by adding additional experiments with LLM-based agnets and updating the implementation.

8.2.7 Value stability in Large Language Models

Participants: Grgur Kovač [correspondant], Masataka Sawayama, Remy Portelas, Peter Ford Dominey, Pierre-Yves Oudeyer.

There has been a growing body of research using Large Language Models (LLMs) to simulate individuals or human populations. Those studies usually focus on how a model can express some behavior or values and often overlook the underlying problem that LLMs are highly context-depenant. This problem is even more exacarbated by the use of psychological questionnares, which were created with the assumption of human-like context-independence. In this project 74, we robustness of LLMs to seemingly unrelated perturbations in the context. Instead of evaluating models on some behavior (and testing robustness along the way), we study how a model's behavior changes over contexts as a primary question. In other words, we study and compare the value stability of LLMs along different contexts.

We leverage the PVQ questionnaire 106 associated with the Schwartz Theory of Basic Values 189, which defines ten values: Self-Direction, Stimulation, Hedonism, Achievement, Power, Security, Conformity, Tradition, Benevolence, Universalism. We study the LLMs abilities to simualte an individual (i.e. without defining a specific persona to simulate) and populations (by instructing the model to simulate various well-known fictional and real-world individuals). Different contexts are induced by simulating conversations on different topics: correcting grammar, writing a poem, playing chess, answering a history question and inventing a joke.

Following psychological methodology we evaluate three types of value stability: mean-level, rank-order, ipsative. We observe that LLMs exhibit low value stability - trivial context changes induce similar or bigger changes in value expression, than those observed in humans as a consequence of much more aggressive circumstances (e.g. 10 years of development or priming in humans compared to a topic change in LLMs). These results push furhter the point that psychological questionnaires administered to LLMs cannot be used and interpreted in the same way as with humans, i.e. much more attention must be given to studing context-dependance. Most importantly, they imply that rather than evaluating many questions from a single context (as is currenlty common), one should also evaluate same questions from many differt contexts. We propose metrics based on these types of value stability, and systematically compare models in terms of their value stability (see figure 22). To our knowledge we propose the first systematic comparison of many different models on their value stability.

Rank-order stabilty of different Large Language Models

8.3.1 Research perspective: The Ecology of Open-Ended skill Acquisition

Participants: Clément Moulin-Frier [correspondant], Eleni Nisioti, Pierre-Yves Oudeyer.

An intriguing feature of the human species is our ability to continuously invent new problems and to proactively acquiring new skills in order to solve them: what is called Open-Ended Skill Acquisition (OESA). Understanding the mechanisms underlying OESA is an important scientific challenge in both cognitive science (e.g. by studying infant cognitive development) and in artificial intelligence (aiming at computational architectures capable of open-ended learning). Both fields, however, mostly focus on cognitive and social mechanisms at the scale of an individual’s life. It is rarely acknowledged that OESA, an ability that is fundamentally related to the characteristics of human intelligence, has been necessarily shaped by ecological, evolutionary and cultural mechanisms interacting at multiple spatiotemporal scales.

The ORIGINS framework identifies central components (boxes) and their interactions (arrows) driving Open-Ended Skill Acquisition, both in terms of its evolution from environmental complexity (roughly: left to right arrows) as well its open-ended aspect through feedback mechanisms (right to left arrows). The employed terminology reflects a diversity of mechanisms considered in both Artificial Intelligence and Human Behavioral Ecology.

We have recently initiated a new research direction aiming at understanding, modeling and simulating the dynamics of OESA in artificial systems, grounded in theories studying its eco-evolutionary bases in the human species. For this aim, we have proposed a conceptual framework, called ORIGINS (illustrated Fig. 23 and developed in 157), expressing the complex interactions between environmental, adaptive, multi-agent and cultural dynamics. This framework raises three main research questions:

• What are the ecological conditions favoring the evolution of autotelic agents?
• How to bootstrap the formation of a cultural repertoire in populations of adaptive agents?
• What is the role of cultural feedback effects in the open-ended dynamics of human skill acquisition?

The contributions described below are addressing some aspects of these research questions. Note that there might be a thematic overlap between the two last research questions outlined above and the previous section on Models of Cultural Evolution 8.2, where we also present related results.

8.3.2 Evolution of plasticity and evolvability in variable environments

Participants: Eleni Nisioti [correspondant], Clément Moulin-Frier.

The diversity and quality of natural systems have been a puzzle and inspiration for communities studying artificial life. It is now widely admitted that the adaptation mechanisms enabling these properties are largely influenced by the environments they inhabit. Organisms facing environmental variability have two alternative adaptation mechanisms operating at different timescales: plasticity, the ability of a phenotype to survive in diverse environments and evolvability, the ability to adapt through mutations. Although vital under environmental variability, both mechanisms are associated with fitness costs hypothesized to render them unnecessary in stable environments. In this work, we aimed at studying the interplay between environmental dynamics and adaptation in a minimal model of the evolution of plasticity and evolvability.

To achieve this we designed a simulation environment that attempts to capture the spatial and temporal heterogeneity of real-world environments while keeping the computational complexity low: the user can choose the number of niches, which are arranged in a simple longitudinal model, and a climate function that captures the temporal variation of environmental conditions, which are arranged based on a simple longitudinal model (see left of figure 24 for an illustration of the environment). We defined the evolvability of an agent as its mutation rate and capture plasticity using tolerance curves, a tool developed in ecology 126. Tolerance curves (which we visualize on the right of Figure 24) have the form of a Gaussian whose mean shows the preferred environmental state of an individual and the variance its plasticity, i.e., its ability to survive under different environmental conditions. This figure also illustrates the cost and benefit of plasticity. If both individuals are at their preferred niche, which coincides with the environmental state, then the plastic individual has lower fitness that the non-plastic (cost of plasticity). If the actual environmental state differs significantly from the preferred one, the plastic individual has higher fitness (benefit of plasticity).

 

We conducted an extensive empirical study in this environment that aimed at disentangling the effects of different mechanisms: we studied three types of climate functions (stable, sinusoid, noisy) and two types of evolutionary selection pressures (survival of the fittest and niche-limited competition) and environments were the number of niches varies from 1 to 100. Through these simulations we showed that environmental dynamics affect plasticity and evolvability differently and that the selection mechanism matters: a) in stable environments with a large number of niches when both selection-based fitness and niche-limited competition (we call this method NF-selection) are activated, plasticity remains high despite its cost (see left plot in Figure 25) ; b) in a noisy environment introducing niche-limited competition (N-selection and NF-selection) makes populations capable of resisting larger amounts of noise (see right plot in Figure 25). We presented our work at GECCO 2022 164 and open-sourced the software for reproducing our simulations in this repository. A follow-up of this work has been published at the ALIFE 2023 conference 53, where we introduced mechanisms of niche constrution to this model.

 

8.3.3 Open-ended recipe crafting through meta reinforcement learning

Participants: Gautier Hamon [correspondant], Eleni Nisioti, Clément Moulin-Frier.

As a first step towards studying the evolution of open-ended skill acquisition in artificial agents, we studied the environmental conditions favoring the systematic exploration of combinatorial recipes involving various objects. By combinatorial recipe, we mean the ability of agents to combine objects in the environment in order to create new ones (in the spirit of the Minecraft video game), some of these crafted objects being associated with a reward. In this work, the training of an agent uses meta reinforcement learning where an outer loop, equivalent to an evolutionary mechanism, meta-learns the parameters of an inner loop which can be seen as a developmental mechanism (where the agent acquire skills during its lifetime by interacting with the environment). In the current setup we use $R{L}^2$ as our meta-learning algorithm 120, 206 which has already been used for acquiring behaviors efficiently balancing exploration and exploitation in a simple navigation task. Other work studied how different conditions in a bandit favor the evolution of innate vs learned behaviors 143.

Our experiments with recipe crafting are inspired by the little alchemy game. The difference with previous works in similar environments (e.g. 8.2.1) is that at every episode the structure of the recipe is randomly chosen. The agent therefore cannot predict what recipes will be rewarding and have to explore different combinations of objects in order to find the rewarding ones. The agent should also memorize the successful and unsuccessful combinations in order to explore and exploit efficiently.

Our preliminary results use both in a vectorized version of the game (where the agents action are only to choose the 2 objects to combine) and an embodied gridworld version (where the agent has to move, grab objects and put them on top of others in order to craft new ones). In both of these cases, the training efficiently meta learns an exploration/exploitation strategy which is to try new recipes (most of the time it does not try non working recipes more than once) until it finds the rewarding ones and then simply exploits them by making them over and over.

Further work will study how we can change the environment/training in order to evolve open-ended exploration strategies where an agent will continuously explore new recipes even if it has already found rewarding ones, as a way to be better prepared for future changes in the recipe structure. We hypothesize that such an intrinsic motivation to explore for the sake of acquiring knowledge of the environment, even in the absence of external rewards, could evolve by introducing drastic changes of recipes which the agent has to anticipate in order to survive. During the project, we switched from evolutionary algorithm and Recurrent neural networks to reinforcement learning and transformers. This allowed for more complex environments with more possibilities. We also obtained preliminary results with agents exploring the environment to gain information for the future.

This work uses the JAX python library for both the model/machine learning part and the environment simulation. JAX allows easy parallelization and fast GPU computation and so learning it through this project will be useful for later projects.

We plan to submit a paper on these experiments in 2024.

8.3.4 Evolving Reservoirs for Meta Reinforcement Learning

Participants: Corentin Leger [correspondant], Gautier Hamon, Eleni Nisioti, Xavier Hinaut, Clément Moulin-Frier.

This contribution was realized in the context of the Master internship of Corentin Léger in 2023, as a collaboration between C. Moulin-Frier from the Flowers team and Xavier Hinaut from the Mnemosyne team. It led to a paper which has been accepted at the Evostar conference 75.

Animals demonstrate remarkable adaptability to their environments, a trait honed through the evolution of their morphological and neural structures 199174. Animals are born equipped with both hard-wired behavior routines (e.g. breathing, motor babbling) and learning capabilities to adapt from experience. The costs and benefits of evolving hard-wired behaviors vs. learning capabilities depends on different factors, a central one being the level of unpredictability of environmental conditions across generations 195138. Phenotypic traits addressing environmental challenges that are shared across many generations are more likely to evolve hard-wired (e.g. breathing), while traits whose utility can hardly be predicted from its utility in previous generations are likely to be learned through individual development (e.g. learning a specific language).

This prompts an intriguing question: How can neural structures, optimized at an evolutionary scale, enhance the capabilities of agents to learn complex tasks at a developmental scale? To address this question, we propose to model the interplay between evolution and development as two nested adaptive loops: evolution optimizes the generation of neural structures through natural selection over generations, shaping developmental learning during an agent’s lifetime (Fig. 26).

More precisely, at the evolutionary scale (the outer loop), we use an evolutionary algorithm to optimize a genome specifying Hyper Parameters of Reservoirs 185. At a developmental scale (the inner loop), a RL agent equipped with a generated reservoir learns an action policy to maximize cumulative reward in a simulated environment. Thus, the objective of the outer evolutionary loop is to optimize macro properties of reservoirs in order to facilitate the learning of an action policy in the inner developmental loop. See Fig.27 for an overview of the method.

Using this computational model, we run experiments in diverse simulated environments, e.g. 2D environments where the agent learns how to balance a pendulum and 3D environments where the agent learns how to control complex morphologies. These experiments provide support to three main hypotheses for how evolved reservoirs can affect intralife learning. First, they can facilitate solving partially-observable tasks, where the agent lacks access to all the information necessary to solve the task. In this case, we test the hypothesis that the recurrent nature of the reservoir will enable learning to infer the unobservable information. Second, it can generate oscillatory dynamics useful for solving locomotion tasks. In this case, the reservoir acts as a meta-learned CPG. Third, it can facilitate the generalization of learned behaviors to new tasks unknown during the evolution phase.

This work was accepted at EvoApplications 27th European Conference on the Applications of Evolutionary and bio-inspired Computation (EvoApps 2024).

8.3.5 Emergence of Collective Open-Ended Exploration from Decentralized Meta-Reinforcement Learning

Participants: Richard Bornemann, Gautier Hamon [correspondant], Eleni Nisioti, Clément Moulin-Frier.

In this work we want to further investigate the emergence of cooperative exploration strategies of decentralized agents by training them on a open ended distribution of tasks. To this end we introduce a novel environment 28 which is conceptually simple yet allows for a complex open ended procedurally generated task space by dynamically combining multiple subtasks sampled from five task types to form a task tree which needs to be solved sequentially, akin to the notion of recipes in 93. We train two agents parametarized by independent recurrent neural networks and optimized using standard proximal policy optimization. As no information is given to the agents about which subtasks have been sampled or how and in which order they should be solved, the agents have to develop general strategies for exploring the environment, effectively learning how to learn from the information obtained by interacting with the environment throughout the episode, in order to solve novel tasks. We show that training independent decentralized agents on only multi agent episodes leads to sub-optimal behavior of the agents, primarily due to the problem of credit assignment when rewards are shared between agents. We propose to include single agent episodes during training to force the agents to learn to solve tasks on their own without relying on any help from other agents. We find that training on a mixture of single and multi agent episodes increases the agents individual performance while simultaneously decreasing the individual performance differences between the agents, leading to a strong improvement in performance in multi agent tasks.

Using this approach we find that decentralized agents trained in our procedurally generated environment learn a powerful collective exploration strategy, allowing them to solve over 70 percent of task trees encountered during training. Moreover, these powerful exploration capabilities lead to strong generalization performance when confronted with objects unseen during training, as well as on novel tasks which require complex coordination to be solved successfully at test time. Additionally we show that the learned collective exploration strategies extend to the open ended task setting, enabling the agents to effectively generalize to task trees with a depth of six, featuring an increased complexity of subtasks, despite being initially trained on task trees comprising only three subtasks.

This work was presented as a poster at the Agent Learning in Open-Endedness (ALOE) workshop at NeurIPS 2023. Videos of the agents behaviors can be found on our companion website

8.3.6 Cooperative control of environmental extremes by artificial intelligent agents

Participants: Martí Sànchez Fibla, Clément Moulin-Frier [correspondant], Ricard Solé.

This contribution is the result of a collaboration between Ricard Solé and Martí Sànchez-Fibla from the University Pompeu Fabra (Barcelona, Spain) and Clément Moulin-Frier (Flowers, Inria). A preprint is available 78 and it a paper has been submitted to PNAS.

Humans have been able to tackle biosphere complexities by acting as ecosystem engineers, profoundly changing the flows of matter, energy and information. This includes major innovations that allowed to reduce and control the impact of extreme events. Modelling the evolution of such adaptive dynamics can be challenging given the potentially large number of individual and environmental variables involved. This paper shows how to address this problem by using fire as the source of external, bursting and wide fluctuations. Fire propagates on a spatial landscape where a group of agents harvest and exploit trees while avoiding the damaging effects of fire spreading. The agents need to solve a conflict to reach a group-level optimal state: while tree harvesting reduces the propagation of fires, it also reduces the availability of resources provided by trees. It is shown that the system displays two major evolutionary innovations that end up in an ecological engineering strategy that favours high biomass along with the suppression of large fires. The implications for potential A.I. management of complex ecosystems are discussed.

The computational model is illustrated Fig. 29 and the main results in Fig. 30.

8.3.7 Eco-evolutionary Dynamics of Non-episodic Neuroevolution in Large Multi-agent Environments

Participants: Gautier Hamon [correspondant], Eleni Nisioti, Clément Moulin-Frier.

This work focuses on eco-evolutionary dynamics where "organisms are not solely products but, by modifying their niche and therefore its associated fitness landscape, are also causes of evolution" 142. The main objective of this paper is to propose a method for studying large-scale eco-evolutionary dynamics in agent-based simulations with a reasonable level of biological and ecological plausibility. For this aim, we implement a system with the following properties (see Fig. 31 for illustration):

• Non-episodic simulation environment with complex intrinsic dynamics. We model our environment after common-pool resource (CPR) appropriation problems, where a group of agents competes for finite resources. We extend an existing environment of CPR appropriation  175 with the presence of multiple niches, where resources regrow proportionally to the density of nearby resources at different rates in different regions of the environment (Fig 31). We prevent any environment or population reset during a whole simulation run, enabling coupled environmental and population dynamics leading to complex eco-evolutionary feedback effects.
• Continuous neuroevolution in a large, size-varying agent population The environment contains thousands of agents, each controlled by a neural network whose weights are optimized using neuroevolution 192
• Physiology-driven death and reproduction There is no notion of rewards, agents are instead equipped with a physiological system modulating their energy level according to the resources they consume, in a non-linear way. At the evolutionary scale, agents reproduce as long as they are able to maintain their energy level within a reasonable range and die if this level goes below a minimum threshold. This is departure from the notion of fitness-based selection and more in line with a minimal criterion selection 96. Note that the population size can vary with time.

In addition to experiments conducted in the large environment presented, we also conduct experiments in "lab environment" (as opposed to the "natural environment") to isolate the study of certain behavior (which are often intertwined with a lot of dynamics in the natural environment).

One interesting results of these simulation is the emergence of sustainable foragers which as shown in lab environment Fig.32 tends to not overconsume when there is enough resource in their neighbourhood. This allows to keep a certain amount of resource to spread which is therefore beneficial for their future survival as well as the survival of their offspring. (as there is no reset of the environment)

This work was presented as a poster to the Genetic and Evolutionary Computation Conference (GECCO) 2023.

8.4.1 Fostering curiosity and meta-cognition in children using LLM-based conversational agents

Participants: Pierre-Yves Oudeyer, Hélène Sauzéon [correspondant], Mehdi Alami, Rania Abdelghani, Didier Roy, Edith Law, Chloé Desvaux.

Since 2019 (Idex cooperation fund between the University of Bordeaux and the University of Waterloo, Canada) and the recent creation of CuriousTECH associate team in 2022 (led by the Flowers team and involving F. Lotte from the Potioc team and M. Fernendes and E. Law from the Waterloo University), we continue our work on the development of new curiosity-driven interaction systems . Substantial progress has been made in this area of application of FLOWERS works (see the website of CuriousTECH team : https://flowers.inria.fr/curioustech-associate-team/).

Curiosity-driven educational technologies : Combining conversational agents and LLM as lever for field scalling up

At the end of 2020, we started an industrial collaboration project with EvidenceB on this topic (CIFRE, contract of Rania Abdelghani validated by the ANRT). The overall objective of the thesis is to propose new educational technologies driven by epistemic curiosity, and allowing childre,n to express themselves more and learn better. To this end, a central question of the work will be to specify the impact of self-questioning aroused by states of curiosity about student performance. Another objective will be to create and study the pedagogical impact of new educational technologies in real situations (schools) promoting an active education of students based on their curiosity. To this end, a web platform called 'Kids Ask' has been designed, developed and tested in three primary schools. The tool offers an interaction with a conversational agent that trains children's abilities to generate curiosity-driven questions and use these questions to explore a learning environment and acquire new knowledge. The results suggest that the configuration helped enhance children's questioning and exploratory behaviors; they also show that learning progress differences in children can be explained by the differences in their curiosity-driven behaviors 85.

Illustration of a conversational agent's strategies in the different work spaces of the "Kids Ask" platform

Despite showing pedagogical efficiency, the method used in the first study of this PhD is still very limited since it relies on generating curiosity-prompting cues by hand for each educational resource in order to feed the "discussion" with the agent, which can be a very long and costly process. For this reason, a logical follow-up to scale-up and generalize this study was to explore ways to automate the said conversational agents' behaviors in order to facilitate their implementation on a larger scale and for different learning tasks. More particularly, we move towards the natural language processing (NLP) field and the large language models (LLMs) that showed an impressive ability in generating text that resembles the way people write.

In this context, we study using the recent LLM GPT-3 to implement conversational agents that can prompt children's curiosity about a given text-based educational content, by proposing some specific cues. We investigate the validity of this automation method by comparing its impact on children's divergent question-asking skills with respect to the hand-crafted condition we had in our previous work. In a second step, we explore using GPT-3 to propose a new curiosity-prompting behavior for our agent that aims to better support the children's needs of competence, autonomy and relatedness during the question-asking training.

The study was conducted in two primary schools with 75 children aged between 9 and 11. Our first results suggest the validity of using GPT-3 to facilitate the implementation of curiosity-stimulating learning technologies. Indeed, children's performance was similar between the conditions where they had hand-generated or GPT-3-gene,rated cues. In a second step, we also found that GPT-3 can be efficient in proposing the relevant cues that leave children with more autonomy to express their curiosity  33 (publication in process).

Left: Participants from the thre,e, conditions were able to improve their divergent QA abilities after the ”Kids Ask” interaction, as shown by the divergent QA fluency test pre- and post-training. ,Right: Chil,dren’s perception of their QA self-efficacy changed more positively with the intervention for those who interacted with the automated agents.

Finally, as a follow-up direction to this line of work, we design new digital interventions, that focus on eliciting the metacognitive mechanisms involved in the stimulation and continuity of curiosity, and not just giving the tools to pursue it as done in the previous studies. For this, we use findings from the theories explaining curiosity in order to break this latter up into a set of relevant metacognitive skills. We then take an operational approach to propose adequate digital metacognitive exercises for each one of the said skills (i.e. exercises to identify uncertainty, generate hypotheses etc). We aim to implement this set of metacognitive exercises and investigate its impact on children's abilities to initiate and maintain curious behaviors. We would also be interested in investigating the impact of such training on the learning progress children can achieve. A first study has been conducted with two classrooms to evaluate the accessibility of this new training and and the impact on metacognitive efficiency, curiosity-driven question-asking and learning. Our first results being rather positive, we aim to recruit a bigger sample size to validate them 41.

Thanks to the building of primary schools networks (Submission of Léa-Ifé project to ENS-Lyon Call), the next step of this work is to study this digital metacognitive intervention in more ecological settings, specifically when administered by teachers, since this reflects the classical classroom setting. The aim is therefore to support teachers in assimilating the intervention to facilitate its transfer into real classrooms. This has already been initiated in collaboration with the Académie de Bordeaux and elementary school teachers of Bordeaux Métropole, as the first steps of the thesis project that started in October (Chloé Desvaux - Université de Bordeaux). The efficacy of the ecological digital intervention is to be compared with previous results. Another objective of this thesis project is to assess the characteristics of children’s curious behaviors more closely. The primary focus being divergent and creative properties and their implication in learning.

On another subject, we also started investigating the importance of curiosity-related metacognitive skills on students' use of the GenAI (Generative AI) tools during learning. Indeed, in  59 , we argue about the importance of developing children's sense of critical thinking, epistemic vigilence, etc in order to allow a more active and informed use of these tools during learning. Such skills can help have a more realistic expectations of such tools and evaluate their outputs before integrating them in one's beliefs. A study aiming to understand how children use these tools to solve learning problems is in progress (piloting stage).

8.4.2 Supporting Qualitative Analysis with Large Language Models: Combining Codebook with GPT-3 for Deductive Coding

Qualitative analysis of textual contents unpacks rich and valuable information by assigning labels to the data. However, this process is often labor-intensive, particularly when working with large datasets. While recent AI-based tools demonstrate utility, researchers may not have readily available AI resources and expertise, let alone be challenged by the limited generalizability of those task-specific models. In this study, we explored the use of large language models (LLMs) in supporting deductive coding, a major category of qualitative analysis where researchers use pre-determined codebooks to label the data into a fixed set of codes. Instead of training task-specific models, a pre-trained LLM could be used directly for various tasks without fine-tuning through prompt learning. Using a curiosity-driven questions coding task as a case study, we found, by combining GPT-3 with expert-drafted codebooks, our proposed approach achieved fair to substantial agreements with expert-coded results. We lay out challenges and opportunities in using LLMs to support qualitative coding and beyond. This work was published in 56 and involved a collaboration with Z. Xiao, V. Liao, E. Yuan from Microsoft Research Montreal.

8.5.1 Machine Learning for Adaptive Personalization in Intelligent Tutoring Systems

Participants: Pierre-Yves Oudeyer [correspondant], Benjamin Clément, Didier Roy, Hélène Sauzeon.

8.5.2 Machine learning for adaptive cognitive training

Participants: Pierre-Yves Oudeyer, Hélène Sauzéon [correspondant], Masataka Sawayama, Benjamin Clément, Maxime Adolphe, Marion Pech.

Because of its cross-cutting nature to all cognitive activities such as learning tasks, attention is a hallmark of good cognitive health throughout life and more particularly in the current context of societal crisis of attention. Recent works have shown the great potential of computerized attention training for an example of attention training, with efficient training transfers to other cognitive activities, and this, over a wide spectrum of individuals (children, elderly, individuals with cognitive pathology such as Attention Deficit and Hyperactivity Disorders). Despite this promising result, a major hurdle is challenging: the high inter-individual variability in responding to such interventions. Some individuals are good responders (significant improvement) to the intervention, others respond variably, and finally some respond poorly, not at all, or occasionally. A central limitation of computerized attention training systems is that the training sequences operate in a linear, non-personalized manner: difficulty increases in the same way and along the same dimensions for all subjects. However, different subjects require in principle a progression at a different, personalized pace according to the different dimensions that characterize attentional training exercises.

To tackle the issue of inter-individual variability, the present project proposes to apply some principles from intelligent tutoring systems (ITS) to the field of attention training. In this context, we have already developed automatic curriculum learning algorithms such as those developed in the KidLearn project, which allow to customize the learner's path according to his/her progress and thus optimize his/her learning trajectory while stimulating his/her motivation by the progress made. ITS are widely identified in intervention research as a successful way to address the challenge of personalization, but no studies to date have actually been conducted for attention training. Thus, whether ITS, and in particular personalization algorithms, can optimize the number of respondents to an attention training program remains an open question.

Randomized and controlled Trial in Young and Olders adults : Predifined vs. ZPDES condition

Utilizing these tools, a pilot study campaign was conducted to evaluate the impact of our AI-based personalized cognitive training program. The first pilot experiment involved n=27 participants and aimed to compare the effectiveness of a cognitive training program using a linear difficulty management procedure (staircase procedure) to a program using an ITS for difficulty manipulation. The online training lasted for 10 hours over a period of 2 weeks. The results indicated that the ITS-based intervention produced diverse learning trajectories compared to the linear procedure 35, leading to broader improvements in pre-post cognitive assessment. However, no significant differences were observed in subjective measures of motivation and engagement between the two groups. Subsequent to this initial experiment, two pilot studies (n=11 and n=10, respectively) were conducted with the goal of enhancing motivation and engagement in the game. The first study implemented gamified components such as scores and feedback, while the second study examined hyperparameter updates to the ITS. The analysis of learning trajectories, learning outcomes, and subjective measures yielded promising results in favor of the AI-based personalized procedure.

Building on the preliminary findings, we expanded our research scope with a more comprehensive experimental setup involving two distinct studies. The first study encompassed 64 young adults, sourced through the Prolific platform, while the second study consisted of 49 older adults, recruited from the "Université du temps libre". Our experimental methodology mirrored that of our initial pilot studies, with a notable enhancement: the integration of new gamified elements (including mini-story creation and new visual content) aimed at boosting participant motivation and engagement.

The data analysis encompassed three primary dimensions: initially, an exploratory phase to delineate learning trajectories between control and intervention groups; subsequently, a comparative analysis of pre- and post-test performance on the cognitive battery; and lastly, an examination of participants' self-reported experiences during training, providing insights into their subjective perceptions of the experiment.

The pilot studies' preliminary outcomes were corroborated in these larger sample groups. Notably, learning trajectories exhibited greater diversity in the group undergoing the intervention procedure. This group also demonstrated a more pronounced improvement across a wider range of cognitive assessment tasks. Although participants engaging in the personalized cognitive training reported a higher cognitive load via questionnaires, the levels of engagement and frustration did not significantly differ between the two groups.

Different learning trajectories for a selected participant in the staircase group (left) and the ITS group (right). The color of a dot indicates the initial presentation of the parameter value, while the size of the dot represents the frequency of the parameter value.

8.5.3 ToGather : Interactive website to foster collaboration among stakeholders of school inclusion for pupils with neurodevelopmental disorders

Participants: Hélène Sauzéon [correspondant], Cécile Mazon, Eric Meyer, Isabeau Saint-Supery, Christelle Maillart [Uni. Liège, Belgium], Kamélia Belassel, Mathieu Périé, Valentin Strahm.

Sustain and support the follow-up of the school inclusion of children with neurodevelopmental disorders (e.g., autism, attention disorders, intellectual deficiencies) has become an emergency : the higher is the school level, the lower is the amount of schooled pupils with cognitive disabilities.

Technology-based interventions to improve school inclusion of children with neurodevelopmental disorders have mostly been individual centered, focusing on their socio-adaptive, and cognitive impairments and implying they have to adapt themselves in order to fit in our society's expectations. Although this approach centered on the normalization of the person has some advantages (reduction of clinical, symptoms), it carries social stereotypes and misconceptions of cognitive disability that are not respectful of the cognitive diversity and intrinsic motivations of the person, and in particular of the student's wishes in terms of school curriculum to achieve his or her future life project  51.

The "ToGather" project aims at enlightening the field of educational technologies for special education by proposing an approach centered on the educational needs of the students and bringing a concerted and informed answer between all the stakeholders including the student and all their support spheres (family, school, medico-social care). To this end, ToGather project that emanates from participatory design methods, primarily consists of having developed a pragmatic tool (interactive website) to help students with cognitive disability and their caregivers to formalize and to visualize the repertoire of academic skills of the student and to make it evolve according to his or her proximal zone of development (in the sense of Vygotsky) on the one hand, and to the intrinsic motivations of the student (his or her own educational and life project) on the other 152.

This project is in partnership with the School Academy of Bordeaux of the French Education Minestery, the ARI association, the Centre of Autism of Aquitaine. It is funded by the FIRAH (foundation) and the Nouvelle-Aquitaine Region (see the dedicated webpages : https://flowers.inria.fr/projet-tous-ensemble/).

First, usability studies have been conducted for evaluating ergonomic qualities of the ToGather website, yielding positive resultats in French and Belgian contexts. Then, we conducted a large field-study to assess the effectiveness of the tool in helping stakeholders to support children with neurodevelopmental disorders (NDD)   180   83   55.

The study protocol consisted in a longitudinal non-randomized controlled trial, with baseline, 3-months, and 6-months fllow-up assessments. The recruitment was conducted across the entire French territory. Our local partners facilitated the dissemination of the call for participation in Gironde and provided us with contacts to extend it to other regions. Additionally, a recruitment campaign through social media was carried out to communicate about the study and encourage participants to test the ToGather tool.

As the tool was designed to support co-educational process between parents and professionals, a support team had to consist of at least two stakeholders, including at least one of the parents. Initially, 157 participants were recruited in 37 support teams, but 30 individuals did not answer to baseline questionnaire, leading to the exclusion of 11 support teams. After baseline assessment, 13 support teams were allocated to the experimental condition (ToGather app) and 11 to the control condition (usual follow-up).

Primary outcomes measures covered stakeholders’ relationships, self-efficacy, and attitudes towards inclusive education, while secondary outcomes measures were related to stakeholders’ burden and quality of life, as well as children’s school well-being and quality of life.

As the study ended in July 2023, data analysis is still ongoing. Preliminary results after 3 months of use showed encouraging results with an improvement in communication between stakeholders and their respective quality of life (paper in progress)

8.5.4 Curious and therefore not overloaded : Study of the links between curiosity and cognitive load in learning mediated by immersive technologies

Participants: Hélène Sauzéon [correspondant], Matisse Poupard, André Tricot [Cosupervisor - Univ. Montpellier], Florian Larrue [Industrialist - Le Catie].

With the ever-increasing interest in digital technologies in education, many questions about their use and effectiveness are emerging. In this context, this project focus on the relationships between three key dimensions of technology-mediated learning: the learner's internal learning processes, the instructional design, and the educational technology used.

In partnership with CATIE (industrial partner) and the EPSYLON laboratory of the University of Montpellier (PR. André Tricot), two main objectives are targeted in this research program started in April 2022:

• To establish connections between the theory of cognitive load and models of curiosity-driven learning.
• To experimentally assess the influence of the choice of educational technology on the associations between pedagogical choices (guided instruction vs. exploration) and learner expertise.

To this end, the program includes 3 main phases of study:

Experimentation

Based on these results, two experiments involving 140 second-year undergraduate students in the field of medicine were conducted. In the first experiment, we use spatial augmented reality and mixed reality (HoloLens 2) to investigate whether guiding students' drawings during their lectures can reduce their cognitive load and enhance their motivation to learn.

Experimental conditions for experiment 1 : Augmented reality-assisted drawing note-taking

Our hypotheses are as follows:

• We plan to demonstrate that drawing guidance will reduce learners' extrinsic cognitive load, thereby promoting learning.
• We assume that reducing cognitive load will stimulate learners' motivation and engagement, considering cognitive load as a motivational cost.
• However, we assume that the presence of a 3D model (RM + 3D model) will add an additional extrinsic cognitive load, potentially leading to overload and reduced learning.
• We expect to observe a positive correlation between drawing quality and learning.
• We anticipate better acceptability and lower cognitive load for the projector compared with HoloLens (due to the weight of the headset and convergence and interaction issues).
• In addition, this experiment can also be used to understand the effect of learners' prior knowledge on their performance.

In the second experiment, we change the learning paradigm, using virtual reality with different levels of interaction and guidance to examine how exploration and embodied interaction with a 3D model can have a positive impact on learning, cognitive load, and curiosity.

Experimental conditions for experiment 2 : Embodied learning in virtual reality, effect of interactivity

We make the following assumptions:

• Interaction with the system, i.e. the manipulation of information, promotes better learning and stimulates intrinsic motivation. Thus, the passive VR condition should lead to poorer results than the other conditions in terms of learning and motivation.
• Embodied manipulation, in VR, presents an advantage for learning and learner engagement. Manipulating information through the senses and one's own body makes for a more immersive and engaging learning experience.
• Guided, motor-only interaction is less interesting in terms of learning and motivation than free interaction, which we can describe as cognitive.
• Free exploration encourages inquisitive behavior, but can overload learners, especially if they are new to the subject.
• Intrinsic motivation minimizes perceived cognitive effort.

8.6.1 Design of an Interactive Software for Automated Discovery in Complex Systems

Participants: Clément Romac [correspondant], Jesse Lin, Mathieu Périé, Mayalen Etcheverry, Clément Moulin-Frier, Pierre-Yves Oudeyer.

We further developed our Automated Discovery software and started experimenting with it. First, we released a new version of our standalone Python library. We improved how experiments could be saved and reloaded.

Second, we focused on implementing tools and interfaces for users to give feedback or instructions to the automated discovery algorithm that explores the complex system. As identified by 16, empowering experiments to collaborate with automated discovery methods can be key to obtain interesting discoveries. Integrating such a collaborative process in our tool came with several engineering challenges and we are currently experimenting with our solution and working on making it user-friendly for non-experts end users.

Finally, we released a first open version of our software. We provide a documentation and installation tools using Docker.

Technical architecture of our software.

8.6.2 Learning Sensorimotor Agency in Cellular Automata

Participants: Gautier Hamon [correspondant], Mayalen Etcheverry, Bert Chan, Clément Moulin-Frier, Pierre-Yves Oudeyer.

As a continuation of the previous projects in Automated Discovery in Self-Organizing Systems, we have been working on expanding the set of discoveries of possible structures in continuous CAs such as Lenia  105, 104, and in particular we have been interested to search for emerging agents with sensorimotor capabilities. Understanding what has led to the emergence of life and sensorimotor agency as we observe in living organisms is a fundamental question. In our work, we initially only assume environments made of low-level elements of matter (called atoms, molecules or cells) locally interacting via physics-like rules. There is no predefined notion of agent embodiment and yet we aim to answer the following scientific question: is it possible to find environments in which there exists/emerge a subpart that could be called a sensorimotor agent?

We use Lenia continuous cellular automaton as our artificial "world"  104. We introduce a novel method based on gradient descent and curriculum learning combined within an intrinsically-motivated goal exploration process (IMGEP) to automatically search parameters of the CA rule that can self-organize spatially localized 2 and moving patterns 3 within Lenia. The IMGEP defines an outer exploratory loop (generation of training goal/loss) and an inner optimization loop (goal-conditioned). We use a population-based version of IMGEP 17, 114 but introduce two novel elements compared to previous papers in the IMGEP literature. First, whereas previous work in 29 and 16 used a very basic nearest-neighbor goal-achievement strategy, our work relies on gradient descent for the local optimization of the (sensitive) parameters of the complex system, which has shown to be very powerful. To do so we made a differentiable version of the Lenia framework, which is also a contribution of this work. Secondly, we propose to control subparts of the environmental dynamics with functional constraints (through predefined channels and kernels in Lenia) to build a curriculum of tasks; and to integrate this stochasticity in the inner optimization loop. This has shown central to train the system to emerge sensorimotor agents that are robust to stochastic perturbations in the environment. In particular, we focus on modeling obstacles in the environment physics and propose to probe the agent sensorimotor capability as its performance to move forward under a variety of obstacle configurations. We also provide in this work tests and metrics to measure the robustness of the obtained agents.

Robustness test to harder/unseen obstacle configurations: straight wall, bigger obstacle, dead ends.

Change of scale changing the kernel size and initialization, the grid is the same size in both

While many complex behaviors have already been observed in Lenia, among which some could qualify as sensorimotor behaviors, they have so far been discovered "by chance" as the result of time-consuming manual search or with simple evolutionary algorithms. Our method provides a more systematic way to automatically learn the CA rules leading to the emergence of basic sensorimotor structures, as shown in Figure 41. Moreover, we investigated and provided ways to measure the (zero-shot) generalization of the discovered sensorimotor agents to several out-of-distribution perturbations that were not encountered during training. Impressively, even though the agents still fail to preserve their integrity in certain configurations, they show very strong robustness to most of the tested variations. The agents are able to navigate in unseen and harder environmental configurations while self-maintaining their individuality (Figure 39). Not only the agents are able to recover their individuality when subjected to external perturbations but also when subjected to internal perturbations: they resist variations of the morphogenetic processes such that less frequent cell updates, quite drastic changes of scales as well as changes of initialization (Figure 40). Furthermore, when tested in a multi-entity initialization and despite hav,ing been trained alone, not only the agents are able to preserve their individuality but they show forms of coordinated interactions (attractiveness and reproduction). Our results sug,gest that, contrary to the (still predominant) mechanistic view on embodiment, biologically-inspired embodiment could pave the way toward agents with strong coherence and generalization to out-of-distribution changes, mimicking the remarkable robustness of living systems to maintain specific functions despite environmental and body perturbations 139. Searching for rules at the cell-level in order to give rise to higher-level cognitive processes at the level of the organism and at the level of the group of organisms opens many exciting opportunities to the development of embodied approaches in AI in general.

Scatter plot of the agents as their measured performances of robustness to obstacles (y axis) and speed in obstacles (x axis) obtained by IMGEP (red), random search with the same compute resources as IMGEP(blue) and the one from the original lenia paper (green)

The work has been released in 2022 as a distill-like article which is currently hosted at this link. This article contains an interactive demo in webGL and javascript, as well as many videos and animations of the results. A colab notebook with the source code of the work is publicly available at.

In 2023, additional quantitative experiments were conducted as well as ablations. And this work was submitted to the Proceedings of the National Academy of Sciences (PNAS) journal.

8.6.3 Flow lenia: Mass conservation for the study of virtual creatures in continuous cellular automata

Participants: Erwan Plantec, Gautier Hamon [correspondant], Mayalen Etcheverry, Pierre-Yves Oudeyer, Clément Moulin-Frier, Bert Chan.

Following our work on trying to find sensorimotor cabapabilities in cellular automata such as Lenia   105, 104, we kept exploring the search for low level cognition in continuous cellular automata. This led to preliminary search on trying to emerge memory in self-organizing agents as well as work on trying to implement other environmental constraints in the CA in order to emerge interesting behavior. To implement more easily those environmental constraints as well as to ease the emergence of spatially localized patterns (and thus have the optimization/search to focus more on the cognitive ability, removing the need to optimize to prevent uncontrollable growth/explosion of the pattern), we worked on adding mass conservation to the Lenia system.

We propose in this work a mass-conservative (i.e the sum of the CA’s activations remains constant over time) extension to Lenia called Flow Lenia 54. We hypothesize that such conservation laws will help in the search for artificial life-forms by constraining emerging patterns to spatially localized ones. It ,also allows to implement more easily environmental constraints on the self-organizing agents such as a need for food to grow, etc.

Furthermore, we show that this new model allows for the integration of the update rule parameters within the CA dynamics enabling the emergence of creatures with different parameters and so different properties in the same environment/grid. This leads to multi-species simulation where the grid is filled with agents with different behaviors and properties 42. Such a feature opens up research perspectives towards the achievement of open-ended intrinsic evolution inside continuous CAs, which means that all the evolutionary part would be a result of the dynamic of the CA (without any external loop/system). We hypothesize that this open-ended instrinsic evolution could, through the competition/cooperation, lead to the emergence of interesting low level cognition in those system.

, , ,

Multi-species simulation in Flow Lenia where each colour represents different parameters. Left to right shows the evolution of the system over time with some species stealing the mass of others.

Multi-species simulation in Flow Lenia where each colour represents different parameters. Left to right shows the evolution of the system over time with some species stealing the mass of others.

Simple evolutionary strategy (with an evolutionary loop outside the system) was also used to optimized for pattern with directional and rotational movement.

You can find some examples of the system and pattern in this companion website, including the ones trained for movement, random parameters, food in flow Lenia, and multi species s,imulations: see. Notebook with the system can be found : here.

This work led to an oral presentation to the WIVACE 2022, 15th International Workshop on Artificial Life and Evolutionary Computation.

In 2023, final quantitative experiments on optimizing the parameters with evolutionary strategies and writing was conducted, as well as some additional exploratory experiments on large simulations for open ended evolution.

This work got the best paper award at the ALIFE 2023 conference where it got an oral presentation.

8.6.4 Exploration of Gene Regulatory Network Behaviors Using Automated Discovery Tools

Participants: Mayalen Etcheverry [correspondent], Clément Moulin-Frier, Pierre-Yves Oudeyer, Michael Levin.

In the context of project "Automated Discovery in Self-Organizing Systems", it has been demonstrated that modern tools leveraging computational models of curiosity developed in the Flowers team can be transposed to form efficient AI-driven "discovery assistants." These tools can assist scientists in mapping and navigating the space of possible outcomes in complex systems 29, 16, 128. In 2022, we initiated a collaboration with Dr. Michael Levin, a renowned biologist at Tufts University, through a 5-month academic exchange with Mayalen Etcheverry in his lab in Boston. This collaboration laid the foundation for continued collaboration throughout 2023, resulting in the submission of one paper 73 (currently under review) and another accepted at the NeurIPS 2023 AI for Science workshop 61.

The primary focus of this collaboration was to leverage curiosity-driven exploration algorithms as tools to empower scientific exploration and analysis of basal cognition in biological systems, specifically numerical models of gene regulatory networks (GRNs). Understanding, mapping, predicting, and controlling the complex behavior of these networks is crucial for applications in biomedicine and synthetic bioengineering. However, there are few quantitative tools that facilitate exploration of these networks, especially when their complexity makes unguided exploration infeasible.

To address these challenges in practice, we proposed an experimental framework summarized in Figure 43. In this framework, we formalized and investigated a view of gene regulatory networks as agents navigating a problem space. We developed automated tools to efficiently map the repertoire of robust goal states that GRNs can reach despite perturbations. These tools rely on two main contributions that we made in this work: (1) The use of curiosity-driven exploration algorithms, originating from the AI community, to explore the range of behavioral abilities of a given system, which we adapted and leveraged to automatically discover the range of reachable goal states of GRNs, and (2) The use of a battery of empirical tests inspired by implementation-agnostic behaviorist approaches that we leveraged to assess the navigation competencies of GRNs.

Our data revealed that models inferred from real biological data can reach a surprisingly wide spectrum of steady states, showcasing various competencies that living agents often exhibit in physiological network dynamics and that do not require structural changes to network properties or connectivity. Furthermore, we investigated the applicability of the discovered “behavioral catalogs” for comparing the evolved competencies across classes of evolved biological networks, as well as for the design of drug interventions in biomedical contexts or for the design of synthetic gene networks in bioengineering. Altogether, these automated tools and the resulting emphasis on behavior-shaping and exploitation of innate competencies can open the path to better interrogation platforms for exploring the complex behavior of biological networks in an efficient and cost-effective manner.

To encourage broader adoption and development of the tools and algorithms, we have released two software packages: SBMLtoODEJax (https://­github.­com/­flowersteam/­sbmltoodejax) 7.1.21 and AutoDiscJax (https://­github.­com/­flowersteam/­autodiscjax). SBMLtoODEJax converts Systems Biology Markup Language (SBML) models into Python classes written in JAX, enabling easy simulation and manipulation. AutoDiscJax, built upon JAX and SBMLtoODEJax, facilitates automated discovery and exploration of complex systems, specifically organizing the exploration of computational models of biological GRNs.

Research on lifelong Deep Reinforcement Learning of multiple tasks (Microsoft

Participants: Pierre-Yves Oudeyer [correspondant], Laetitia Teodorescu.

Financing of the PhD grant of Laetitia Teodorescu.

Automated Discovery of Self-Organized Structures (Poïetis)

Participants: Pierre-Yves Oudeyer [correspondant], Mayalen Etcheverry.

Financing of the CIFRE PhD grant of Mayalen Etcheverry by Poietis.

Machine learning for adaptive cognitive training (OnePoint)

Participants: Hélène Sauzéon [correspondant], Pierre-Yves Oudeyer, Maxime Adolphe.

Financing of the CIFRE PhD grant of Maxime Adolphe by Onepoint.

Curiosity-driven interaction system for learning (evidenceB)

Participants: Hélène Sauzéon [correspondant], Pierre-Yves Oudeyer, Rania Abdelghani.

Financing of the CIFRE PhD grant of Rania Abdelghani by EvidenceB.

Curious and therefore not overloaded : Study of the links between curiosity and cognitive load in learning mediated by immersive technologies (CATIE)

Participants: Hélène Sauzéon [correspondant], Matisse Poupard, André Tricot [Cosupervisor - Univ. Montpellier], Florian Larrue [Industrialist - Le Catie].

Financing of a PhD grant of Matisse Poupard with CATIE and EPSYLON Lab (Univ. Montpellier).

Augmenting curiosity-driven exploration with very large language models in deep reinforcement learning agents (Hugging Face)

Participants: Pierre-Yves Oudeyer [correspondant], Clément Romac.

Financing of the PhD grant of Clément Romac by Hugging Face.

Autonomous Driving Commuter Car (Renault)

Participants: David Filliat [correspondant], Emmanuel Battesti.

We developed planning algorithms for a autonomous electric car for Renault SAS in the continuation of the previous ADCC project. We improved our planning algorithm in order to go toward navigation on open roads, in particular with the ability to reach higher speed than previously possible, deal with more road intersection case (roundabouts), and with multiple lane roads (overtake, insertion...).

School+ /ToGather project (FIRAH and Region Nouvelle-Aquitaine)

Participants: Hélène Sauzéon [correspondant], Cécile Mazon, Isabeau Saint-supery, Eric Meyer.

Financing of one year-postdoctoral position and the app. development by the International Foundation for Applied Research on Disability (FIRAH). The School+ project consists of a set of educational technologies to promote inclusion for children with Autism Spectrum Disorder (ASD). School+ primary aims at encouraging the acquisition of socio-adaptive behaviours at school while promoting self-determination (intrinsic motivation), and has been created according to the methods of the User-Centered Design (UCD). Requested by the stakeholders (child, parent, teachers, and clinicians) of school inclusion, Flowers team works to the adding of an interactive tool for a collaborative and shared monitoring of school inclusion of each child with ASD. This new app will be assessed in terms of user experience (usability and elicited intrinsic motivation), self-efficacy of each stakeholder and educational benefit for child. This project includes the Academie de Bordeaux –Nouvelle Aquitaine, the CRA (Health Center for ASD in Aquitania), and the ARI association.

CLEMENCE Cohort (Fondation de France and Théa Pharma)

Participants: Hélène Sauzéon [correspondant], Cécile Mazon, Cécile Delcourt.

The project "Cohorte LongitudinalE sur la Myopie et le développement oculaire dans l’ENfanCE(CLEMENCE) is led by C. Delcourt from the lab of Bordeaux Populational Health (2M€). Hélène Sauzéon and Cécile Mazon participate to the research program with the study of developemental changes due to Myopa in visual attention.

10.1.1 Inria associate team not involved in an IIL or an international program

We created a new three years- Inria associate team since april 2023, namely CuriousTECH team (see the website of CuriousTECH team : https://flowers.inria.fr/curioustech-associate-team/). Its entails two team of Inria (Flowers and Potioc) and two labs of the Waterloo University (HCI Lab, David R. Cheriton School of Computer Science and the Cognitive neuroscience lab. of Pyschology Department). This prosed associate team aims to develop an original, cross-disciplinary approach, joining together two perspectives:

1) The fundamental study of curiosity-driven learning across life-span (in children, young adults and older adults) and

2) The study of how new (re)educational technologies, using both curiosity-related models and artificial intelligence techniques [3, 8, 9], can personalize learning sequences for each individual, maximizing curiosity and learning efficiency in real world contexts.

Our proposed research will produce new understanding of the role of curiosity in education and healthy aging, through the design and the field assessment of new interactive educational technologies or health-related technologies. Beyond academic contributions, we expect our findings to inform the broader societal challenges inherent to the School of the 21st Century, ranging from helping children (and their teachers) to develop cross-domain skills for learning such as curiosity and meta-cognition, while improving inclusivity in schools (learners with disabilities, especially cognitive disabilities) as well as promoting lifelong learning in older adults (successful aging), using cognitive-based research findings.

Another outcome of our joint program is to use applied research to accelerate the transfer of results to industries and public institutions related to education and healthy aging in both countries. The mixed method approach used in our proposed project (user-centered methods, digital technologies, artificial intelligence, and field assessment) will help demonstrate the effectiveness of our developed technology, and facilitate adoption by industry partners and market stakeholders from various education and health care organizations.

10.2.1 Visits of international scientists

10.2.2 Visits to international teams

10.2.3 Horizon Europe

ANR Chaire Individuelle Deep Curiosity

- PY Oudeyer continued to work on the research program of this Chaire, funding 2 PhDs and 3 postdocs for five years (until 2025).

ANR JCJC ECOCURL

- C. Moulin-Frier obtained an ANR JCJC grant. The project is entitled "ECOCURL: Emergent communication through curiosity-driven multi-agent reinforcement learning". The project starts in Feb 2021 for a duration of 48 months. It will fund a PhD student (36 months) and a Research Engineer (18 months) as well as 4 Master internships (one per year).

Projet AIxIA: "Analyse d’Interférences par Intelligence Artificielle".

Pierre-Yves Oudeyer and Clément Moulin-Frier obtained a grant from the call for project AIRSTRIP "L'intelligence Artificielle au service de l'IngénieRie des SysTèmes aéRonautIques et sPatiaux", in collaboration with the IRT Saint Exupery. The project was accepted in 2023 and will fund 18 months of a research engineer position starting in 2024.

Inria Exploratory Action AIDE

- Didier Roy is collaborator of the Inria Exploratory Action AIDE "Artificial Intelligence Devoted to Education", ported by Frédéric Alexandre (Inria Mnemosyne Project-Team), Margarida Romero (LINE Lab) and Thierry Viéville (Inria Mnemosyne Project-Team, LINE Lab). The aim of this Exploratory Action consists to explore to what extent approaches or methods from cognitive neuroscience, linked to machine learning and knowledge representation, could help to better formalize human learning as studied in educational sciences. AIDE is a four year project started middle 2020 until 2024 see.

Inria Exploratory Action I'AM

- Hélène Sauzéon is co-PI with P. Dragicevic of the Inria Exploratory Action I'AM "Impact of Augmented Reality on Autobiographical Memory: Examining Involuntary Memories and False Memories" (174,5k€). Starting in last september, the aim of this Exploratory Action consists to explore to what extent augmented reality based devices can produce erroneous autobiographical memories, and more particularly in vulnerable people (Children and older adults or yound adults with low memory abilities of source monitoring).

New collaboration with Maxime Derex from IAST Toulouse

for the co-direction of the PhD thesis of Jeremy Perez with Clément Moulin-Frier and Pierre-Yves Oudeyer on "Interactions between intrinsically motivated goal-exploration processes and cummulative cultural evolution" (see section 8.2.3).

France 2030 - PPR AUTONOMIE : Vieillissement Et Situations De Handicap - Projet INNOVCare (Lechevalier S., 3,5M€) (2023-26)

- Hélène Sauzéon and AS Rigaud will supervize the WP5 dedicated to two care-led innovation experiments with assistive technologies (400k € for Bordeaux).

VBHI project(Vascular Brain Health Institute -IHU, led by S. Debette, 5M€)) (2023-26)

- Hélène Sauzéon will supervize the WP4.3 dedicated to "Explore Digital Therapeutics To Slow Down Cognitive Decline In Covert Csvd" (150k€)

10.3.1 Adaptiv'Math

• Adaptiv'Math
• Program: PIA
• Duration: 2019 - 2020
• Coordinator: EvidenceB
• Partners:
  • EvidenceB
  • Nathan
  • APMEP
  • LIP6
  • INRIA
  • ISOGRAD
  • Daesign
  • Schoolab
  • BlueFrog

The solution Adaptiv'Math comes from an innovation partnership for the development of a pedagogical assistant based on artificial intelligence. This partnership is realized in the context of a call for projects from the Ministry of Education to develop a pedagogical plateform to propose and manage mathematical activities intended for teachers and students of cycle 2. The role of Flowers team is to work on the AI of the proposed solution to personalize the pedagogical content to each student. This contribution is based on the work done during the Kidlearn Project and the thesis of Benjamin Clement 108, in which algorithms have been developed to manage and personalize sequence of pedagogical activities. One of the main goal of the team here is to transfer technologies developed in the team in a project with the perspective of industrial scaling.

11.1.1 Scientific events: organisation

11.1.2 Scientific events: selection

11.1.3 Journal

11.1.4 Invited talks

Clément Moulin-Frier gave an invited talk at the seminar Marge, exception, déviation at Université Bordeaux Montaigne on February 24th, 2023. Title of the talk: Écologie de la cognition humaine.

Clément Moulin-Frier gave an invited talk at the "2nd réunion scientifique de la Société Psychédélique Aquitaine" at the Hopital Saint André in Bordeaux on October 17th, 2023. Title of the talk: The Ecology of Open-Ended Skill Acquisition: Eco-evolutionary, developmental and socio-cultural perspectives.

Clément Moulin-Frier gave an invited talk at the symposium Intelligence: natural,artificial and synthetic at the Barcelona Collaboratorium on October 5th, 2023. Title of the talk: Promoting behavioral diversity in artificial agents through eco-evolutionary and socio-cultural dynamics.

Clément Moulin-Frier gave an invited talk at the seminar of the GIPSA-Lab (Grenoble, France) on December 15th, 2023. Title of the talk: Modelling the eco-evolutionary, developmental and socio-cultural origins of open-ended skill acquisition.

Cédric Colas gave an invited talk at the ENS in June 2023 during a Lab meeting of Stefano Palminteri. Title of the talk: Towards Social Autotelic Agents.

Cédric Colas gave an invited talk at ICRA in the Life-Long Learning with Human Help Workshop, in July 2023. Title of the talk: Towards Social Autotelic Agents.

Cédric Colas gave an invited talk at Brown University in the USA, during Lab meeting of George Konidaris, in September 2023. Title of the talk: Towards Social Autotelic Agents.

Thomas Carta gave an invited talk online during a reading group with Glen Berseth on RL focused on effective generalization and pre-training strategies for control in MILA and ServiceNow Research, in April 2023. Title of the talk: Grounding Large Language Models in Interactive Environments with Online Reinforcement Learning.

Thomas Carta gave an invited talk online at Naver Labs Europe, in May 2023. Title of the talk: Grounding Large Language Models in Interactive Environments with Online Reinforcement Learning.

Clément Romac gave an invited talk at the COLT team'seminar at Universitat Pompeu Fabra (Barcelona) in May 2023. Title of the talk: Grounding Large Language Models in Interactive Environments with Online Reinforcement Learning.

Hélène Sauzéon gave 4 invited talks :

• Digital technology to support healthy aging : examples in the fields of smart home and computerized cognitive training” Symposium on Research on the Biology and Diseases of Ageing in Bordeaux - Disorders of tissue or cellular homeostasis during ageing and understanding of mechanisms and pathological consequences», 14th of November 2023, University of Bordeaux, Bordeaux
• Que fait le Centre Inria de l’université de Bordeaux pour l’enseignement et l’éducation ? , Journées scientifiques d’Inria, August 30th- September, 1rst 2023 Bordeaux
• Participation to TABLE RONDE - Communautés, espaces sociaux, impact de l’IA sur la restructuration des sociétés Forum NAIA -Robocup, July, 6-7 th, Bordeaux
• Les interventions numériques auprès des personnes âgées : le cas des maisons intelligentes, une révolution en marche?, Scientific day on Digital technologie for aging in place, co-organised by The « Conseil de l’Age » (https://www.hcfea.fr/) , October 16th, 2023, Paris (Online participation)

Cécile Mazon gave an invited talk for Inria Disability conferences cycle (cycles de conférences Handicap) with Cathy Hémon (Autism Resource center, trainer and specialized teacher) on Autism Spectrum Disorders, coeducation, and participative methods to address field issues

PY Oudeyer gave several invited presentations:

• Keynote talk at the annual conference of the Vision Science Society of Japan, "Developmental AI: machines that learn like children" (Jan 2023);
• Public lecture of the Life, Structure and Cognition conference, "La curiosité: apprentissage, évolution et intelligence artificielle” (June 2023);
• Invited talk at the IMOL Conference, Paris, Autotelic agents, open-endedness and applications (Sept. 2023)
• Invited talk at the NeuroEconomics seminar of DEC, ENS, "Open-ended learning in humans and AI: the roles of curiosity and language"
• Invited talk at the NeuroDevRob workshop, Cergy Paris University, Autotelic agents, open-endedness and applications (Dec. 2023)

11.1.5 Leadership within the scientific community

Helene Sauzéon is the proxis of the Inria center of the University of Bordeaux réseau RT (GDR) CNRS Education since july 2022

Cédric Colas was member of the board of the IMOL community https://­www.­imol-community.­org/­community/

11.1.6 Scientific expertise

- Clément Moulin-Frier is on active discussion with the start-up Pontos in view of a future collaboration in 2024.

- Clément Moulin-Frier was a member of the CRCN/ISFP Jury of Inria Bordeaux on May 9th, 2023.

- Helene Sauzeon was member of the selection committe of DR2 09 (Univ. G. Eiffel) in 2023

- Helene Sauzéon was Member of the selection committe of PhD of MSCA COFUND SOUND.AI program (European Prog. at Sorbonne Univ) since 2023

- Hélène Sauzéon is member of the ANR committee - CES 38 (Interdisciplinary research section)- since November 2023.

- Helene Sauzéon was member of scientific committee of MAVIE-II -Calyxis link.

- PY Oudeyer was a member of the jury selecting grants for PhDs in AI in the context of SoundAI project at Sorbonne University

- PY Oudeyer reviewed projects for the European Commission (ERC, Marie Curie grants, EU Pathfinder), for the US/Israel Binational Science Foundation, for ANR, for RIF Cyprus, Leverhulme Trust.

11.1.7 Research administration

• Hélène Sauzéon and Cécile Mazon are members of directory committee of LILLAB (https://¬www.¬lillabneurodev.¬fr/) which is a living and learning lab funded by the “délégation interministérielle à la stratégie nationale à l’autisme et troubles neurodéveloppementaux” and aiming the dissemination of knowledge in connection with the 3 centers of excellence for autism and Neurodevelopmental syndromes; since 2020.

• Hélène Sauzéon is member of directory committee of IFHR / FEDRAH (https://¬ifr-handicap.-inserm.¬fr/) which is a national institute on disability funded by Inserm aiming the researcher networking and dissemination of knowledge on multidisciplinary research on disability; since 2018.

• Hélène Sauzéon is the head of the Innovations and Transfer Committee of the BIND Center of Excellence link in Bordeaux and member of the BIND directory Committee since 2018

• Cécile Mazon is co-responsible of the WP digital tools of the PIA AtypieFriendly (ex - AspieFriendly)

• Pierre-Yves Oudeyer is head of the Flowers project-team, Inria/Univ. Bordeaux/ENSTA ParisTech

11.2.1 Teaching

Teaching Responsibilities:

• Clément Moulin-Frier is responsible professor of the "System Design, Integration and Control" course at the University Pompeu Fabra in Barcelona, Spain.
• Cécile Mazon is responsible of the second year of the curriculum in Technology, Ergonomics, Cognition and Handicap (TECH, Cognitive Sciences - University of Bordeaux) since sept. 2021.
• Cécile Mazon is responsible of the curriculum in Technology, Ergonomics, Cognition and Handicap (TECH, Cognitive Sciences - University of Bordeaux) since sept. 2022.
• David Filliat is in charge since 2012 of the "Robotics and autonomous systems" third year speciality at ENSTA Paris.
• Sao Mai Nguyen is in charge of the "Robot Learning" third year course at ENSTA Paris.

Teaching Involvement in Computer / Engineer science or in cognitive science:

• BS & Master: Cognitive Science, Univ. of Bordeaux- , 37h, Marion Pech
• BS & Master: Neuropsychology, Univ. of Caen-,2H, Marion Pech
• ENSC/ENSEIRB Presentation of developmental artificial intelligence and the Flowers Lab, 2h, Option Robot (Laetitia Teodorescu)
• ENSC Introduction to bayesian analysis, 8h, Option AI (Adolphe Maxime)
• ENSC Transformers and Large Language Models, 8h, Option AI (Clément Romac)
• BS & Master: Cognitive Science, Univ. of Bordeaux- , 192h, Cécile Mazon
• Master: Navigation for Robotics, 21 h, M2, ENSTA Paris, David Filliat
• Master: Navigation for Robotics, 24 h, M2 DataAI, IP Paris - Paris, David Filliat
• Université de Bordeaux - MIASHS Bachelor : Models and measures of high-level cognitive functions - knowledge and representation, 22h, (Poupard Matisse)
• Université de Bordeaux - TECH Master: Virtual reality, interaction and healthcare applications, 7h, (Poupard Matisse)
• Université de Bordeaux - TECH Master: Scientific basis, 2h, (Poupard Matisse)
• 2nd year : Deep Learning, 12h, IMT Atlantique (Sao Mai Nguyen).
• Université de Bordeaux - TECH Master: IT Project Management, 18h (Isabeau Saint-Supery)
• Master UPF-Barcelona: Robotics and AI, 10h (Clément Moulin-Frier)
• PY Oudeyer gave a course on developmental reinforcement learning at ENSEIRB (2h), dec. 2023.
• PY Oudeyer gave a course on developmental reinforcement learning at Cogmaster at Sorbonne Univ./ENS (2h+4h project jury), dec. 2023.
• Master: Cognitive Science, 24h (Eric Meyer)
• Teacher training at the Rectorat of Bordeaux : digital technologies for students with special educational needs , 6h (Eric Meyer)
• 2nd year Master in cognitive science : Assistive technologies (20h), Rania Abdelghani.
• University Degree in Neuropsychological Sciences : Cognitive aging and Digital technologies (3h), Helene Sauzéon
• Clément Moulin-Frier gave a class in the Parcours Robotique at ENSEIRB-MATMECA (Bordeaux INP). 2 hours, November 2023.

11.2.2 Supervision

• PhD defended: Tristan Karch (defended in 2023), "Language acquisition in curiosity-driven Deep RL", beg. in sept. 2019 (supervisors: PY. Oudeyer and C. Moulin-Frier)

• PhD defended: Mayalen Etcheverry (defended in 2023), "Automated discovery with intrinsically motivated goal exploration processes", beg. in sept. 2020 (supervisors: PY. Oudeyer and C. Moulin-Frier)

• PhD defended: Laetitia Teodorescu, (defended in 2023), "Graph Neural Networks in Curiosity-driven Exploring Agents", beg. in sept. 2020 (supervisors: PY. Oudeyer and K. Hoffman)

• PhD in progress: J Grgur Kovac (in progress), "Developmental training of socio-cognitive abilities in AI systems", (supervisors:PF. Dominey and PY. Oudeyer)

• PhD in progress: J Julien Pourcel (in progress), "Autotelic LLMs that learn how to code", (supervisors: C. Moulin-Frier and PY. Oudeyer)

• PhD in progress: J Thomas Carta (in progress), "LLM-based Autotelic deep reinforcement learning agents", (supervisors: O. Sigaud, S. Lamprier and PY. Oudeyer)

• PhD in progress: J Clément Romac (in progress), "Grounding LLMs with online RL", (supervisors: T. Wolf and PY. Oudeyer)

• PhD in progress: Jeremy Perez, supervized by Clément Moulin-Frier, which started in October 2023

• PhD in progress : Chloé Desvaux "Design and experiment new metacognitive trainings for fostering curiosity and creativity among children in a school setting: a lever for intrinsically motivated learning? ",beg. in October. 2023 (supervised by H. Sauzéon and PY Oudeyer).

• PhD in progress : Leana Petiot " Study of Augmented reality on the functioning of Implicit autobiographical memory",beg. in October. 2023 (supervised by H. Sauzéon and P. Dragicevic from Potioc team).

• PhD in progress: Maxime Adolphe, "Adaptive personalization in attention training systems", beg. in sept. 2020 (supervisors: H. Sauzéon and PY. Oudeyer)

• PhD in progress: Rania Abdelgani, "Fostering curiosity and meta-cognitive skills in educational technologies", beg. in dec. 2020 (supervisors: H. Sauzéon and PY. Oudeyer).

• PhD in progress: Isabeau Saint-Supery, "Designing and Assessing a new interactive tool fostering stakeholders' cooperation for school inclusion", supervised by H. Sauzéon and C. Mazon.

• PhD in progress : Matisse Poupard "Optimize learning in a digital environment according to learners' level of expertise, epistemic curiosity and mode of instruction" ",beg. in Ap. 2022 (supervised by H. Sauzéon and A. Tricot from Univ. Montpellier).

Gautier Hamon and Clément Moulin-Frier supervised the Master internships of Richard Bornemann and of Corentin Léger (in collaboration with Xavier Hinaut from Mnemosyne) in 2023.

Maxime Adolphe and Hélène Sauzéon supervised the Master internships of Stéphanie Mortemousque in 2023.

Cécile Mazon and Isabeau Saint-Supery supervised the Master internship of Valentin Strahm in 2023.

Rania Abdelghani and Hélène Sauzéon supervised the Master internship of Chloé Desvaux in 2023.

Pierre Dragicevic and Hélène Sauzéon supervised the Master internship of Léana Petiot in 2023.

11.2.3 Juries

H. Sauzéon has been member of scientific board of HDR degree of P. Dragicevic

Clément Moulin-Frier was a member of the PHD jury of Joachim Winther Pedersen (thesis director: Sebastian Risi, Univeristy of Copenhagen).

H. Sauzéon has been president of PhD jury of Axelle Gelineau on " Projet RGS@HOME : Evaluation de l’acceptabilité d’un système de télé-réhabilitation membre supérieur basé sur la réalité virtuelle auprès des patients post-AVC. " at the university of Limoges, December, 8th 2023, Limoges.

H. Sauzéon has been reviewer in the PhD jury of Marine Saba on " Efficacité d'un programme d'entraînement des ressources attentionnelles et de la mémoire de travail chez les personnes âgées avec un trouble cognitif léger : Effets sur les fonctions cognitives et une situation écologique évaluée avec la réalité virtuelle " at the university of Paris, December, 15th 2023, Paris.

Clément Moulin-Frier was a member of the jury of the Premier hackathon consacré à l’enquête journalistique sur les algorithmes at IJBA (Bordeaux) on November 30th, 2023.

Clément Moulin-Frier is a member of the "comité de suivi de thèse" of Nathan Trouvain (Mnemosyne, Inria) and Camille Charrier (LPNC, Grenoble).

Hélène sauzéon is a member of the "comité de suivi de thèse" of Hugo Fournier (Lab Psychology, Bordeaux).

Hélène sauzéon is academic tutor for 2 PhD students of the Doctoral School SP2.

Cécile Mazon organized and was president of the jury of the defense of Cognitive Sciences Master students (M1 and M2)

PY Oudeyer was a reviewer in the HdR of Erik Gustaffsson (Univ. Bourgogne Franche Comté), and examiner in the PhD of Enrique Donancio (INSA Rouen Normandie) and Lina Mezghani (Univ. Grenoble).

PY Oudeyer was in the PhD "comité de suivi" of Marc Welter (Univ. Bordeaux), Jean-Baptiste Gaya (Université Paris-Sorbonne), Marie Martin (Univ. Paris Saclay), Elias Najarro (Univ. Copenhagen), Matthis Poupard (Univ. Bordeaux),

11.3.1 Internal or external Inria responsibilities

• Hélène Sauzéon was member of extented office of Project-team committee of the centre of Inria of university of Bordeaux.

• Hélène Sauzéon was student in the Inria MasterClass since sep. 2022.

• PY Oudeyer contributed several internal notes on AI in society, helping Inria direction answer several requests on this topic from governmental organizations.

11.3.2 Articles and contents

• Clément Romac, Thomas Carta and PY Oudeyer wrote down "Quand les IA font l’expérience de la réalité" explaining the recently published paper "Grounding Large Language Models in Interactive Environments with Online Reinforcement Learning" (to be published in "Pour La Science" march 2024)
• Alexandre Torres–Leguet, Clément Romac, Thomas Carta and PY Oudeyer created the (on-going) series of videos "ChatGPT expliqué en 5 minutes", aimed at training AI litteracy in a wide diversity of students (e.g. high school), available here: https://­developmentalsystems.­org/­chatgpt_en_5_minutes/. It was already integrated in the MOOC AI4T (https://­www.­ai4t.­eu).
• Eleni Nisioti, Clément Moulin-Frier and Pierre-Yves Oudeyer published a Blog post on An ecological approach to Artificial Intelligence.
• Rania Abdelghani, Pierre-Yves Oudeyer and Hélène Sauzéon published a blog post for the ministry of education on Curiosité, numérique et apprentissage: que dit la recherche?.
• Rania Abdelghani, Pierre-Yves Oudeyer, and Maxime Adolphe were interviewed for a TV program about AI during the ROBOCUP 2023 event.
• Hélène Sauzéon gave an interview for Sud-Ouest Newspaper on "Assistants vocaux : faut-il laisser les enfants utiliser seuls une enceinte connectée ?, December, 14th, 2023 https://­www.­sudouest.­fr/­societe/­parentalite/­assistants-vocaux-faut-il-laisser-les-enfants-utiliser-seul-une-enceinte-connectee-17789838.­php
• PY Oudeyer gave an interview on the societal dimensions of AI at France Bleu/France 3 gironde (Dec 2023)
• PY Oudeyer gave an interview on cultural and social stakes of AI in Les Echos Judiciaires
• PY Oudeyer gave an interview on large language models/ChatGPT at Sud-Ouest.

11.3.3 Education

• 3 "Stagiaire de 3e", Presentation of topics of the team (generative AI, Large language models ,Cellular automata, Reinforcement learning) February 2023 (Clément Romac, Maxime Adolphe)
• "Stagiaire de 3e", Presentation of topics of the team (generative AI, Large language models ,Cellular automata, Reinforcement learning, ) July 2023 (Gautier Hamon, Matisse Poupart)
• 2 "Stagiaire de 3e", Presentation of topics of the team (generative AI, Large language models ,Cellular automata, Reinforcement learning, Curiosity and Learning in Human Cognition) November 2023 (Gautier Hamon, Matisse Poupard, Maxime Adolphe)
• 3 "Stagiaire de 3e", Presentation of topics of the team (generative AI, Large language models ,Cellular automata, Reinforcement learning, Curiosity and Learning in Human Cognition) December 2023 (Clément Romac, Matisse Poupard, Chloé Desvaux, Marion Pech, Hélène Sauzéon)
• 2 Classrooms for the CHICHE - One researcher -One classroom- Lycée Václav Havel, Bègles, febuary 2023
• Podcast - "Etudes scientifiques pluridisciplinaires", Bègles, febuary 2023
• The 1st edition of the Hackathon Hack1Robo at the Fablab of Cap Science (June 2-4, 2023) was co-organized by Clément Moulin-Frier.

11.3.4 Interventions

• Clément Moulin-Frier, Hélène Sauzeon and Clement Romac have participated to the Workshop (4h) on AI and its impact for French teachers at Boussoles du Numérique (sep. 2023)
• Workshop (2h) on generative AIs for education at INSPE (nov. 2023)
• Clément Moulin-Frier was interviewed by Antoine Beauchamp for the radio program La Science, CQFD. The program theme was "Origine de la vie : raconte-moi ta LYFE" and it was broadcasted on France Culture on June 21st, 2023.
• Sauzéon gave a talk on « Outils numériques et Maladies Neurodégénératives : une révolution en marche ? » for the "Journée Grand Public - Chercheurs, patients, aidants : Tous ensemble !, organisez by the patients' association, la Maison du Cerveau, March 9th 2023, Bordeaux.
• Table ronde à NAIA-R, Bordeaux, "IA Génératives: ChatGPT and co" (July 2023)
• Invited presentation at a Cafe Pedagogique seminar, « L’intelligence artificielle peut-elle contribuer à stimuler la curiosité et la méta-cognition chez les enfants ? » (Sept. 2023)
• Invited presentation at FrenchTech Days in Bordeaux, « Entrainer la curiosité chez les enfants avec ChatGPT, c’est possible ? »

International journals

• 33 articleR.Rania Abdelghani, Y.-H.Yen-Hsiang Wang, X.Xingdi Yuan, T.Tong Wang, P.Pauline Lucas, H.Hélène Sauzéon and P.-Y.Pierre-Yves Oudeyer. GPT-3-driven pedagogical agents for training children's curious question-asking skills.International Journal of Artificial Intelligence in EducationJune 2023HALDOIback to textback to text
• 34 articleX.Xerxes Arsiwalla, R.Ricard Solé, C.Clément Moulin-Frier, I.Ivan Herreros, M.Martí Sánchez-Fibla and P.Paul Verschure. The Morphospace of Consciousness: Three Kinds of Complexity for Minds and Machines.NeuroSci42March 2023, 79-102HALDOIback to text
• 35 articleD.David Brellmann, D.David Filliat and G.Goran Frehse. Fourier Features in Reinforcement Learning with Neural Networks.Transactions on Machine Learning Research JournalOctober 2023HAL
• 36 articleL.Levin Brinkmann, F.Fabian Baumann, J.-F.Jean-François Bonnefon, M.Maxime Derex, T. F.Thomas F Müller, A.-M.Anne-Marie Nussberger, A.Agnieszka Czaplicka, A.Alberto Acerbi, T.Thomas Griffiths, J.Joseph Henrich, J.Joel Leibo, R.Richard Mcelreath, P.-Y.Pierre-Yves Oudeyer, J.Jonathan Stray and I.Iyad Rahwan. Machine culture.Nature Human Behaviour711November 2023, 1855-1868HALDOIback to text
• 37 articleP.-A.Pierre-Antoine Cinquin, P.Pascal Guitton and H.Hélène Sauzéon. Towards Truly Accessible MOOCs for Persons with Cognitive Impairments: A Field Study.Human-Computer Interaction385-6May 2023HALDOI
• 38 articleA.Aurélien Frick, S.S. Fay, B.Badiâa Bouazzaoui, H.Hélène Sauzéon, L.L. Angel, S.Sandrine Vanneste and L.Laurence Taconnat. The respective contribution of cognitive control and working memory to semantic and subjective organization in aging.Psychology and Aging2023HALDOI
• 39 articleM.Maxime Gasse, D.Damien Grasset, G.Guillaume Gaudron and P.-Y.Pierre-Yves Oudeyer. Using Confounded Data in Latent Model-Based Reinforcement Learning.Transactions on Machine Learning Research JournalAugust 2023HAL

National journals

• 40 articleJ.Jérôme Cuadrado, É.Éric Meyer, J.Jenna Maire, C.Charlotte Legigan, C.Charlie Sentenac, M.Mélissa Athane, M.Marie Demont, C.Cassandra Langa, A.Axel Mence and G.Grégory Michel. ‘‘Gifted’’ children, challenges of an integrative psychological assessment in the adolescent clinic. A case study..Annales Médico-Psychologiques, Revue PsychiatriqueMay 2023HALDOI

International peer-reviewed conferences

• 41 inproceedingsR.Rania Abdelghani, E.Edith Law, C.Chloé Desvaux, P.-Y.Pierre-Yves Oudeyer and H.Hélène Sauzéon. Interactive environments for training children’s curiosity through the practice of metacognitive skills : a pilot study.IDC 2023 - The 22nd annual ACM Interaction Design and Children ConferenceChicago IL, United StatesACM; ACMNovember 2023, 495-501HALDOIback to textback to text
• 42 inproceedingsL.Louis Annabi, Z.Ziqi Ma and S. M.Sao Mai Nguyen. Unsupervised Motion Retargeting for Human-Robot Imitation.Companion of the 2024 ACM/IEEE International Conference on Human-Robot InteractioBoulder (CO), United StatesACMMarch 2024HALDOI
• 43 inproceedingsL.Louis Annabi and S. M.Sao Mai Nguyen. Prerequisite Structure Discovery in Intelligent Tutoring Systems.2023 IEEE International Conference on Development and Learning (ICDL)Macau, ChinaIEEENovember 2023, 176-181HALDOI
• 44 inproceedingsT.Thomas Carta, C.Clément Romac, T.Thomas Wolf, S.Sylvain Lamprier, O.Olivier Sigaud and P.-Y.Pierre-Yves Oudeyer. Grounding Large Language Models in Interactive Environments with Online Reinforcement Learning.International Conference on Machine Learning 20232023676-3713Honololu, Hawaii, United States2023HALback to text
• 45 inproceedingsC.Cédric Colas, L.Laetitia Teodorescu, P.-Y.Pierre-Yves Oudeyer, E. X.Eric Xingdi Yuan and M.-A.Marc-Alexandre Côté. Augmenting Autotelic Agents with Large Language Models.CoLLAs 2023 - Conference on Lifelong Learing AgentsMontreal, CanadaAugust 2023HALback to text
• 46 inproceedingsT.Tom Dupuis, J.Jaonary Rabarisoa, Q.-C.Quoc-Cuong Pham and D.David Filliat. VIBR: Learning View-Invariant Value Functions For Robust Visual Control.CoLLAs 2023 - Conference on Lifelong Learning AgentsMontreal, CanadaAugust 2023HAL
• 47 inproceedingsG.Gianni Franchi, X.Xuanlong Yu, M.Marwane Hariat, A.Antoine Manzanera and D.David Filliat. InfraParis: A multi-modal and multi-task autonomous driving dataset.IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)Waikoloa, United States2024HAL
• 48 inproceedingsG.Gautier Hamon, E.Eleni Nisioti and C.Clément Moulin-Frier. Eco-evolutionary Dynamics of Non-episodic Neuroevolution in Large Multi-agent Environments.GECCO '23 Companion - Companion Conference on Genetic and Evolutionary ComputationLisbon Portugal, FranceACM; ACMJuly 2023, 143-146HALDOIback to text
• 49 inproceedingsG.Grgur Kovač, R.Rémy Portelas, P. F.Peter Ford Dominey and P.-Y.Pierre-Yves Oudeyer. The SocialAI School: Insights from Developmental Psychology Towards Artificial Socio-Cultural Agents.TOM 2023 - Theory-of-Mind Workshop at PMLR-202Honolulu (Hawaii), United StatesNovember 2023HAL
• 50 inproceedingsA.Aleksa Marusic, S. M.Sao Mai Nguyen and A.Adriana Tapus. Evaluating Kinect, OpenPose and BlazePose for Human Body Movement Analysis on a Low Back Pain Physical Rehabilitation Dataset.HRI 2023 - ACM/IEEE International Conference on Human-Robot InteractionStockholm, SwedenACMMarch 2023, 587-591HALDOI
• 51 inproceedingsE.Eric Meyer, H.Hélène Sauzéon, I.Isabeau Saint-Supery and C.Cécile Mazon. Systematic review of technologies to collaborate and co-educate students with special educational needs and supporting their schooling.IHIET 2023 - 10th International Conference on Human Interaction and Emerging Technologies111Nice, FranceAHFE InternationalJuly 2023, 1-12HALDOIback to textback to text
• 52 inproceedingsE.Eleni Nisioti, E.Elías Masquil, G.Gautier Hamon and A. C.And Clément Moulin-Frier. Autotelic Reinforcement Learning in Multi-Agent Environments.CoLLAs 2023, Conference on Lifelong Learning AgentsMontréal, CanadaAugust 2023HALback to textback to text
• 53 inproceedingsE.Eleni Nisioti and C.Clément Moulin-Frier. Dynamics of niche construction in adaptable populations evolving in diverse environments.ALIFE 2023 - The 2023 Conference on Artificial LifeSapporo, JapanMIT Press2023HALDOIback to textback to text
• 54 inproceedingsE.Erwan Plantec, G.Gautier Hamon, M.Mayalen Etcheverry, P.-Y.Pierre-Yves Oudeyer, C.Clément Moulin-Frier and B.-C. W.Bert Wang-Chak Chan. Flow-Lenia: Towards open-ended evolution in cellular automata through mass conservation and parameter localization.The 2023 Conference on Artificial LifeTokyo, JapanMIT Press; MIT PressJuly 2023HALDOIback to textback to textback to text
• 55 inproceedingsI.Isabeau Saint-Supery, H.Hélène Sauzéon, C.Christelle Maillart, N.Nicolas Neu, E.Eric Meyer and C.Cécile Mazon. Cross-cultural evaluation of a web application to support communication and collaboration among stakeholders of the school inclusion of children with ASD.AAATE 2023 - The 17h International Conference of the Association for the Advancement of Assistive Technology in EuropeParis, France2023HALback to textback to text
• 56 inproceedingsZ.Ziang Xiao, X.Xingdi Yuan, Q. V.Q. Vera Liao, R.Rania Abdelghani and P.-Y.Pierre-Yves Oudeyer. Supporting Qualitative Analysis with Large Language Models: Combining Codebook with GPT-3 for Deductive Coding.IUI 2023 - 28th International Conference on Intelligent User InterfacesSydney, AustraliaACMMarch 2023, 75-78HALDOIback to textback to text
• 57 inproceedingsM.Mehdi Zadem, S.Sergio Mover and S. M.Sao Mai Nguyen. Goal Space Abstraction in Hierarchical Reinforcement Learning via Set-Based Reachability Analysis.2023 IEEE International Conference on Development and Learning (ICDL)Macau, ChinaIEEENovember 2023, 423-428HALDOI
• 58 inproceedingsK.Kai Zhang, E.Eric Lucet, J.Julien Alexandre Dit Sandretto and D.David Filliat. Navigation among movable obstacles using machine learning based total time cost optimization.IROS 2023 - IEEE/RSJ International Conference on Intelligent Robots and SystemsDetroit (MI), United StatesOctober 2023HAL

Conferences without proceedings

• 59 inproceedings R.Rania Abdelghani, H.Hélène Sauzéon and P.-Y.Pierre-Yves Oudeyer. Generative AI in the Classroom: Can Students Remain Active Learners? NeurIPS 2023 - GAIED Workshop - Conference on Neural Information Processing Systems New orleans, USA, United States arXiv 2023 HAL DOI back to text back to text
• 60 inproceedingsR.Richard Bornemann, G.Gautier Hamon, E.Eleni Nisioti and C.Clément Moulin-Frier. Emergence of Collective Open-Ended Exploration from Decentralized Meta-Reinforcement Learning.ALOE 2023 - Agent Learning in Open-Endedness workshop at NeurIPS 2023New Orleans, United StatesDecember 2023HALback to text
• 61 inproceedingsM.Mayalen Etcheverry, M.Michael Levin, C.Clément Moulin-Frier and P.-Y.Pierre-Yves Oudeyer. SBMLTOODEJAX: Efficient Simulation and Optimization of Biological Network Models in JAX.NeurIPS 2023 - AI4Science WorkshopNew Orleans (LA), FranceOctober 2023HALback to textback to text
• 62 inproceedingsE.Eric Meyer, I.Isabeau Saint-Supery, H.Hélène Sauzéon and C.Cécile Mazon. A review of technologies to collaborate and co-educate students with special educational needs and support their schooling.Colloque international Education et InégalitésParis, FranceMay 2023HAL
• 63 inproceedingsI.Isabeau Saint-Supery, H.Hélène Sauzéon, C.Christelle Maillart, N.Nicolas Neu, E.Eric Meyer and C.Cécile Mazon. Cross-cultural evaluation of a web application to support communication andcollaboration among stakeholders of the school inclusion of children with ASD.AAATE 2023 - The 17h International Conference of the Association for the Advancement of Assistive Technology in EuropeParis ( France), FranceAugust 2023HAL
• 64 inproceedingsL.Laetitia Teodorescu, C.Cédric Colas, M.Matthew Bowers, T.Thomas Carta and P.-Y.Pierre-Yves Oudeyer. Codeplay: Autotelic Learning through Collaborative Self-Play in Programming Environments.IMOL 2023 - Intrinsically Motivated Open-ended Learning workshop at NeurIPS 2023New Orleans, United StatesDecember 2023HALback to text
• 65 inproceedingsM.Mehdi Zadem, S.Sergio Mover and S. M.Sao Mai Nguyen. Goal Space Abstraction in Hierarchical Reinforcement Learning via Reachability Analysis.IMOL 2023 - The 6th International Workshop on Intrinsically Motivated Open-ended LearningParis, FranceSeptember 2023HAL

Scientific books

• 66 bookD.Didier Roy. Manuel d'éducation numérique - Cycle 2 56 - Collection Décodage (ouvrage collectif).December 2023HAL
• 67 bookD.Didier Roy. Manuel d'éducation numérique - Cycle 2 78 - Collection Décodage (ouvrage collectif).January 2024HAL

Scientific book chapters

• 68 inbookK.Kai Zhang, E.Eric Lucet, J. A.Julien Alexandre Dit Sandretto, S.Selma Kchir and D.David Filliat. A review of classical and learning based approaches in task and motion planning.LNCS-836Informatics in Control, Automation and RoboticsLecture Notes in Networks and SystemsSpringer International PublishingNovember 2023, 83-99HALDOI

Doctoral dissertations and habilitation theses

• 69 thesisM.Mayalen Etcheverry. Curiosity-driven AI for Science: Automated Discovery of Self-Organized Structures.Inria & Labri, Université BordeauxNovember 2023HALback to textback to text
• 70 thesisT.Tristan Karch. Towards Social Autotelic Artificial Agents : Formation and Exploitation of Cultural Conventions in Autonomous Embodied Artificial Agents.Université de BordeauxMay 2023HALback to textback to text
• 71 thesisL.Laetitia Teodorescu. Endless minds most beautiful: building open-ended linguistic autotelic agents with deep reinforcement learning and language models.Université de bordeauxNovember 2023HALback to text

Reports & preprints

• 72 miscM.Maxime Adolphe, M.Marion Pech, M.Masataka Sawayama, D.Denis Maurel, A.Alexandra Delmas, P.-Y.Pierre-Yves Oudeyer and H.Hélène Sauzéon. Exploring the Potential of Artificial Intelligence in Individualized Cognitive Training: a Systematic Review.December 2023HALDOIback to text
• 73 miscM.Mayalen Etcheverry, C.Clément Moulin-Frier, P.-Y.Pierre-Yves Oudeyer and M.Michael Levin. AI-driven Automated Discovery Tools Reveal Diverse Behavioral Competencies of Biological Networks.January 2024HALDOIback to textback to textback to text
• 74 miscG.Grgur Kovač, M.Masataka Sawayama, R.Rémy Portelas, C.Cédric Colas, P. F.Peter Ford Dominey and P.-Y.Pierre-Yves Oudeyer. Large Language Models as Superpositions of Cultural Perspectives.November 2023HALback to textback to text
• 75 miscC.Corentin Léger, G.Gautier Hamon, E.Eleni Nisioti, X.Xavier Hinaut and C.Clément Moulin-Frier. Evolving Reservoirs for Meta Reinforcement Learning.2023, 143-146HALback to textback to text
• 76 miscJ.Jérémy Perez, M.Martí Sánchez-Fibla and C.Clément Moulin-Frier. Cultural evolution in populations with heterogeneous and variable preferences.December 2023HAL
• 77 miscJ.Julien Pourcel, C.Cédric Colas, P.-Y.Pierre-Yves Oudeyer and L.Laetitia Teodorescu. ACES: Generating diverse programming puzzles with autotelic language models and semantic descriptors.October 2023HALback to text
• 78 miscM.Martí Sánchez-Fibla, C.Clément Moulin-Frier and R.Ricard Solé. Cooperative control of environmental extremes by artificial intelligent agents.2022HALDOIback to textback to text
• 79 miscY.Yadurshana Sivashankar, M.Myra Fernandes, P.-Y.Pierre-Yves Oudeyer and H.Hélène Sauzéon. The Beneficial Role of Curiosity on Route memory in Children.October 2023HALDOIback to textback to textback to text

Other scientific publications

• 80 inproceedingsL.Louis Annabi and S. M.Sao Mai Nguyen. Prerequisite structure discovery for an intelligent tutoring system based on intrinsic motivation.IMOL 2023 - The 6th Internatinal Workshop on Intrinsically Motivated Open-ended LearningParis, FranceSeptember 2023HAL
• 81 inproceedingsJ.Julien Pourcel, C.Cédric Colas, P.-Y.Pierre-Yves Oudeyer and L.Laetitia Teodorescu. ACES: generating diverse programming puzzles with autotelic language models and semantic descriptors.Neurips 2023 - The 37th Annual Conference on Neural Information Processing SystemsNouvelle Orleans, United StatesDecember 2023HAL
• 82 inproceedingsM.Mehdi Zadem, S.Sergio Mover and S. M.Sao Mai Nguyen. Emergence of a Symbolic Goal Representation with an Intelligent Tutoring System based on Intrinsic Motivation.NeurIPS 2023 - IMOL Workshop "Intrinsically-Motivated and Open-Ended Learning"New Orleans (Louisiana), United StatesIEEEDecember 2023, 423-428HAL

Scientific popularization

• 83 inbookI.Isabeau Saint-Supery, C.Cécile Mazon, M.Meyer Eric and H.Hélène Sauzéon. Design of a web application to support coeducation for school inclusion of pupils with ASD.Éthiques inclusives en éducation. Recherches, contextes et pratiques (p. 145-160)Parentalité & HandicapChamps Social2023, 260HALback to text