2025Activity report​​​‌Project-TeamSCOOL

7.1 Bandits​​ and RL theory

Best-Arm​​​‌ Identification in Unimodal Bandits‌, 39

We study‌​‌ the fixed-confidence best-arm identification​​ problem in unimodal bandits,​​​‌ in which the means‌ of the arms increase‌​‌ with the index of​​ the arm up to​​​‌ their maximum, then decrease.‌ We derive two lower‌​‌ bounds on the stopping​​ time of any algorithm.​​​‌ The instance-dependent lower bound‌ suggests that due to‌​‌ the unimodal structure, only​​ three arms contribute to​​​‌ the leading confidence-dependent cost.‌ However, a worst-case lower‌​‌ bound shows that a​​ linear dependence on the​​​‌ number of arms is‌ unavoidable in the confidence-independent‌​‌ cost. We propose modifications​​ of Track-and-Stop and a​​​‌ Top Two algorithm that‌ leverage the unimodal structure.‌​‌ Both versions of Track-and-Stop​​ are asymptotically optimal for​​​‌ one-parameter exponential families. The‌ Top Two algorithm is‌​‌ asymptotically near-optimal for Gaussian​​ distributions and we prove​​​‌ a non-asymptotic guarantee matching‌ the worse-case lower bound.‌​‌ The algorithms can be​​ implemented efficiently and we​​​‌ demonstrate their competitive empirical‌ performance.

The Batch Complexity‌​‌ of Bandit Pure Exploration​​, 40

In a​​​‌ fixed-confidence pure exploration problem‌ in stochastic multi-armed bandits,‌​‌ an algorithm iteratively samples​​ arms and should stop​​​‌ as early as possible‌ and return the correct‌​‌ answer to a query​​ about the arms distributions.​​​‌ We are interested in‌ batched methods, which change‌​‌ their sampling behaviour only​​ a few times, between​​​‌ batches of observations. We‌ give an instance-dependent lower‌​‌ bound on the number​​ of batches used by​​​‌ any sample efficient algorithm‌ for any pure exploration‌​‌ task. We then give​​ a general batched algorithm​​​‌ and prove upper bounds‌ on its expected sample‌​‌ complexity and batch complexity.​​ We illustrate both lower​​​‌ and upper bounds on‌ best-arm identification and thresholding‌​‌ bandits.

Pareto Set Identification​​​‌ With Posterior Sampling,​ 33

The problem of​‌ identifying the best answer​​ among a collection of​​​‌ items having real-valued distribution​ is well-understood. Despite its​‌ practical relevance for many​​ applications, fewer works have​​​‌ studied its extension when​ multiple and potentially conflicting​‌ metrics are available to​​ assess an item's quality.​​​‌ Pareto set identification (PSI)​ aims to identify the​‌ set of answers whose​​ means are not uniformly​​​‌ worse than another. This​ paper studies PSI in​‌ the transductive linear setting​​ with potentially correlated objectives.​​​‌ Building on posterior sampling​ in both the stopping​‌ and the sampling rules,​​ we propose the PSIPS​​​‌ algorithm that deals simultaneously​ with structure and correlation​‌ without paying the computational​​ cost of existing oraclebased​​​‌ algorithms. Both from a​ frequentist and Bayesian perspective,​‌ PSIPS is asymptotically optimal.​​ We demonstrate its good​​​‌ empirical performance in real-world​ and synthetic instances.

FraPPE:​‌ Fast and Efficient Preference-based​​ Pure Exploration, 30​​​‌

Preference-based Pure Exploration (PrePEx)​ aims to identify with​‌ a given confidence level​​ the set of Pareto​​​‌ optimal arms in a​ vector-valued (aka multi-objective) bandit,​‌ where the reward vectors​​ are ordered via a​​​‌ (given) preference cone $\mathrm{𝒞}$. Though PrePEx and​‌ its variants are well-studied,​​ there does not exist​​​‌ a computationally efficient algorithm​ that can optimally track​‌ the existing lower bound​​ for arbitrary preference cones.​​​‌ We successfully fill this​ gap by efficiently solving​‌ the minimisation and maximisation​​ problems in the lower​​​‌ bound. First, we derive​ three structural properties of​‌ the lower bound that​​ yield a computationally tractable​​​‌ reduction of the minimisation​ problem. Then, we deploy​‌ a Frank-Wolfe optimiser to​​ accelerate the maximisation problem​​​‌ in the lower bound.​ Together, these techniques solve​‌ the maxmin optimisation problem​​ in $𝒪\left(\mathrm{K‌}{L}^2\right)$ time​ for a bandit instance​‌ with $K$ arms and​​ $L$ dimensional reward, which​​​‌ is a significant acceleration​ over the literature. We​‌ further prove that our​​ proposed PrePEx algorithm, FraPPE,​​​‌ asymptotically achieves the optimal​ sample complexity. Finally, we​‌ perform numerical experiments across​​ synthetic and real datasets​​​‌ demonstrating that FraPPE achieves​ the lowest sample complexities​‌ to identify the exact​​ Pareto set among the​​​‌ existing algorithms.

Leveraging Priors​ on Distribution Functions for​‌ Multi-Arm Bandits, 21​​

We introduce Dirichlet Process​​​‌ Posterior Sampling (DPPS), a​ Bayesian non-parametric algorithm for​‌ multi-arm bandits based on​​ Dirichlet Process (DP) priors.​​​‌ Like Thompson-sampling, DPPS is​ a probability-matching algorithm, i.e.,​‌ it plays an arm​​ based on its posterior​​​‌ probability of being optimal.​ Instead of assuming a​‌ parametric class for the​​ reward generating distribution of​​​‌ each arm, and then​ putting a prior on​‌ the parameters, in DPPS​​ the reward generating distribution​​​‌ is directly modeled using​ DP priors. DPPS provides​‌ a principled approach to​​ incorporate prior belief about​​​‌ the bandit environment, and​ in the noninformative limit​‌ of the DP priors​​ (i.e. Bayesian Bootstrap), we​​​‌ recover Non Parametric Thompson​ Sampling (NPTS), a popular​‌ non-parametric bandit algorithm, as​​ a special case of​​​‌ DPPS. We employ stick-breaking​ representation of the DP​‌ priors, and show excellent​​ empirical performance of DPPS​​ in challenging synthetic and​​​‌ real world bandit environments.‌ Finally, using an information-theoretic‌​‌ analysis, we show non-asymptotic​​ optimality of DPPS in​​​‌ the Bayesian regret setup.‌

Towards Blackwell Optimality: Bellman‌​‌ Optimality Is All You​​ Can Get, 48​​​‌

Although average gain optimality‌ is a commonly adopted‌​‌ performance measure in Markov​​ Decision Processes (MDPs), it​​​‌ is often too asymptotic.‌ Further incorporating measures of‌​‌ immediate losses leads to​​ the hierarchy of bias​​​‌ optimalities, all the way‌ up to Blackwell optimality.‌​‌ In this paper, we​​ investigate the problem of​​​‌ identifying policies of such‌ optimality orders. To that‌​‌ end, for each order,​​ we construct a learning​​​‌ algorithm with vanishing probability‌ of error. Furthermore, we‌​‌ characterize the class of​​ MDPs for which identification​​​‌ algorithms can stop in‌ finite time. That class‌​‌ corresponds to the MDPs​​ with a unique Bellman​​​‌ optimal policy, and does‌ not depend on the‌​‌ optimality order considered. Lastly,​​ we provide a tractable​​​‌ stopping rule that when‌ coupled to our learning‌​‌ algorithm triggers in finite​​ time whenever it is​​​‌ possible to do so.‌

7.2 Bandits and RL‌​‌ under Real-life constraints

Constrained​​ Pareto Set Identification with​​​‌ Bandit Feedback, 35‌

In this paper, we‌​‌ address the problem of​​ identifying the Pareto Set​​​‌ under feasibility constraints in‌ a multivariate bandit setting.‌​‌ Specifically, given a K-armed​​ bandit with unknown means​​​‌ in ${ℝ}^d$ ,‌ the goal is to‌​‌ identify the set of​​ arms whose mean is​​​‌ not uniformly worse than‌ that of another arm‌​‌ (i.e., not smaller for​​ all objectives), while satisfying​​​‌ some known set of‌ linear constraints, expressing, for‌​‌ example, some minimal performance​​ on each objective. Our​​​‌ focus lies in fixed-confidence‌ identification, for which we‌​‌ introduce an algorithm that​​ significantly outperforms racing-like algorithms​​​‌ and the intuitive two-stage‌ approach that first identifies‌​‌ feasible arms and then​​ their Pareto Set. We​​​‌ further prove an information-theoretic‌ lower bound on the‌​‌ sample complexity of any​​ algorithm for constrained Pareto​​​‌ Set identification, showing that‌ the sample complexity of‌​‌ our approach is near-optimal.​​ Our theoretical results are​​​‌ supported by an extensive‌ empirical evaluation on a‌​‌ series of benchmarks.

Bandit​​ Pareto Set Identification in​​​‌ a Multi-Output Linear Model‌, 34

We study‌​‌ the Pareto Set Identification​​ (PSI) problem in a​​​‌ structured multi-output linear bandit‌ model. In this setting,‌​‌ each arm is associated​​ a feature vector belonging​​​‌ to ${ℝ}^h$ ,‌ and its mean vector‌​‌ in ${ℝ}^d$ linearly​​ depends on this feature​​​‌ vector through a common‌ unknown matrix $\Theta \in ‌‌{ℝ}^{h\times \mathrm{d}}$ . The goal is​​​‌ to identify the set‌ of non-dominated arms by‌​‌ adaptively collecting samples from​​ the arms. We introduce​​​‌ and analyze the first‌ optimal design-based algorithms for‌​‌ PSI, providing nearly optimal​​ guarantees in both the​​​‌ fixed-budget and the fixed-confidence‌ settings. Notably, we show‌​‌ that the difficulty of​​ these tasks mainly depends​​​‌ on the sub-optimality gaps‌ of h arms only.‌​‌ Our theoretical results are​​ supported by an extensive​​​‌ benchmark on synthetic and‌ real-world datasets.

Kriging and‌​‌ Gaussian Process Interpolation for​​​‌ Georeferenced Data Augmentation,​ 51

Data augmentation is​‌ a crucial step in​​ the development of robust​​​‌ supervised learning models, especially​ when dealing with limited​‌ datasets. This study explores​​ interpolation techniques for the​​​‌ augmentation of geo-referenced data,​ with the aim of​‌ predicting the presence of​​ Commelina benghalensis L. in​​​‌ sugarcane plots in La​ Réunion. Given the spatial​‌ nature of the data​​ and the high cost​​​‌ of data collection, we​ evaluated two interpolation approaches:​‌ Gaussian processes (GPs) with​​ different kernels and kriging​​​‌ with various variograms. The​ objectives of this work​‌ are threefold: (i) to​​ identify which interpolation methods​​​‌ offer the best predictive​ performance for various regression​‌ algorithms, (ii) to analyze​​ the evolution of performance​​​‌ as a function of​ the number of observations​‌ added, and (iii) to​​ assess the spatial consistency​​​‌ of augmented datasets. The​ results show that GP-based​‌ methods, in particular with​​ combined kernels (GP-COMB), significantly​​​‌ improve the performance of​ regression algorithms while requiring​‌ less additional data. Although​​ kriging shows slightly lower​​​‌ performance, it is distinguished​ by a more homogeneous​‌ spatial coverage, a potential​​ advantage in certain contexts.​​​‌

Optimal Regret of Bandits​ under Differential Privacy,​‌ 25

As sequential learning​​ algorithms are increasingly applied​​​‌ to real life, ensuring​ data privacy while maintaining​‌ their utilities emerges as​​ a timely question. In​​​‌ this context, regret minimisation​ in stochastic bandits under​‌ $ϵ$-global Differential Privacy​​ (DP) has been widely​​​‌ studied. Unlike bandits without​ DP, there is a​‌ significant gap between the​​ best-known regret lower and​​​‌ upper bound in this​ setting, though they ”match”​‌ in order. Thus, we​​ revisit the regret lower​​​‌ and upper bounds of​ $ϵ$-global DP algorithms​‌ for Bernoulli bandits and​​ improve both. First, we​​​‌ prove a tighter regret​ lower bound involving a​‌ novel information-theoretic quantity characterising​​ the hardness of $\mathrm{ϵ‌}$-global DP in stochastic​ bandits. Our lower bound​‌ strictly improves on the​​ existing ones across all​​​‌ $ϵ$ values. Then, we​ choose two asymptotically optimal​‌ bandit algorithms, i.e. DP-KLUCB​​ and DP-IMED, and propose​​​‌ their DP versions using​ a unified blueprint, i.e.,​‌ (a) running in arm-dependent​​ phases, and (b) adding​​​‌ Laplace noise to achieve​ privacy. For Bernoulli bandits,​‌ we analyse the regrets​​ of these algorithms and​​​‌ show that their regrets​ asymptotically match our lower​‌ bound up to a​​ constant arbitrary close to​​​‌ 1. This refutes the​ conjecture that forgetting past​‌ rewards is necessary to​​ design optimal bandit algorithms​​​‌ under global DP. At​ the core of our​‌ algorithms lies a new​​ concentration inequality for sums​​​‌ of Bernoulli variables under​ Laplace mechanism, which is​‌ a new DP version​​ of the Chernoff bound.​​​‌ This result is universally​ useful as the DP​‌ literature commonly treats the​​ concentrations of Laplace noise​​​‌ and random variables separately,​ while we couple them​‌ to yield a tighter​​ bound.

FLIPHAT: Joint Differential​​​‌ Privacy for High Dimensional​ Sparse Linear Bandits,​‌ 28

High dimensional sparse​​ linear bandits serve as​​​‌ an efficient model for​ sequential decision-making problems (e.g.​‌ personalized medicine), where high​​ dimensional features (e.g. genomic​​ data) on the users​​​‌ are available, but only‌ a small subset of‌​‌ them are relevant. Motivated​​ by data privacy concerns​​​‌ in these applications, we‌ study the joint differentially‌​‌ private high dimensional sparse​​ linear bandits, where both​​​‌ rewards and contexts are‌ considered as private data.‌​‌ First, to quantify the​​ cost of privacy, we​​​‌ derive a lower bound‌ on the regret achievable‌​‌ in this setting. To​​ further address the problem,​​​‌ we design a computationally‌ efficient bandit algorithm, F‌​‌orgetfuLIterative​​ Private HArd​​​‌ Thresholding (FLIPHAT). Along‌ with doubling of episodes‌​‌ and episodic forgetting, FLIPHAT​​ deploys a variant of​​​‌ Noisy Iterative Hard Thresholding‌ (N-IHT) algorithm as a‌​‌ sparse linear regression oracle​​ to ensure both privacy​​​‌ and regret-optimality. We show‌ that FLIPHAT achieves optimal‌​‌ regret up to logarithmic​​ factors. We analyze the​​​‌ regret by providing a‌ novel refined analysis of‌​‌ the estimation error of​​ N-IHT, which is of​​​‌ parallel interest.

Stochastic Online‌ Instrumental Variable Regression: Regrets‌​‌ for Endogeneity and Bandit​​ Feedback, 31

The​​​‌ independence of noise and‌ covariates is a standard‌​‌ assumption in online linear​​ regression with unbounded noise​​​‌ and linear bandit literature.‌ This assumption and the‌​‌ following analysis are invalid​​ in the case of​​​‌ endogeneity, i.e., when the‌ noise and covariates are‌​‌ correlated. In this paper,​​ we study the online​​​‌ setting of Instrumental Variable‌ (IV) regression, which is‌​‌ widely used in economics​​ to identify the underlying​​​‌ model from an endogenous‌ dataset. Specifically, we upper‌​‌ bound the identification and​​ oracle regrets of the​​​‌ popular Two-Stage Least Squares‌ (2SLS) approach to IV‌​‌ regression but in the​​ online setting. Our analysis​​​‌ shows that Online 2SLS‌ (O2SLS) achieves $𝒪(‌‌d^2{log}^{\mathrm{2}}T)$ identification and​​​‌ $𝒪(\gamma \sqrt{\mathrm{d‌}TlogT})‌‌$ oracle regret after $\mathrm{T}$ interactions, where $d$ is​​​‌ the dimension of covariates‌ and $\gamma$ is the‌​‌ bias due to endogeneity.​​ Then, we leverage O2SLS​​​‌ as an oracle to‌ design OFUL-IV, a linear‌​‌ bandit algorithm. OFUL-IV can​​ tackle endogeneity and achieves​​​‌ $𝒪\left(d\sqrt{\mathrm{T‌}}logT\right)$ regret.‌​‌ For different datasets with​​ endogeneity, we experimentally show​​​‌ efficiencies of O2SLS and‌ OFUL-IV.

Unifying (Federated) (Private)‌​‌ High-Dimensional Bandits via ADMM​​, 45

We study​​​‌ all possible variants of‌ the high dimensional stochastic‌​‌ linear contextual bandit problem​​ in federated and private​​​‌ settings. We propose a‌ unifying algorithm design and‌​‌ analysis framework built on​​ ADMM. Our method achieves​​​‌ existing state-of-the art guarantees‌ in either setting for‌​‌ the central model. For​​ the federated model, our​​​‌ results are entirely new‌ and near-optimal in either‌​‌ setting. We also establish​​ a novel lower bound​​​‌ on privacy-utility trade-off for‌ the federated model in‌​‌ the private setting and​​ demonstrate on suitable numerical​​​‌ experiments for all problem‌ variants.

The Confusing Instance‌​‌ Principle for Online Linear​​ Quadratic Control, 19​​​‌

We revisit the problem‌ of controlling linear systems‌​‌ with quadratic costunder unknown​​ dynamics with model-based reinforcement​​​‌ learning. Traditional methods like‌ Optimism in the Face‌​‌ of Uncertainty and Thompson​​​‌ Sampling, rooted in multi-armed​ bandits (MABs), face practical​‌ limitations. In contrast, we​​ propose an alternative based​​​‌ on the Confusing Instance​ (CI) principle, which underpins​‌ regret lower bounds in​​ MABsand discrete Markov Decision​​​‌ Processes (MDPs) and is​ central to the Minimum​‌ Empirical Divergence (MED) family​​ of algorithms, known fortheir​​​‌ asymptotic optimality in various​ settings. By leveraging the​‌ structure of LQR policies​​ along with sensitivity and​​​‌ stability analysis, we develop​ MED-LQ. This novel control​‌ strategy extends the principles​​ of CI and MED​​​‌ beyond small-scale settings. Our​ benchmarks on a comprehensive​‌ control suite demonstrate that​​ MED-LQ achieves competitive performance​​​‌ in various scenarios while​ highlighting its potential for​‌ broader applications in large-scale​​ MDPs.

7.3 Bandits and​​​‌ RL for real-life: Deep​ RL and Applications

Breiman​‌ meets Bellman: Non-Greedy Decision​​ Trees with MDPs,​​​‌ 32

In supervised learning,​ decision trees are valued​‌ for their inter-pretability and​​ performance. While greedy decision​​​‌ tree algorithms like CART​ remain widely used due​‌ to their computational efficiency,they​​ often produce sub-optimal solutions​​​‌ with respect to a​ regularized training loss. Conversely,​‌ optimal decision tree method​​ scan find better solutions​​​‌ but are computationally intensive​ and typically limited to​‌ shallow trees or binary​​ features. We present Dynamic​​​‌ Programming Decision Trees (DPDT),​ a framework that bridges​‌ the gap between greedy​​ and optimal approaches. DPDT​​​‌ relies on a Markov​ Decision Process formulation combined​‌ with heuristic split generation​​ to construct near-optimal decision​​​‌ trees with significantly reduced​ computational complexity. Our approach​‌ dynamically limits the set​​ of admissible splits at​​​‌ each node while directly​ optimizing the tree regularized​‌ training loss. Theoretical analysis​​ demonstrates that DPDT can​​​‌ minimize regularized training losses​ at least as well​‌ as CART. Our empirical​​ study shows on multiple​​​‌ datasets that DPDT achieves​ near-optimal loss with orders​‌ of magnitude fewer operations​​ than existing optimal solvers.​​​‌ More importantly, extensive benchmarking​ suggests statistically significant improvements​‌ of DPDT over both​​ CART and optimal decision​​​‌ trees in terms of​ generalization to unseen data.​‌ We demonstrate DPDT practicality​​ through applications to boosting,​​​‌ where it consistently outperforms​ baselines. Our framework provides​‌ a promising direction for​​ developing efficient, near-optimal decision​​​‌ tree algorithms that scale​ to practical applications.

How​‌ Hard is it to​​ Confuse a World Model?​​​‌, 53

In reinforcement​ learning (RL) theory, the​‌ concept of most confusing​​ instances is central to​​​‌ establishing regret lower bounds,​ that is, the minimal​‌ exploration needed to solve​​ a problem. Given a​​​‌ reference model and its​ optimal policy, a most​‌ confusing instance is the​​ statistically closest alternative model​​​‌ that makes a suboptimal​ policy optimal. While this​‌ concept is well-studied in​​ multi-armed bandits and ergodic​​​‌ tabular Markov decision processes,​ constructing such instances remains​‌ an open question in​​ the general case. In​​​‌ this paper, we formalize​ this problem for neural​‌ network world models as​​ a constrained optimization: finding​​​‌ a modified model that​ is statistically close to​‌ the reference one, while​​ producing divergent performance between​​​‌ optimal and suboptimal policies.​ We propose an adversarial​‌ training procedure to solve​​ this problem and conduct​​ an empirical study across​​​‌ world models of varying‌ quality. Our results suggest‌​‌ that the degree of​​ achievable confusion correlates with​​​‌ uncertainty in the approximate‌ model, which may inform‌​‌ theoretically-grounded exploration strategies for​​ deep model-based RL.

Hierarchical​​​‌ Subspaces of Policies for‌ Continual Offline Reinforcement Learning‌​‌, 43

We consider​​ a Continual Reinforcement Learning​​​‌ setup, where a learning‌ agent must continuously adapt‌​‌ to new tasks while​​ retaining previously acquired skill​​​‌ sets, with a focus‌ on the challenge of‌​‌ avoiding forgetting past gathered​​ knowledge and ensuring scalability​​​‌ with the growing number‌ of tasks. Such issues‌​‌ prevail in autonomous robotics​​ and video game simulations,​​​‌ notably for navigation tasks‌ prone to topological or‌​‌ kinematic changes. To address​​ these issues, we introduce​​​‌ HiSPO, a novel hierarchical‌ framework designed specifically for‌​‌ continual learning in navigation​​ settings from offline data.​​​‌ Our method leverages distinct‌ policy subspaces of neural‌​‌ networks to enable flexible​​ and efficient adaptation to​​​‌ new tasks while preserving‌ existing knowledge. We demonstrate,‌​‌ through a careful experimental​​ study, the effectiveness of​​​‌ our method in both‌ classical MuJoCo maze environments‌​‌ and complex video game-like​​ navigation simulations, showcasing competitive​​​‌ performances and satisfying adaptability‌ with respect to classical‌​‌ continual learning metrics, in​​ particular regarding the memory​​​‌ usage and efficiency.

A‌ Continual Offline Reinforcement Learning‌​‌ Benchmark for Navigation Tasks​​, 42

Autonomous agents​​​‌ operating in domains such‌ as robotics or video‌​‌ game simulations must adapt​​ to changing tasks without​​​‌ forgetting about the previous‌ ones. This process called‌​‌ Continual Reinforcement Learning poses​​ non-trivial difficulties, from preventing​​​‌ catastrophic forgetting to ensuring‌ the scalability of the‌​‌ approaches considered. Building on​​ recent advances, we introduce​​​‌ a benchmark providing a‌ suite of video-game navigation‌​‌ scenarios, thus filling a​​ gap in the literature​​​‌ and capturing key challenges‌ : catastrophic forgetting, task‌​‌ adaptation, and memory efficiency.​​ We define a set​​​‌ of various tasks and‌ datasets, evaluation protocols, and‌​‌ metrics to assess the​​ performance of algorithms, including​​​‌ state-of-the-art baselines. Our benchmark‌ is designed not only‌​‌ to foster reproducible research​​ and to accelerate progress​​​‌ in continual reinforcement learning‌ for gaming, but also‌​‌ to provide a reproducible​​ framework for production pipelines​​​‌ – helping practitioners to‌ identify and to apply‌​‌ effective approaches.

StaQ it!​​ Growing neural networks for​​​‌ Policy Mirror Descent,‌ 44

In Reinforcement Learning‌​‌ (RL), regularization has emerged​​ as a popular tool​​​‌ both in theory and‌ practice, typically based either‌​‌ on an entropy bonus​​ or a Kullback-Leibler divergence​​​‌ that constrains successive policies.‌ In practice, these approaches‌​‌ have been shown to​​ improve exploration, robustness and​​​‌ stability, giving rise to‌ popular Deep RL algorithms‌​‌ such as SAC and​​ TRPO. Policy Mirror Descent​​​‌ (PMD) is a theoretical‌ framework that solves this‌​‌ general regularized policy optimization​​ problem, however the closed-form​​​‌ solution involves the sum‌ of all past Q-functions,‌​‌ which is intractable in​​ practice. We propose and​​​‌ analyze PMD-like algorithms that‌ only keep the last‌​‌ $M$ Q-functions in memory,​​ and show that for​​​‌ finite and large enough‌ $M$, a convergent‌​‌ algorithm can be derived,​​​‌ introducing no error in​ the policy update, unlike​‌ prior deep RL PMD​​ implementations. StaQ, the resulting​​​‌ algorithm, enjoys strong theoretical​ guarantees and is competitive​‌ with deep RL baselines,​​ while exhibiting less performance​​​‌ oscillation, paving the way​ for fully stable deep​‌ RL algorithms and providing​​ a testbed for experimentation​​​‌ with Policy Mirror Descent.​

PB${}^2$: Preference​‌ Space Exploration via Population-Based​​ Methods in Preference-Based Reinforcement​​​‌ Learning, 41

Preference-based​ reinforcement learning (PbRL) has​‌ emerged as a promising​​ approach for learning behaviors​​​‌ from human feedback without​ predefined reward functions. However,​‌ current PbRL methods face​​ a critical challenge in​​​‌ effectively exploring the preference​ space, often converging prematurely​‌ to suboptimal policies that​​ satisfy only a narrow​​​‌ subset of human preferences.​ In this work, we​‌ identify and address this​​ preference exploration problem through​​​‌ population-based methods. We demonstrate​ that maintaining a diverse​‌ population of agents enables​​ more comprehensive exploration of​​​‌ the preference landscape compared​ to single-agent approaches. Crucially,​‌ this diversity improves reward​​ model learning by generating​​​‌ preference queries with clearly​ distinguishable behaviors, a key​‌ factor in real-world scenarios​​ where humans must easily​​​‌ differentiate between options to​ provide meaningful feedback. Our​‌ experiments reveal that current​​ methods may fail by​​​‌ getting stuck in local​ optima, requiring excessive feedback,​‌ or degrading significantly when​​ human evaluators make errors​​​‌ on similar trajectories, a​ realistic scenario often overlooked​‌ by methods relying on​​ perfect oracle teachers. Our​​​‌ population-based approach demonstrates robust​ performance when teachers mislabel​‌ similar trajectory segments and​​ shows significantly enhanced preference​​​‌ exploration capabilities, particularly in​ environments with complex reward​‌ landscapes

Lagrangian-based Equilibrium Propagation:​​ generalisation to arbitrary boundary​​​‌ conditions & equivalence with​ Hamiltonian Echo Learning,​‌ 52

Equilibrium Propagation (EP)​​ is a learning algorithm​​​‌ for training Energy-based Models​ (EBMs) on static inputs​‌ which leverages the variational​​ description of their fixed​​​‌ points. Extending EP to​ time-varying inputs is a​‌ challenging problem, as the​​ variational description must apply​​​‌ to the entire system​ trajectory rather than just​‌ fixed points, and careful​​ consideration of boundary conditions​​​‌ becomes essential. In this​ work, we present Generalized​‌ Lagrangian Equilibrium Propagation (GLEP),​​ which extends the variational​​​‌ formulation of EP to​ time-varying inputs. We demonstrate​‌ that GLEP yields different​​ learning algorithms depending on​​​‌ the boundary conditions of​ the system, many of​‌ which are impractical for​​ implementation. We then show​​​‌ that Hamiltonian Echo Learning​ (HEL) – which includes​‌ the recently proposed Recurrent​​ HEL (RHEL) and the​​​‌ earlier known Hamiltonian Echo​ Backpropagation (HEB) algorithms –​‌ can be derived as​​ a special case of​​​‌ GLEP. Notably, HEL is​ the only instance of​‌ GLEP we found that​​ inherits the properties that​​​‌ make EP a desirable​ alternative to backpropagation for​‌ hardware implementations: it operates​​ in a ”forward-only” manner​​​‌ (i.e. using the same​ system for both inference​‌ and learning), it scales​​ efficiently (requiring only two​​​‌ or more passes through​ the system regardless of​‌ model size), and enables​​ local learning.

Studying Exploration​​​‌ in RL: An Optimal​ Transport Analysis of Occupancy​‌ Measure Trajectories, 18​​

The rising successes of​​ RL are propelled by​​​‌ combining smart algorithmic strategies‌ and deep architectures to‌​‌ optimize the distribution of​​ returns and visitations over​​​‌ the state-action space. A‌ quantitative framework to compare‌​‌ the learning processes of​​ these eclectic RL algorithms​​​‌ is currently absent but‌ desired in practice. We‌​‌ address this gap by​​ representing the learning process​​​‌ of an RL algorithm‌ as a sequence of‌​‌ policies generated during training,​​ and then studying the​​​‌ policy trajectory induced in‌ the manifold of state-action‌​‌ occupancy measures. Using an​​ optimal transport-based metric, we​​​‌ measure the length of‌ the paths induced by‌​‌ the policy sequence yielded​​ by an RL algorithm​​​‌ between an initial policy‌ and a final optimal‌​‌ policy. Hence, we first​​ define the Effort of​​​‌ Sequential Learning (ESL). ESL‌ quantifies the relative distance‌​‌ that an RL algorithm​​ travels compared to the​​​‌ shortest path from the‌ initial to the optimal‌​‌ policy. Furthermore, we connect​​ the dynamics of policies​​​‌ in the occupancy measure‌ space and regret (another‌​‌ metric to understand the​​ suboptimality of an RL​​​‌ algorithm), by defining the‌ Optimal Movement Ratio (OMR).‌​‌ OMR assesses the fraction​​ of movements in the​​​‌ occupancy measure space that‌ effectively reduce an analogue‌​‌ of regret. Finally, we​​ derive approximation guarantees to​​​‌ estimate ESL and OMR‌ with a finite number‌​‌ of samples and without​​ access to an optimal​​​‌ policy. Through empirical analyses‌ across various environments and‌​‌ algorithms, we demonstrate that​​ ESL and OMR provide​​​‌ insights into the exploration‌ processes of RL algorithms‌​‌ and the hardness of​​ different tasks in discrete​​​‌ and continuous MDPs.

7.4‌ Others

Improving Diffusion Models‌​‌ for the Traveling Salesman​​ Problem (TSP) by Leveraging​​​‌ the Structure of the‌ Solution Space, 26‌​‌

In this paper we​​ show how leveraging the​​​‌ structure of the solution‌ space of the Traveling‌​‌ Salesman Problem (TSP) can​​ lead to a dramatic​​​‌ improvement of the performance‌ of state of the‌​‌ art diffusion based neural​​ solvers. Building on recent​​​‌ approaches of DIFUSCO and‌ T2TCO which pipeline a‌​‌ diffusion-based solution generation with​​ a local search procedure,​​​‌ we propose IDEQ (constrained‌ Inverse Diffusion and EQuivalence‌​‌ class-based training of diffusion​​ models for combinatorial optimization).​​​‌ IDEQ improves the quality‌ of the solutions by‌​‌ leveraging the constrained structure​​ of the TSP state​​​‌ space. Indeed, the solution‌ space consists of locally‌​‌ optimal Hamiltonian tours which​​ is a much smaller​​​‌ space than the space‌ of adjacency matrices used‌​‌ in previous works. Also,​​ the local search procedure​​​‌ defines an equivalence class‌ of Hamiltonian tours: all‌​‌ elements of this equivalence​​ class reach the same​​​‌ local optimum after the‌ application of the local‌​‌ search. This should be​​ aligned with the supervised​​​‌ training objective of the‌ diffusion. IDEQ addresses these‌​‌ two points. Our experiments​​ show that IDEQ achieves​​​‌ 0.3% to 0.4% optimality‌ gap on TSP instances‌​‌ made of 500 cities,​​ and 0.5% to 0.6%​​​‌ optimality gap on TSP‌ instances with 1000 cities.‌​‌ This sets a new​​ SOTA for neural based​​​‌ methods solving the TSP.‌ IDEQ also performs well‌​‌ on the instances of​​​‌ the TSPlib, a reference​ benchmark in the TSP​‌ community, outside of the​​ training distribution, with optimality​​​‌ gaps ranging from 0.9​ to 1.1 %.

Yara:​‌ An Ocean Virtual Environment​​ for Research and Development​​​‌ of Autonomous Sailing Robots​ and Other Unmanned Surface​‌ Vessels, 20

Overall,​​ a big challenge in​​​‌ building a sailboat USV​ relies on the development​‌ of an autonomous system​​ for guidance, navigation, and​​​‌ control (GNC) because both​ sail and rudder angle​‌ must be cooperatively adjusted​​ to correct the navigation​​​‌ direction -traditional propelled boats​ can be more easily​‌ controlled with a straightforward​​ control task to set​​​‌ the rudder angle. Moreover,​ sailing upwind requires special​‌ maneuvers to reach a​​ given target in that​​​‌ unfeasible direction. Reinforcement learning​ emerges as a promising​‌ technique for building autonomous​​ GNCs for sailing robots,​​​‌ but training the neural​ network with a real​‌ sailboat is impractical due​​ to long periods of​​​‌ training and safety reasons.​ Even traditional control-based approaches​‌ are mainly tested in​​ simulated environments due to​​​‌ the difficulties in building​ and operating a real​‌ sailboat. The issue that​​ arises is the fidelity​​​‌ of these simulated environments.​ In this context, we​‌ propose Yara, an oceanic​​ virtual environment with a​​​‌ reliable physics simulation for​ developing, training, and evaluating​‌ autonomous agents to operate​​ digital twins of sailing​​​‌ robots in reinforcement learning​ and other paradigms. An​‌ autonomous sailing robot digital​​ twin is available within​​​‌ the virtual environment, with​ the foil dynamics constructed​‌ based on a real​​ sailing robot. We coupled​​​‌ these foil dynamics in​ Gazebo's physics engine to​‌ compute the lift and​​ drag forces acting on​​​‌ the sail, rudder, and​ keel. The simulated world​‌ feeds sensors such as​​ cameras, wind sensors, and​​​‌ GPS. The Robot Operating​ System communicates these sensors'​‌ data through topics, facilitating​​ users' implementation and testing​​​‌ of new GNC solutions.​ Yara provides a reliable​‌ solution for foil dynamic​​ simulated physics that achieves​​​‌ a simulation speedup of​ 300 times on an​‌ i7 laptop with 8​​ GB of RAM, powered​​​‌ by a Nvidia RTX​ 3060 and running Ubuntu​‌ 20.04. With this speedup,​​ it is possible to​​​‌ complete a million time​ steps of deep reinforcement​‌ learning training in approximately​​ eight hours. Evaluation scenarios​​​‌ were presented to highlight​ specific features of the​‌ simulator, like the maneuverability​​ of the sailing robot​​​‌ digital twin and applications​ to train, evaluate, and​‌ compare reinforcement learning agents​​ and other control solutions.​​​‌

Efficient Active Imitation Learning​ with Random Network Distillation​‌, 27

Developing agents​​ for complex and underspecified​​​‌ tasks, where no clear​ objective exists, remains challenging​‌ but offers many opportunities.​​ This is especially true​​​‌ in video games, where​ simulated players (bots) need​‌ to play realistically, and​​ there is no clear​​​‌ reward to evaluate them.​ While imitation learning has​‌ shown promise in such​​ domains, these methods often​​​‌ fail when agents encounter​ out-of-distribution scenarios during deployment.​‌ Expanding the training dataset​​ is a common solution,​​​‌ but it becomes impractical​ or costly when relying​‌ on human demonstrations. This​​ article addresses active imitation​​ learning, aiming to trigger​​​‌ expert intervention only when‌ necessary, reducing the need‌​‌ for constant expert input​​ along training. We introduce​​​‌ Random Network Distillation DAgger‌ (RND-DAgger), a new active‌​‌ imitation learning method that​​ limits expert querying by​​​‌ using a learned state-based‌ out-of-distribution measure to trigger‌​‌ interventions. This approach avoids​​ frequent expert-agent action comparisons,​​​‌ thus making the expert‌ intervene only when it‌​‌ is useful. We evaluate​​ RND-DAgger against traditional imitation​​​‌ learning and other active‌ approaches in 3D video‌​‌ games (racing and third-person​​ navigation) and in a​​​‌ robotic locomotion task and‌ show that RND-DAgger surpasses‌​‌ previous methods by reducing​​ expert queries. Link

Exploring​​​‌ Flow-Lenia Universes with a‌ Curiosity-driven AI Scientist: Discovering‌​‌ Diverse Ecosystem Dynamics,​​ 37

We present a​​​‌ method for the automated‌ discovery of system-level dynamics‌​‌ in Flow-Lenia-a continuous cellular​​ automaton (CA) with mass​​​‌ conservation and parameter localization-using‌ a curiosity-driven AI scientist.‌​‌ This method aims to​​ uncover processes leading to​​​‌ self-organization of evolutionary and‌ ecosystemic dynamics in CAs.‌​‌ We build on previous​​ work which uses diversity​​​‌ search algorithms in Lenia‌ to find self-organized individual‌​‌ patterns, and extend it​​ to large environments are​​​‌ that support distinct interacting‌ patterns. We adapt Intrinsically‌​‌ Motivated Goal Exploration Processes​​ (IMGEPs) to drive exploration​​​‌ of diverse Flow-Lenia environments‌ using simulation-wide metrics, such‌​‌ as evolutionary activity, compression-based​​ complexity, and multi-scale entropy.​​​‌ We test our method‌ in two experiments, showcasing‌​‌ its ability to illuminate​​ significantly more diverse dynamics​​​‌ compared to random search.‌ We show qualitative results‌​‌ illustrating how ecosystemic simulations​​ enable self-organization of complex​​​‌ collective behaviors not captured‌ by previous individual pattern‌​‌ search and analysis. We​​ complement automated discovery with​​​‌ an interactive exploration tool,‌ creating an effective human-AI‌​‌ collaborative workflow for scientific​​ investigation. Though demonstrated specifically​​​‌ with Flow-Lenia, this methodology‌ provides a framework potentially‌​‌ applicable to other parameterizable​​ complex systems where understanding​​​‌ emergent collective properties is‌ of interest.

8.1‌ Bilateral contracts with industry‌​‌

• contract with Ubisoft, 2023–2026:​​ PI: Odalric-Ambrym Maillard .​​​‌

  This contract is related‌ to Anthony Kobanda s‌​‌ Ph.D. “Continual Reinforcement Learning​​ with changing environments: Application​​​‌ to Video Games”

• contract‌ with Lilly Group, 2023–2026,‌​‌ PI: Philippe Preux .​​

  This contract is related​​​‌ to Mickael Basson 's‌ Ph.D. “Reinforcement learning to‌​‌ solve combinatorial optimization problems”.​​

• contract with Saint-Gobain Research,​​​‌ 2023–2026, PI: Philippe Preux‌ .

  This contract is‌​‌ related to Yann Berthelot​​ 's Ph.D. “Reinforcement learning​​​‌ for advanced control of‌ industrial processus”.

9.1‌​‌ International initiatives

9.2 International​​ research visitors

9.3 National initiatives

10.1​​ Promoting scientific activities

10.2 Teaching​ - Supervision - Juries​‌ - Educational and pedagogical​​ outreach

• Juliette Achddou :​​​‌ “Introduction au Deep Learning”,​ M2 Informatique et Statistiques,​‌ Polytech' Lille.
• Juliette Achddou​​ : “Machine Learning”, M1​​​‌ Informatique et Statistiques, Polytech'​ Lille.
• Juliette Achddou :​‌ “Algorithmique numérique pour l'optimisation”,​​ M1 Informatique et Statistiques,​​​‌ Polytech' Lille.
• Juliette Achddou​ : “Classification supervisée”, L3​‌ Informatique et Statistiques, Polytech'​​ Lille.
• Juliette Achddou :​​​‌ “Régression Linéaire”, L3 Informatique​ et Statistiques, Polytech' Lille.​‌
• Juliette Achddou : “Data​​ Mining”, L3 Informatique et​​​‌ Statistiques, Polytech' Lille.
• Juliette​ Achddou : “Algèbre Linéaire​‌ Numérique”, L3 Informatique et​​ Statistiques, Polytech' Lille.
• Juliette​​​‌ Achddou : “Introduction aux​ Environnements Virtuels”, M1 Informatique​‌ et Statistiques, Polytech' Lille.​​
• Riadh Akrour : “Option​​​‌ Machine Learning”, L3 Informatique,​ Université de Lille.
• Riadh​‌ Akrour : “Sequential Decision​​ Making”, M2 Data Science,​​​‌ Ecole Centrale de Lille.​
• Debabrota Basu : “Sequential​‌ Decision Making”, M2 in​​ Data Science, Centrale Lille​​​‌ and Université de Lille.​
• Debabrota Basu : “Research​‌ Reading Group”, M2 in​​ Data Science, Centrale Lille​​​‌ and Université de Lille.​
• Debabrota Basu : “Advanced​‌ Machine Learning and Decision​​ Making”, Centrale Lille
• Remy​​​‌ Degenne : “Sequential Learning”,​ Master MVA, ENS Paris-Saclay.​‌
• Remy Degenne : “Sequential​​ Learning”, Centrale Lille.
• Emilie​​​‌ Kaufmann : “Statistics 2”,​ M1 Data Science, Ecole​‌ Centrale de Lille.
• Odalric-Ambrym​​ Maillard : “Reinforcement Learning​​​‌ Research Challenges”, Executive Master​ Ecole Polytechnique.
• Odalric-Ambrym Maillard​‌ : “Reinforcement Learning for​​ the Industry”, Inria Academy.​​​‌
• Philippe Preux : “Prise​ de décision séquentielle dans​‌ l'incertain”, M2 in Computer​​ Science, Université de Lille.​​​‌
• Philippe Preux : “Apprentissage​ par renforcement”, M2 in​‌ Computer Science, Université de​​ Lille.
• Philippe Preux :​​​‌ “Science des données II”,​ L3 MIASHS, Université de​‌ Lille.
• Philippe Preux :​​ “Science des données III”,​​​‌ L3 MIASHS, Université de​ Lille.
• Philippe Preux :​‌ “Réseaux de neurones”, L1​​ Maths-Informatique, Université de Lille.​​​‌
• Philippe Preux : “IA​ et apprentissage automatique”, DU​‌ IA & Santé, Université​​ de Lille.
• Adrienne Tuynman​​​‌ : “Cryptographie” (practical sessions),​ M1 in Applied Mathematics,​‌ University of Lille

10.3 Popularization‌​‌

11.1 Major publications

• 1‌​‌ inproceedingsB.Borja Balle​​​‌ and O.-A.Odalric-Ambrym Maillard​. Spectral Learning from​‌ a Single Trajectory under​​ Finite-State Policies.International​​​‌ conference on Machine Learning​Proceedings of the International​‌ conference on Machine Learning​​Sidney, FranceJuly 2017​​​‌HAL
• 2 inproceedingsD.​Dorian Baudry, R.​‌Romain Gautron, E.​​Emilie Kaufmann and O.-A.​​​‌Odalric-Ambrym Maillard. Optimal​ Thompson Sampling strategies for​‌ support-aware CVaR bandits.​​38th International Conference on​​​‌ Machine Learningproceedings of​ machine learning researchVirtual,​‌ United StatesJuly 2021​​HAL
• 3 inproceedingsL.​​​‌Lilian Besson and E.​Emilie Kaufmann. Multi-Player​‌ Bandits Revisited.Algorithmic​​ Learning TheoryMehryar Mohri​​​‌ and Karthik SridharanLanzarote,​ SpainApril 2018HAL​‌
• 4 articleG.Gabriel​​ Dulac-Arnold, L.Ludovic​​​‌ Denoyer, P.Philippe​ Preux and P.Patrick​‌ Gallinari. Sequential approaches​​ for learning datum-wise sparse​​​‌ representations.Machine Learning​891-2October 2012​‌, 87-122HALDOI​​
• 5 inproceedingsY.Yannis​​​‌ Flet-Berliac and P.Philippe​ Preux. Only Relevant​‌ Information Matters: Filtering Out​​ Noisy Samples to Boost​​​‌ RL.IJCAI 2020​ - International Joint Conference​‌ on Artificial IntelligenceYokohama,​​ JapanJuly 2020HAL​​​‌DOI
• 6 inproceedingsA.​Aurélien Garivier and E.​‌Emilie Kaufmann. Optimal​​ Best Arm Identification with​​​‌ Fixed Confidence.29th​ Annual Conference on Learning​‌ Theory (COLT)49JMLR​​ Workshop and Conference Proceedings​​​‌New York, United States​June 2016HAL
• 7​‌ inproceedingsB.Bishwamittra Ghosh​​, D.Debabrota Basu​​​‌ and K. S.Kuldeep​ S. Meel. Justicia:​‌ A Stochastic SAT Approach​​ to Formally Verify Fairness​​​‌.Proceedings of the​ AAAI Conference on Artificial​‌ IntelligenceAAAI Conference on​​ Artificial Intelligence35Proceedings​​​‌ of the AAAI Conference​ on Artificial Intelligence9​‌Virtual, CanadaFebruary 2021​​, 7554-7563HAL
• 8​​​‌ inproceedingsM.Marc Jourdan​, R.Rémy Degenne​‌, D.Dorian Baudry​​, R.Rianne de​​​‌ Heide and E.Emilie​ Kaufmann. Top Two​‌ Algorithms Revisited.NeurIPS​​ 2022 - 36th Conference​​​‌ on Neural Information Processing​ SystemAdvances in Neural​‌ Information Processing SystemsNew​​ Orleans, United StatesNovember​​​‌ 2022HAL
• 9 article​H.Hachem Kadri,​‌ E.Emmanuel Duflos,​​ P.Philippe Preux,​​​‌ S.Stéphane Canu,​ A.Alain Rakotomamonjy and​‌ J.Julien Audiffren.​​ Operator-valued Kernels for Learning​​​‌ from Functional Response Data​.Journal of Machine​‌ Learning Research1720​​2016, 1-54HAL​​​‌
• 10 inproceedingsE.Emilie​ Kaufmann and W. M.​‌Wouter M. Koolen.​​ Monte-Carlo Tree Search by​​​‌ Best Arm Identification.​NIPS 2017 - 31st​‌ Annual Conference on Neural​​ Information Processing SystemsAdvances​​​‌ in Neural Information Processing​ SystemsLong Beach, United​‌ StatesDecember 2017,​​ 1-23HAL
• 11 inproceedings​​​‌E.Emilie Kaufmann,​ P.Pierre Ménard,​‌ O.Omar Darwiche Domingues​​, A.Anders Jonsson​​​‌, E.Edouard Leurent​ and M.Michal Valko​‌. Adaptive reward-free exploration​​.Algorithmic Learning Theory​​​‌Paris, France2021HAL​
• 12 articleO.-A.Odalric-Ambrym​‌ Maillard. Boundary Crossing​​ Probabilities for General Exponential​​​‌ Families.Mathematical Methods​ of Statistics272018​‌HAL
• 13 inproceedingsO.-A.​​Odalric-Ambrym Maillard, H.​​Hippolyte Bourel and M.​​​‌ S.Mohammad Sadegh Talebi‌. Tightening Exploration in‌​‌ Upper Confidence Reinforcement Learning​​.International Conference on​​​‌ Machine LearningVienna, Austria‌July 2020HAL
• 14‌​‌ articleT.Timothée Mathieu​​, R.Riccardo Della​​​‌ Vecchia, A.Alena‌ Shilova, M.Matheus‌​‌ Medeiros Centa, H.​​Hector Kohler, O.-A.​​​‌Odalric-Ambrym Maillard and P.‌Philippe Preux. AdaStop:‌​‌ adaptive statistical testing for​​ sound comparisons of Deep​​​‌ RL agents.Transactions‌ on Machine Learning Research‌​‌ Journal2024HAL
• 15​​ inproceedingsF.Fabien Pesquerel​​​‌ and O.-A.Odalric-Ambrym Maillard‌. IMED-RL: Regret optimal‌​‌ learning of ergodic Markov​​ decision processes.NeurIPS​​​‌ 2022 - Thirty-sixth Conference‌ on Neural Information Processing‌​‌ SystemsThirty-sixth Conference on​​ Neural Information Processing Systems​​​‌New-Orleans, United StatesNovember‌ 2022HAL

11.2 Publications‌​‌ of the year

11.3​‌ Cited publications

• 54 book​​M.M. Puterman.​​​‌ Markov Decision Processes: Discrete​ Stochastic Dynamic Programming.​‌John Wiley & Sons​​1994back to text​​​‌
• 55 unpublishedB.B.​ Recht. A Tour​‌ of Reinforcement Learning: The​​ View from Continuous Control​​​‌.2018, arxiv​ preprint 1806.09460back to​‌ text
• 56 bookR.​​R.S. Sutton and A.​​​‌A. Barto. Reinforcement​ Learning: an Introduction.​‌http://incompleteideas.net/book/the-book-2nd.htmlMIT Press2018​​back to text
• 57​​​‌ bookC.C. Szepesvári​ and T.T. Lattimore​‌. Bandit Algorithms.​​Cambridge University press2019​​​‌back to text