2025‌Activity reportProject-TeamSIERRA‌​‌

2.1 Statement

Machine​​ learning is a recent​​​‌ scientific domain, positioned between‌ applied mathematics, statistics and‌​‌ computer science. Its goals​​ are the optimization, control,​​​‌ and modeling of complex‌ systems from examples. It‌​‌ applies to data from​​ numerous engineering and scientific​​​‌ fields (e.g., vision, bioinformatics,‌ neuroscience, audio processing, text‌​‌ processing, economy, finance, etc.),​​ the ultimate goal being​​​‌ to derive general theories‌ and algorithms allowing advances‌​‌ in each of these​​ domains. Machine learning is​​​‌ characterized by the high‌ quality and quantity of‌​‌ the exchanges between theory,​​ algorithms and applications: interesting​​​‌ theoretical problems almost always‌ emerge from applications, while‌​‌ theoretical analysis allows the​​ understanding of why and​​​‌ when popular or successful‌ algorithms do or do‌​‌ not work, and leads​​ to proposing significant improvements.​​​‌

Our academic positioning is‌ exactly at the intersection‌​‌ between these three aspects—algorithms,​​ theory and applications—and our​​​‌ main research goal is‌ to make the link‌​‌ between theory and algorithms,​​ and between algorithms and​​​‌ high-impact applications in various‌ engineering and scientific fields.‌​‌

6.1 Awards

• Election of​​​‌ Michael Jordan at the​ Chinese Academy of Sciences​‌
• PhD award for Baptiste​​ Goujaud: 2025 PhD award​​​‌ from department of mathematics,​ Ecole Polytechnique.
• PhD​‌ award for Antoine Bambade:​​ 2025 Paul Caseau PhD​​​‌ award (from EDF R&D).​

6.2 Invited talks

• Plenary​‌ talk at ICCOPT 2025​​ for Alexandre d'Aspremont
• Plenary​​​‌ talk at COLT 2025​ for Francis Bach
• Plenary​‌ talk at the France​​ AI summit for Michael​​​‌ Jordan

7.1 Latest software developments​​

7.2‌​‌ Open data

8.1 A PAC-Bayesian​​​‌ Link Between Generalisation and‌ Flat Minima

Modern machine‌​‌ learning usually involves predictors​​ in the overparametrised setting​​​‌ (number of trained parameters‌ greater than dataset size),‌​‌ and their training yield​​ not only good performances​​​‌ on training data, but‌ also good generalisation capacity.‌​‌ This phenomenon challenges many​​ theoretical results, and remains​​​‌ an open problem. To‌ reach a better understanding,‌​‌ in 14 we provide​​ novel generalisation bounds involving​​​‌ gradient terms. To do‌ so, we combine the‌​‌ PAC-Bayes toolbox with Poincaré​​ and Log-Sobolev inequalities, avoiding​​​‌ an explicit dependency on‌ dimension of the predictor‌​‌ space. Our results highlight​​ the positive influence of​​​‌ flat minima (being minima‌ with a neighbourhood nearly‌​‌ minimising the learning problem​​ as well) on generalisation​​​‌ performances, involving directly the‌ benefits of the optimisation‌​‌ phase.

8.2 Heavy-Tailed Diffusion​​ with Denoising Lévy Probabilistic​​​‌ Models

Exploring noise distributions‌ beyond Gaussian in diffusion‌​‌ models remains an open​​ challenge. While Gaussian-based models​​​‌ succeed within a unified‌ SDE framework, recent studies‌​‌ suggest that heavy-tailed noise​​ distributions, like α-stable distributions,​​​‌ may better handle mode‌ collapse and effectively manage‌​‌ datasets exhibiting class imbalance,​​ heavy tails, or prominent​​​‌ outliers. Recently, Yoon et‌ al. (NeurIPS 2023), presented‌​‌ the Lévy-Itô model (LIM),​​ directly extending the SDE-based​​​‌ framework to a class‌ of heavy-tailed SDEs, where‌​‌ the injected noise followed​​ an α-stable distribution, a​​​‌ rich class of heavy-tailed‌ distributions. However, the LIM‌​‌ framework relies on highly​​ involved mathematical techniques with​​​‌ limited flexibility, potentially hindering‌ broader adoption and further‌​‌ development. In 30,​​ instead of starting from​​​‌ the SDE formulation, we‌ extend the denoising diffusion‌​‌ probabilistic model (DDPM) by​​ replacing the Gaussian noise​​​‌ with α-stable noise. By‌ using only elementary proof‌​‌ techniques, the proposed approach,​​ Denoising Lévy Probabilistic Models​​​‌ (DLPM), boils down to‌ vanilla DDPM with minor‌​‌ modifications. As opposed to​​ the Gaussian case, DLPM​​​‌ and LIM yield different‌ training algorithms and different‌​‌ backward processes, leading to​​ distinct sampling algorithms. These​​​‌ fundamental differences translate favorably‌ for DLPM as compared‌​‌ to LIM: our experiments​​ show improvements in coverage​​​‌ of data distribution tails,‌ better robustness to unbalanced‌​‌ datasets, and improved computation​​​‌ times requiring smaller number​ of backward steps.

8.3​‌ Don't Be Greedy, Just​​ Relax! Pruning LLMs via​​​‌ Frank-Wolfe

Pruning is a​ common technique to reduce​‌ the compute and storage​​ requirements of Neural Networks.​​​‌ While conventional approaches typically​ retrain the model to​‌ recover pruning-induced performance degradation,​​ state-of-the-art Large Language Model​​​‌ (LLM) pruning methods operate​ layer-wise, minimizing the per-layer​‌ pruning error on a​​ small calibration dataset to​​​‌ avoid full retraining, which​ is considered computationally prohibitive​‌ for LLMs. However, finding​​ the optimal pruning mask​​​‌ is a hard combinatorial​ problem and solving it​‌ to optimality is intractable.​​ Existing methods hence rely​​​‌ on greedy heuristics that​ ignore the weight interactions​‌ in the pruning objective.​​ In 74, we​​​‌ instead consider the convex​ relaxation of these combinatorial​‌ constraints and solve the​​ resulting problem using the​​​‌ Frank-Wolfe (FW) algorithm. Our​ method drastically reduces the​‌ per-layer pruning error, outperforms​​ strong baselines on state-of-the-art​​​‌ GPT architectures, and remains​ memory-efficient. We provide theoretical​‌ justification by showing that,​​ combined with the convergence​​​‌ guarantees of the FW​ algorithm, we obtain an​‌ approximate solution to the​​ original combinatorial problem upon​​​‌ rounding the relaxed solution​ to integrality.

8.4 Algorithm-​‌ and Data-Dependent Generalization Bounds​​ for Score-Based Generative Models​​​‌

Score-based generative models (SGMs)​ have emerged as one​‌ of the most popular​​ classes of generative models.​​​‌ A substantial body of​ work now exists on​‌ the analysis of SGMs,​​ focusing either on discretization​​​‌ aspects or on their​ statistical performance. In the​‌ latter case, bounds have​​ been derived, under various​​​‌ metrics, between the true​ data distribution and the​‌ distribution induced by the​​ SGM, often demonstrating polynomial​​​‌ convergence rates with respect​ to the number of​‌ training samples. However, these​​ approaches adopt a largely​​​‌ approximation theory viewpoint, which​ tends to be overly​‌ pessimistic and relatively coarse.​​ In particular, they fail​​​‌ to fully explain the​ empirical success of SGMs​‌ or capture the role​​ of the optimization algorithm​​​‌ used in practice to​ train the score network.​‌ To support this observation,​​ in 10, we​​​‌ first present simple experiments​ illustrating the concrete impact​‌ of optimization hyperparameters on​​ the generalization ability of​​​‌ the generated distribution. Then,​ this paper aims to​‌ bridge this theoretical gap​​ by providing the first​​​‌ algorithmic- and data-dependent generalization​ analysis for SGMs. In​‌ particular, we establish bounds​​ that explicitly account for​​​‌ the optimization dynamics of​ the learning algorithm, offering​‌ new insights into the​​ generalization behavior of SGMs.​​​‌ Our theoretical findings are​ supported by empirical results​‌ on several datasets.

8.5​​ The surprising agreement between​​​‌ convex optimization theory and​ learning-rate scheduling for large​‌ model training

In 28​​, we show that​​​‌ learning-rate schedules for large​ model training behave surprisingly​‌ similar to a performance​​ bound from non-smooth convex​​​‌ optimization theory. We provide​ a bound for the​‌ constant schedule with linear​​ cooldown; in particular, the​​​‌ practical benefit of cooldown​ is reflected in the​‌ bound due to the​​ absence of logarithmic terms.​​​‌ Further, we show that​ this surprisingly close match​‌ between optimization theory and​​ practice can be exploited​​ for learning-rate tuning: we​​​‌ achieve noticeable improvements for‌ training 124M and 210M‌​‌ Llama-type models by (i)​​ extending the schedule for​​​‌ continued training with optimal‌ learning-rate, and (ii) transferring‌​‌ the optimal learning-rate across​​ schedules.

8.6 Augmented Lagrangian​​​‌ methods for infeasible convex‌ optimization problems and diverging‌​‌ proximal-point algorithms

In 2​​, we investigate the​​​‌ convergence behavior of augmented‌ Lagrangian methods (ALMs) when‌​‌ applied to convex optimization​​ problems that may be​​​‌ infeasible. ALMs are a‌ popular class of algorithms‌​‌ for solving constrained optimization​​ problems. We establish progressively​​​‌ stronger convergence results, ranging‌ from basic sequence convergence‌​‌ to precise convergence rates,​​ under a hierarchy of​​​‌ assumptions.

In particular, we‌ demonstrate that, under mild‌​‌ assumptions, the sequences of​​ iterates generated by ALMs​​​‌ converge to solutions of‌ the “closest feasible problem”.‌​‌ This study leverages the​​ classical relationship between ALMs​​​‌ and the proximal-point algorithm‌ applied to the dual‌​‌ problem. A key technical​​ contribution is a set​​​‌ of concise results on‌ the behavior of the‌​‌ proximal-point algorithm when applied​​ to functions that may​​​‌ not have minimizers. These‌ results pertain to its‌​‌ convergence in terms of​​ its subgradients and of​​​‌ the values of the‌ convex conjugate.

8.7 A‌​‌ constructive approach to strengthen​​ algebraic descriptions of function​​​‌ and operator classes

It‌ is well known that‌​‌ functions (resp. operators) satisfying​​ a property $p$ on​​​‌ a subset $Q\subset ‌{ℝ}^d$ cannot necessarily‌​‌ be extended to a​​ function (resp. operator) satisfying​​​‌ $p$ on the whole‌ of ${ℝ}^d$.‌​‌ Given $Q\subseteq {\mathrm{ℝ}}^d$, this work​​​‌ considers the problem of‌ obtaining necessary and ideally‌​‌ sufficient conditions to be​​ satisfied by a function​​​‌ (resp. operator) on $\mathrm{Q‌}$, ensuring the existence‌​‌ of an extension of​​ this function (resp. operator)​​​‌ satisfying $p$ on ${\mathrm{ℝ‌}}^d$.

More precisely,‌​‌ given some property $\mathrm{p}$, we present in​​​‌ 26 a refinement procedure‌ to obtain stronger necessary‌​‌ conditions to be imposed​​ on $Q$. This​​​‌ procedure can be applied‌ iteratively until the stronger‌​‌ conditions are also sufficient.​​ We illustrate the procedure​​​‌ on a few examples,‌ including the strengthening of‌​‌ existing descriptions for the​​ classes of smooth functions​​​‌ satisfying a Łojasiewicz condition,‌ convex blockwise smooth functions,‌​‌ Lipschitz monotone operators, strongly​​ monotone cocoercive operators, and​​​‌ uniformly convex functions.

In‌ most cases, these strengthened‌​‌ descriptions can be represented,​​ or relaxed, to semi-definite​​​‌ constraints, which can be‌ used to formulate tractable‌​‌ optimization problems on functions​​ (resp. operators) within those​​​‌ classes.

8.8 Optimized projection-free‌ algorithms for online learning:‌​‌ construction and worst-case analysis​​

In 33, we​​​‌ study and develop projection-free‌ algorithms for online learning‌​‌ with linear optimization oracles​​ (a.k.a. Frank–Wolfe) for handling​​​‌ the constraint set. More‌ precisely, this work (i)‌​‌ shows how to exploit​​ semidefinite programming to jointly​​​‌ design and analyze online‌ Frank–Wolfe-type algorithms numerically in‌​‌ a variety of settings,​​ (ii) leverage those design​​​‌ techniques to propose an‌ improved (optimized) variant of‌​‌ an online Frank–Wolfe algorithm​​ along with its conceptually​​​‌ simple potential-based proof, and‌ (iii) its anytime version‌​‌ which benefits from similar​​​‌ $O\left(T^{\mathrm{3}/4}\right)$ regret​‌ rate without requiring to​​ know the time horizon​​​‌ $T$ in advance. We​ are not aware of​‌ other direct regret guarantees​​ for anytime version of​​​‌ online Frank–Wolfe without using​ the classical doubling trick.​‌

Based on the semidefinite​​ technique, we conclude with​​​‌ strong numerical evidence suggesting​ that no pure online​‌ Frank–Wolfe algorithm within our​​ model class can have​​​‌ a regret guarantee better​ than $O({\mathrm{T‌}}^{3/4})$ without additional assumptions, that​​​‌ the current algorithms do​ not have optimal constants,​‌ and that multiple linear​​ optimization rounds do not​​​‌ generally help to obtain​ better regre

8.9 Large​‌ Stepsizes Accelerate Gradient Descent​​ for Regularized Logistic Regression​​​‌

In 35, we​ investigate the convergence dynamics​‌ of gradient descent (GD)​​ with constant stepsizes for​​​‌ $l_2$-regularized logistic​ regression on linearly separable​‌ data. While classical optimization​​ theory prescribes small stepsizes​​​‌ to ensure monotonic objective​ reduction, yielding a convergence​‌ rate linear in the​​ condition number $\kappa$,​​​‌ this study demonstrates that​ large stepsizes can accelerate​‌ this rate to $\widetilde{\mathrm{𝒪}}(\sqrt{\kappa })‌$. This acceleration leverages​ the "Edge of Stability"​‌ regime, where the objective​​ evolves non-monotonically, effectively matching​​​‌ the optimal rates of​ Nesterov's momentum without explicit​‌ acceleration terms. We extend​​ prior analyses from unregularized​​​‌ convex settings to the​ strongly convex case with​‌ finite minimizers. Furthermore, the​​ study establishes that these​​​‌ benefits extend to generalization​ bounds, improving the best-known​‌ bounds for minimizing population​​ risk under separable distribution.​​​‌ Finally, the work provides​ a sharp characterization of​‌ the maximum stepsize threshold​​ for local convergence.

8.10​​​‌ Statistical Advantage of Softmax​ Attention: Insights from Single-Location​‌ Regression

In 11,​​ we provide a theoretical​​​‌ grounding for the prevalence​ of softmax attention over​‌ linear alternatives in Large​​ Language Models. Focusing on​​​‌ the "Single-Location Regression" task,​ where the output depends​‌ on a single token​​ at a random position,​​​‌ we employ statistical physics​ techniques to analyze the​‌ learning dynamics in the​​ high-dimensional limit. We prove​​​‌ that softmax attention achieves​ the optimal Bayes risk,​‌ whereas linear attention fundamentally​​ falls short due to​​​‌ inherent approximation limitations.

In​ particular, the study characterizes​‌ generalization performance through a​​ small set of order​​​‌ parameters, demonstrating that both​ the exponential nonlinearity and​‌ the normalization scheme are​​ critical for this optimality.​​​‌ We further derive self-consistent​ equations to describe the​‌ regularized empirical risk minimizer​​ and extend their analysis​​​‌ to the finite-sample regime.​ In this regime, while​‌ softmax is no longer​​ strictly Bayes-optimal, it is​​​‌ shown to consistently outperform​ linear attention, offering robust​‌ statistical evidence for its​​ practical dominance.

8.11 Phase​​​‌ Diagram of Dropout for​ Two-Layer Neural Networks in​‌ the Mean-Field Regime

In​​ 6, we investigate​​​‌ the training dynamics of​ two-layer neural networks trained​‌ with dropout in the​​ large-width mean-field regime. We​​​‌ derive a rich asymptotic​ phase diagram comprising five​‌ distinct nondegenerate phases, determined​​ by the relative scalings​​​‌ of width, learning rate,​ and dropout rate. A​‌ key finding is that​​ the conventional “penalty” interpretation​​ of dropout as an​​​‌ implicit regularizer only persists‌ for impractically small learning‌​‌ rates of order $\mathrm{O}(1/\text{width‌})$. In the‌ more practical regime of‌​‌ larger learning rates, the​​ study demonstrates that dropout​​​‌ acts instead as a‌ "random geometry" modification, equivalent‌​‌ to a random block-coordinate​​ descent. In this limit,​​​‌ the dynamics are described‌ by mean-field jump processes‌​‌ driven by Poisson or​​ Bernoulli clocks. The analysis​​​‌ employs a combination of‌ coupling techniques for mean-field‌​‌ particle systems and martingale​​ methods to establish convergence​​​‌ in both path and‌ distribution spaces.

8.12 Convergence‌​‌ of Shallow ReLU Networks​​ on Weakly Interacting Data​​​‌

We analyse in 50‌ the convergence of one-hidden-layer‌​‌ ReLU networks trained by​​ gradient flow on $\mathrm{n‌}$ data points. Our main‌ contribution leverages the high‌​‌ dimensionality of the ambient​​ space, which implies low​​​‌ correlation of the input‌ samples, to demonstrate that‌​‌ a network with width​​ of order $log(‌n)$ neurons suffices‌ for global convergence with‌​‌ high probability. Our analysis​​ uses a Polyak–Łojasiewicz viewpoint​​​‌ along the gradient-flow trajectory,‌ which provides an exponential‌​‌ rate of convergence of​​ $1/n$ .​​​‌ When the data are‌ exactly orthogonal, we give‌​‌ further refined characterizations of​​ the convergence speed, proving​​​‌ its asymptotic behavior lies‌ between the orders $\frac{\mathrm{1‌‌}}{n}$ and $1/\sqrt{n}$ , and exhibiting​​​‌ a phase-transition phenomenon in‌ the convergence rate, during‌​‌ which it evolves from​​ the lower bound to​​​‌ the upper, and in‌ a relative time of‌​‌ order $1/log\left(n\right)$.​​​‌

8.13 Convergence of Deterministic‌ and Stochastic Diffusion-Model Samplers:‌​‌ A Simple Analysis in​​ Wasserstein Distance

We provide​​​‌ in 54 new convergence‌ guarantees in Wasserstein distance‌​‌ for diffusion-based generative models,​​ covering both stochastic (DDPM-like)​​​‌ and deterministic (DDIM-like) sampling‌ methods. We introduce a‌​‌ simple framework to analyze​​ discretization, initialization, and score​​​‌ estimation errors. Notably, we‌ derive the first Wasserstein‌​‌ convergence bound for the​​ Heun sampler and improve​​​‌ existing results for the‌ Euler sampler of the‌​‌ probability flow ODE. Our​​ analysis emphasizes the importance​​​‌ of spatial regularity of‌ the learned score function‌​‌ and argues for controlling​​ the score error with​​​‌ respect to the true‌ reverse process, in line‌​‌ with denoising score matching.​​ We also incorporate recent​​​‌ results on smoothed Wasserstein‌ distances to sharpen initialization‌​‌ error bounds.

8.14 Adaptive​​ Coverage Policies in Conformal​​​‌ Prediction

Traditional conformal prediction‌ methods construct prediction sets‌​‌ such that the true​​ label falls within the​​​‌ set with a user-specified‌ coverage level. However, poorly‌​‌ chosen coverage levels can​​ result in uninformative predictions,​​​‌ either producing overly conservative‌ sets when the coverage‌​‌ level is too high,​​ or empty sets when​​​‌ it is too low.‌ Moreover, the fixed coverage‌​‌ level cannot adapt to​​ the specific characteristics of​​​‌ each individual example, limiting‌ the flexibility and efficiency‌​‌ of these methods. In​​ this work, we leverage​​​‌ recent advances in e-values‌ and post-hoc conformal inference,‌​‌ which allow the use​​ of data-dependent coverage levels​​​‌ while maintaining valid statistical‌ guarantees. We propose in‌​‌ 66 to optimize an​​​‌ adaptive coverage policy by​ training a neural network​‌ using a leave-one-out procedure​​ on the calibration set,​​​‌ allowing the coverage level​ and the resulting prediction​‌ set size to vary​​ with the difficulty of​​​‌ each individual example. We​ support our approach with​‌ theoretical coverage guarantees and​​ demonstrate its practical benefits​​​‌ through a series of​ experiments.

8.15 Fast kernel​‌ methods: Sobolev, physics-informed, and​​ additive models

Physics-informed machine​​​‌ learning typically integrates physical​ priors into the learning​‌ process by minimizing a​​ loss function that includes​​​‌ both a data-driven term​ and a partial differential​‌ equation (PDE) regularization. Building​​ on the formulation of​​​‌ the problem as a​ kernel regression task, we​‌ use in 62 Fourier​​ methods to approximate the​​​‌ associated kernel, and propose​ a tractable estimator that​‌ minimizes the physics-informed risk​​ function. We refer to​​​‌ this approach as physics-informed​ kernel learning (PIKL). This​‌ framework provides theoretical guarantees,​​ enabling the quantification of​​​‌ the physical prior’s impact​ on convergence speed. We​‌ demonstrate the numerical performance​​ of the PIKL estimator​​​‌ through simulations, both in​ the context of hybrid​‌ modeling and in solving​​ PDEs. In particular, we​​​‌ show that PIKL can​ outperform physics-informed neural networks​‌ in terms of both​​ accuracy and computation time.​​​‌ Additionally, we identify cases​ where PIKL surpasses traditional​‌ PDE solvers, particularly in​​ scenarios with noisy boundary​​​‌ conditions.

8.16 On the​ Effectiveness of the z-Transform​‌ Method in Quadratic Optimization​​

The z-transform of a​​​‌ sequence is a classical​ tool used within signal​‌ processing, control theory, computer​​ science, and electrical engineering.​​​‌ It allows for studying​ sequences from their generating​‌ functions, with many operations​​ that can be equivalently​​​‌ defined on the original​ sequence and its z-transform.​‌ In particular, the z-transform​​ method focuses on asymptotic​​​‌ behaviors and allows the​ use of Taylor expansions.​‌ We present a sequence​​ of results of increasing​​​‌ significance and difficulty for​ linear models and optimization​‌ algorithms, demonstrating the effectiveness​​ and versatility of the​​​‌ z-transform method in deriving​ new asymptotic results. Starting​‌ from the simplest gradient​​ descent iterations in an​​​‌ infinite-dimensional Hilbert space, we​ show in 51 how​‌ the spectral dimension characterizes​​ the convergence behavior. We​​​‌ then extend the analysis​ to Nesterov acceleration, averaging​‌ techniques, and stochastic gradient​​ descent.

8.17 Rethinking Early​​​‌ Stopping: Refine, Then Calibrate​

Machine learning classifiers often​‌ produce probabilistic predictions that​​ are critical for accurate​​​‌ and interpretable decision-making in​ various domains. The quality​‌ of these predictions is​​ generally evaluated with proper​​​‌ losses like cross-entropy, which​ decompose into two components:​‌ calibration error assesses general​​ under/overconfidence, while refinement error​​​‌ measures the ability to​ distinguish different classes. In​‌ 52, we provide​​ theoretical and empirical evidence​​​‌ that these two errors​ are not minimized simultaneously​‌ during training. Selecting the​​ best training epoch based​​​‌ on validation loss thus​ leads to a compromise​‌ point that is suboptimal​​ for both calibration error​​​‌ and, most importantly, refinement​ error. To address this,​‌ we introduce a new​​ metric for early stopping​​​‌ and hyperparameter tuning that​ makes it possible to​‌ minimize refinement error during​​ training. The calibration error​​ is minimized after training,​​​‌ using standard techniques. Our‌ method integrates seamlessly with‌​‌ any architecture and consistently​​ improves performance across diverse​​​‌ classification tasks.

8.18 Conditional‌ Coverage Diagnostics for Conformal‌​‌ Prediction

Evaluating conditional coverage​​ remains one of the​​​‌ most persistent challenges in‌ assessing the reliability of‌​‌ predictive systems. Although conformal​​ methods can give guarantees​​​‌ on marginal coverage, no‌ method can guarantee to‌​‌ produce sets with correct​​ conditional coverage, leaving practitioners​​​‌ without a clear way‌ to interpret local deviations.‌​‌ To overcome sample-inefficiency and​​ overfitting issues of existing​​​‌ metrics, we cast in‌ 58 conditional coverage estimation‌​‌ as a classification problem.​​ Conditional coverage is violated​​​‌ if and only if‌ any classifier can achieve‌​‌ lower risk than the​​ target coverage. Through the​​​‌ choice of a (proper)‌ loss function, the resulting‌​‌ risk difference gives a​​ conservative estimate of natural​​​‌ miscoverage measures such as‌ L1 and L2 distance,‌​‌ and can even separate​​ the effects of over-​​​‌ and under-coverage, and non-constant‌ target coverages. We call‌​‌ the resulting family of​​ metrics excess risk of​​​‌ the target coverage (ERT).‌ We show experimentally that‌​‌ the use of modern​​ classifiers provides much higher​​​‌ statistical power than simple‌ classifiers underlying established metrics‌​‌ like CovGap. Additionally, we​​ use our metric to​​​‌ benchmark different conformal prediction‌ methods. Finally, we release‌​‌ an open-source package for​​ ERT as well as​​​‌ previous conditional coverage metrics.‌ Together, these contributions provide‌​‌ a new lens for​​ understanding, diagnosing, and improving​​​‌ the conditional reliability of‌ predictive systems.

8.19 Functional‌​‌ protein mining with conformal​​ guarantees

Molecular structure prediction​​​‌ and homology detection offer‌ promising paths to discovering‌​‌ protein function and evolutionary​​ relationships. However, current approaches​​​‌ lack statistical reliability assurances,‌ limiting their practical utility‌​‌ for selecting proteins for​​ further experimental and in-silico​​​‌ characterization. To address this‌ challenge, we introduce a‌​‌ statistically principled approach to​​ protein search leveraging principles​​​‌ from conformal prediction, offering‌ a framework that ensures‌​‌ statistical guarantees with user-specified​​ risk and provides calibrated​​​‌ probabilities (rather than raw‌ ML scores) for any‌​‌ protein search model. Our​​ method (1) lets users​​​‌ select many biologically-relevant loss‌ metrics (i.e. false discovery‌​‌ rate) and assigns reliable​​ functional probabilities for annotating​​​‌ genes of unknown function;‌ (2) achieves state-of-the-art performance‌​‌ in enzyme classification without​​ training new models; and​​​‌ (3) robustly and rapidly‌ pre-filters proteins for computationally‌​‌ intensive structural alignment algorithms.​​ Our framework enhances the​​​‌ reliability of protein homology‌ detection and enables the‌​‌ discovery of uncharacterized proteins​​ with likely desirable functional​​​‌ properties.

8.20 Gradient equilibrium‌ in online learning: Theory‌​‌ and applications

We present​​ a new perspective on​​​‌ online learning that we‌ refer to as gradient‌​‌ equilibrium: a sequence of​​ iterates achieves gradient equilibrium​​​‌ if the average of‌ gradients of losses along‌​‌ the sequence converges to​​ zero. In general, this​​​‌ condition is not implied‌ by, nor implies, sublinear‌​‌ regret. It turns out​​ that gradient equilibrium is​​​‌ achievable by standard online‌ learning methods such as‌​‌ gradient descent and mirror​​ descent with constant step​​​‌ sizes (rather than decaying‌ step sizes, as is‌​‌ usually required for no​​​‌ regret). Further, as we​ show through examples, gradient​‌ equilibrium translates into an​​ interpretable and meaningful property​​​‌ in online prediction problems​ spanning regression, classification, quantile​‌ estimation, and others. Notably,​​ we show that the​​​‌ gradient equilibrium framework can​ be used to develop​‌ a debiasing scheme for​​ black-box predictions under arbitrary​​​‌ distribution shift, based on​ simple post hoc online​‌ descent updates. We also​​ show that post hoc​​​‌ gradient updates can be​ used to calibrate predicted​‌ quantiles under distribution shift,​​ and that the framework​​​‌ leads to unbiased Elo​ scores for pairwise preference​‌ prediction.

8.21 Universal log-optimality​​ for general classes of​​​‌ e-processes and sequential hypothesis​ tests

We consider the​‌ problem of sequential hypothesis​​ testing by betting. For​​​‌ a general class of​ composite testing problems –​‌ which include bounded mean​​ testing, equal mean testing​​​‌ for bounded random tuples,​ and some key ingredients​‌ of two-sample and independence​​ testing as special cases​​​‌ – we show that​ any e-process satisfying a​‌ certain sublinear regret bound​​ is adaptively, asymptotically, and​​​‌ almost surely log-optimal for​ a composite alternative. This​‌ is a strong notion​​ of optimality that has​​​‌ not previously been established​ for the aforementioned problems​‌ and we provide explicit​​ test supermartingales and e-processes​​​‌ satisfying this notion in​ the more general case.​‌ Furthermore, we derive matching​​ lower and upper bounds​​​‌ on the expected rejection​ time for the resulting​‌ sequential tests in all​​ of these cases. The​​​‌ proofs of these results​ make weak, algorithm-agnostic moment​‌ assumptions and rely on​​ a general-purpose proof technique​​​‌ involving the aforementioned regret​ and a family of​‌ numeraire portfolios. Finally, we​​ discuss how all of​​​‌ these theorems hold in​ a distribution-uniform sense, a​‌ notion of log-optimality that​​ is stronger still and​​​‌ seems to be new​ to the literature.

8.22​‌ The statistical fairness-accuracy frontier​​

Machine learning models must​​​‌ balance accuracy and fairness,​ but these goals often​‌ conflict, particularly when data​​ come from multiple demographic​​​‌ groups. A useful tool​ for understanding this trade-off​‌ is the fairness-accuracy (FA)​​ frontier, which characterizes the​​​‌ set of models that​ cannot be simultaneously improved​‌ in both fairness and​​ accuracy. Prior analyses of​​​‌ the FA frontier provide​ a full characterization under​‌ the assumption of complete​​ knowledge of population distributions​​​‌ – an unrealistic ideal.​ We study the FA​‌ frontier in the finite-sample​​ regime, showing how it​​​‌ deviates from its population​ counterpart and quantifying the​‌ worst-case gap between them.​​ In particular, we derive​​​‌ minimax-optimal estimators that depend​ on the designer's knowledge​‌ of the covariate distribution.​​ For each estimator, we​​​‌ characterize how finite-sample effects​ asymmetrically impact each group's​‌ risk, and identify optimal​​ sample allocation strategies. Our​​​‌ results transform the FA​ frontier from a theoretical​‌ construct into a practical​​ tool for policymakers and​​​‌ practitioners who must often​ design algorithms with limited​‌ data.

9.1 Bilateral grants with​ industry

• Chaire “Marchés et​‌ Apprentissage”, portée par Michael​​ Jordan au sein de​​​‌ la Fondation Inria, et​ lancée en Juillet 2024.​‌ En partenariat avec Air​​ Liquide, BNP Paribas Asset​​ Management Europe, EDF, Orange​​​‌ et la SNCF.

• Francis‌ Bach: Co-advised PhD student‌​‌ with Meta.
• Pierre Marion:​​ Co-advised PhD student with​​​‌ Meta.
• Pierre Marion: Gift‌ from Google.org.

10.1 International​​ initiatives

10.2 European initiatives​​

10.3 National initiatives‌

• Alexandre d'Aspremont, Francis Bach,‌​‌ Michael Jordan: Chairs from​​ the PRAIRIE-PSAI Cluster.

10.4​​​‌ Regional initiatives

• Pierre Marion:‌ Tremplin Chair from the‌​‌ PRAIRIE-PSAI Cluster.
  • Title:
    Mathematical​​ Foundations of Modern Deep​​​‌ Learning
  • Duration:
    From September‌ 1, 2025 to September‌​‌ 30, 2029
  • Summary:
    Recent​​ years have witnessed breakthroughs​​​‌ across many fields of‌ artificial intelligence (AI), largely‌​‌ driven by rapid advances​​ in deep learning techniques.​​​‌ At the same time,‌ modern AI models also‌​‌ present fundamental flaws: hallucinations,​​ copyright infringements, biases, brittleness​​​‌ to adversarial attacks, economic‌ and ecological cost. On‌​‌ the theoretical side, many​​​‌ fundamental questions regarding the​ striking effectiveness of deep​‌ learning remain open. While​​ general theories of deep​​​‌ learning have provided valuable​ insights, they do not​‌ always capture the wide​​ variety of settings encompassed​​​‌ in practice. My overarching​ research goal is to​‌ address some of these​​ challenges, by leveraging a​​​‌ mathematically-grounded approach to understand​ and improve modern AI​‌ techniques. My research​​ proposal is structured around​​​‌ three complementary axes towards​ advancing this goal: (i)​‌ Theoretical insights on generative​​ models. The first axis​​​‌ explores core methodologies underpinning​ modern generative AI, particularly​‌ denoising diffusion models and​​ Transformers, which form the​​​‌ backbone of large language​ models (LLMs). I seek​‌ to analyze how specific​​ architectural choices and training​​​‌ procedures impact performance, robustness,​ and efficiency. (ii) Deep​‌ learning optimization. The remarkable​​ effectiveness of stochastic gradient​​​‌ descent at finding good​ solutions in deep learning​‌ settings—large non-convex optimization problems—remains​​ only partially understood. My​​​‌ research in this axis​ focuses on the role​‌ of regularization, especially through​​ the lens of optimization​​​‌ dynamics. (iii) LLMs for​ formal mathematical reasoning. AI-assisted​‌ formal reasoning is a​​ rapidly emerging field, recently​​​‌ achieving successes at the​ level of Olympiad mathematics.​‌ These advances bring us​​ closer to AI-assisted theorem​​​‌ proving, with the potential​ to revolutionize the practice​‌ of mathematical research, while​​ also serving as a​​​‌ testbed for the reasoning​ abilities of LLMs. A​‌ particularly promising route involves​​ using LLMs to generate​​​‌ proofs in a formal​ language such as Lean​‌ or Rocq. This raises​​ crucial methodological questions that​​​‌ for now have been​ little investigated. For instance:​‌ Is it advantageous to​​ represent proofs as trees​​​‌ rather than unstructured sequences​ of text? If so,​‌ how can we guide​​ LLMs in exploring the​​​‌ proof tree efficiently? And​ can reinforcement learning be​‌ used to train LLMs​​ in this context, despite​​​‌ the absence of a​ standard two-player game framework​‌ (as in chess or​​ Go)?

11.1​​​‌ Promoting scientific activities

11.2 Teaching -​ Supervision - Juries -​‌ Educational and pedagogical outreach​​

• Adrien Taylor: Convex Optimization​​​‌ (M1, ENS; 21h)
• Adrien​ Taylor: Convex Optimization (MVA;​‌ 3h)
• Adrien Taylor: Optimization​​ & deep learning (M1,​​​‌ X/HEC; 30h)
• Alexandre d'Aspremont:​ Convex Optimization (MVA; 21h)​‌
• Umut Simsekli: Introduction to​​ Machine Learning (ENS, L3;​​​‌ 12h)
• Francis Bach: Learning​ Theory from First Principles​‌ (M2 IASD; 27h)

11.3 Popularization

12.1 Major​ publications

• 1 inproceedingsR.​‌Rayna Andreeva, B.​​Benjamin Dupuis, R.​​Rik Sarkar, T.​​​‌Tolga Birdal and U.‌Umut Şimşekli. Topological‌​‌ Generalization Bounds for Discrete-Time​​ Stochastic Optimization Algorithms.​​​‌PMLRAdvances in Neural‌ Information Processing SystemsVancouver,‌​‌ Canada2024HAL
• 2​​ miscR.Roland Andrews​​​‌, J.Justin Carpentier‌ and A.Adrien Taylor‌​‌. Augmented Lagrangian methods​​ for infeasible convex optimization​​​‌ problems and diverging proximal-point‌ algorithms.June 2025‌​‌HALback to text​​
• 3 articleA.Armin​​​‌ Askari, A.Alexandre‌ d'Aspremont and L. E.‌​‌Laurent El Ghaoui.​​ Approximation Bounds for Sparse​​​‌ Programs.SIAM Journal‌ on Mathematics of Data‌​‌ Science42June​​ 2022, 514-530HAL​​​‌DOI
• 4 inproceedingsA.‌Andrea Bertazzi, D.‌​‌Dario Shariatian, U.​​Umut Simsekli, E.​​​‌Eric Moulines and A.‌Alain Durmus. Piecewise‌​‌ deterministic generative models.​​PMLRAdvances in Neural​​​‌ Information Processing SystemsVancouver,‌ Canada2024HAL
• 5‌​‌ miscT.Théophile Cantelobre​​, C.Carlo Ciliberto​​​‌, B.Benjamin Guedj‌ and A.Alessandro Rudi‌​‌. Measuring dissimilarity with​​ diffeomorphism invariance.February​​​‌ 2022HALDOI
• 6‌ miscL.Lénaïc Chizat‌​‌, P.Pierre Marion​​ and Y.Yerkin Yesbay​​​‌. Phase Diagram of‌ Dropout for Two-Layer Neural‌​‌ Networks in the Mean-Field​​ Regime.October 2025​​​‌HALback to text‌
• 7 articleR.-A.Radu-Alexandru‌​‌ Dragomir, A.Adrien​​ Taylor, A.Alexandre​​​‌ d'Aspremont and J.Jérôme‌ Bolte. Optimal Complexity‌​‌ and Certification of Bregman​​ First-Order Methods.Mathematical​​​‌ Programming1941July‌ 2022, 41-83HAL‌​‌DOI
• 8 inproceedingsB.​​Benjamin Dupuis, G.​​​‌George Deligiannidis and U.‌Umut Şimşekli. Generalization‌​‌ Bounds using Data-Dependent Fractal​​ Dimensions.Proceedings of​​​‌ Machine Learning ResearchInternational‌ Conference on Machine Learning‌​‌ (ICML 2023)Honolulu, United​​ StatesJuly 2023HAL​​​‌
• 9 inproceedingsB.Benjamin‌ Dupuis, D.Dario‌​‌ Shariatian, M.Maxime​​ Haddouche, A.Alain​​​‌ Durmus and U.Umut‌ Simsekli. Algorithm- and‌​‌ Data-Dependent Generalization Bounds for​​ Score-Based Generative Models.​​​‌Advances in Neural Information‌ Processing SystemsSan Diego,‌​‌ United States2025HAL​​
• 10 inproceedingsB.Benjamin​​​‌ Dupuis and U.Umut‌ Şimşekli. Generalization Bounds‌​‌ for Heavy-Tailed SDEs through​​ the Fractional Fokker-Planck Equation​​​‌.PMLRInternational Conference‌ on Machine LearningVienna,‌​‌ Austria2024HALback​​ to text
• 11 misc​​​‌O.Odilon Duranthon,‌ P.Pierre Marion,‌​‌ C.Claire Boyer,​​ B.Bruno Loureiro and​​​‌ L.Lenka Zdeborová.‌ Statistical Advantage of Softmax‌​‌ Attention: Insights from Single-Location​​ Regression.October 2025​​​‌HALback to text‌
• 12 articleM.Mert‌​‌ Gurbuzbalaban, Y.Yuanhan​​ Hu, U.Umut​​​‌ Simsekli, K.Kun‌ Yuan and L.Lingjiong‌​‌ Zhu. Heavy-Tail Phenomenon​​ in Decentralized SGD.​​​‌IISE Transactions2024HAL‌
• 13 articleM.Mert‌​‌ Gürbüzbalaban, Y.Yuanhan​​ Hu, U.Umut​​​‌ Şimşekli and L.Lingjiong‌ Zhu. Cyclic and‌​‌ Randomized Stepsizes Invoke Heavier​​ Tails in SGD than​​​‌ Constant Stepsize.Transactions‌ on Machine Learning Research‌​‌ Journal2023HAL
• 14​​ inproceedingsM.Maxime Haddouche​​​‌, P.Paul Viallard‌, U.Umut Şimşekli‌​‌ and B.Benjamin Guedj​​​‌. A PAC-Bayesian Link​ Between Generalisation and Flat​‌ Minima.ALT 2025​​ - 36th International Conference​​​‌ on Algorithmic Learning Theory​Milan, Italy2025,​‌ 1-31HALback to​​ text
• 15 inproceedingsL.​​​‌Liam Hodgkinson, U.​Umut Şimşekli, R.​‌Rajiv Khanna and M.​​ W.Michael W. Mahoney​​​‌. Generalization Bounds using​ Lower Tail Exponents in​‌ Stochastic Optimizers.International​​ Conference on Machine Learning​​​‌Baltimore, United States2022​HAL
• 16 inproceedingsS.​‌Soheil Kolouri, K.​​Kimia Nadjahi, S.​​​‌Shahin Shahrampour and U.​Umut Simsekli. Generalized​‌ Sliced Probability Metrics.​​ICASSP 2022 - 2022​​​‌ IEEE International Conference on​ Acoustics, Speech and Signal​‌ Processing (ICASSP)Singapore, Singapore​​IEEEMay 2022,​​​‌ 4513-4517HALDOI
• 17​ articleT.Thomas Lauvaux​‌, C.Clément Giron​​, M.Matthieu Mazzolini​​​‌, A.Alexandre d'Aspremont​, R.Riley Duren​‌, D.Daniel Cusworth​​, D.Drew Shindell​​​‌ and P.Philippe Ciais​. Global assessment of​‌ oil and gas methane​​ ultra-emitters.Science375​​​‌6580February 2022,​ 557-561HALDOI
• 18​‌ inproceedingsS. H.Soon​​ Hoe Lim, Y.​​​‌Yijun Wan and U.​Umut Şimşekli. Chaotic​‌ Regularization and Heavy-Tailed Limits​​ for Deterministic Gradient Descent​​​‌.Advances in Neural​ Processing SystemsNew Orleans,​‌ United States2022HAL​​
• 19 unpublishedU.Ulysse​​​‌ Marteau-Ferey, F.Francis​ Bach and A.Alessandro​‌ Rudi. Non-parametric Models​​ for Non-negative Functions.​​​‌July 2020, working​ paper or preprintHAL​‌
• 20 inproceedingsS.Sejun​​ Park, U.Umut​​​‌ Şimşekli and M. A.​Murat A. Erdogdu.​‌ Generalization Bounds for Stochastic​​ Gradient Descent via Localized​​​‌ $$-Covers.Advances in​ Neural Processing SystemsBaltimore,​‌ United StatesSeptember 2022​​HAL
• 21 proceedingsK.​​​‌ L.Krunoslav Lehman Pavasovic​, A.Alain Durmus​‌ and U.Umut Simsekli​​, eds. Approximate Heavy​​​‌ Tails in Offline (Multi-Pass)​ Stochastic Gradient Descent.​‌Advances in Neural Information​​ Processing SystemsOctober 2023​​​‌HAL
• 22 inproceedingsA.​Anant Raj, M.​‌Melih Barsbey, M.​​Mert Gürbüzbalaban, L.​​​‌Lingjiong Zhu and U.​Umut Şimşekli. Algorithmic​‌ Stability of Heavy-Tailed Stochastic​​ Gradient Descent on Least​​​‌ Squares.Algorithmic Learning​ TheorySingapore, Singapore2023​‌HAL
• 23 proceedingsA.​​Anant Raj, U.​​​‌Umut Şimşekli and A.​Alessandro Rudi, eds.​‌ Efficient Sampling of Stochastic​​ Differential Equations with Positive​​​‌ Semi-Definite Models.Advances​ in Neural Information Processing​‌ Systems2023HAL
• 24​​ inproceedingsA.Anant Raj​​​‌, L.Lingjiong Zhu​, M.Mert Gürbüzbalaban​‌ and U.Umut Şimşekli​​. Algorithmic Stability of​​​‌ Heavy-Tailed SGD with General​ Loss Functions.International​‌ Conference on Machine Learning​​Honolulu, United States2023​​​‌HAL
• 25 articleV.​Vincent Roulet and A.​‌Alexandre D'Aspremont. Sharpness,​​ Restart and Acceleration.​​​‌SIAM Journal on Optimization​301October 2020​‌, 262-289HALDOI​​
• 26 miscA.Anne​​​‌ Rubbens, J. M.​Julien M. Hendrickx and​‌ A.Adrien Taylor.​​ A constructive approach to​​​‌ strengthen algebraic descriptions of​ function and operator classes​‌.September 2025HAL​​back to text
• 27​​ inproceedingsS.Sarah Sachs​​​‌, T.Tim van‌ Erven, L.Liam‌​‌ Hodgkinson, R.Rajiv​​ Khanna and U.Umut​​​‌ Simsekli. Generalization Guarantees‌ via Algorithm-dependent Rademacher Complexity‌​‌.Conference on Learning​​ TheoryBangalore (Virtual event),​​​‌ IndiaJuly 2023HAL‌
• 28 inproceedingsF.Fabian‌​‌ Schaipp, A.Alexander​​ Hägele, A.Adrien​​​‌ Taylor, U.Umut‌ Simsekli and F.Francis‌​‌ Bach. The Surprising​​ Agreement Between Convex Optimization​​​‌ Theory and Learning-Rate Scheduling‌ for Large Model Training‌​‌.ICML 2025 -​​ 42nd International Conference on​​​‌ Machine LearningVancouver (BC),‌ CanadaJuly 2025HAL‌​‌back to text
• 29​​ inproceedingsM.Milad Sefidgaran​​​‌, A.Amin Gohari‌, G.Gael Richard‌​‌ and U.Umut Şimşekli​​. Rate-Distortion Theoretic Generalization​​​‌ Bounds for Stochastic Learning‌ Algorithms.COLT 2022‌​‌ - 35th Annual Conference​​ on Learning Theory178​​​‌Proceedings of Machine Learning‌ ResearchLondon, United Kingdom‌​‌July 2022HAL
• 30​​ inproceedingsD.Dario Shariatian​​​‌, U.Umut Simsekli‌ and A.Alain Durmus‌​‌. Heavy-Tailed Diffusion with​​ Denoising Lévy Probabilistic Models​​​‌.International Conference on‌ Learning RepresentationsSingapore, Singapore‌​‌2025HALback to​​ text
• 31 inproceedingsP.​​​‌Paul Viallard, M.‌Maxime Haddouche, U.‌​‌Umut Şimşekli and B.​​Benjamin Guedj. Learning​​​‌ via Wasserstein-Based High Probability‌ Generalisation Bounds.NeurIPS‌​‌ 2023 - Thirty-seventh Conference​​ on Neural Information Processing​​​‌ SystemsNew Orleans, United‌ StatesJune 2023HAL‌​‌DOI
• 32 inproceedingsY.​​Yijun Wan, M.​​​‌Melih Barsbey, A.‌Abdellatif Zaidi and U.‌​‌Umut Simsekli. Implicit​​ Compressibility of Overparametrized Neural​​​‌ Networks Trained with Heavy-Tailed‌ SGD.PMLRInternational‌​‌ Conference on Machine Learning​​Vienna, Austria2024HAL​​​‌
• 33 miscJ.Julien‌ Weibel, P.Pierre‌​‌ Gaillard, W. M.​​Wouter M. Koolen and​​​‌ A.Adrien Taylor.‌ Optimized projection-free algorithms for‌​‌ online learning: construction and​​ worst-case analysis.June​​​‌ 2025HALback to‌ text
• 34 miscB.‌​‌Blake Woodworth, F.​​Francis Bach and A.​​​‌Alessandro Rudi. Non-Convex‌ Optimization with Certificates and‌​‌ Fast Rates Through Kernel​​ Sums of Squares.​​​‌April 2022HALDOI‌
• 35 inproceedingsJ.Jingfeng‌​‌ Wu, P.Pierre​​ Marion and P. L.​​​‌Peter L Bartlett.‌ Large Stepsizes Accelerate Gradient‌​‌ Descent for Regularized Logistic​​ Regression.NeurIPS 2025​​​‌ - 39th Annual Conference‌ on Neural Information Processing‌​‌ SystemsAdvances in Neural​​ Information Processing Systems38​​​‌San Diego (CA), United‌ StatesDecember 2025HAL‌​‌back to text
• 36​​ proceedingsL.Lingjiong Zhu​​​‌, M.Mert Gurbuzbalaban‌, A.Anant Raj‌​‌ and U.Umut Simsekli​​, eds. Uniform-in-Time Wasserstein​​​‌ Stability Bounds for (Noisy)‌ Stochastic Gradient Descent.‌​‌Advances in Neural Information​​ Processing Systems2023HAL​​​‌

12.2 Publications of the‌ year

12.3‌​‌ Cited publications

• 80 phdthesis​​L.Lawrence Stewart.​​​‌ Understanding and Formulating Training‌ Objectives: Key Insights for‌​‌ Deep Learning.INRIA​​June 2025HALback​​​‌ to text