Section: New Results
Non-parametric Stochastic Approximation with Large Step sizes
Participants : Aymeric Dieuleveut, Francis Bach.
We consider the random-design least-squares regression problem within the reproducing kernel Hilbert space (RKHS) framework. Given a stream of independent and identically distributed input/output data, we aim to learn a regression function within an RKHS , even if the optimal predictor (i.e., the conditional expectation) is not in . In a stochastic approximation framework where the estimator is updated after each observation, we show that the averaged unregularized least-mean-square algorithm (a form of stochastic gradient), given a sufficient large step-size, attains optimal rates of convergence for a variety of regimes for the smoothnesses of the optimal prediction function and the functions in .