Paper detail

Computer-aided analyses of stochastic first-order methods, via interpolation conditions for stochastic optimization

This work proposes a framework, embedded within the Performance Estimation framework (PEP), for obtaining worst-case performance guarantees on stochastic first-order methods. Given a first-order method, a function class, and a noise model with prescribed expectation and variance properties, we present a semidefinite program (SDP), whose size grows linearly with $N$, the number of iterations analyzed, and whose solution yields a convergence guarantee on the problem. The framework accommodates a wide range of stochastic settings, with finite or infinite support, including the unstructured noise model with bounded variance, finite-sum optimization, and block-coordinate methods, in a unified manner, as guarantees apply to any setting consistent with the noise model, i.e., its expectation and variance. It covers both non-variance-reduced and variance-reduced methods. Using the framework, we analyze the stochastic gradient method under several noise models, and illustrate how the resulting numerical and analytical convergence rates connect with existing results. In particular, we provide improved convergence rates on the unstructured noise model with bounded variance and in the block-coordinate setting.