Stochastic Modified Flows, Mean-Field Limits and Dynamics of Stochastic Gradient Descent

• Authors: Benjamin Gess, Sebastian Kassing, Vitalii Konarovskyi
• Preprint year: 2023
• First public date: 2023-02-14
• arXiv: 2302.07125
• Status: Published
• Publication type: Journal article
• Publication year: 2024
• Journal: Journal of machine learning research, 25 (2024) 30, pp. 1-27

Abstract

We propose new limiting dynamics for stochastic gradient descent in the small learning rate regime called stochastic modified flows. These SDEs are driven by a cylindrical Brownian motion and improve the so-called stochastic modified equations by having regular diffusion coefficients and by matching the multi-point statistics. As a second contribution, we introduce distribution dependent stochastic modified flows which we prove to describe the fluctuating limiting dynamics of stochastic gradient descent in the small learning rate – infinite width scaling regime.

Associated SAiS members

• Benjamin Gess

Research areas

• Stochastic Dynamics
• Numerics
• Machine Learning
• Non-equilibrium Statistical Mechanics, Interacting Particle Systems and Fluctuating Hydrodynamics

BibTeX

@article{arxiv230207125,
  title = {Stochastic Modified Flows, Mean-Field Limits and Dynamics of Stochastic Gradient Descent},
  author = {Benjamin Gess and Sebastian Kassing and Vitalii Konarovskyi},
  year = {2024},
  journal = {Journal of machine learning research, 25 (2024) 30, pp. 1-27},
  eprint = {2302.07125},
  archivePrefix = {arXiv},
  primaryClass = {math.PR}
}