Self-Averaging and On-line Learning

Conditions are given under which one may prove that the stochastic dynamics of on-line learning can be described by the deterministic evolution of a finite set of order parameters in the thermodynamic limit. A global constraint on the average magnitude of the increments in the stochastic process is necessary to ensure self-averaging. In the absence of such a constraint, convergence may only be in probability.