Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Optimal-transport CBO trains neural networks and, in the mean-field limit, provably shrinks particle variance.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

CBO can train small two-layer networks, a hybrid CBO-Adam method improves convergence and stability, and a Wasserstein mean-field model of CBO has monotonically decreasing variance.

T0 review reviewed 2026-08-03 challenge →

load-bearing objection Solid incremental OT-CBO paper with a clean variance decay bound, but the claimed N→∞ mean-field limit drops the Gibbs weights without justification. the 3 major comments →

arxiv 2511.21466 v3 pith:H3C7LZEO submitted 2025-11-26 cs.LG math.OC

Mean-Field Model for Two-Layer Neural Networks Trained with Consensus-Based Optimization

William De Deyn , Michael Herty , Giovanni Samaey This is my paper
classification cs.LG math.OC MSC 49Q2268T0782C22
keywords consensus-based optimizationmean-field limittwo-layer neural networksoptimal transportWasserstein barycentervariance decaymulti-task learningparticle methods
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Consensus-Based Optimization (CBO) is a gradient-free, particle-based global optimizer; this paper applies it to training two-layer neural networks and compares it with Adam. It proposes a hybrid CBO+Adam scheme that converges faster than CBO alone and a multi-task variant that recycles one particle population across related tasks, lowering memory use. The theoretical core is a reformulation of CBO in optimal-transport language: each particle is a probability measure over network parameters, and the consensus point becomes a Wasserstein barycenter. Taking both the network width and the particle number to infinity, the authors derive a time-discrete mean-field limit on a Wasserstein-over-Wasserstein space and prove the population variance contracts as V(ρ^{k+1}) ≤ (1−Δt)^2 V(ρ^k), establishing consensus in the limit. Numerical experiments confirm the mean-field models converge.

Core claim

The paper's central claim is that the optimal-transport reformulation of CBO admits a clean mean-field description: with particles as measures, the update is a pushforward along the Wasserstein geodesic toward the barycenter, and in the N→∞ limit the law of particles evolves by ρ^{k+1} = (Ψ_Δt)#ρ^k. Proposition 3 then shows the variance, measured as half the squared Wasserstein distance to the barycenter, decays by the factor (1−Δt)^2 each step. The paper argues this proves the OT-CBO scheme reaches consensus, and that the same dynamics train arbitrarily wide two-layer networks when paired with the M→∞ mean-field representation of the network as a measure.

What carries the argument

The central object is the time-discrete map Ψ_Δt(μ) = ((1−Δt) Id + Δt T)#μ, where T pushes μ to the consensus barycenter; it interpolates along a constant-speed Wasserstein geodesic between μ and the barycenter. The barycenter is the weighted Fréchet mean minimizing Σ β(μ_n) W_2^2(ν, μ_n), with Gibbs weights β(μ_n) ∝ exp(−α R̂(μ_n)). In the mean-field limit the barycenter is replaced by the unweighted minimizer of F_ρ(ν) = (1/2)∫ W_2^2(μ,ν)dρ(μ), and the variance functional V(ρ) = (1/2)∫ W_2^2(μ,μ̄)dρ(μ) is what contracts. The geodesic interpolation property is what turns the two-step argument into a one-step contraction bound.

Load-bearing premise

In passing to the infinite-particle limit, the paper silently drops the risk-dependent Gibbs weights from the consensus point and uses an unweighted mean-field functional instead; if those weights matter in the limit, the mean-field model does not describe the finite-particle CBO actually run in the experiments.

What would settle it

Run the finite-particle OT-CBO scheme with a large number of particles on a benchmark where particle risks differ strongly; compute the empirical consensus barycenter with Gibbs weights and compare it to the unweighted barycenter of the particle distribution. If the two diverge as α increases or as N grows, the mean-field model (which uses the unweighted functional) is falsified. Alternatively, simulate the time-discrete mean-field dynamics directly and check whether the measured variance violates V(ρ^{k+1}) ≤ (1−Δt)^2 V(ρ^k) for any Δt ∈ (0,1].

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the mean-field limit is valid, training an infinitely wide two-layer network reduces to a deterministic gradient flow on the space of measures, giving a global, gradient-free training model.
  • The variance bound V(ρ^{k+1}) ≤ (1−Δt)^2 V(ρ^k) implies exponential decay of particle spread in the time-discrete mean-field dynamics, i.e., consensus.
  • On the MNIST test case, the hybrid CBO+Adam method reaches lower risk than CBO and is more stable than Adam, suggesting a practical recipe for stabilizing gradient training.
  • Multi-task CBO can optimize many related empirical risks with one shared particle ensemble, reducing memory overhead compared to per-task ensembles.
  • The optimal-transport CBO scheme trains neural networks of increasing width M, and wider networks reach lower empirical risk.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the Gibbs weights do not vanish in the N→∞ limit, the true mean-field barycenter would be a weighted average over risk-dependent weights, not the unweighted F_ρ minimizer; a natural test is to derive the limit retaining β(μ) and compare consensus points at finite N.
  • Because the OT dynamics are deterministic (no Brownian motion), the clean variance contraction likely holds only for the noise-free scheme; the paper's numerical noise schedule may reintroduce variance growth, so the bound should not be expected for the noisy finite-particle scheme without modification.
  • The hybrid method's stability suggests formalizing the consensus point as a distributed filter that dampens simultaneous exploding gradients; a testable extension is to analyze the Jacobian of the hybrid update to quantify this damping.
  • The same optimal-transport reformulation could apply to other consensus-based sampling or estimation algorithms, where a variance contraction on the Wasserstein-over-Wasserstein space might yield new convergence guarantees.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies Consensus-Based Optimization (CBO) for training two-layer neural networks. It proposes a hybrid CBO-Adam scheme and a multi-task CBO variant, reports numerical experiments on sine regression, MNIST classification, and multi-task sine regression, and then develops an optimal-transport reformulation of CBO. The theoretical part introduces a Wasserstein-over-Wasserstein time-discrete dynamics and proves a variance contraction inequality (Prop. 3). The paper claims that this dynamics is the N → ∞ mean-field limit of the OT-CBO scheme and that the variance decreases monotonically toward consensus.

Significance. The numerical contributions — the hybrid CBO-Adam scheme and the multi-task particle-recycling idea — are potentially useful and are supported by reproducible experiments with a referenced code repository. The optimal-transport reformulation of CBO for neural-network measures is an interesting direction. However, the central theoretical claim, namely that Eq. (64) is the N → ∞ mean-field limit of the finite-particle CBO dynamics and that Prop. 3 therefore describes CBO, is not established. Because the analytical mean-field result is a main advertised contribution, the gap is load-bearing.

major comments (3)
  1. [§5.2, Eq. (61) vs. Eq. (32)] The finite-particle barycenter in Eq. (32) is a risk-weighted Fréchet mean with Gibbs weights β(μ_n^k)=exp(-α R(μ_n^k))/Σ exp(-α R(μ_n^k)). The mean-field functional F_ρ(ν) in Eq. (61) is unweighted. No argument is provided that the Gibbs weights converge to constants, factorize in the limit, or can be absorbed into ρ. In classical CBO mean-field limits on R^d the weights persist in the limiting consensus point (a Gibbs-weighted average), and here they should enter a weighted barycenter problem. Since Prop. 3's proof uses the unweighted functional F_{ρ^{k+1}}(·), it proves contraction for a different process, not for the CBO scheme defined by Eq. (32). Example 4 (Eqs. 53–55) uses the weighted barycenter, so its numerics do not validate the unweighted mean-field model either.
  2. [§5.2, Eq. (64)] The paper labels Eq. (64) the 'time-discrete mean-field limit', but no rigorous N → ∞ derivation is given. There is no statement of the convergence of the empirical particle law to ρ, no propagation-of-chaos argument, and no discussion of how the finite-particle optimal maps T_n in Eq. (57), which depend on the full ensemble through the barycenter ¯μ^k of Eq. (32), converge to the single map in Eq. (62). Moreover, Assumption 2 requires each measure μ ∈ X to be absolutely continuous, whereas the implemented scheme and Example 4 use atomic empirical measures as in Eq. (49). Thus Eq. (64) is best read as a definition of a new deterministic process, not as a proven limit of the CBO dynamics studied in the paper.
  3. [§5.2, Eq. (69)] The proof of Prop. 3 uses the identity W_2^2(Ψ_{Δt}(μ), ¯μ^k) = (1-Δt)^2 W_2^2(μ, ¯μ^k), which requires Ψ_{Δt} to be the constant-speed geodesic from μ to the current barycenter ¯μ^k. In the finite-particle dynamics, the barycenter is recomputed at each iteration and depends on all particles and on the Gibbs weights; Ψ_{Δt} in Eq. (62) is not indexed by k and its target measure is ambiguous. Even granting the absolute-continuity assumption, the contraction inequality is an internal-consistency statement for the unweighted one-step process, not for the coupled weighted iterative scheme.
minor comments (5)
  1. [§3.3, Eq. (21)] The optimization dimension is written as R^{M(d+2)} here, while d_o = M(d+2) is used earlier. Please use the notation consistently.
  2. [Example 4, Eqs. (53)–(55)] The numerical scheme adds a manually scheduled noise term and a drift parameter λ, while the mean-field model Eq. (64) is deterministic and contains no λ. The connection between the analyzed dynamics and the implemented dynamics should be clarified; otherwise the experiment does not directly validate Eq. (64).
  3. [Proposition 2, Eq. (48)] The denominator Σ_n β(ˆμ_n) Σ_i (π*_n)_{j,i} is omitted in the displayed expression. Since β is normalized the final formula is correct, but the intermediate step should be shown.
  4. [§4.2 and §4.3] Typos: 'hybird' in §4.2, 'wether' and 'stratety' in §4.3, 'predicition' in §3.1, 'fullfills' in Prop. 1, and 'differet' in the caption of Fig. 8.
  5. [§5.2, Eqs. (62)–(63)] The notation ¯μ is used both for the barycenter of ρ in Assumption 1 and for the target measure satisfying ¯μ = T#μ. This ambiguity should be resolved, and the dependence on the time index k should be made explicit.

Circularity Check

2 steps flagged

Variance contraction is a definitional consequence of the geodesic update, and the N→∞ limit drops CBO's Gibbs weights.

specific steps
  1. self definitional [§5.2, Assumption 1 / Eqs. (61)–(66), (69)]
    "Let the variance of the measure ρ^k be given by V(ρ^k)=1/2∫_X W_2^2(µ,¯µ^k)dρ^k(µ). ... The map Ψ_Δt defines a constant-speed Wasserstein geodesic between μ and ¯μ ... W_2^2(Ψ_Δt(μ),¯μ^k)=(1−Δt)^2 W_2^2(μ,¯μ^k)."

    Ψ_Δt is defined to move every μ along a geodesic to the same barycenter ¯μ (Eqs. 62–63), and V is defined as the squared W_2 distance to that same ¯μ (Eq. 65). Hence Prop. 3's inequality is an identity: the dynamics and the observable share the same target, so the variance contraction is contained in the definitions. It does not use the Gibbs weights β(µ_n^k)=exp(−αR(µ_n^k))/Σ exp(−αR(µ_n^k)) of Eq. (32), and it is not a statement about convergence to an R-minimizer.

  2. other [§5.2, Eq. (61) vs §5.1 Eq. (32); Eq. (64)]
    "Fρ(ν)=1/2∫_X W_2^2(µ,ν)dρ(µ). ... β(µ^k_n)=exp(−αR(µ^k_n))/Σ exp(−αR(µ^k_n)). ... This is the time-discrete mean-field limit."

    The N→∞ limit is asserted by replacing the weighted finite-N barycenter (Eq. 32) with the unweighted functional (Eq. 61) and then labeling Eq. (64) the mean-field limit. No argument is given that the risk weights factor or vanish. Consequently Prop. 3's consensus result is a property of the unweighted process introduced as an assumption, not of the CBO dynamics whose selection mechanism is the Gibbs weighting; the central claim reduces to an assumption rather than following from the particle model.

full rationale

Prop. 3 is a valid inequality for the unweighted Wasserstein barycenter process, so this is not a case of a false theorem or a pure self-citation chain. The circularity is partial: the mean-field dynamics are defined by geodesic interpolation toward the same unweighted barycenter that defines the variance, so the monotone decay of V is an immediate identity (Eq. 69) rather than an emergent property of CBO. The step from the finite-particle weighted barycenter Eq. (32) to the unweighted Assumption 1/Eq. (61) is asserted without proof; the risk-dependent Gibbs weights, which are the optimization mechanism of CBO, play no role in the contraction. Example 4 tests the finite-particle weighted OT-CBO, not the ρ-equation, so it does not supply external validation of the mean-field model. The self-citations to Refs. [7] and [9] are not load-bearing. Because the central theoretical claim about consensus is largely built into the definitions, but the paper also contains independent numerical examples and a valid (if definitional) contraction proof, the circularity score is moderate.

Axiom & Free-Parameter Ledger

6 free parameters · 6 axioms · 0 invented entities

The central theoretical claim rests on standard Wasserstein/OT facts plus two domain assumptions (existence of the barycenter and absolute continuity). The most fragile premise is the implicit omission of Gibbs weights in the mean-field limit, which is not derived or justified. The algorithm hyperparameters α, σ~, λ, Δt, and γ are hand-chosen, and the artificial noise schedule in Example 4 is an additional ad hoc component.

free parameters (6)
  • inverse temperature α = 1e4–1e5, annealed to 1e7
    Controls the concentration of the Gibbs weights used to form the consensus point; set by hand and annealed every 100 epochs in the experiments.
  • diffusion coefficient σ~ = sqrt(1.2), sqrt(1.4), sqrt(1.6), sqrt(1.8); manually decayed in Example 4
    Noise level is chosen per experiment; in Example 4 it is reduced every 100 iterations to compensate for the missing multiplicative noise in the OT formulation.
  • drift parameter λ = 1
    Set to 1 in all experiments; no sensitivity analysis is provided.
  • time step Δt = 0.1 or 0.2
    Learning rate / time step is shared between optimizers and chosen by hand per experiment.
  • hybrid mixing weight γ = 0.7
    Controls the balance between the Adam step and the CBO step in the hybrid method; a single value is used with no study of its effect.
  • particle count N = 200, 1000
    Number of CBO particles is a computational choice that affects the quality of the stochastic approximation; chosen per experiment.
axioms (6)
  • domain assumption The barycenter μ̄ of ρ, the global minimizer of F_ρ(ν) = (1/2) ∫ W_2^2(μ,ν) dρ(μ), exists.
    Assumption 1 in Section 5.2; required to define the consensus point in the mean-field regime.
  • domain assumption All relevant measures μ and μ̄ are absolutely continuous with respect to the Lebesgue measure.
    Assumption 2 in Section 5.2; needed for the existence of optimal transport maps and the Wasserstein geodesic identity used in Prop. 3.
  • ad hoc to paper The risk-dependent Gibbs weights β(μ_n) present in the finite barycenter Eq. (32) can be omitted in the N → ∞ limit.
    The mean-field functional Eq. (61) has no exp(−α R(μ)) factor, while the finite-particle barycenter Eq. (32) does. No convergence argument is supplied.
  • domain assumption Global convergence of CBO requires the global minimizer θ* to lie in the support of the initial particle distribution ρ_0.
    Stated in Section 3.3 and used to justify the multi-task recycling strategy in Section 3.4.2.
  • standard math The Laplace principle holds, so the Gibbs measure concentrates exponentially around the global minimizer as α → ∞.
    Invoked in Eq. (24) to argue that the consensus point approaches the global minimizer.
  • domain assumption There is no Brownian motion on P_2(R^{d+2}), so the OT-CBO dynamics are deterministic and noise is added artificially after discretization.
    Stated in Section 5.1; this is an acknowledged modeling limitation that also introduces hand-scheduled noise parameters.

reviewed 2026-08-03 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Mean-Field Model for Two-Layer Neural Networks Trained with Consensus-Based Optimization." pith.science (2026). https://pith.science/paper/H3C7LZEO

@misc{pith2026251121466,
  author       = {Pith},
  title        = {Pith review of: Mean-Field Model for Two-Layer Neural Networks Trained with Consensus-Based Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/H3C7LZEO}},
  note         = {Machine review of arXiv:2511.21466}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

We study Consensus-Based Optimization (CBO) for two-layer neural network training. We compare the performance of CBO against Adam on two test cases and demonstrate how a hybrid approach, combining CBO with Adam, provides faster convergence than CBO. Additionally, in the context of multi-task learning, we recast CBO into a formulation that offers less memory overhead. The CBO method allows for a mean-field model formulation, which we couple with the mean-field model of the neural network. To this end, we first reformulate CBO within the optimal transport framework. As the number of particles tends to infinity, we lift the corresponding dynamics to the Wasserstein-over-Wasserstein space and show that the variance decreases monotonically. We confirm numerically that both mean-field models converge.

Figures

Figures reproduced from arXiv: 2511.21466 by Giovanni Samaey, Michael Herty, William De Deyn.

Figure 1
Figure 1. Figure 1: Conceptual illustration of single-task versus Multi-Task CBO. In single-task CBO a single [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 3
Figure 3. Figure 3: Approximations of the sine function sin(2πx) obtained with a two-layer neural network with M = 100 trained with CBO and Adam. The plot only contains a subset of the training dataset to improve the clarity. 4.2 Example 2: MNIST In the second experiment, we further investigate the applicability of CBO in training two-layer neural networks to classify the MNIST dataset [40]. This is a standard classification … view at source ↗
Figure 4
Figure 4. Figure 4: Empirical risk Rˆ(θ) as a function of training epochs for a two-layer neural network trained with Adam, CBO and the hybrid method (Adam + CBO) on the MNIST dataset. The instablility of the Adam method in [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗
Figure 6
Figure 6. Figure 6: Approximation of five different sine functions obtained with five different two-layer neural networks, each with M = 100. The neural networks are trained with Multi-Task CBO. 5 Mean-field Limits This section provides a study of mean-field limits, as indicated in [PITH_FULL_IMAGE:figures/full_fig_p011_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Diagram illustrating possible derivations of the mean-field limits. In this paper, we [PITH_FULL_IMAGE:figures/full_fig_p011_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Empirical risk Rˆ(µ) as a function of training epochs for four different neural networks, each with a different width M. Each neural network is represent by a measure and trained with the optimal transport formulation of CBO. The figure displays the mean empirical risks taken over 10 differet simulations. 5.2 The mean-field limit for infinitely many particles (N → ∞) In Subsection 5.1, we introduced the op… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Exploiting Structure with Anisotropic Consensus-Based Optimization

    math.OC 2026-07 accept novelty 6.0

    Anisotropic CBO's computational complexity depends exponentially only on the intrinsic dimension of an additively separable objective, not the ambient dimension, under aligned anisotropic noise.

Reference graph

Works this paper leans on

57 extracted references · 4 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Ambrosio, N

    L. Ambrosio, N. Gigli, and G. Savar´ e.Gradient Flows: In Metric Spaces and in the Space of Probability Measures. Lectures in Mathematics. ETH Z¨ urich. Birkh¨ auser Verlag, Basel, second edition, 2008

    2008

  2. [2]

    Bach and L

    F. Bach and L. Chizat. Gradient Descent on Infinitely Wide Neural Networks: Global Convergence and Generalization. InInternational Congress of Mathematicians, Saint-Petersbourg, Russia, July 2022

    2022

  3. [3]

    A. Barron. Universal approximation bounds for superpositions of a sigmoidal function.IEEE Transactions on Information Theory, 39(3):930–945, May 1993

    1993

  4. [4]

    Beiglb¨ ock, G

    M. Beiglb¨ ock, G. Pammer, and S. Schrott. A Brenier Theorem on (P 2(P2(Rd)), W2) and Appli- cations to Adapted Transport.arXiv:2509.03506, Sept. 2025

    arXiv 2025

  5. [5]

    Bengio, P

    Y. Bengio, P. Simard, and P. Frasconi. Learning long-term dependencies with gradient descent is difficult.IEEE Transactions on Neural Networks, 5(2):157–166, Mar. 1994

    1994

  6. [6]

    Bonet, C

    C. Bonet, C. Vauthier, and A. Korba. Flowing datasets with wasserstein over wasserstein gradient flows. In42nd International Conference on Machine Learning, 2025

    2025

  7. [7]

    Borghi, M

    G. Borghi, M. Herty, and L. Pareschi. A Consensus-Based Algorithm for Multi-Objective Opti- mization and Its Mean-Field Description. In2022 IEEE 61st Conference on Decision and Control (CDC), pages 4131–4136, Dec. 2022

    2022

  8. [8]

    Borghi, M

    G. Borghi, M. Herty, and L. Pareschi. Constrained Consensus-Based Optimization.SIAM Journal on Optimization, 33(1):211–236, Mar. 2023

    2023

  9. [9]

    Borghi, M

    G. Borghi, M. Herty, and A. Stavitskiy. Dynamics of Measure-Valued Agents in the Space of Probabilities.SIAM Journal on Mathematical Analysis, 57(5):5107–5134, Oct. 2025

    2025

  10. [10]

    B¨ ottcher and T

    L. B¨ ottcher and T. Asikis. Near-optimal control of dynamical systems with neural ordinary differential equations.Machine Learning: Science and Technology, 3(4):045004, Dec. 2022

    2022

  11. [11]

    L. Bottou. Online algorithms and stochastic approximations. In D. Saad, editor,Online Learning and Neural Networks. Cambridge University Press, Cambridge, UK, 1998

    1998

  12. [12]

    J. A. Carrillo, Y.-P. Choi, C. Totzeck, and O. Tse. An analytical framework for consensus-based global optimization method.Mathematical Models and Methods in Applied Sciences, 28(06):1037– 1066, June 2018

    2018

  13. [13]

    J. A. Carrillo, F. Hoffmann, A. M. Stuart, and U. Vaes. Consensus-based sampling.Studies in Applied Mathematics, 148(3):1069–1140, Apr. 2022. 17

    2022

  14. [14]

    J. A. Carrillo, S. Jin, L. Li, and Y. Zhu. A consensus-based global optimization method for high dimensional machine learning problems.ESAIM: Control, Optimisation and Calculus of Variations, 27:S5, 2021

    2021

  15. [15]

    J. A. Carrillo, C. Totzeck, and U. Vaes.Consensus-Based Optimization and Ensemble Kalman Inversion for Global Optimization Problems with Constraints, volume 40, pages 195–230. WORLD SCIENTIFIC, Feb. 2023

    2023

  16. [16]

    R. Caruana. Multitask Learning.Machine Learning, 28(1):41–75, July 1997

    1997

  17. [17]

    R. T. Q. Chen, Y. Rubanova, J. Bettencourt, and D. Duvenaud. Neural ordinary differential equations. InProceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS’18, page 6572–6583, Red Hook, NY, USA, 2018. Curran Associates Inc

    2018

  18. [18]

    Z. Chen, V. Badrinarayanan, C.-Y. Lee, and A. Rabinovich. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. InProceedings of the 35th International Conference on Machine Learning (ICML), pages 794–803. PMLR, 2018

    2018

  19. [19]

    L. Chizat. Mean-field langevin dynamics: Exponential convergence and annealing.Transactions on Machine Learning Research, Aug. 2022

    2022

  20. [20]

    Chizat and F

    L. Chizat and F. Bach. Implicit bias of gradient descent for wide two-layer neural networks trained with the logistic loss. InProceedings of the Thirty-Third Conference on Learning Theory (COLT), volume 125 ofProceedings of Machine Learning Research, pages 1305–1338. PMLR, July 2020

    2020

  21. [21]

    Cuturi and A

    M. Cuturi and A. Doucet. Fast Computation of Wasserstein Barycenters.Proceedings of the 31st International Conference on Machine Learning, 32(2):685–693, June 2014

    2014

  22. [22]

    W. De Deyn. Code repository for: Mean-Field Limits for Two-Layer Neural Net- works Trained with Consensus-Based Optimization.https://git.rwth-aachen.de/wdedeyn/ mean-field-limits-for-neural-networks, 2025. Accessed: 05-10-2025

    2025

  23. [23]

    Dembo and O

    A. Dembo and O. Zeitouni.Large Deviations Techniques and Applications. Number 38 in Stochas- tic Modelling and Applied Probability. Springer, Berlin New York, 2nd ed., corr. print edition, 2010

    2010

  24. [24]

    W. E, C. Ma, and L. Wu. The Barron Space and the Flow-Induced Function Spaces for Neural Network Models.Constructive Approximation, 55(1):369–406, Feb. 2022

    2022

  25. [25]

    Fornasier, T

    M. Fornasier, T. Klock, and K. Riedl. Convergence of Anisotropic Consensus-Based Optimization in Mean-Field Law. InApplications of Evolutionary Computation, volume 13224, pages 738–754. Springer International Publishing, Cham, 2022

    2022

  26. [26]

    Fornasier, T

    M. Fornasier, T. Klock, and K. Riedl. Consensus-Based Optimization Methods Converge Globally. SIAM Journal on Optimization, 34(3):2973–3004, Sept. 2024

    2024

  27. [27]

    Fornasier, L

    M. Fornasier, L. Pareschi, H. Huang, and P. S¨ unnen. Consensus-based optimization on the sphere: Convergence to global minimizers and machine learning.Journal of Machine Learning Research, 22(237):1–55, 2021

    2021

  28. [28]

    Gerber, F

    N. Gerber, F. Hoffmann, and U. Vaes. Mean-field limits for Consensus-Based Optimization and Sampling.ESAIM: Control, Optimisation and Calculus of Variations, July 2025

    2025

  29. [29]

    Glorot and Y

    X. Glorot and Y. Bengio. Understanding the difficulty of training deep feedforward neural net- works. InProceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pages 249–256. JMLR Workshop and Conference Proceedings, Mar. 2010

    2010

  30. [30]

    Goodfellow, Y

    I. Goodfellow, Y. Bengio, and A. Courville.Deep Learning. MIT Press, 2016

    2016

  31. [31]

    S.-Y. Ha, S. Jin, and D. Kim. Convergence of a first-order consensus-based global optimization algorithm.Mathematical Models and Methods in Applied Sciences, 30(12):2417–2444, Nov. 2020. 18

    2020

  32. [32]

    S.-Y. Ha, S. Jin, and D. Kim. Convergence and error estimates for time-discrete consensus-based optimization algorithms.Numerische Mathematik, 147(2):255–282, Feb. 2021

    2021

  33. [33]

    Hochreiter and J

    S. Hochreiter and J. Schmidhuber. Long Short-Term Memory.Neural Computation, 9(8):1735– 1780, Nov. 1997

    1997

  34. [34]

    Hornik, M

    K. Hornik, M. Stinchcombe, and H. White. Multilayer feedforward networks are universal ap- proximators.Neural Networks, 2(5):359–366, Jan. 1989

    1989

  35. [35]

    Huang and J

    H. Huang and J. Qiu. On the mean-field limit for the consensus-based optimization.Mathematical Methods in the Applied Sciences, 45(12):7814–7831, Aug. 2022

    2022

  36. [36]

    D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. InProceedings of the 3rd International Conference on Learning Representations (ICLR), 2015

    2015

  37. [37]

    M. Koß, S. Weissmann, and J. Zech. On the mean-field limit of consensus based methods. arXiv:2409.03518, Sept. 2024

    Pith/arXiv arXiv 2024

  38. [38]

    Krizhevsky, I

    A. Krizhevsky, I. Sutskever, and G. E. Hinton. ImageNet classification with deep convolutional neural networks.Communications of the ACM, 60(6):84–90, May 2017

    2017

  39. [39]

    LeCun, Y

    Y. LeCun, Y. Bengio, and G. Hinton. Deep learning.Nature, 521(7553):436–444, May 2015

    2015

  40. [40]

    LeCun, C

    Y. LeCun, C. Cortes, and C. J. Burges. The MNIST Database of handwritten digits. Courant Institute, NYU, Google Labs, New York, Microsoft Research, Redmond, 1998

    1998

  41. [41]

    S. Mei, A. Montanari, and P.-M. Nguyen. A mean field view of the landscape of two-layer neural networks.Proceedings of the National Academy of Sciences, 115(33), Aug. 2018

    2018

  42. [42]

    K. P. Murphy.Machine Learning: A Probabilistic Perspective. Adaptive Computation and Machine Learning. MIT Press, Cambridge, 2014

    2014

  43. [43]

    Nguyen and H

    P.-M. Nguyen and H. T. Pham. A rigorous framework for the mean field limit of multilayer neural networks.Mathematical Statistics and Learning, 6(3):201–357, Oct. 2023

    2023

  44. [44]

    Nocedal and S

    J. Nocedal and S. J. Wright.Numerical Optimization. Springer Series in Operations Research and Financial Engineering. Springer New York, 2006

    2006

  45. [45]

    Pascanu, T

    R. Pascanu, T. Mikolov, and Y. Bengio. On the difficulty of training recurrent neural networks. In S. Dasgupta and D. McAllester, editors,Proceedings of the 30th International Conference on Machine Learning, volume 28 ofProceedings of Machine Learning Research, pages 1310–1318, Atlanta, Georgia, USA, 17–19 Jun 2013. PMLR

    2013

  46. [46]

    Paszke, S

    A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style, high-performance deep learning library. InAdvances in Neural Information Processing Syst...

    2019

  47. [47]

    Philipp, D

    G. Philipp, D. Song, and J. G. Carbonell. The exploding gradient problem demystified - definition, prevalence, impact, origin, tradeoffs, and solutions.arXiv:1712.05577, 2017

    Pith/arXiv arXiv 2017

  48. [48]

    Pinnau, C

    R. Pinnau, C. Totzeck, O. Tse, and S. Martin. A consensus-based model for global optimization and its mean-field limit.Mathematical Models and Methods in Applied Sciences, 27(01):183–204, Jan. 2017

    2017

  49. [49]

    Pinzi and G

    A. Pinzi and G. Savar´ e. Totally convex functions,L 2-Optimal transport for laws of random measures, and solution to the Monge problem.arXiv:2509.01768, Sept. 2025

    Pith/arXiv arXiv 2025

  50. [50]

    Raissi, P

    M. Raissi, P. Perdikaris, and G. Karniadakis. Physics-informed neural networks: A deep learn- ing framework for solving forward and inverse problems involving nonlinear partial differential equations.Journal of Computational Physics, 378:686–707, Feb. 2019. 19

    2019

  51. [51]

    D. E. Rumelhart, G. E. Hinton, and R. J. Williams. Learning representations by back-propagating errors.Nature, 323(6088):533–536, Oct. 1986

    1986

  52. [52]

    Sirignano and K

    J. Sirignano and K. Spiliopoulos. Mean field analysis of neural networks: A central limit theorem. Stochastic Processes and their Applications, 130(3):1820–1852, Mar. 2020

    2020

  53. [53]

    Sirignano and K

    J. Sirignano and K. Spiliopoulos. Mean Field Analysis of Neural Networks: A Law of Large Numbers.SIAM Journal on Applied Mathematics, 80(2):725–752, Jan. 2020

    2020

  54. [54]

    V. Vapnik. An overview of statistical learning theory.IEEE Transactions on Neural Networks, 10(5):988–999, 1999

    1999

  55. [55]

    J. Wei, F. Wu, and W. Bian. A Consensus-Based Optimization Method for Nonsmooth Nonconvex Programs with Approximated Gradient Descent Scheme.arXiv.2501.08906, Jan. 2025

    Pith/arXiv arXiv 2025

  56. [56]

    L. Yang, Z. Zhang, Y. Song, S. Hong, R. Xu, Y. Zhao, W. Zhang, B. Cui, and M.-H. Yang. Diffusion Models: A Comprehensive Survey of Methods and Applications.ACM Computing Surveys, 56(4):1–39, Apr. 2024

    2024

  57. [57]

    T. Yu, S. Kumar, A. Gupta, S. Levine, K. Hausman, and C. Finn. Gradient surgery for multi-task learning. InAdvances in Neural Information Processing Systems, 2020. 20

    2020

This paper was first reviewed by deepseek-v4-flash on August 3, 2026.