Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

The Butterfly Effect: Neural Network Training Trajectories Are Highly Sensitive to Initial Conditions

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A single perturbed weight early in training can send identical neural networks to different loss basins.

desk verdict A genuinely useful measurement framework for training sensitivity, with a real empirical counterexample about pre-training stability, but the deterministic-to-stochastic lower-bound claim is too strong as stated. read the letter →

arxiv 2506.13234 v1 pith:3F2ACKBU submitted 2025-06-16 cs.LG

Devin Kwok , Gül Sena Alt{i}ntaş , Colin Raffel , David Rolnick This is my paper
classification cs.LG
keywords butterflyeffectneuralnetworktrainingstabilitylossbarrierslinearmodeconnectivitypermutationalignmentfine-tuningmodelensemblingchaoticphase
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Neural network training is often blamed on stochastic gradient descent noise when two runs diverge. This paper argues that the training dynamics themselves are chaotic near the start: during an initial "chaotic" phase, perturbing as little as a single weight in an otherwise identically initialized network reliably sends the two copies to different loss basins, separated by a measurable loss barrier along the linear path between their final weights. The effect decays rapidly with training time, so the same perturbation applied after roughly the first half percent of training leaves the trajectories essentially together. These barriers survive permutation alignment and are visible in representational similarity, which means the two networks are genuinely different in function, not just in how neurons are ordered. If the claim holds, training instability is an inherent property of early optimization rather than a byproduct of noise, with direct consequences for fine-tuning, model merging, and ensemble diversity.

What carries the argument

The central instrument is the spawn-and-perturb experiment. A parent network is trained deterministically from $\theta_0$ to time $t$, split into two copies, one copy is given a perturbation $\varepsilon$ of controlled magnitude $\sigma$, and both copies are trained identically to time $T$; instability is the size of $d(\theta_T, \theta'_T)$ under four dissimilarity measures: $L^2$ parameter distance, the training-loss barrier $\sup_\alpha[\ell(\alpha\theta_T+(1-\alpha)\theta'_T)-\alpha\ell(\theta_T)-(1-\alpha)\ell(\theta'_T)]$, the same barrier after weight matching over neuron permutations, and Angular CKA between penultimate-layer activations. The load-bearing theoretical object is the deterministic training map $T^T(\theta_0; \xi)$ with fixed noise $\xi$, together with the claim that divergence after a single controlled perturbation lower-bounds divergence after independent stochastic noise: $E[d(T(\theta,\xi), T(\theta+\varepsilon,\xi'))]\ge E[d(T(\theta,\xi), T(\theta+\varepsilon,\xi))]$. The perturbations are chosen either in the direction of a batch gradient or as initialization-scaled Gaussian noise, so that stability is probed along both training-relevant and generic directions.

What would settle it

A direct test is to run the spawn-and-perturb protocol at initialization with a single perturbed weight, but after the perturbation train the two copies with independently sampled minibatches rather than identical ones; if the resulting barrier at convergence is not at least as large as the barrier from the identical-noise run, the claimed lower bound fails. A second test is to search even one architecture, task, or optimizer for which the same single-weight perturbation at $t=0$ produces a zero training-loss barrier, which would contradict the claim that early training is universally chaotic.

Watch

Extended reading notes

Core claim

Viewed as a dynamical system, training is an iterated map $T$ on the parameter space; with the random seed fixed, it is deterministic. The paper claims that this map exhibits a butterfly effect: a perturbation $\varepsilon$ applied at $\theta_0$, scaled to as little as one weight (roughly $10^{-4}$ relative to the initialization scale), grows under training so that the final network $\theta'_T$ has a large training-loss barrier from $\theta_T$, even though both copies saw identical batches, augmentations, and seeds. Applying the same perturbation after only $0.5\%$ of training yields a much smaller or negligible barrier, and later perturbations require orders-of-magnitude larger $\varepsilon$ to produce any barrier at all. Weight matching that searches over neuron permutations fails to reduce the barrier, and Angular CKA shows the diverged networks have different intermediate representations, so the divergence is functional rather than a symmetry artifact. The paper presents this as evidence that early training is chaotic in the dynamical-systems sense, not merely noisy.

Load-bearing premise

The load-bearing premise is that a single deterministic perturbation cannot make two identically trained copies diverge less than independent random batches would; if independent noise can sometimes push the two copies closer together, then the deterministic butterfly experiments would not bound the instability of ordinary stochastic training.

Editorial extensions

If this is right

  • A model that is stable to training noise can still be unstable to perturbations larger than that noise, so stability measured by seed agreement is not a guarantee of convergence to a single basin.
  • Hyperparameter choices that increase early stability, such as longer learning-rate warm-up and wider or shallower architectures, shift but do not eliminate the chaotic phase.
  • Fine-tuning stability is not a monotone function of pre-training: for several language-model checkpoints, the most pre-trained model is the most sensitive to perturbations during fine-tuning.
  • Because permutation alignment does not remove the barriers, the diverged networks are not equivalent up to neuron reordering, and permutation-based merging methods will not reconnect them by themselves.
  • In the vision settings tested, ensembling the original and perturbed networks yields accuracy gains that grow with representational dissimilarity, so deliberate perturbation is a candidate mechanism for ensemble diversity.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the lower-bound inequality holds in general, then every stochastic training run is at least as unpredictable as these deterministic experiments show, and reproduction efforts should focus on avoiding the chaotic phase (for example, by starting pre-trained or extending warm-up) rather than on controlling noise alone.
  • The rapid stabilization within 0.5% of training suggests a sharp dynamical transition; estimating the local Lyapunov exponent (the rate at which nearby trajectories separate) along the training path would locate this transition and could predict the minimum perturbation that survives to convergence.
  • The task dependence of stability (CIFAR-100 to CIFAR-10 versus the reverse, and vision versus language) hints that the controlling factor is the geometry of the loss landscape near the starting checkpoint, not the architecture class; a testable prediction is that stability correlates with local curvature at the checkpoint.
  • Since weight matching cannot remove the barriers, a natural follow-up the paper leaves open is whether richer alignment methods, such as layer-wise or activation-normalized permutations, could reconnect the divergent networks, or whether the functional differences are irreducible.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes a deterministic 'spawn-and-perturb' protocol to measure the sensitivity of neural network training trajectories to small perturbations of the weights. From an initial state, a network is trained to time t, one copy is perturbed by noise of controlled magnitude and direction, and both copies are then trained with identical noise; divergence is measured by L2 parameter distance, training-loss barriers, permutation-aligned barriers, and Angular CKA. Across ResNet, ViT, BERT, and OLMo experiments, the authors report that perturbations as small as a single weight applied at initialization reliably cause large barriers, that stability increases rapidly during early training, that certain hyperparameters (warm-up, wider architectures) improve stability, that pre-trained networks are generally more stable but longer language-model pre-training can reduce fine-tuning stability, and that barrier and L2 growth are not exponential. The paper interprets the results as evidence for a 'butterfly effect' in neural network training with implications for fine-tuning, model merging, and ensembles.

Significance. If the claims hold, this is a clean empirical characterization of trajectory sensitivity that complements prior noise-based spawning experiments and extends them to controlled perturbations of arbitrary size and timing. Strengths of the paper include the carefully designed protocol, the use of multiple complementary divergence measures, the breadth of architectures and tasks, reproducibility-oriented practices (deterministic training, released code), and the absence of fitted free parameters in the reported measurements. The central 'single-weight perturbation causes divergence' finding is striking and falsifiable. However, the paper's stated bridge from deterministic to stochastic training rests on an unproved and in general false inequality, and some headline claims (non-exponential growth, several comparative fine-tuning findings) lack statistical support. These issues are addressable but currently weaken the interpretation of otherwise valuable measurements.

major comments (3)
  1. [§3.1, displayed inequality after Eq. (1)] The inequality E[d(T(θ,ξ),T(θ+ε,ξ'))] ≥ E[d(T(θ,ξ),T(θ+ε,ξ))] is stated without proof as a general lower bound connecting deterministic same-noise experiments to stochastic training. It is not true for arbitrary stochastic maps and distance metrics: for scalar θ, T(θ,ξ)=ξθ with ξ,ξ' iid uniform ±1, and θ=−ε/2, the same-noise distance is always |ε| whereas the independent-noise expected distance is |ε|/2. Since this inequality is the only stated formal justification for claiming that the deterministic experiments 'lower bound' the instability of regular stochastic training, and since that claim is invoked in the abstract and in the practical fine-tuning/merging discussion, it must either be proved under explicit assumptions on T and d, or reformulated as an empirical observation restricted to the settings verified in Figure 10.
  2. [§5, Figure 6] The claim that 'neither barrier nor L2 increase exponentially over training' is based on visual inspection of curves that are averaged over perturbation magnitudes, with no statistical test or fitted growth model. This is contribution #5 of the paper and directly contradicts the linearized-dynamics expectation derived in Section B.4, so it needs quantitative support: for example, slopes of log(L2) and log(barrier) versus step with confidence intervals, or a comparison of exponential versus sub-exponential models, and a statement of whether the non-exponential pattern holds per perturbation magnitude or only after averaging.
  3. [Main figures, especially Figures 2, 4, 5, 6, 7] The main figures do not show error bars or other variance information even though Appendix B reports that evaluations are averaged over only three seeds (two for fine-tuning) per configuration. Several comparative claims in Section 4.4, such as the final MultiBERT checkpoint (2000k) having the largest QNLI barriers and the ImageNet-1k-fine-tuned ViT-Base having larger barriers than its in21k counterpart, depend on differences that may be within seed variance. Adding error bars or shaded confidence regions, or reporting effect sizes, is necessary for the reader to assess the reliability of these quantitative comparisons.
minor comments (6)
  1. [Abstract] The phrase 'even extremely small perturbations reliably causes otherwise identical training trajectories to diverge' has a subject-verb agreement error ('perturbations ... causes'); please change 'causes' to 'cause'.
  2. [Equation (5)] There is an extra closing parenthesis in 'dCKA(θT, θ′T)) = CKA [...]'; also, the notation dCKA is used for a distance while CKA is defined as arccos(...), so please make the composition explicit.
  3. [§3.1, Eq. (1)] The symbol T is overloaded: T denotes both the single-step stochastic map and the total number of training steps in T^T, and the displayed inequality uses a single T while referring to full training. Please use distinct notation for the number of steps and the map.
  4. [Figure 10] In the right panel, the y-axis label 'Noise L2' is not defined in the caption; please specify whether this is the L2 norm of the perturbation, the expected SGD noise magnitude, or something else.
  5. [Section C.3 and Section D.2] There are typos: 'corraborate' should be 'corroborate' in Section C.3, and 'tarnsformer' should be 'transformer' in Section D.2.
  6. [Tables 4 and 5] Barrier values are reported as '0.00 ± 0.00'; please state whether these are exactly zero or simply below the rounding threshold, and if so, give the threshold or report more significant digits.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claims are direct measurements, and the load-bearing inequality is an unproved but empirically checked assumption rather than a fitted input.

full rationale

The paper's central claims—that tiny deterministic perturbations early in training cause trajectories to diverge into different loss basins—are direct measurements of a fixed training map under controlled perturbations. No parameter is fitted to the claimed outcome and then presented as a prediction; the reported barriers, L2 distances, and CKA values are measurements. The linearized dynamical system in Section B.4 (Eqs. 7 and 8) is presented as a benchmark and is explicitly falsified by the experiments in Figure 6, so it is not an input used to force the result. The inequality in Section 3.1 connecting deterministic same-noise divergence to stochastic training instability is unproved and may be a correctness risk, as the skeptical note observes, but it is not circular: it is an asserted transfer claim that the paper checks empirically for one setting in Figure 10, and it is not derived from the conclusion. Self-citations such as Altıntaş et al. (2023), Sharma et al. (2024), and Lange et al. (2023) are used for prior context, permutation-alignment conventions, and the CKA implementation, respectively; none of these define the measured barriers or the divergence phenomenon itself. No equation in the paper defines its output in terms of its input, and no fitted parameter is relabeled as a prediction. The core empirical findings are therefore self-contained against external benchmarks and do not exhibit circular reasoning.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces no fitted free parameters: perturbation magnitudes and times are experiment inputs. It relies on four domain assumptions: the barrier-converse interpretation, the deterministic lower-bound inequality, deterministic reproducibility, and the validity of the chosen functional similarity measures. The inequality in Section 3.1 is the least standard and is not proven.

assumptions (4)
  • domain assumption Networks with zero barrier are in the same loss basin
    Footnote 2 ('Following Entezari et al. (2022); Neyshabur et al. (2020), we make the extra assumptions needed to assume the converse, i.e. that networks with zero barrier are in the same loss basin') is needed to interpret nonzero barriers as divergence into distinct basins.
  • ad hoc to paper E[d(T(θ,ξ),T(θ+ε,ξ'))] >= E[d(T(θ,ξ),T(θ+ε,ξ))] for independent noise ξ, ξ'
    Section 3.1 states this inequality without proof and uses it to claim deterministic perturbations lower-bound stochastic training instability. It is not generally true for arbitrary distance metrics and stochastic maps; Figure 10 provides empirical support only for one setting.
  • domain assumption Deterministic training with identical seeds reproduces identical networks
    Section B states 'training with no perturbations results in exactly identical networks as expected', an empirical verification rather than a mathematical guarantee, and the basis for the spawn-and-perturb design.
  • domain assumption Barriers and Angular CKA are valid measures of functional divergence
    Section 3.4 adopts loss barriers (Frankle et al. 2020; Neyshabur et al. 2020) and Angular CKA (Williams et al. 2021) as functional similarity measures; this relies on the standard interpretation of these metrics in prior literature.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Butterfly Effect: Neural Network Training Trajectories Are Highly Sensitive to Initial Conditions." pith.science (2026). https://pith.science/paper/3F2ACKBU

@misc{pith2026250613234,
  author       = {Pith},
  title        = {Pith review of: The Butterfly Effect: Neural Network Training Trajectories Are Highly Sensitive to Initial Conditions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3F2ACKBU}},
  note         = {Machine review of arXiv:2506.13234}
}
abstract

Neural network training is inherently sensitive to initialization and the randomness induced by stochastic gradient descent. However, it is unclear to what extent such effects lead to meaningfully different networks, either in terms of the models' weights or the underlying functions that were learned. In this work, we show that during the initial "chaotic" phase of training, even extremely small perturbations reliably causes otherwise identical training trajectories to diverge-an effect that diminishes rapidly over training time. We quantify this divergence through (i) $L^2$ distance between parameters, (ii) the loss barrier when interpolating between networks, (iii) $L^2$ and barrier between parameters after permutation alignment, and (iv) representational similarity between intermediate activations; revealing how perturbations across different hyperparameter or fine-tuning settings drive training trajectories toward distinct loss minima. Our findings provide insights into neural network training stability, with practical implications for fine-tuning, model merging, and diversity of model ensembles.

Figures

Figures reproduced from arXiv: 2506.13234 by the authors.

Figure 1
Figure 1. Left: illustration of the “butterfly effect”: a network θ0 is trained until time t and perturbed by ε early (εt) or later (εt∗ ) in training. Both copies are trained deterministically until T and their divergence is measured (purple loss basins). Right: barriers (training cross-entropy loss) at T versus perturbation magnitude (σ = 1 is the network’s initialization scale). A perturbation of as little as one weight (l… view at source ↗
Figure 2
Figure 2. Stability of ResNet-20 trained on CIFAR-10 with SGD (details in Section A). Loss barriers on training data at the end of training (y-axis) are plotted against perturbation magnitude (x-axis) and perturbation step (color indicates fraction of total training time). Left: barriers due to batch perturbation. Middle: batch perturbation barriers after accounting for permutations. Right: barriers due to Gaussian perturbati… view at source ↗
Figure 3
Figure 3. Left: same as [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (24 more)
Figure 4
Figure 4. Figure 4: Same as [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Stability of transfer learning on vision tasks: a ResNet-50 is pre-trained on CIFAR-100 and fine-tuned on CIFAR-10 (left) or vice versa (middle). Barriers (y-axis) are plotted against perturbation magnitudes (x-axis) for various pre-training durations and perturbation …
Figure 6
Figure 6. Figure 6: Evolution of barriers (left) and L 2 (right) over training for standard ResNet20 trained on CIFAR-10. Each colored line averages over all perturbation magnitudes, as they are nearly indis￾tinguishable. 5. L 2 Divergence and Barriers Barriers and L 2 divergence do not e…
Figure 8
Figure 8. Figure 8: Train loss barriers against test accuracy (left) and training cross entropy loss (right) of the perturbed model at the end of training. A.2. Finetuning Experiments CIFAR pre-training We pre-trained two ResNet-50 models with different initializations on on both CIFAR-10…
Figure 9
Figure 9. Figure 9: Train loss barriers before and after permutations when perturbing only normalization layers. Results are shown for batch (left) and Gaussian (right) perturbations on ResNet-20 trained with SGD (momentum, no weight decay), using a learning rate of 0.1, 2% warm-up, and a…
Figure 10
Figure 10. Figure 10: Left: Comparison of training loss barriers between our butterfly setting and Frankle et al.’s spawning setting. In the spawning setting, each network is trained with different non-determinism after perturbation step t, while our method applies a single perturbation. A…
Figure 11
Figure 11. Figure 11: Left: absolute L 2 norm of the noise as a function of the fraction of perturbed weights. Right: train loss barriers as a function of the fraction of perturbed weights. C.3. Additional Hyperparameter Settings 10 3 10 1 0 10 20 C E ResNet-20 (Adam) - Batch 10 3 10 1 0 1…
Figure 12
Figure 12. Figure 12: Same as [PITH_FULL_IMAGE:figures/full_fig_p020_12.png]
Figure 13
Figure 13. Figure 13: Representational similarity distance measured via Angular CKA for ResNet-20 with weight decay (left) or 10x warm-up (right). 0.2 0.4 Train Angular CKA 10 2 10 1 10 0 C E ResNet-8 (10x Warm-up) - Batch 0.25 0.50 0.75 1.00 Train Angular CKA 10 2 10 0 C E ResNet-20 (Weig…
Figure 14
Figure 14. Figure 14: Train loss barrier versus representational similarity distance on ResNet-20 with weight decay (left), ResNet-8 with 10x warm-up (middle) with perturbation time indicated by color, and including the standard setting from [PITH_FULL_IMAGE:figures/full_fig_p021_14.png]
Figure 15
Figure 15. Figure 15: Test accuracy of an ensemble of the original and perturbed models after training, versus representational similarity distance, on ResNet-20 with weight decay (left), ResNet-8 with 10x warm-up (middle) with perturbation time indicated by color, nd including the standar…
Figure 16
Figure 16. Figure 16: suggests a weak correlation, where the number of fixed points is inversely proportional to both the barrier heights and the L 2 distance between the networks before alignment in the ResNet settings. Unfortunately, this observation does not extend to our ViT or BERT se…
Figure 17
Figure 17. Figure 17: Same as [PITH_FULL_IMAGE:figures/full_fig_p023_17.png]
Figure 18
Figure 18. Figure 18: Same as [PITH_FULL_IMAGE:figures/full_fig_p023_18.png]
Figure 19
Figure 19. Figure 19: Same as Figures 5, 17 and 18 (left panel of each), but with randomly initialized ResNet-50 instead of networks pre-trained on CIFAR-100. Train loss barriers are reported for batch perturbations during fine-tuning on CIFAR-10 using the recipe from Section A.2. 23 [PIT…
Figure 20
Figure 20. Figure 20: we observe that earlier and larger perturbations cause more pronounced barriers. Interestingly, the google/vit-base￾patch16-224 variant, which underwent additional fine-tuning on ImageNet-1K after its initial ImageNet-21K pre-training, effectively represents a longer …
Figure 21
Figure 21. Figure 21: Left: representational similarity distance measured via Angular CKA for ViT-base. Middle: test ensembling accuracy against Angular CKA. Right: training loss barrier against Angular CKA. between barriers and L 2 divergence ( [PITH_FULL_IMAGE:figures/full_fig_p026_21.png]
Figure 22
Figure 22. Figure 22: Same as [PITH_FULL_IMAGE:figures/full_fig_p027_22.png]
Figure 23
Figure 23. Figure 23: Top: representational similarity distance measured via Angular CKA for MultiBERT on MRPC (left) and QNLI (right). Bottom: barriers vs. angular CKA on MRPC (left) and QNLI (right). 27 [PITH_FULL_IMAGE:figures/full_fig_p027_23.png]
Figure 24
Figure 24. Figure 24: Stability of fine-tuning OLMo-1B on GSM8K mathematical reasoning tasks. We fine-tune OLMo-1B checkpoints from different pre-training stages (early, middle, and final checkpoints) on GSM8K with batch perturbations applied at various training steps. Left: Loss barriers …
Figure 25
Figure 25. Figure 25: Train loss barriers vs. L 2 distance between the perturbed and original models at the end of training for fine-tuning vision models: ResNet-50 transferring from CIFAR-100 to CIFAR-10 (left), ResNet-50 transferring from CIFAR-10 to CIFAR-100 (middle), and ViT-base fine…
Figure 26
Figure 26. Figure 26: Train loss barriers vs. L 2 distance between the perturbed and original models at the end of training for MRPC, RTE, SST-2, and COLA. 28 [PITH_FULL_IMAGE:figures/full_fig_p028_26.png]
Figure 27
Figure 27. Figure 27: Same data as [PITH_FULL_IMAGE:figures/full_fig_p029_27.png]
Figure 28
Figure 28. Figure 28: Same as [PITH_FULL_IMAGE:figures/full_fig_p029_28.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Quasi-Monte Carlo Initialization for Meta-Reinforcement Learning

    cs.LG 2026-07 conditional novelty 5.0 of 10

    Quasi-Monte Carlo weight priors give a small convergence boost over orthogonal initialization on similar continuous-control tasks, but hurt on dissimilar tasks.

Reference graph

Works this paper leans on

73 extracted references · 56 canonical work pages · cited by 1 Pith paper

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Layer-wise linear mode connectivity

    Adilova, L., Andriushchenko, M., Kamp, M., Fischer, A., and Jaggi, M. Layer-wise linear mode connectivity. In International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=LfmZh91tDI

    work page 2024

  3. [3]

    Git re-basin: Merging models modulo permutation symmetries

    Ainsworth, S., Hayase, J., and Srinivasa, S. Git re-basin: Merging models modulo permutation symmetries. In International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=CQsmMYmlP5T

    work page 2023

  4. [4]

    S., Bachmann, G., Noci, L., and Hofmann, T

    Alt nta s , G. S., Bachmann, G., Noci, L., and Hofmann, T. Disentangling linear mode connectivity. In UniReps: the First Workshop on Unifying Representations in Neural Models, 2023. URL https://openreview.net/forum?id=PbvPwiySXz

    work page 2023

  5. [5]

    L., Kiros, J

    Ba, J. L., Kiros, J. R., and Hinton, G. E. Layer normalization, 2016. URL https://arxiv.org/abs/1607.06450

    arXiv 2016

  6. [6]

    V., Akram, Y., Zucchet, N., Aitchison, L., and Steger, A

    Benzing, F., Schug, S., Meier, R., Oswald, J. V., Akram, Y., Zucchet, N., Aitchison, L., and Steger, A. Random initialisations performing above chance and how to find them. In OPT 2022: Optimization for Machine Learning (NeurIPS 2022 Workshop), 2022. URL https://openreview.net/forum?id=HS5zuN_qFI

    work page 2022

  7. [7]

    Shift-Curvature, SGD, and Generalization

    Bradley, A. V., Gomez-Uribe, C. A., and Vuyyuru, M. R. Shift-curvature, SGD , and generalization, July 2022. URL http://arxiv.org/abs/2108.09507. arXiv:2108.09507 [stat]

    work page Pith review arXiv 2022

  8. [9]

    Training verifiers to solve math word problems

    Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems. CoRR, abs/2110.14168, 2021 b . URL https://arxiv.org/abs/2110.14168

    arXiv 2021

Show all 73 references
  1. [10]

    M., Kaur, S., Li, Y., Kolter, J

    Cohen, J. M., Kaur, S., Li, Y., Kolter, J. Z., and Talwalkar, A. Gradient descent on neural networks typically occurs at the edge of stability, 2021. URL http://arxiv.org/abs/2103.00065

    2021 arXiv

  2. [11]

    Why do we need weight decay in modern deep learning?, 2024

    D'Angelo, F., Andriushchenko, M., Varre, A., and Flammarion, N. Why do we need weight decay in modern deep learning?, 2024. URL http://arxiv.org/abs/2310.04415. arXiv:2310.04415 [cs] version: 2

    2024 arXiv

  3. [12]

    BERT : Pre-training of deep bidirectional transformers for language understanding

    Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. BERT : Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technol...

    2019 doi

  4. [13]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on...

    2021

  5. [14]

    Essentially No Barriers in Neural Network Energy Landscape

    Draxler, F., Veschgini, K., Salmhofer, M., and Hamprecht, F. Essentially No Barriers in Neural Network Energy Landscape . In Proceedings of the 35th International Conference on Machine Learning , pp.\ 1309--1318. PMLR, July 2018. URL https://proceedings.mlr.press/v80/draxler18...

    2018

  6. [15]

    The role of permutation invariance in linear mode connectivity of neural networks

    Entezari, R., Sedghi, H., Saukh, O., and Neyshabur, B. The role of permutation invariance in linear mode connectivity of neural networks. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=dNigytemkL

    2022

  7. [16]

    Deep ensembles: A loss landscape perspective, 2019

    Fort, S., Hu, H., and Lakshminarayanan, B. Deep ensembles: A loss landscape perspective, 2019

    2019

  8. [17]

    K., Paul, M., Kharaghani, S., Roy, D

    Fort, S., Dziugaite, G. K., Paul, M., Kharaghani, S., Roy, D. M., and Ganguli, S. Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the neural tangent kernel. Advances in Neural Information Processing Systems, 33: 0 5...

    2020

  9. [18]

    and Carbin, M

    Frankle, J. and Carbin, M. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=rJl-b3RcF7

    2019

  10. [19]

    K., Roy, D., and Carbin, M

    Frankle, J., Dziugaite, G. K., Roy, D., and Carbin, M. Linear mode connectivity and the lottery ticket hypothesis. In Proceedings of the 37th International Conference on Machine Learning, volume 119, pp.\ 3259--3269. PMLR, 2020 a . URL https://proceedings.mlr.press/v119/frankl...

    2020

  11. [20]

    J., and Morcos, A

    Frankle, J., Schwab, D. J., and Morcos, A. S. The early phase of neural network training. In International Conference on Learning Representations, 2020 b . URL https://openreview.net/forum?id=Hkl1iRNFwS

    2020

  12. [21]

    P., and Wilson, A

    Garipov, T., Izmailov, P., Podoprikhin, D., Vetrov, D. P., and Wilson, A. G. Loss Surfaces , Mode Connectivity , and Fast Ensembling of DNNs . In Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018. URL https://proceedings.neurips.cc/pa...

    2018

  13. [22]

    J., Vinyals, O., and Saxe, A

    Goodfellow, I. J., Vinyals, O., and Saxe, A. M. Qualitatively characterizing neural network optimization problems, 2015. URL https://arxiv.org/abs/1412.6544

    2015 arXiv

  14. [23]

    H., Ivison, H., Magnusson, I., Wang, Y., Arora, S., Atkinson, D., Authur, R., Chandu, K

    Groeneveld, D., Beltagy, I., Walsh, P., Bhagia, A., Kinney, R., Tafjord, O., Jha, A. H., Ivison, H., Magnusson, I., Wang, Y., Arora, S., Atkinson, D., Authur, R., Chandu, K. R., Cohan, A., Dumas, J., Elazar, Y., Gu, Y., Hessel, J., Khot, T., Merrill, W., Morrison, J., Muennigh...

    2024 arXiv

  15. [24]

    Delving deep into rectifiers: Surpassing human-level performance on imagenet classification

    He, K., Zhang, X., Ren, S., and Sun, J. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), December 2015. URL https://openaccess.thecvf.com/content_iccv_201...

    2015

  16. [25]

    Deep residual learning for image recognition

    He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016

    2016

  17. [26]

    E., and Weinberger, K

    Huang, G., Li, Y., Pleiss, G., Liu, Z., Hopcroft, J. E., and Weinberger, K. Q. Snapshot ensembles: Train 1, get M for free. In International Conference on Learning Representations, 2017. URL https://openreview.net/forum?id=BJYwwY9ll

    2017

  18. [27]

    T., Wortsman, M., Schmidt, L., Hajishirzi, H., and Farhadi, A

    Ilharco, G., Ribeiro, M. T., Wortsman, M., Schmidt, L., Hajishirzi, H., and Farhadi, A. Editing models with task arithmetic. In International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=6t0Kwf8-jrj

    2023

  19. [28]

    Analysis of linear mode connectivity via permutation-based weight matching

    Ito, A., Yamada, M., and Kumagai, A. Analysis of linear mode connectivity via permutation-based weight matching. In International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=lYRkGZZi9D

    2025

  20. [29]

    Maximal initial learning rates in deep R e LU networks

    Iyer, G., Hanin, B., and Rolnick, D. Maximal initial learning rates in deep R e LU networks. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedi...

    2023

  21. [30]

    Averaging weights leads to wider optima and better generalization

    Izmailov, P., Podoprikhin, D., Garipov, T., Vetrov, D., and Wilson, A. Averaging weights leads to wider optima and better generalization. In 34th Conference on Uncertainty in Artificial Intelligence 2018, UAI 2018, pp.\ 876--885, 2018

    2018

  22. [31]

    Neural tangent kernel: Convergence and generalization in neural networks

    Jacot, A., Gabriel, F., and Hongler, C. Neural tangent kernel: Convergence and generalization in neural networks. In Advances in Neural Information Processing Systems, volume 31, 2018. URL https://proceedings.neurips.cc/paper_files/paper/2018/file/5a4be1fa34e62bb8a6ec6b91d2462...

    2018

  23. [32]

    The break-even point on optimization trajectories of deep neural networks

    Jastrzebski, S., Szymczak, M., Fort, S., Arpit, D., Tabor, J., Cho*, K., and Geras*, K. The break-even point on optimization trajectories of deep neural networks. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=r1g87C4KwB

    2020

  24. [33]

    REPAIR : RE normalizing P ermuted A ctivations for I nterpolation R epair

    Jordan, K., Sedghi, H., Saukh, O., Entezari, R., and Neyshabur, B. REPAIR : RE normalizing P ermuted A ctivations for I nterpolation R epair. In International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=gU5sJ6ZggcX

    2023

  25. [34]

    Linear connectivity reveals generalization strategies

    Juneja, J., Bansal, R., Cho, K., Sedoc, J., and Saphra, N. Linear connectivity reveals generalization strategies. In International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=hY6M0JHl3uL

    2023

  26. [35]

    S., Mudigere, D., Nocedal, J., Smelyanskiy, M., and Tang, P

    Keskar, N. S., Mudigere, D., Nocedal, J., Smelyanskiy, M., and Tang, P. T. P. On large-batch training for deep learning: Generalization gap and sharp minima. In International Conference on Learning Representations, 2017. URL https://openreview.net/forum?id=H1oyRlYgg

    2017

  27. [36]

    Similarity of neural network representations revisited

    Kornblith, S., Norouzi, M., Lee, H., and Hinton, G. Similarity of neural network representations revisited. In Proceedings of the 36th International Conference on Machine Learning, volume 97, pp.\ 3519--3529. PMLR, 09--15 Jun 2019. URL https://proceedings.mlr.press/v97/kornbli...

    2019

  28. [37]

    Learning multiple layers of features from tiny images, 2009

    Krizhevsky, A. Learning multiple layers of features from tiny images, 2009. URL https://www.cs.toronto.edu/ kriz/learning-features-2009-TR.pdf

    2009

  29. [38]

    D., Kwok, D., Matelsky, J

    Lange, R. D., Kwok, D., Matelsky, J. K., Wang, X., Rolnick, D., and Kording, K. Deep networks as paths on the manifold of neural representations. In Proceedings of 2nd Annual Workshop on Topology, Algebra, and Geometry in Machine Learning (TAG-ML), volume 221, pp.\ 102--133. P...

    2023

  30. [39]

    Wide neural networks of any depth evolve as linear models under gradient descent

    Lee, J., Xiao, L., Schoenholz, S., Bahri, Y., Novak, R., Sohl-Dickstein, J., and Pennington, J. Wide neural networks of any depth evolve as linear models under gradient descent. In Advances in Neural Information Processing Systems, volume 32, 2019. URL https://proceedings.neur...

    2019

  31. [40]

    Exploring neural network landscapes: Star-shaped and geodesic connectivity

    Lin, Z., Li, P., and Wu, L. Exploring neural network landscapes: Star-shaped and geodesic connectivity. arXiv preprint arXiv:2404.06391, 2024

    2024 arXiv

  32. [41]

    How good is a single basin? In Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, volume 238, pp.\ 4015--4023

    Lion, K., Noci, L., Hofmann, T., and Bachmann, G. How good is a single basin? In Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, volume 238, pp.\ 4015--4023. PMLR, 02--04 May 2024. URL https://proceedings.mlr.press/v238/lion24a.html

    2024

  33. [42]

    and Hutter, F

    Loshchilov, I. and Hutter, F. Fixing weight decay regularization in adam. CoRR, abs/1711.05101, 2017. URL http://arxiv.org/abs/1711.05101

    2017 arXiv

  34. [43]

    S., Bigelow, E

    Lubana, E. S., Bigelow, E. J., Dick, R. P., Krueger, D., and Tanaka, H. Mechanistic mode connectivity. In Proceedings of the 40th International Conference on Machine Learning, volume 202, pp.\ 22965--23004. PMLR, 23--29 Jul 2023. URL https://proceedings.mlr.press/v202/lubana23a.html

    2023

  35. [44]

    I., Farajtabar, M., Gorur, D., Pascanu, R., and Ghasemzadeh, H

    Mirzadeh, S. I., Farajtabar, M., Gorur, D., Pascanu, R., and Ghasemzadeh, H. Linear mode connectivity in multitask and continual learning, 2021. URL https://openreview.net/forum?id=Fmg_fQYUejf

    2021

  36. [45]

    Equivariant deep weight space alignment

    Navon, A., Shamsian, A., Fetaya, E., Chechik, G., Dym, N., and Maron, H. Equivariant deep weight space alignment. In Proceedings of the 41st International Conference on Machine Learning, volume 235, pp.\ 37376--37395. PMLR, 21--27 Jul 2024. URL https://proceedings.mlr.press/v2...

    2024

  37. [46]

    What is being transferred in transfer learning? In Advances in Neural Information Processing Systems, volume 33, pp.\ 512--523, 2020

    Neyshabur, B., Sedghi, H., and Zhang, C. What is being transferred in transfer learning? In Advances in Neural Information Processing Systems, volume 33, pp.\ 512--523, 2020. URL https://proceedings.neurips.cc/paper_files/paper/2020/file/0607f4c705595b911a4f3e7a127b44e0-Paper.pdf

    2020

  38. [47]

    Do wide and deep networks learn the same things? Uncovering how neural network representations vary with width and depth

    Nguyen, T., Raghu, M., and Kornblith, S. Do wide and deep networks learn the same things? Uncovering how neural network representations vary with width and depth. International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=KJNcAkY8tY4

    2021

  39. [48]

    Pe \ n a, F. A. G., Medeiros, H. R., Dubail, T., Aminbeidokhti, M., Granger, E., and Pedersoli, M. Re-basin via implicit S inkhorn differentiation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 20237--20246, June 2023. URL https://o...

    2023

  40. [49]

    SVCCA : Singular vector canonical correlation analysis for deep learning dynamics and interpretability

    Raghu, M., Gilmer, J., Yosinski, J., and Sohl-Dickstein, J. SVCCA : Singular vector canonical correlation analysis for deep learning dynamics and interpretability. In Advances in Neural Information Processing Systems, volume 30, pp.\ 6076--6085, 2017. URL https://proceedings.n...

    2017

  41. [50]

    T., Bello-Rivas, J

    Redman, W. T., Bello-Rivas, J. M., Fonoberova, M., Mohr, R., Kevrekidis, Y., and Mezic, I. Identifying equivalent training dynamics. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=bOYVESX7PK

    2024

  42. [51]

    C., and Fei-Fei, L

    Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A. C., and Fei-Fei, L. ImageNet Large Scale Visual Recognition Challenge . International Journal of Computer Vision, 115 0 (3): 0 211--252, December 2...

    2015 doi

  43. [52]

    P., and Lobacheva, E

    Sadrtdinov, I., Pozdeev, D., Vetrov, D. P., and Lobacheva, E. To stay or not to stay in the pre-train basin: Insights on ensembling in transfer learning. In Advances in Neural Information Processing Systems, volume 36, pp.\ 15936--15964, 2023. URL https://proceedings.neurips.c...

    2023

  44. [53]

    Random teachers are good teachers

    Sarnthein, F., Bachmann, G., Anagnostidis, S., and Hofmann, T. Random teachers are good teachers. In Proceedings of the 40th International Conference on Machine Learning, volume 202, pp.\ 30022--30041. PMLR, 23--29 Jul 2023. URL https://proceedings.mlr.press/v202/sarnthein23a.html

    2023

  45. [54]

    The MultiBERTs : BERT reproductions for robustness analysis

    Sellam, T., Yadlowsky, S., Tenney, I., Wei, J., Saphra, N., D'Amour, A., Linzen, T., Bastings, J., Turc, I., Eisenstein, J., Das, D., and Pavlick, E. The MultiBERTs : BERT reproductions for robustness analysis. In International Conference on Learning Representations, 2022

    2022

  46. [55]

    M., Rolnick, D., and Dziugaite, G

    Sharma, E., Kwok, D., Denton, T., Roy, D. M., Rolnick, D., and Dziugaite, G. K. Simultaneous linear connectivity of neural networks modulo permutation. In Bifet, A., Davis, J., Krilavi c ius, T., Kull, M., Ntoutsi, E., and Z liobait \. e , I. (eds.), Machine Learning and Knowl...

    2024

  47. [56]

    Geometry of the loss landscape in overparameterized neural networks: Symmetries and invariances

    Simsek, B., Ged, F., Jacot, A., Spadaro, F., Hongler, C., Gerstner, W., and Brea, J. Geometry of the loss landscape in overparameterized neural networks: Symmetries and invariances. In Proceedings of the 38th International Conference on Machine Learning, volume 139, pp.\ 9722-...

    2021

  48. [57]

    Singh, S. P. and Jaggi, M. Model fusion via optimal transport. In Advances in Neural Information Processing Systems, volume 33, pp.\ 22045--22055, 2020. URL https://proceedings.neurips.cc/paper/2020/hash/fb2697869f56484404c8ceee2985b01d-Abstract.html

    2020

  49. [58]

    P., Adilova, L., Kamp, M., Fischer, A., Schölkopf, B., Tübingen, M.-I., Hofmann, T., and Ch, E

    Singh, S. P., Adilova, L., Kamp, M., Fischer, A., Schölkopf, B., Tübingen, M.-I., Hofmann, T., and Ch, E. Landscaping Linear Mode Connectivity . In High-dimensional Learning Dynamics 2024: The Emergence of Structure and Reasoning, 2024

    2024

  50. [59]

    Smith, L. N. A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decay, April 2018. URL http://arxiv.org/abs/1803.09820. arXiv:1803.09820 [cs]

    2018 arXiv

  51. [60]

    Smith, S. L. and Le, Q. V. A B ayesian perspective on generalization and stochastic gradient descent. In International Conference on Learning Representations, 2018. URL https://openreview.net/forum?id=BJij4yg0Z

    2018

  52. [61]

    The boundary of neural network trainability is fractal, 2024

    Sohl-Dickstein, J. The boundary of neural network trainability is fractal, 2024. URL https://arxiv.org/abs/2402.06184

    2024 arXiv

  53. [62]

    Sonthalia, A., Rubinstein, A., Abbasnejad, E., and Oh, S. J. Do deep neural network solutions form a star domain?, 2024. URL https://arxiv.org/abs/2403.07968

    2024 arXiv

  54. [63]

    M., Goyal, S., Wen, K., Kumar, T., Yue, X., Malladi, S., Neubig, G., and Raghunathan, A

    Springer, J. M., Goyal, S., Wen, K., Kumar, T., Yue, X., Malladi, S., Neubig, G., and Raghunathan, A. Overtrained language models are harder to fine-tune, 2025. URL https://arxiv.org/abs/2503.19206

    2025 arXiv

  55. [64]

    Nonlinear dynamics and chaos: with applications to physics, biology, chemistry, and engineering

    Strogatz, S. Nonlinear dynamics and chaos: with applications to physics, biology, chemistry, and engineering. A Chapman & Hall book. CRC Press, Boca Raton London New York, second edition, first issued in hardback edition, 2019. ISBN 978-0-8133-4910-7 978-0-367-09206-1

    2019

  56. [65]

    W., Thiery, A

    Teh, Y. W., Thiery, A. H., and Vollmer, S. J. Consistency and fluctuations for stochastic gradient langevin dynamics. J. Mach. Learn. Res., 17 0 (1): 0 193–225, January 2016. ISSN 1532-4435

    2016

  57. [66]

    Weight averaging for neural networks and local resampling schemes

    Utans, J. Weight averaging for neural networks and local resampling schemes. In Proc. AAAI-96 Workshop on Integrating Multiple Learned Models. AAAI Press, pp.\ 133--138, 1996

    1996

  58. [67]

    Vlaar, T. J. and Frankle, J. What can linear interpolation of neural network loss landscapes tell us? In Proceedings of the 39th International Conference on Machine Learning, volume 162, pp.\ 22325--22341. PMLR, 2022. URL https://proceedings.mlr.press/v162/vlaar22a.html

    2022

  59. [68]

    Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. GLUE : A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=rJ4km2R5t7

    2019

  60. [69]

    Federated learning with matched averaging

    Wang, H., Yurochkin, M., Sun, Y., Papailiopoulos, D., and Khazaeni, Y. Federated learning with matched averaging. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=BkluqlSFDS

    2020

  61. [70]

    H., Kunz, E., Kornblith, S., and Linderman, S

    Williams, A. H., Kunz, E., Kornblith, S., and Linderman, S. Generalized shape metrics on neural representations. In Advances in Neural Information Processing Systems, volume 34, pp.\ 4738--4750, 2021. URL https://proceedings.neurips.cc/paper_files/paper/2021/file/252a3dbaeb32e...

    2021

  62. [71]

    C., Guestrin, C., Farhadi, A., and Rastegari, M

    Wortsman, M., Horton, M. C., Guestrin, C., Farhadi, A., and Rastegari, M. Learning neural network subspaces. In Proceedings of the 38th International Conference on Machine Learning, volume 139, pp.\ 11217--11227. PMLR, 2021. URL https://proceedings.mlr.press/v139/wortsman21a.html

    2021

  63. [72]

    How SGD selects the global minima in over-parameterized learning: A dynamical stability perspective

    Wu, L., Ma, C., and E, W. How SGD selects the global minima in over-parameterized learning: A dynamical stability perspective. In Advances in Neural Information Processing Systems, volume 31, 2018. URL https://proceedings.neurips.cc/paper_files/paper/2018/file/6651526b6fb8f29a...

    2018

  64. [73]

    K., Savarese, P

    Yunis, D., Patel, K. K., Savarese, P. H. P., Vardi, G., Frankle, J., Walter, M., Livescu, K., and Maire, M. On convexity and linear mode connectivity in neural networks. In OPT 2022: Optimization for Machine Learning (NeurIPS 2022 Workshop), 2022. URL https://openreview.net/fo...

    2022

  65. [74]

    Going beyond linear mode connectivity: The layerwise linear feature connectivity

    Zhou, Z., Yang, Y., Yang, X., Yan, J., and Hu, W. Going beyond linear mode connectivity: The layerwise linear feature connectivity. In Advances in Neural Information Processing Systems, volume 36, pp.\ 60853--60877, 2023. URL https://proceedings.neurips.cc/paper_files/paper/20...

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.