Connecting Independently Trained Modes via Layer-Wise Connectivity

Maksim Kitsak; Peter Hofstee; Yongding Tian; Zaid Al-Ars

REVIEW 4 major objections 5 minor 20 references

Connecting Independently Trained Modes via Layer-Wise Connectivity

T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Layer-wise variance spheres can connect independently trained modern neural networks.

desk verdict A genuinely new layer-wise path-finding method with the broadest architecture coverage in mode connectivity so far, though its load-bearing variance-sphere premise and comparative claims are thinner than the paper's tone suggests. read the letter →

arxiv 2505.02604 v5 pith:5D7RUK6U submitted 2025-05-05 cs.LG

Yongding Tian , Zaid Al-Ars , Maksim Kitsak , Peter Hofstee This is my paper · ORCID

classification cs.LG

keywords modeconnectivitylow-losspathvariancespherelayer-wiseindependentlytrainedmodelslosslandscapedeeplearning

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes Low-Loss Path Finding (LLPF), a layer-wise algorithm for constructing continuous paths of low training loss between two independently trained neural networks. Its central claim is that this works not only on older small CNNs but across modern architectures such as MobileNet, ShuffleNet, EfficientNet, RegNet, DLA, and CCT, and does so reproducibly from random seed to random seed. The paper also claims LLPF can connect modes trained with different hyperparameters, which existing mode-connectivity methods have not addressed. A reader would care because the result suggests mode connectivity is a general, engineerable property of modern trained networks, not a quirk of a few classic architectures.

What carries the argument

The central object is the layer-wise variance sphere $S_{v} = \{P_{l_x} \in \mathbb{R}^{d_{l_x}} : \operatorname{Var}(P_{l_x}) = v\}$, the set of layer parameter vectors whose variance equals a given value; because trained layer means are near zero, squared distance to the origin is approximately proportional to variance. On this geometry, VarianceCorrection rescales a layer's weights to a target variance while preserving its mean, countering the vanishing variance that occurs when averaging uncorrelated networks. The Follow Data Flow (FDF) ordering applies the correction and training one layer group at a time, in the direction of data flow, which the paper argues is what makes the method work on modern branched architectures.

What would settle it

Train several pairs of a modern architecture, say EfficientNet-B0 or CCT-7, on CIFAR-10 with two different seeds, record per-layer weight variances, and run LLPF without any per-pair hyperparameter tuning; if any layer's variance of the two modes differs by more than a small relative tolerance, or if the resulting path's maximum training loss exceeds the stated tolerance, the central claim of consistent, architecture-general connectivity fails, and a test starting from sharp minima would be decisive.

Watch

Extended reading notes

Core claim

Two independently trained modes of the same architecture and dataset tend to have similar per-layer parameter variance, with layer means near zero, so each layer's parameters sit on a high-dimensional 'variance sphere' centered near the origin. LLPF exploits this geometry: the model-to-model variant repeatedly moves an intermediate model partway toward the destination, rescales each layer back onto the starting variance sphere (VarianceCorrection), and trains a few steps to recover low loss, while the model-to-origin variant moves a model toward the origin with a learning rate rescaled by the shrinking variance (AngleConformal). The claimed discovery is that this variance-sphere control, applied layer by layer following data flow, yields continuous low-loss paths between arbitrary well-trained pairs across a broad set of modern architectures, with the final point converging to the destination mode; the paper demonstrates maximum training loss below $0.006$ on ResNet-18 and consistent trajectories across repeated runs.

Load-bearing premise

The load-bearing premise is that independently trained modes with identical hyperparameters lie on approximately the same per-layer variance sphere with near-zero means and start in flat low-loss regions, while sharp minima or mismatched layer variances break the algorithm, even though this premise is empirically justified mainly on ResNet-18.

Editorial extensions

If this is right

Mode connectivity is not confined to LeNet, VGG, and ResNet; it can be constructed systematically in modern convolutional and compact transformer architectures.
The discovered paths are reproducible: repeated experiments with different random seeds give nearly identical loss and accuracy trajectories.
Modes trained under different hyperparameters, including different weight decay, can be connected by combining the model-to-origin and model-to-model phases.
Paths can be made very tight: with tuned hyperparameters, maximum training loss along the ResNet-18 connection stays below 0.006.
The path is continuous in practice: linear interpolation between consecutive generated points maintains low loss.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If per-layer variance spheres are a stable organizing structure of trained networks, they may give a tractable coordinate system for theories of the loss landscape and for model-merging and ensembling beyond permutations.
The flat-minima prerequisite suggests the method would not survive sharp, early-stopped, or adversarially sharpened solutions; testing this boundary could define the method's true scope.
Because paths are found in training-loss space, the method says nothing about preserving test accuracy; using validation-aware loss thresholds or sharpness-aware training during path construction could plausibly fix generalization drift.
Connecting modes across variance spheres may provide an empirical way to study how regularization strength, such as weight decay, reorganizes the loss landscape, since the path passes through intermediate variance values.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is real work with a genuine algorithmic contribution and the broadest architecture coverage in the mode-connectivity literature to date. The central claim—that a layer-wise Move/VarianceCorrection/retrain loop finds low training-loss paths across ResNet, DLA, CCT7, MobileNet, ShuffleNet, EfficientNet, and RegNet—is supported by repeated-seed experiments in the paper. I would not desk-reject it.

What's new: the LLPF recipe itself. Combining layer-wise movement with per-layer variance normalization and short retraining, plus AngleConformal for crossing variance spheres, is not present in Adilova et al., Zhou et al., or the AutoNEB/FGE/SPRO line. The paper also reports successful connectivity on modern architectures that previous empirical work did not test. Given the field's habit of staying on VGG/ResNet, that is a useful extension.

Where it is soft: first, the algorithm's key premise—that independently trained modes with identical hyperparameters sit on approximately equal per-layer variance spheres (Eq. 4)—is empirically supported only by 100 ResNet-18 runs in Appendix A.1, and BatchNorm layers are excluded there even though the phase schedules in Table 2 include BN weights. The variance-sphere premise is exactly what makes the projection step safe; if it fails for EfficientNet or CCT-7, the claimed generality does not follow. Second, the flat-minima prerequisite is stated as a failure mode but never measured; the claim that near-zero training loss implies flatness is not established and conflicts with the known existence of sharp zero-loss minima. Third, the comparisons in Table 1 are not backed by rerunning AutoNEB, FGE, or SPRO on the same architectures and mode pairs; the table's worst-case losses come from different papers and models, so "more consistent" and "broader" are not directly tested. Fourth, code is promised for the final version but not available now, and the algorithm has many free hyperparameters (step sizes, layer order, phase schedule), so independent replication would be painful without code.

None of these soft spots kills the paper. The central evidence—low training loss along paths, repeated over seeds, with distance-to-destination converging—is plausible and honestly presented. The authors also state the generalization limitation clearly, and the self-citations to their earlier vanishing-variance work are used appropriately.

Audience: loss-landscape researchers and anyone using interpolation-based ensembling. A serious referee should see it; conditional acceptance with code release and at least one direct baseline comparison would be the right outcome.

Referee Report

4 major / 5 minor

Summary. The paper proposes LLPF, an empirical algorithm for constructing low-loss paths between independently trained neural network modes. The M2M variant alternates small moves toward a destination with a VarianceCorrection step that rescales each layer's parameter variance back to the starting model's variance sphere, interleaved with short training rounds; the M2O variant moves a model toward the origin with an angle-conformal learning-rate adjustment. Experiments report training losses below 0.1 along paths for LeNet-5, VGG-11, DenseNet, ResNet-18, MobileNet-V2, ShuffleNet-V2, EfficientNet-B0, RegNet, DLA, and CCT-7, with repeated-seed runs, continuity checks via linear interpolation, and a cross-variance-sphere experiment using different weight decay. The paper claims broader architecture coverage, more consistent paths across seeds, and support for modes trained with different hyperparameters compared with prior methods such as AutoNEB, FGE, and SPRO.

Significance. If the empirical claims hold, LLPF would be a practically useful contribution: it would extend mode connectivity to several modern architectures and offer a more reproducible procedure than prior methods. The paper's strengths include the breadth of the experimental evaluation, explicit repetition over independently trained mode pairs, and a clearly specified algorithmic template with hyperparameter tables. However, the central generality claim rests on variance-sphere assumptions that are validated only on ResNet-18, and the comparative claims are not tested by running prior methods on the same architectures. The contribution is therefore plausible but not yet established at the claimed scope.

major comments (4)

[Appendix A.1/A.2, Table 2, Algorithm 1] Equation (4) is the premise that makes VarianceCorrection safe, but it is empirically demonstrated only for 100 ResNet-18 models on CIFAR-10 (Figure 7), and that figure explicitly excludes BatchNorm layers. Table 2 nonetheless includes bn1.weight/bias in the ResNet-18 phase-2 layer set, and the paper's main claim extends this variance-sphere projection to MobileNet, ShuffleNet, EfficientNet, RegNet, DLA, and CCT without reporting per-architecture variance-sphere measurements. Because a violation of Eq. (4) means that projecting M1 or M3 onto P0's sphere can move the iterate outside the low-loss region, the architecture-generality claim is not yet supported by the evidence.
[Section 4, Eq. (5), Appendix A.2] The zero-mean approximation is load-bearing for the geometric picture (Eq. (6), Figure 1), but Figure 8 shows visibly nonzero means for some layers, and the paper gives no quantitative tolerance for how close Mean(theta_lx) must be to zero. VarianceCorrection (Algorithm 1, lines 20-24) rescales around the empirical mean of the layer being processed, so if that mean is not small relative to the standard deviation, the 'variance sphere' centered at the origin is not the sphere on which the algorithm actually operates. The paper should either state and verify a tolerance or revise the geometric justification.
[Section 4, Prerequisite] The flat-minima prerequisite is conceded as a failure mode but never measured. The statement that minima with near-zero training loss are generally flat is an unverified empirical claim, and it conflicts with the known existence of sharp minima with zero training loss. Since the authors state that sharp starting modes cause LLPF to fail, the reported seed-consistency results (Section 5, Table 1) are conditional on an unmeasured property of the starting modes; reporting a sharpness proxy, such as the maximum Hessian eigenvalue or a SAM-style sharpness value, for the modes used in the experiments would be necessary to support the reproducibility claim.
[Table 1 and Section 5] The comparison with AutoNEB, FGE, and SPRO is not a controlled comparison: those methods were not run on the same architectures or mode pairs, and the worst-case loss entries are taken from different models (ResNet-20 and ResNet-158) with the paper's own note that they are not strictly comparable. Consequently, the claims that LLPF is 'more consistent' and 'broader than prior methods' are not directly tested. Running AutoNEB, FGE, and SPRO on at least one common architecture, such as ResNet-18 or DLA on CIFAR-10, with the same mode pairs would support the comparative statements.

minor comments (5)

[Table 2] The entry 'bn1.{weigt+bias}' contains a typo ('weigt' should be 'weight'); in addition, Table 3's 'A1 L45' reference is confusing because Algorithm 2, not Algorithm 1, contains the iteration at line 45.
[Appendix A.6] The main text says additional cross-variance-sphere experiments were performed on ResNet18@CIFAR10 and CCT7@CIFAR10, but Figure 11's caption names ResNet18@CIFAR10 and DLA@CIFAR10; the discrepancy should be corrected.
[Appendix A.7] Figure 12's caption states that the panels are for CCT7@CIFAR10 and ResNet18@CIFAR100, while the visible panel labels are DLA@CIFAR10 and ResNet18@CIFAR100; the caption or the panel labels should be made consistent.
[Appendix A.4] The caption of Figure 10 refers to 'the corresponding architectures in Figure A.4', but no Figure A.4 exists in the appendix; it should refer to Figure 10 or another numbered figure.
[Appendix A.8] Calling the argument a 'proof' is misleading because the derivation depends on the unproven approximation in Eq. (5); renaming it a 'derivation under the zero-mean approximation' would be more accurate.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation: the algorithm's low-loss paths are empirically validated, and the only same-author citation is a non-load-bearing motivation for the variance-correction step.

full rationale

The paper's central claim is an empirical algorithm and its experimental evaluation, not a predicted quantity derived from the model's own definitions. Equation 4 is stated as an empirical tendency and is supported by the paper's own Appendix A.1 (100 ResNet-18 runs) plus an external reference; Equation 5 is checked empirically in Appendix A.2. Equation 6 is a mathematical identity under the zero-mean approximation: with mean approximately zero, squared Euclidean norm equals n times variance, so it does not smuggle the conclusion into the premise. Algorithm 1's VarianceCorrection does force each projected point to lie on the chosen variance sphere by scaling, but this only guarantees the variance constraint; it does not by construction guarantee low training loss, convergence of Pi to D, or low test loss, all of which are verified experimentally against baselines. The one self-citation, Tian et al. (2024), is used only to name the 'vanishing variance problem' and motivate why averaging uncorrelated networks shrinks variance; the correction itself is an algebraic operation and the paths are empirically validated independently of that prior result, so the self-citation is minor and not load-bearing. The paper's own limitations—flat-minima prerequisite, BatchNorm excluded from the variance validations, and the fact that Equations 4 and 5 are only empirically justified on ResNet-18—are real gaps in support for the claimed architectural generality, but they are correctness risks, not circular reductions. No fitted parameter is renamed as a prediction, and no result reduces to its input by construction.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The variance-sphere geometry and flat-minima assumptions are supplied as prior/empirical postulates, not derived in the paper; the path-finding algorithm's success on new architectures depends on them.

free parameters (4)

step_a, step_c, step_f (move step sizes) = per-architecture values, e.g. step_a=1e-3, step_c=2e-3 for DLA@CIFAR10; defaults 0
Control how far each path iteration moves toward the destination (Algorithm 1, line 14; Table 2).
path training round r and loss thresholds = e.g. train until loss<0.04 or r>200 for ResNet-18 phase 1; loss<0.05 for CCT-7
Determine when an intermediate point is accepted as low-loss (Algorithm 1, line 7; Table 2).
layer selection and phase schedule = per-architecture phases in Table 2 (ResNet-18 has 6 phases, DLA 21, CCT-7 8/9)
Only a subset of layers is trained at each phase; the FDF strategy reduces but does not eliminate manual ordering.
path optimizer learning rate and batch size = SGD eta=0.001, batch 64-256 depending on model
Standard optimizer hyperparameters for the Train step in Algorithms 1 and 2 (Tables 2 and 3).

assumptions (5)

domain assumption Independently trained modes with identical hyperparameters lie on approximately the same per-layer variance spheres (Eq. 4)
Load-bearing for Algorithm 1; validated empirically in Appendix A.1 only for ResNet-18 on CIFAR-10, cited to Chen et al. 2024 for support.
domain assumption Layer parameter means are approximately zero after training (Eq. 5)
Used to identify squared distance with variance in Eq. 6; validated for ResNet-18 in Appendix A.2. Biases and normalization layers are treated as exceptions.
domain assumption Starting modes must lie in flat low-loss regions
Stated in Section 4 Prerequisite; sharp minima cause LLPF to fail. This restricts the population of models to which the method applies.
domain assumption Layer-wise mode connectivity framework (Adilova et al., 2024) applies
Justifies sequential per-layer processing, drawn from prior literature, not proven here.
domain assumption Small parameter perturbations do not substantially change outputs (Neyshabur et al., 2018)
Used in the Mode-Connection Continuity Check to argue that linear interpolation between path points stays low-loss.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Connecting Independently Trained Modes via Layer-Wise Connectivity." pith.science (2026). https://pith.science/paper/5D7RUK6U

@misc{pith2026250502604,
  author       = {Pith},
  title        = {Pith review of: Connecting Independently Trained Modes via Layer-Wise Connectivity},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5D7RUK6U}},
  note         = {Machine review of arXiv:2505.02604}
}

read the original abstract

Empirical studies have shown that continuous low-loss paths can be constructed between independently trained neural network models. This phenomenon, known as mode connectivity, refers to the existence of such paths between distinct modes-i.e., well-trained solutions in parameter space. However, existing empirical methods do not reliably connect independently trained modes and have been evaluated mainly on a narrow set of architectures (e.g., basic CNNs, VGG, and ResNet), leaving their effectiveness on newer models unclear. In this work, we propose a new empirical algorithm for connecting independently trained modes that generalizes beyond traditional architectures and supports a broader range of networks, including MobileNet, ShuffleNet, EfficientNet, RegNet, Deep Layer Aggregation (DLA), and Compact Convolutional Transformers (CCT). In addition to broader applicability, the proposed method yields more consistent connectivity paths across independently trained mode pairs and supports connecting modes obtained with different training hyperparameters.

Figures

Figures reproduced from arXiv: 2505.02604 by the authors.

**Figure 2.** Results of Algorithm 1. Each column corresponds to a different model and dataset: [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

**Figure 3.** Results of Algorithm 2, following the same layout and postprocessing steps as in Figure 2, [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Figures from the paper (9 more)

**Figure 4.** Figure 4: Connecting two modes on different variance spheres. Iterations 0–1000 (“m2o”) use Algorithm 2 to reach the target variance sphere; the subsequent “m2m” phase uses Algorithm 1 to reach the destination mode. Training accuracy remains high throughout, indicating a vali…

**Figure 5.** Figure 5: Training-set loss along linear interpolations between Pt and Pt+1; 50 interpolation samples per segment. The training loss curve differs from [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]

**Figure 6.** Figure 6: Training loss along a modeconnection path for ResNet18@CIFAR10. Hyperparameters are tuned so that the maximum training loss along the path stays below 0.006. The major limitation of our approach is that it does not guarantee model generalization ability along the dis…

**Figure 7.** Figure 7: Layer-wise weights variance across 100 independently trained ResNet-18 models on the [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]

**Figure 8.** Figure 8: Layer-wise weights mean across 100 independently trained ResNet-18 models on the [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]

**Figure 9.** Figure 9: Additional results for Algorithm 1 across diverse architectures: LeNet-5, VGG-11, [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]

**Figure 10.** Figure 10: Additional results for Algorithm 2 across the corresponding architectures in Figure A.4. [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]

**Figure 11.** Figure 11: Extensions of the results in Figure 4 for ResNet18@CIFAR10 and DLA@CIFAR10 [PITH_FULL_IMAGE:figures/full_fig_p018_11.png]

**Figure 12.** Figure 12: Extensions of the results in Figure 5 for CCT7@CIFAR10 and ResNet18@CIFAR100 [PITH_FULL_IMAGE:figures/full_fig_p018_12.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 6 canonical work pages

[2]

10 Published as a conference paper at ICLR 2026 Chenhang

URLhttps://openreview.net/forum? id=Tcft2V63Vd. 10 Published as a conference paper at ICLR 2026 Chenhang. GitHub - chenhang98/mobileNet-v2 cifar10: a pytorch implement of mobileNet v2 on cifar10 — github.com.https://github.com/chenhang98/mobileNet-v2_ cifar10, a. [Accessed 15-05-2025]. Chenhang. GitHub - chenhang98/shuffleNet-cifar10: a pytorch implement ...

work page 2026
[7]

Applied Layers

The results indicate that the mean values of most layers are close to zero. 13 Published as a conference paper at ICLR 2026 A.3 ADDITIONALRESULTS FORALGORITHM1 0.0 0.5 1.0 1.5 2.0 2.5 iteration i ×104 0.00 0.05Training Loss LeNet5 @MNIST training loss 0 2 4 6 8 iteration i ×103 0.000 0.025 0.050 VGG11 @CIFAR10 training loss 0 1 2 3 4 iteration i ×103 0.01...

work page 2026
[8]

Quynh Nguyen and Matthias Hein

URLhttps://arxiv.org/ abs/1707.09564. Quynh Nguyen and Matthias Hein. The loss surface of deep and wide neural networks. InPro- ceedings of the 34th International Conference on Machine Learning - Volume 70, ICML’17, pp. 2603––2612,

arXiv
[9]

URLhttps://arxiv.org/abs/2003.13678. H. Robbins and S. Monro. A stochastic approximation method.Annals of Mathematical Statistics, 22:400–407,

arXiv 2003
[12]

Mingxing Tan and Quoc V

URLhttps://arxiv.org/abs/1708.07120. Mingxing Tan and Quoc V . Le. Efficientnet: Rethinking model scaling for convolutional neural networks,

arXiv
[13]

Yongding Tian, Zaid Al-Ars, Maksim Kitsak, and Peter Hofstee

URLhttps://arxiv.org/abs/1905.11946. Yongding Tian, Zaid Al-Ars, Maksim Kitsak, and Peter Hofstee. Vanishing variance problem in fully decentralized neural-network systems,

arXiv 1905
[14]

Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun

URL https://arxiv.org/abs/1707.06484. Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices,

arXiv
[18]

16 Published as a conference paper at ICLR 2026 Table 3: Hyperparameters used for applying Algorithm 2 to find the low-loss path toward origin

dall layers in the first bottleneck layer. 16 Published as a conference paper at ICLR 2026 Table 3: Hyperparameters used for applying Algorithm 2 to find the low-loss path toward origin. Applied layers Iteration T (A1 L45) Train roundr (A1 L7) Batch sizeB (A1 L7) Optimizer and hyperparameter x (A1 L7) step a (A1 L14) step c (A1 L14) LeNet5 @MNIST all laye...

work page 2026

Show all 20 references

[19]

Data pre-precessing (augmentation) steps Step 1 Step 2 Step 3 MNIST Random-rotate for 5 degrees Random-crop to 28x28 with padding size 2 Normalize with dataset mean (0.1307) and standard derivation (0.3081) CIFAR10 Random-crop to 32x32 with padding size 4 Random-horizontal-fli...

2026
[1998]

ISBN 3540653112

Springer-Verlag. ISBN 3540653112. Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicol `o Cesa-Bianchi, and Roman Garnett (eds.),Advances in...

2018 arXiv
[2015]

Leslie N

URLhttps://arxiv.org/abs/1409.1556. Leslie N. Smith and Nicholay Topin. Super-convergence: Very fast training of neural networks using large learning rates,

arXiv
[2017]

Zhanpeng Zhou, Yongyi Yang, Xiaojiang Yang, Junchi Yan, and Wei Hu

URLhttps://arxiv.org/abs/ 1707.01083. Zhanpeng Zhou, Yongyi Yang, Xiaojiang Yang, Junchi Yan, and Wei Hu. Going beyond linear mode connectivity: The layerwise linear feature connectivity,

arXiv
[2018]

11 Published as a conference paper at ICLR 2026 Kuangliu

URLhttps://arxiv.org/abs/1608.06993. 11 Published as a conference paper at ICLR 2026 Kuangliu. GitHub - kuangliu/pytorch-cifar: 95.47https://github.com/kuangliu/ pytorch-cifar/tree/master. [Accessed 21-09-2025]. Yann LeCun, L ´eon Bottou, Genevieve B. Orr, and Klaus-Robert M ¨...

2026 arXiv
[2019]

Karen Simonyan and Andrew Zisserman

URLhttps://arxiv.org/abs/ 1801.04381. Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition,

arXiv
[2020]

org/abs/2010.15110

URLhttps://arxiv. org/abs/2010.15110. Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In Hal Daum ´e III and Aarti Singh (eds.),Pro- ceedings of the 37th International Conference on Machi...

2010 arXiv
[2021]

Behnam Neyshabur, Srinadh Bhojanapalli, and Nathan Srebro

URLhttps://arxiv.org/abs/1902.04742. Behnam Neyshabur, Srinadh Bhojanapalli, and Nathan Srebro. A pac-bayesian approach to spectrally-normalized margin bounds for neural networks,

1902 arXiv
[2022]

org/abs/2104.05704

URLhttps://arxiv. org/abs/2104.05704. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In2015 IEEE International Conference on Computer Vision (ICCV), pp. 1026–1034, 2015a. doi: ...

arXiv
[2023]

URLhttps://arxiv.org/ abs/2307.08286. 12 Published as a conference paper at ICLR 2026 A APPENDIX A.1 EMPIRICALVALIDATION OFEQUATION4 0 20 40 60 80 100 Model Index 10 5 10 4 10 3 10 2 Layer Variance conv1.weight layer1.0.conv1.weight layer1.0.conv2.weight layer1.1.conv1.weight ...

2026 arXiv
[2024]

Samuel Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa

URLhttps://arxiv.org/abs/2307.06966. Samuel Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re-basin: Merging models mod- ulo permutation symmetries. InThe Eleventh International Conference on Learning Representa- tions,

arXiv
[5090]

A.6 ADDITIONALRESULTS FORCONNECTINGMODES ONDIFFERENTVARIANCESPHERE 0 1 2 3 4 5 6 7 8 iteration i ×103 10 1 training & test loss m2o m2m ResNet18 @CIFAR10 training loss test loss training accuracy test accuracy 0 1 2 3 4 5 iteration i ×103 10 4 10 3 10 2 10 1 100 training & tes...

2026

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.

[1] [2]

10 Published as a conference paper at ICLR 2026 Chenhang

URLhttps://openreview.net/forum? id=Tcft2V63Vd. 10 Published as a conference paper at ICLR 2026 Chenhang. GitHub - chenhang98/mobileNet-v2 cifar10: a pytorch implement of mobileNet v2 on cifar10 — github.com.https://github.com/chenhang98/mobileNet-v2_ cifar10, a. [Accessed 15-05-2025]. Chenhang. GitHub - chenhang98/shuffleNet-cifar10: a pytorch implement ...

work page 2026

[2] [7]

Applied Layers

The results indicate that the mean values of most layers are close to zero. 13 Published as a conference paper at ICLR 2026 A.3 ADDITIONALRESULTS FORALGORITHM1 0.0 0.5 1.0 1.5 2.0 2.5 iteration i ×104 0.00 0.05Training Loss LeNet5 @MNIST training loss 0 2 4 6 8 iteration i ×103 0.000 0.025 0.050 VGG11 @CIFAR10 training loss 0 1 2 3 4 iteration i ×103 0.01...

work page 2026

[3] [8]

Quynh Nguyen and Matthias Hein

URLhttps://arxiv.org/ abs/1707.09564. Quynh Nguyen and Matthias Hein. The loss surface of deep and wide neural networks. InPro- ceedings of the 34th International Conference on Machine Learning - Volume 70, ICML’17, pp. 2603––2612,

arXiv

[4] [9]

URLhttps://arxiv.org/abs/2003.13678. H. Robbins and S. Monro. A stochastic approximation method.Annals of Mathematical Statistics, 22:400–407,

arXiv 2003

[5] [12]

Mingxing Tan and Quoc V

URLhttps://arxiv.org/abs/1708.07120. Mingxing Tan and Quoc V . Le. Efficientnet: Rethinking model scaling for convolutional neural networks,

arXiv

[6] [13]

Yongding Tian, Zaid Al-Ars, Maksim Kitsak, and Peter Hofstee

URLhttps://arxiv.org/abs/1905.11946. Yongding Tian, Zaid Al-Ars, Maksim Kitsak, and Peter Hofstee. Vanishing variance problem in fully decentralized neural-network systems,

arXiv 1905

[7] [14]

Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun

URL https://arxiv.org/abs/1707.06484. Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices,

arXiv

[8] [18]

16 Published as a conference paper at ICLR 2026 Table 3: Hyperparameters used for applying Algorithm 2 to find the low-loss path toward origin

dall layers in the first bottleneck layer. 16 Published as a conference paper at ICLR 2026 Table 3: Hyperparameters used for applying Algorithm 2 to find the low-loss path toward origin. Applied layers Iteration T (A1 L45) Train roundr (A1 L7) Batch sizeB (A1 L7) Optimizer and hyperparameter x (A1 L7) step a (A1 L14) step c (A1 L14) LeNet5 @MNIST all laye...

work page 2026