{"id":"70e961f8-75be-4571-9053-8b18e9c9d72d","arxiv_id":"2505.02604","paper_version":5,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"LLPF connects independently trained models by moving toward the target, rescaling each layer's weights back to a fixed variance sphere, and retraining; it works on modern architectures and across training hyperparameters.","lead":"This paper introduces an algorithm called LLPF that builds a continuous low-loss path between two separately trained neural networks. It is evaluated on EfficientNet, MobileNet, RegNet, ShuffleNet, DLA, and compact vision transformers, and it is more repeatable across random seeds than earlier methods.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"LLPF's central mechanism depends on per-layer variance-sphere and flat-minima prerequisites that are validated only on ResNet-18, with BatchNorm excluded; this is the weakest support for the claimed generality across modern architectures.","rationale":"The paper's central claim is empirical and plausible: LLPF is simple, the figures show low training-loss paths for many architectures, continuity is checked by interpolation, and the across-variance-sphere experiment is a valuable addition. The reader's CONDITIONAL verdict is appropriate. My stress-test pass identifies the same load-bearing point: the algorithm is an engine for walking along an assumed shared per-layer variance sphere, but that sphere premise and the flatness prerequisite are only tested on ResNet-18, and even there the validation excludes BatchNorm while the algorithm includes it. The consistency-across-seeds claim and the broad-architecture claim both inherit this assumption, so if it fails the contribution shrinks to a ResNet-family technique. The concrete check would settle the question by quantifying variance, mean, and sharpness per architecture and measuring failure rates under a fixed schedule. I see no internal contradiction or reason to reject the paper; the conditional acceptance should require code release, per-architecture premise checks, and preferably direct endpoint-distance evidence for the different-hyperparameter connections.","tokens_in":19569,"tokens_out":8727,"duration_ms":120296,"concrete_test":"Run a cross-architecture version of Appendix A.1/A.2: for each architecture in Table 2 (at least EfficientNet-B0, MobileNet-V2, CCT7, and DLA), train 20 independently seeded modes with the paper's training hyperparameters; compute per-layer variance and mean for every layer Algorithm 1 would touch, including BatchNorm/LayerNorm layers, and report the maximum relative spread of variances across seeds and the maximum |mean|/std per layer. Also compute a sharpness proxy (e.g., top Hessian eigenvalue or SAM m-sharpness) for the trained modes. Then run LLPF M2M on 10 random mode pairs with a fixed FDF layer schedule and record the failure rate under the paper's own criteria (max training loss below 0.05 and final |Pi-D| below 0.1). If the variance spread or mean magnitude exceeds the ResNet-18 baseline, or if any pair fails, the premises are not general enough to support the central claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Algorithm 1's VarianceCorrection and the geometric argument of Eqs. 2-6 and Figure 1 require three premises: (i) same-hyperparameter modes lie on nearly equal per-layer variance spheres (Eq. 4); (ii) layer means are approximately zero so the sphere center is the origin (Eq. 5); and (iii) starting modes lie in flat low-loss regions (Section 4, Prerequisite). These premises are what make the projection step safe rather than destructive. However, Eq. 4 is empirically justified only in Appendix A.1 for 100 ResNet-18 models, and Figure 7 explicitly excludes BatchNorm layers even though Table 2's ResNet phase schedules include BN weight/bias layers in Algorithm 1. Eq. 5 is shown only approximately in Figure 8, with visibly nonzero means for some layers and no quantitative tolerance stated. The flat-minima prerequisite is conceded as a failure mode but never measured; the paper's statement that near-zero training loss implies flatness conflicts with the known existence of sharp zero-loss minima. If any of these premises fails for EfficientNet, MobileNet, CCT7, DLA, or RegNet, VarianceCorrection can push intermediate points off the low-loss region, so the claimed reproducibility over seeds and the claimed architecture generality would not follow. The paper's own stated prerequisites therefore mark exactly where the central claim is most exposed.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LLPF, an empirical algorithm for constructing low-loss paths between independently trained neural network modes. The M2M variant alternates small moves toward a destination with a VarianceCorrection step that rescales each layer's parameter variance back to the starting model's variance sphere, interleaved with short training rounds; the M2O variant moves a model toward the origin with an angle-conformal learning-rate adjustment. Experiments report training losses below 0.1 along paths for LeNet-5, VGG-11, DenseNet, ResNet-18, MobileNet-V2, ShuffleNet-V2, EfficientNet-B0, RegNet, DLA, and CCT-7, with repeated-seed runs, continuity checks via linear interpolation, and a cross-variance-sphere experiment using different weight decay. The paper claims broader architecture coverage, more consistent paths across seeds, and support for modes trained with different hyperparameters compared with prior methods such as AutoNEB, FGE, and SPRO.","tokens_in":19866,"tokens_out":5948,"duration_ms":67397,"significance":"If the empirical claims hold, LLPF would be a practically useful contribution: it would extend mode connectivity to several modern architectures and offer a more reproducible procedure than prior methods. The paper's strengths include the breadth of the experimental evaluation, explicit repetition over independently trained mode pairs, and a clearly specified algorithmic template with hyperparameter tables. However, the central generality claim rests on variance-sphere assumptions that are validated only on ResNet-18, and the comparative claims are not tested by running prior methods on the same architectures. The contribution is therefore plausible but not yet established at the claimed scope.","major_comments":[{"comment":"Equation (4) is the premise that makes VarianceCorrection safe, but it is empirically demonstrated only for 100 ResNet-18 models on CIFAR-10 (Figure 7), and that figure explicitly excludes BatchNorm layers. Table 2 nonetheless includes bn1.weight/bias in the ResNet-18 phase-2 layer set, and the paper's main claim extends this variance-sphere projection to MobileNet, ShuffleNet, EfficientNet, RegNet, DLA, and CCT without reporting per-architecture variance-sphere measurements. Because a violation of Eq. (4) means that projecting M1 or M3 onto P0's sphere can move the iterate outside the low-loss region, the architecture-generality claim is not yet supported by the evidence.","section":"Appendix A.1/A.2, Table 2, Algorithm 1"},{"comment":"The zero-mean approximation is load-bearing for the geometric picture (Eq. (6), Figure 1), but Figure 8 shows visibly nonzero means for some layers, and the paper gives no quantitative tolerance for how close Mean(theta_lx) must be to zero. VarianceCorrection (Algorithm 1, lines 20-24) rescales around the empirical mean of the layer being processed, so if that mean is not small relative to the standard deviation, the 'variance sphere' centered at the origin is not the sphere on which the algorithm actually operates. The paper should either state and verify a tolerance or revise the geometric justification.","section":"Section 4, Eq. (5), Appendix A.2"},{"comment":"The flat-minima prerequisite is conceded as a failure mode but never measured. The statement that minima with near-zero training loss are generally flat is an unverified empirical claim, and it conflicts with the known existence of sharp minima with zero training loss. Since the authors state that sharp starting modes cause LLPF to fail, the reported seed-consistency results (Section 5, Table 1) are conditional on an unmeasured property of the starting modes; reporting a sharpness proxy, such as the maximum Hessian eigenvalue or a SAM-style sharpness value, for the modes used in the experiments would be necessary to support the reproducibility claim.","section":"Section 4, Prerequisite"},{"comment":"The comparison with AutoNEB, FGE, and SPRO is not a controlled comparison: those methods were not run on the same architectures or mode pairs, and the worst-case loss entries are taken from different models (ResNet-20 and ResNet-158) with the paper's own note that they are not strictly comparable. Consequently, the claims that LLPF is 'more consistent' and 'broader than prior methods' are not directly tested. Running AutoNEB, FGE, and SPRO on at least one common architecture, such as ResNet-18 or DLA on CIFAR-10, with the same mode pairs would support the comparative statements.","section":"Table 1 and Section 5"}],"minor_comments":[{"comment":"The entry 'bn1.{weigt+bias}' contains a typo ('weigt' should be 'weight'); in addition, Table 3's 'A1 L45' reference is confusing because Algorithm 2, not Algorithm 1, contains the iteration at line 45.","section":"Table 2"},{"comment":"The main text says additional cross-variance-sphere experiments were performed on ResNet18@CIFAR10 and CCT7@CIFAR10, but Figure 11's caption names ResNet18@CIFAR10 and DLA@CIFAR10; the discrepancy should be corrected.","section":"Appendix A.6"},{"comment":"Figure 12's caption states that the panels are for CCT7@CIFAR10 and ResNet18@CIFAR100, while the visible panel labels are DLA@CIFAR10 and ResNet18@CIFAR100; the caption or the panel labels should be made consistent.","section":"Appendix A.7"},{"comment":"The caption of Figure 10 refers to 'the corresponding architectures in Figure A.4', but no Figure A.4 exists in the appendix; it should refer to Figure 10 or another numbered figure.","section":"Appendix A.4"},{"comment":"Calling the argument a 'proof' is misleading because the derivation depends on the unproven approximation in Eq. (5); renaming it a 'derivation under the zero-mean approximation' would be more accurate.","section":"Appendix A.8"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern about the variance-sphere assumptions is largely aligned with my reading: the central mechanism is exposed exactly where the paper's own prerequisites are stated but not measured. I view this as a fixable gap rather than a fatal flaw, since the authors could add per-architecture variance/mean diagnostics, sharpness measurements, and direct baseline comparisons. I recommend major revision on that basis."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is real work with a genuine algorithmic contribution and the broadest architecture coverage in the mode-connectivity literature to date. The central claim—that a layer-wise Move/VarianceCorrection/retrain loop finds low training-loss paths across ResNet, DLA, CCT7, MobileNet, ShuffleNet, EfficientNet, and RegNet—is supported by repeated-seed experiments in the paper. I would not desk-reject it.\n\nWhat's new: the LLPF recipe itself. Combining layer-wise movement with per-layer variance normalization and short retraining, plus AngleConformal for crossing variance spheres, is not present in Adilova et al., Zhou et al., or the AutoNEB/FGE/SPRO line. The paper also reports successful connectivity on modern architectures that previous empirical work did not test. Given the field's habit of staying on VGG/ResNet, that is a useful extension.\n\nWhere it is soft: first, the algorithm's key premise—that independently trained modes with identical hyperparameters sit on approximately equal per-layer variance spheres (Eq. 4)—is empirically supported only by 100 ResNet-18 runs in Appendix A.1, and BatchNorm layers are excluded there even though the phase schedules in Table 2 include BN weights. The variance-sphere premise is exactly what makes the projection step safe; if it fails for EfficientNet or CCT-7, the claimed generality does not follow. Second, the flat-minima prerequisite is stated as a failure mode but never measured; the claim that near-zero training loss implies flatness is not established and conflicts with the known existence of sharp zero-loss minima. Third, the comparisons in Table 1 are not backed by rerunning AutoNEB, FGE, or SPRO on the same architectures and mode pairs; the table's worst-case losses come from different papers and models, so \"more consistent\" and \"broader\" are not directly tested. Fourth, code is promised for the final version but not available now, and the algorithm has many free hyperparameters (step sizes, layer order, phase schedule), so independent replication would be painful without code.\n\nNone of these soft spots kills the paper. The central evidence—low training loss along paths, repeated over seeds, with distance-to-destination converging—is plausible and honestly presented. The authors also state the generalization limitation clearly, and the self-citations to their earlier vanishing-variance work are used appropriately.\n\nAudience: loss-landscape researchers and anyone using interpolation-based ensembling. A serious referee should see it; conditional acceptance with code release and at least one direct baseline comparison would be the right outcome.","headline":"A genuinely new layer-wise path-finding method with the broadest architecture coverage in mode connectivity so far, though its load-bearing variance-sphere premise and comparative claims are thinner than the paper's tone suggests.","tokens_in":20414,"tokens_out":1929,"would_cite":true,"duration_ms":24124,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Layer-wise variance spheres can connect independently trained modern neural networks.","keywords":["mode connectivity","low-loss path","variance sphere","layer-wise connectivity","independently trained models","loss landscape","deep learning"],"falsifier":"Train several pairs of a modern architecture, say EfficientNet-B0 or CCT-7, on CIFAR-10 with two different seeds, record per-layer weight variances, and run LLPF without any per-pair hyperparameter tuning; if any layer's variance of the two modes differs by more than a small relative tolerance, or if the resulting path's maximum training loss exceeds the stated tolerance, the central claim of consistent, architecture-general connectivity fails, and a test starting from sharp minima would be decisive.","tokens_in":19362,"feed_emoji":"🔗","tokens_out":6756,"duration_ms":72528,"temperature":0.7,"pith_summary":"The paper proposes Low-Loss Path Finding (LLPF), a layer-wise algorithm for constructing continuous paths of low training loss between two independently trained neural networks. Its central claim is that this works not only on older small CNNs but across modern architectures such as MobileNet, ShuffleNet, EfficientNet, RegNet, DLA, and CCT, and does so reproducibly from random seed to random seed. The paper also claims LLPF can connect modes trained with different hyperparameters, which existing mode-connectivity methods have not addressed. A reader would care because the result suggests mode connectivity is a general, engineerable property of modern trained networks, not a quirk of a few classic architectures.","feed_headline":"A variance-sphere trick links independently trained modern models","feed_subtitle":"Works on MobileNet, EfficientNet, RegNet, DLA and CCT, and reproduces across random seeds.","key_machinery":"The central object is the layer-wise variance sphere $S_{v} = \\{P_{l_x} \\in \\mathbb{R}^{d_{l_x}} : \\operatorname{Var}(P_{l_x}) = v\\}$, the set of layer parameter vectors whose variance equals a given value; because trained layer means are near zero, squared distance to the origin is approximately proportional to variance. On this geometry, VarianceCorrection rescales a layer's weights to a target variance while preserving its mean, countering the vanishing variance that occurs when averaging uncorrelated networks. The Follow Data Flow (FDF) ordering applies the correction and training one layer group at a time, in the direction of data flow, which the paper argues is what makes the method work on modern branched architectures.","core_discovery":"Two independently trained modes of the same architecture and dataset tend to have similar per-layer parameter variance, with layer means near zero, so each layer's parameters sit on a high-dimensional 'variance sphere' centered near the origin. LLPF exploits this geometry: the model-to-model variant repeatedly moves an intermediate model partway toward the destination, rescales each layer back onto the starting variance sphere (VarianceCorrection), and trains a few steps to recover low loss, while the model-to-origin variant moves a model toward the origin with a learning rate rescaled by the shrinking variance (AngleConformal). The claimed discovery is that this variance-sphere control, applied layer by layer following data flow, yields continuous low-loss paths between arbitrary well-trained pairs across a broad set of modern architectures, with the final point converging to the destination mode; the paper demonstrates maximum training loss below $0.006$ on ResNet-18 and consistent trajectories across repeated runs.","pith_inferences":["If per-layer variance spheres are a stable organizing structure of trained networks, they may give a tractable coordinate system for theories of the loss landscape and for model-merging and ensembling beyond permutations.","The flat-minima prerequisite suggests the method would not survive sharp, early-stopped, or adversarially sharpened solutions; testing this boundary could define the method's true scope.","Because paths are found in training-loss space, the method says nothing about preserving test accuracy; using validation-aware loss thresholds or sharpness-aware training during path construction could plausibly fix generalization drift.","Connecting modes across variance spheres may provide an empirical way to study how regularization strength, such as weight decay, reorganizes the loss landscape, since the path passes through intermediate variance values."],"forward_implications":["Mode connectivity is not confined to LeNet, VGG, and ResNet; it can be constructed systematically in modern convolutional and compact transformer architectures.","The discovered paths are reproducible: repeated experiments with different random seeds give nearly identical loss and accuracy trajectories.","Modes trained under different hyperparameters, including different weight decay, can be connected by combining the model-to-origin and model-to-model phases.","Paths can be made very tight: with tuned hyperparameters, maximum training loss along the ResNet-18 connection stays below 0.006.","The path is continuous in practice: linear interpolation between consecutive generated points maintains low loss."],"supporting_citations":[{"why":"Supplies the layer-wise linear mode connectivity concept that justifies processing one layer group at a time.","marker":"(Adilova et al., 2024)"},{"why":"Introduces the mode-connectivity problem and the FGE baseline that LLPF must beat.","marker":"(Garipov et al., 2018)"},{"why":"Provides the AutoNEB baseline and the observation that linear interpolation between modes has a loss barrier.","marker":"(Draxler et al., 2018)"},{"why":"Extends connectivity to simplices and serves as the SPRO baseline.","marker":"(Benton et al., 2021)"},{"why":"Establishes the sharp-versus-flat minima perspective used as the prerequisite that starting modes lie in flat low-loss regions.","marker":"(Foret et al., 2021)"},{"why":"Identifies the vanishing variance problem that VarianceCorrection is designed to counter.","marker":"(Tian et al., 2024)"},{"why":"Supports path continuity by showing that small weight perturbations do not substantially change network outputs.","marker":"(Neyshabur et al., 2018)"}],"fun_headline_variants":["Variance sphere links independently trained models","Layer-wise variance control unites modern neural nets","Spherical trick connects independent model modes","Variance geometry bridges trained models across architectures","Variance-sphere method connects modern models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that independently trained modes with identical hyperparameters lie on approximately the same per-layer variance sphere with near-zero means and start in flat low-loss regions, while sharp minima or mismatched layer variances break the algorithm, even though this premise is empirically justified mainly on ResNet-18.","fun_headline_variants_meta":{"raw":{"variants":["Variance sphere links independently trained models","Layer-wise variance control unites modern neural nets","Spherical trick connects independent model modes","Variance geometry bridges trained models across architectures","Variance-sphere method connects modern models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000885,"raw_usage":{"total_tokens":3795,"prompt_tokens":890,"completion_tokens":2905,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":506,"completion_tokens_details":{"reasoning_tokens":2840}},"tokens_in":506,"tokens_out":2905,"duration_ms":27773,"temperature":1.0,"reasoning_tokens":2840,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:47:09.952397+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train several pairs of a modern architecture, say EfficientNet-B0 or CCT-7, on CIFAR-10 with two different seeds, record per-layer weight variances, and run LLPF without any per-pair hyperparameter tuning; if any layer's variance of the two modes differs by more than a small relative tolerance, or if the resulting path's maximum training loss exceeds the stated tolerance, the central claim of consistent, architecture-general connectivity fails, and a test starting from sharp minima would be decisive.","supporting_citations":[],"review_version":1}