Pith. sign in

REVIEW 4 major objections 5 minor 41 references

In finetuning, the benefit of manifold-constrained hyper-connections comes from input-dependent read/write routing, not from learned residual mixing.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 15:53 UTC pith:PNC6Z4E5

load-bearing objection A genuine first application of mHC to PEFT with a plausible mechanistic finding, but the evidence is single-seed and the warm-start attribution needs numbers before I'd fully believe it. the 4 major comments →

arxiv 2607.18130 v1 pith:PNC6Z4E5 submitted 2026-07-20 cs.LG

Manifold-Constrained Hyper-Connections for Parameter-Efficient Finetuning

Valentijn Oldenburg , Floris de Kam , Bente Zuijdam , Lieve Eberson , Nicky van Zutphen , Stef de Wildt , Ivo Verhoeven This is my paper
classification cs.LG
keywords hyper-connectionsresidual connectionsparameter-efficient finetuningLoRAfrozen Transformersidentity initializationBirkhoff polytoperesidual routing
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper asks whether manifold-constrained hyper-connections (mHC), a generalization of residual connections that routes hidden states through multiple residual streams, can serve as a parameter-efficient finetuning method for frozen Transformers. It finds that mHC can finetune a frozen backbone, but its role differs fundamentally from pre-training: the learned residual mixing matrix drifts toward identity, and fixing it to identity improves or preserves performance while saving parameters. The main benefit comes instead from the input-dependent read and write gates that decide which streams feed the frozen layer and how its output is written back. As a standalone method mHC does not consistently beat LoRA, but at matched parameter budgets, combining mHC with LoRA improves language-modelling loss and gives task-dependent downstream gains at 1B and 7B scale. The paper identifies residual routing as a distinct and promising new axis for parameter-efficient finetuning.

Core claim

The central claim is that in frozen-backbone finetuning, mHC's useful capacity is concentrated in its input-dependent read/write routing (the gates h_pre and h_post), not in the learned depth-wise residual mixing matrix H_res. Evidence: learned H_res moves toward near-identity in later layers, ablating or fixing H_res to identity improves or preserves loss while removing parameters, and this pattern holds across three parameterisations (Sinkhorn, mHC-lite, and KromHC) at both 1B and 7B scale. The paper concludes that finetuning should preserve the pretrained residual pathway and use residual routing mainly to modulate how pretrained representations are read from and written back to the strea

What carries the argument

The central object is the mHC wrapper: a module that replaces a single residual connection with n parallel residual streams, a doubly stochastic mixing matrix H_res constrained to the Birkhoff polytope (via Sinkhorn projection, Birkhoff-von Neumann decomposition, or Kronecker factors), and sigmoid read/write gates h_pre and h_post. The doubly stochastic constraint keeps repeated products of the mixing matrix bounded, so the residual signal does not amplify or vanish over depth. The identity-preserving variant fixes H_res = I, turning the module into pure input-dependent read/write routing, which the paper shows is the component that carries the finetuning gains.

Load-bearing premise

The entire attribution rests on the warm-start property: at initialization the mHC wrapper is numerically equivalent to the frozen backbone within about 1% per sub-layer, and the paper only verifies this at initialization and for early gradients, not for the full training trajectory; if the per-layer deviation compounds over 16–32 wrapped sub-layers during training, the model may drift from the pretrained minimum and the loss or benchmark gains could stem from that drift rath

What would settle it

Train an mHC-wrapped model while recording, at each training step, the maximum per-layer L2 deviation between the wrapped model's hidden states and the vanilla frozen backbone's hidden states. If the cumulative deviation grows above the bf16 rounding tolerance (roughly 1% per sub-layer) before the language-modelling loss improvement appears, the improvement would be attributable to the perturbation rather than the routing. Alternatively, remove the read/write gates while keeping learned H_res; if loss still improves, the identity-preservation conclusion is wrong.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Residual routing is a distinct, complementary PEFT axis: it changes how pretrained representations move through the model rather than changing weights or activations.
  • Combining mHC with LoRA can beat LoRA alone at matched trainable parameter budgets, improving language-modelling loss (e.g., static mHC n=16 + LoRA r=8 reduces test loss from 1.28 to 1.21 on the 1B backbone) and giving task-dependent benchmark gains at 1B and 7B scale.
  • Fixing the residual mixing matrix to identity removes up to 8.39M trainable parameters (at n=16 on the 1B model) while improving test loss, making identity-preserving mHC the parameter-efficient default.
  • Standalone mHC is competitive on some knowledge and multiple-choice tasks (best PIQA at n=2, best TriviaQA at n=4) but does not consistently outperform LoRA or VeRA; its value is as a complement, not a replacement.
  • The identity-preservation result holds across all tested mHC parameterisations and scales to the 7B model, suggesting it is a general property of finetuning rather than an artefact of one implementation.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the identity-preservation result generalizes across architectures, then any PEFT method that learns transformations on the residual stream will face the same pressure toward identity; this could be tested by applying mHC to a post-norm Transformer or to a different model family.
  • The 'residual memory' hypothesis—that finetuning should modulate access to a pretrained residual pathway rather than rewrite it—predicts that methods constraining residual-stream perturbations will generalize better under distribution shift; this is an extension the paper does not test.
  • The combination results suggest LoRA rank and residual routing occupy different capacity niches; a reader might reasonably expect mHC to complement other PEFT methods such as VeRA or (IA)3 in similar ways, but that is speculative beyond the paper's experiments.
  • The reported carbon cost of the study (~99.45 kgCO2eq) suggests mHC experiments on this setup are comparatively cheap, making broader sweeps across seeds, datasets, and architectures feasible; that implication is ours, not the paper's.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes manifold-constrained hyper-connections (mHC) as a parameter-efficient finetuning method for frozen Transformers. The authors wrap each attention/MLP sub-layer of OLMo-2-1B and OLMo-2-7B with static or dynamic residual-routing modules, comparing Sinkhorn-based mHC with mHC-lite and KromHC parameterizations. The central claim is that, in finetuning, the learned residual mixing matrix H_res tends to move toward the identity, and that fixing H_res=I_n improves or preserves language-modelling loss while reducing trainable parameters; hence mHC's benefit comes mainly from input-dependent read/write routing via h_pre and h_post. As a standalone method mHC does not consistently beat LoRA, but at matched parameter budgets mHC+LoRA combinations yield small loss improvements and task-dependent benchmark gains.

Significance. If the identity-preservation result holds, the paper identifies a distinct and practically useful PEFT axis—residual routing—and provides a strong motivation for simplifying mHC in the finetuning setting by dropping H_res entirely. The study is systematic in several respects: it includes multiple mHC parameterizations, ablations of each routing component, matched parameter-budget comparisons with LoRA, evaluations on eight benchmarks, scaling from 1B to 7B, and public code. These are genuine strengths. However, the headline quantitative claims are supported by small loss differences from single-seed runs, and the mechanistic attribution depends on an empirically underdocumented warm-start property.

major comments (4)
  1. [§3.3 Diagnostics, §4.3.3, Appendix B/C] The central mechanistic claim—that the advantage of mHCidentity comes from learned routing rather than from differences in initialization perturbation—requires that the wrapped and vanilla models start at the same function. Appendix C states that symmetry-breaking noise breaks exact equivalence, that the single-sublayer deviation is 'roughly 1%', and that final logits remain close, but no measured cumulative deviation, initial loss gap, or early gradient norm is reported. A 1% per-sublayer perturbation can compound over 32 wrapped sublayers. Please report the actual final-logit cosine/total-variation distance at initialization, the initial loss gap, and the gradient norms for each variant; without these numbers the learned-H_res vs. mHCidentity comparison is not cleanly attributable.
  2. [Tables 3, 5, 8, 9] All main results are single-seed. Key claims rest on small gaps: Table 5 reports ΔLtest of only -0.01 and -0.02 for n=2 and n=4, and Table 8 reports Ltest 0.980 vs. 0.981 for the 7B combination. With no seeds or error bars, these differences are not distinguishable from noise, and the benchmark deltas in Table 6 and Table 9 are similarly within plausible run-to-run variation. Please provide at least 3-5 seeds for the load-bearing comparisons, or state clearly which conclusions are provisional.
  3. [Appendix F; §4.3.3] Two post-hoc choices affect the interpretation. The 20k-step training cutoff was chosen after observing LoRA's curve (Appendix F), and the mHCidentity variant was selected after observing that trained H_res matrices converge to identity (§4.3.3). This creates a selection bias: the reported comparisons are, in part, the result of inspecting the same validation signals that define the final claim. The authors should either provide a pre-specified evaluation protocol or quantify how sensitive the conclusions are to the cutoff and to the post-hoc selection of identity routing.
  4. [Appendix H.1] The initialization ablation that addresses the warm-start concern is run for only 3,000 steps. This is too short to establish that the choice of H_res initialization is irrelevant to the 20k-step results reported in Table 5. If the identity advantage is instead caused by learned-H_res runs starting with a larger effective perturbation, the initialization ablation would not detect it. Extend at least the identity/uniform/noise initialization comparison to the full training length, or report the perturbation magnitude over the whole trajectory.
minor comments (5)
  1. [Appendix C] Typo: 'Altough' should be 'Although'.
  2. [Appendix B] The sentence 'This property was tested by checking whether the vanilla OLMo and mHC-wrapped OLMo model’s final logits are sufficiently numerically close' is duplicated verbatim.
  3. [§5] The phrase 'LoRA en VeRA' mixes English and Dutch; should be 'LoRA and VeRA'.
  4. [§4.1] The sentence 'KromHC is both the most parameter-efficient and the performs the best too' is ungrammatical. Also, 'Similarily' is a typo.
  5. [§3.1 and Appendix H.3] The main text says the final readout adds n softmax parameters, while Appendix H.3 says a mean readout is used and a softmax was tested as an ablation. Please clarify which readout is used in the main experiments.

Circularity Check

0 steps flagged

No significant circularity; the identity-preservation claim is tested on fresh configurations, with only one non-load-bearing self-citation.

full rationale

The paper is empirical rather than derivational. Its central mechanistic claim — that fixing H_res to identity improves finetuning and that mHC's benefit comes mainly from h_pre/h_post routing — is supported by an interpretability observation (Section 4.3.1, Fig. 3) followed by fresh training runs with H_res := I_n (Section 4.3.3, Table 5) and by component ablations (Section 4.3.2, Table 4). These are new configurations, not re-statements of fitted values: learned-H_res and identity-H_res runs have different parameterizations, parameter counts, and training dynamics, so the comparison is not forced by construction. The only self-citation (Oldenburg et al., 2024, Section 3.4) concerns per-method learning rates in joint optimization and is not load-bearing; removing it would not change any claim. The initialization-equivalence condition (Appendices B and C) is an experimental assumption used for attribution, and its approximate verification ("roughly 1%" per sub-layer, bf16-level) could be more thorough, but this is a robustness/measurement gap, not a circular reduction: the paper does not define 'benefit of routing' in terms of that same 1% tolerance. No uniqueness theorem, ansatz-via-citation, or renaming of a known result occurs. Score 1 reflects only the presence of one minor, non-load-bearing self-citation; otherwise the derivation chain is self-contained against external baselines and held-out benchmarks.

Axiom & Free-Parameter Ledger

5 free parameters · 6 axioms · 0 invented entities

The paper introduces no new mathematical entities; it reuses the mHC framework and its variants. The key load-bearing assumptions are the warm-start/initialization equivalence and the disjointness of the finetuning data from pre-training data. The free parameters are hyperparameters that shape the central empirical claims.

free parameters (5)
  • Expansion rate n = 4 (default); 2, 8, 16 varied
    Controls number of residual streams and trainable parameters; chosen for best loss in Table 3; the central identity-preservation result is reported for several n.
  • Dynamic KromHC learning rate = 1e-3
    Tuned on validation (Appendix H.2); affects whether the identity warm-start is preserved or destabilized; the paper caps the sweep at 1e-3.
  • Training length = 20,000 steps
    Selected because LoRA's curves approached stagnation (Appendix F); all methods compared at this cutoff, which favors methods that converge quickly.
  • Symmetry-breaking noise / dropout = Gaussian σ=10^-2; Bernoulli p=0.1 on H_res
    Hand-chosen to break stream-permutation symmetry while approximately preserving the identity residual; load-bearing for the warm-start property.
  • Sinkhorn iterations / epsilon = 20 iterations; ε=1e-6
    Fixed to match Xie et al. (2026); affects how close the projected matrix is to doubly stochastic.
axioms (6)
  • standard math Sinkhorn-Knopp iteration converges to a doubly stochastic matrix for positive matrices; 20 iterations suffice in practice.
    Used in Eq. (8) and Appendix D to define H_res.
  • standard math Birkhoff-von Neumann theorem: every doubly stochastic matrix is a convex combination of permutation matrices.
    Defines mHC-lite parameterisation, Eq. (15).
  • standard math Kronecker product of doubly stochastic matrices is doubly stochastic.
    Defines KromHC constraint, Eq. (16).
  • ad hoc to paper The mHC wrapper at initialization is functionally equivalent to the frozen backbone within numerical tolerance (~1% per sub-layer deviation).
    Appendix B/C: the symmetry-breaking noise breaks exact equivalence; the paper assumes this deviation is negligible. If it compounds over 16–32 layers, the warm-start property is violated, undermining attribution of improvements to mHC learning.
  • domain assumption OLMo-2 applies pre-norm-style additive updates (Eq. 18) where the residual stream is unnormalized; wrapping each sub-layer with mHC preserves this interface.
    Section 3.1: the wrapper lifts to n streams and averages at the end; this assumes the frozen OLMo-2 architecture is compatible with multi-stream routing without changing the branch computations.
  • domain assumption Tulu-3 SFT mixture is not part of OLMo-2's pre-training data.
    Section 3.3: 'To the best of our knowledge'; if false, finetuning gains may partly reflect exposure to pre-training data, changing interpretation of the results.

pith-pipeline@v1.3.0-alltime-deepseek · 26496 in / 13212 out tokens · 105148 ms · 2026-08-01T15:53:12.620183+00:00 · methodology

0 comments
read the original abstract

Most parameter-efficient finetuning (PEFT) methods adapt weights or activations, thus leaving one of the key Transformer components unchanged: residual connections. This paper investigates Manifold-Constrained Hyper-Connections (mHC), a generalisation of residual connections, as a novel PEFT approach, wrapping frozen OLMo-2 backbones with learned residual routing modules. We find that mHC can finetune frozen Transformers, but that its role differs fundamentally from the original pre-training setting: in finetuning, fixing the residual mixing matrix to identity often improves performance. As a standalone PEFT method, mHC does not consistently outperform LoRA. However, at matched trainable parameter budgets, mHC+LoRA combinations improve language-modelling loss and show task-dependent benchmark gains at both 1B and 7B scale. Overall, our results identify residual routing as a distinct and promising novel PEFT axis.

Figures

Figures reproduced from arXiv: 2607.18130 by Bente Zuijdam, Floris de Kam, Ivo Verhoeven, Lieve Eberson, Nicky van Zutphen, Stef de Wildt, Valentijn Oldenburg.

Figure 1
Figure 1. Figure 1: Comparison of an MLP with a residual connection, Manifold-Constrained Hyper-Connections (mHC), and a LoRA adapter. 1 arXiv:2607.18130v1 [cs.LG] 20 Jul 2026 [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Architecture overview of mHC-wrapped Transformer block. B denotes batch size, T sequence length, D embedding dimension. 3.1. Static mHC as PEFT We adapt the pre-trained Transformer by wrapping each attention and MLP sub-layer with a static mHC module, i.e., where the routing logits h˜l pre, H˜ l res, and h˜l post are learned per layer and kept fixed at inference, and with routing ob￾jects h l pre, Hl res, … view at source ↗
Figure 3
Figure 3. Figure 3: Visualisation of the Hl res mixing matrices within static mHC for the 12th, 13th, 15th layer attention and MLP sub-layer. After the layers before started mixing the routing, layer 13 suddenly jumps back to a strong identity matrix. zero, indicating that the residual streams are propagated almost independently. The finding is in line with earlier interpretability research of mHC for pre-training, where deep… view at source ↗
Figure 4
Figure 4. Figure 4: shows that combining mHC with LoRA can im￾prove the trade-off between perplexity and the number of parameters. The combinations yield improved performance for only a small increase in the number of trainable parame￾ters. In particular, static mHCn=16+LoRAr=8 reduces test loss compared with LoRAr=8 with 1.282 to 1.214, with only 1280 more parameters. The results on benchmarks can be found in [PITH_FULL_IMA… view at source ↗
Figure 5
Figure 5. Figure 5: (a): Ltrain for LoRA on OLMo-2-1B. (b): corresponding Lvalidation. G. Downstream benchmarking framework To test downstream capabilities of mHC against standard PEFT baselines, all experiments are evaluated on benchmarks from the lm-evaluation-harness from (Gao et al., 2023). Every model configuration, as well as the untrained OLMo-2-1B model, is evaluated on the same benchmarks as the paper from (Xie et al… view at source ↗
Figure 6
Figure 6. Figure 6: The heatmap shows the linear Centered Kernel Alignment between all layers. Our results can be found in Figures 6, 7, and 8. Firstly, routing has been found to differ by layer, where most differences can be found in layer 4 and 13. After visualising the weights in these interesting layers from CKA (see [PITH_FULL_IMAGE:figures/full_fig_p022_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: For each prompt, we zero one stream’s matrices routes and measure the change in the correct versus incorrect logit margin. Taller bars indicate stronger causal sensitivity to that stream. Zeroing stream-specific routing weights reveals prompt-dependent causal roles [PITH_FULL_IMAGE:figures/full_fig_p023_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Each panel shows the top-1 predicted token at each layer and position for one KromHC stream. For high-confidence completions, KromHC routing does not induce stream specialisation. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Propagation Instability of HC. This figure illustrates the propagation dynamics of (a) the single-layer mapping of Hl res, here denoted as Hres l , and (b) the composite mapping QL−l i=1 Hl res, here denoted as QL−l i=1 Hres L−i , within the 27B model. The layer index l (x-axis) unrolls each standard Transformer block into two independent layers (Attention and FFN). The Amax Gain Magnitude (y-axis) is calc… view at source ↗
Figure 10
Figure 10. Figure 10: 10000 20000 30000 40000 50000 Steps -0.06 -0.04 -0.02 0.00 Absolute Loss Gap (a) Absolute Training Loss Gap vs: Training Steps Baseline HC mHC 10000 20000 30000 40000 50000 Steps 0.00 0.05 0.10 0.15 0.20 Grad Norm (b) Gradient Norm vs: Training Steps Baseline HC mHC [PITH_FULL_IMAGE:figures/full_fig_p024_10.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

41 extracted references · 27 linked inside Pith

  1. [6]

    Dua, D., Wang, Y ., Dasigi, P., Stanovsky, G., Singh, S., and Gardner, M

    URL https://arxiv.org/abs/2604.02309. Dua, D., Wang, Y ., Dasigi, P., Stanovsky, G., Singh, S., and Gardner, M. DROP: A reading comprehension bench- mark requiring discrete reasoning over paragraphs,

    arXiv

  2. [7]

    URLhttps://arxiv.org/abs/1903.00161. Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., Le Noac’h, A., Li, H., McDonell, K., Muennighoff, N., Ociepa, C., Phang, J., Reynolds, L., Schoelkopf, H., Skowron, A., Sutawika, L., Tang, E., Thite, A., Wang, B., Wang, K., and Zou, A. A framework for few-shot languag...

    Pith/arXiv arXiv 1903

  3. [8]

    org/abs/2403.14608

    URL https://arxiv. org/abs/2403.14608. He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition,

    Pith/arXiv arXiv

  4. [10]

    URL https://arxiv.org/abs/ 1902.00751. Hu, E. J., Shen, Y ., Wallis, P., Allen-Zhu, Z., Li, Y ., Wang, S., Wang, L., and Chen, W. LoRA: Low-rank adap- tation of large language models,

    Pith/arXiv arXiv 1902

  5. [11]

    Joshi, M., Choi, E., Weld, D

    URL https: //arxiv.org/abs/2106.09685. Joshi, M., Choi, E., Weld, D. S., and Zettlemoyer, L. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension,

    Pith/arXiv arXiv

  6. [13]

    doi: 10.1073/pnas.1611835114

    ISSN 1091-6490. doi: 10.1073/pnas.1611835114. URL http://dx.doi. org/10.1073/pnas.1611835114. Kopiczko, D. J., Blankevoort, T., and Asano, Y . M. VeRA: Vector-based random matrix adaptation,

    doi:10.1073/pnas.1611835114

  7. [14]

    Lacoste, A., Luccioni, A., Schmidt, V ., and Dandres, T

    URL https://arxiv.org/abs/2310.11454. Lacoste, A., Luccioni, A., Schmidt, V ., and Dandres, T. CodeCarbon,

    Pith/arXiv arXiv

  8. [16]

    Liu, H., Tam, D., Muqeeth, M., Mohta, J., Huang, T., Bansal, M., and Raffel, C

    URL https://arxiv.org/abs/2104.08691. Liu, H., Tam, D., Muqeeth, M., Mohta, J., Huang, T., Bansal, M., and Raffel, C. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learn- ing,

    Pith/arXiv arXiv

  9. [17]

    Loshchilov, I

    URLhttps://arxiv.org/abs/2603.20896. Loshchilov, I. and Hutter, F. Decoupled weight decay regu- larization,

    arXiv

  10. [18]

    Lyubinin, A

    URL https://arxiv.org/abs/ 1711.05101. Lyubinin, A. Tbp-mhc: full expressivity for manifold- constrained hyper connections through transportation polytopes,

    Pith/arXiv arXiv

  11. [19]

    Modi, S., Vishwakarma, R., Rajeev, R., Parikh, T., Patel, K., and Diamond, H

    URL https://arxiv.org/abs/ 2605.21724. Modi, S., Vishwakarma, R., Rajeev, R., Parikh, T., Patel, K., and Diamond, H. Sparse selective hyper-connections: A unified framework for stable and efficient deep residual learning. InSoutheastCon 2026, pp. 1–7,

    Pith/arXiv arXiv 2026

  12. [20]

    Nowtricity

    doi: 10.1109/SoutheastCon63549.2026.11476522. Nowtricity. CO2 emissions per kwh in the netherlands, May

    arXiv 2026

  13. [21]

    Forecasting smog clouds with deep learning: A proof-of-concept

    Oldenburg, V ., Cardenas-Cartagena, J., and Valdenegro- Toro, M. Forecasting smog clouds with deep learning: A proof-of-concept. InICML 2024 AI for Science Workshop,

    2024

  14. [22]

    URL https://arxiv.org/abs/ 2501.00656. Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., K¨opf, A., Yang, E., DeVito, Z., Rai- son, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S.PyTorch: an imperative style, high-performance deep learn...

    Pith/arXiv arXiv

  15. [23]

    Sengupta, B., Wang, J., and Brunswic, L

    URL https://arxiv.org/ abs/2603.14833. Sengupta, B., Wang, J., and Brunswic, L. Jpmhc dynamical isometry via orthogonal hyper-connections,

    arXiv

  16. [24]

    Sinkhorn, R

    URL https://arxiv.org/abs/2602.18308. Sinkhorn, R. and Knopp, P. Concerning nonnegative ma- trices and doubly stochastic matrices.Pacific Journal of Mathematics, 21(2):343–348,

    arXiv

  17. [25]

    URL https://arxiv.org/abs/2503.19206. Sun, K. and Dredze, M. Amuro and char: Analyzing the relationship between pre-training and fine-tuning of large language models,

    Pith/arXiv arXiv

  18. [26]

    Suzgun, M., Scales, N., Sch ¨arli, N., Gehrmann, S., Tay, Y ., Chung, H

    URL https://arxiv.org/ abs/2408.06663. Suzgun, M., Scales, N., Sch ¨arli, N., Gehrmann, S., Tay, Y ., Chung, H. W., Chowdhery, A., Le, Q. V ., Chi, E. H., Zhou, D., and Wei, J. Challenging BIG-bench tasks and whether chain-of-thought can solve them,

    Pith/arXiv arXiv

  19. [28]

    Veit, A., Wilber, M., and Belongie, S

    URL https://arxiv.org/ abs/1706.03762. Veit, A., Wilber, M., and Belongie, S. Residual networks be- have like ensembles of relatively shallow networks,

    Pith/arXiv arXiv

  20. [30]

    Wang, C., Li, B., Zhang, J., Liu, X., Fan, Y ., Lyu, L., Chen, X., Wang, J., Xiao, T., Pei, P., and Cai, X

    URL https://arxiv.org/abs/2606.07574. Wang, C., Li, B., Zhang, J., Liu, X., Fan, Y ., Lyu, L., Chen, X., Wang, J., Xiao, T., Pei, P., and Cai, X. Span- Norm: Reconciling training stability and performance in deep transformers,

    Pith/arXiv arXiv

  21. [31]

    org/abs/2601.22580

    URL https://arxiv. org/abs/2601.22580. Xie, Z., Wei, Y ., Cao, H., Zhao, C., Deng, C., Li, J., Dai, D., Gao, H., Chang, J., Yu, K., Zhao, L., Zhou, S., Xu, Z., Zhang, Z., Zeng, W., Hu, S., Wang, Y ., Yuan, J., Wang, L., and Liang, W. mHC: Manifold- constrained hyper-connections,

    Pith/arXiv arXiv

  22. [32]

    Xu, G., Wang, X., Wu, X., Leng, X., and Xu, Y

    URL https: //arxiv.org/abs/2512.24880. Xu, G., Wang, X., Wu, X., Leng, X., and Xu, Y . Devel- opment of residual learning in deep neural networks for computer vision: A survey.Engineering Applications of Artificial Intelligence, 142:109890,

    Pith/arXiv arXiv

  23. [33]

    Zellers, R., Holtzman, A., Bisk, Y ., Farhadi, A., and Choi, Y

    URL https://arxiv.org/ abs/2601.05732. Zellers, R., Holtzman, A., Bisk, Y ., Farhadi, A., and Choi, Y . HellaSwag: Can a machine really finish your sentence?,

    arXiv

  24. [34]

    Zhang, B

    URL https://arxiv.org/abs/ 1905.07830. Zhang, B. and Sennrich, R.Root mean square layer nor- malization. Curran Associates Inc., Red Hook, NY , USA,

    Pith/arXiv arXiv 1905

  25. [35]

    14 Manifold-Constrained Hyper-Connections for Parameter-Efficient Finetuning A

    URL https://arxiv.org/abs/2409.19606. 14 Manifold-Constrained Hyper-Connections for Parameter-Efficient Finetuning A. Hyperparameters This appendix gives more detail on the hyperparameter configurations and choices. A.1. High-level GPU memory optimisation For determining the sequence length where we strike a balance between speed and data usage, we derive...

    Pith/arXiv arXiv

  26. [36]

    This threshold ensures that we cover more than 99% of examples in allenai/tulu-3-sft-olmo-2-mixture

    For the experiments, we used a maximum sequence length of 2048 tokens. This threshold ensures that we cover more than 99% of examples in allenai/tulu-3-sft-olmo-2-mixture. We applied the same size to our other experiments. Table

    2048

  27. [37]

    Lengths are computed after applying the OLMo chat formatting and tokenizing with the allenai/OLMo-2-1B tokenizer

    Tokenized sequence length statistics for the T ¨ulu 3 OLMo SFT mixture. Lengths are computed after applying the OLMo chat formatting and tokenizing with the allenai/OLMo-2-1B tokenizer. Fractions are the proportion of examples going over each truncation threshold. Type Statistic Value Dataset Hugging Face IDallenai/tulu-3-sft-olmo-2-mixture Tokenizer Hugg...

    2048

  28. [38]

    with padding to 2048 tokens) setting

    Larger sizes, such as 8, resulted in CUDA out-of-memory errors under the worst-case (i.e. with padding to 2048 tokens) setting. Thus, we use the 4-4 configuration for all main finetuning runs. For some smaller experiments, we adjusted the batch size of 8 and 2 gradient accumulation steps. A.2. Hyperparameters of PEFT methods All experiments use a Hydra-ma...

    2048

  29. [39]

    Random entries are set to −∞ before applying the Sinkhorn normalisation

    For stage 2, during training we additionally apply Bernoulli logit dropout (p= 0.1) exclusively to Hl res. Random entries are set to −∞ before applying the Sinkhorn normalisation. We add a safety check to prevent fully masked rows or columns. This per-forward pass stochastic perturbation directly targets the residual mixing matrix where cross-stream infor...

    1967

  30. [40]

    This diverse layout ensures a robust assessment of each finetuning method

    spanse a diverse range of capabilities, from physical intuition and commonsense reasoning, to tasks such as multi-step logic and advance mathematical problems. This diverse layout ensures a robust assessment of each finetuning method. Table 15.Benchmark configurations and additional explanations/clarifications BENCHMARKCONTEXTTYPEX-SHOTCONFIGURATIONEVALUA...

    2025

  31. [41]

    For high-confidence completions, KromHC routing does not induce stream specialisation

    Each panel shows the top-1 predicted token at each layer and position for one KromHC stream. For high-confidence completions, KromHC routing does not induce stream specialisation. 23 Manifold-Constrained Hyper-Connections for Parameter-Efficient Finetuning J. Earlier reported findings on stability for HC and mHC The propagation instability of Hyper-Connec...

    2026

  32. [2015]

    Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J

    URL https:// arxiv.org/abs/1512.03385. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding, 2021a. URL https: //arxiv.org/abs/2009.03300. Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring math- ematical prob...

    Pith/arXiv arXiv 2009

  33. [2016]

    URLhttps://arxiv.org/abs/1605.06431. Wang, C. and Qiu, Y . Accelerating birkhoff projection for manifold-constrained hyper-connections,

    Pith/arXiv arXiv

  34. [2017]

    Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A

    URL https: //arxiv.org/abs/1705.03551. Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., Hassabis, D., Clopath, C., Kumaran, D., and Hadsell, R. Overcom- ing catastrophic forgetting in neural networks.Pro- ceedings of the National Academy of Sciences, 114 (13):35...

    Pith/arXiv arXiv

  35. [2019]

    Cobbe, K., Kosaraju, V ., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J

    URL https://arxiv.org/abs/ 1911.11641. Cobbe, K., Kosaraju, V ., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems,

    Pith/arXiv arXiv 1911

  36. [2021]

    org/abs/2110.14168

    URL https://arxiv. org/abs/2110.14168. Dandachi, T. and Diggs-Galligan, S. go-mhc: Direct param- eterization of manifold-constrained hyper-connections 12 Manifold-Constrained Hyper-Connections for Parameter-Efficient Finetuning via generalized orthostochastic matrices,

    Pith/arXiv arXiv

  37. [2022]

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A

    URL https://arxiv.org/abs/2210.09261. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need,

    Pith/arXiv arXiv

  38. [2023]

    Bıc ¸ıcı, E

    URL https://arxiv.org/abs/ 2307.13721. Bıc ¸ıcı, E. Residual connections improve click- through rate and conversion rate prediction perfor- mance,

    Pith/arXiv arXiv

  39. [2024]

    Birkhoff, G

    URL https://arxiv.org/abs/2405.14782. Birkhoff, G. Tres observaciones sobre el algebra lineal. Univ. Nac. Tucuman, Ser. A, 5:147–154,

    Pith/arXiv arXiv

  40. [2025]

    Lester, B., Al-Rfou, R., and Constant, N

    URLhttps://arxiv.org/abs/2411.15124. Lester, B., Al-Rfou, R., and Constant, N. The power of scale for parameter-efficient prompt tuning,

    Pith/arXiv arXiv

  41. [2026]

    Awais, M., Naseer, M., Khan, S., Anwer, R

    URL https://arxiv.org/ abs/2606.03483. Awais, M., Naseer, M., Khan, S., Anwer, R. M., Cholakkal, H., Shah, M., Yang, M.-H., and Khan, F. S. Founda- tional models defining a new era in vision: A survey and outlook,

    Pith/arXiv arXiv