Pith. sign in

REVIEW 3 major objections 5 minor 56 references

The paper's central claim is that residual-stream expansion in Transformer language models can be pushed meaningfully beyond N=4 if each layer writes richer information back into the streams and only a sparse subset of streams is updated. O

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 01:48 UTC pith:7KYAPGXP

load-bearing objection xHC is a real step forward for residual-stream expansion — the sparse dense-read design is clever and carefully ablated — but missing training-token counts keep me from fully trusting the headline gains and compute-efficiency ratios. the 3 major comments →

arxiv 2607.14530 v1 pith:7KYAPGXP submitted 2026-07-16 cs.LG cs.CL

xHC: Expanded Hyper-Connections

Xiangdong Zhang , Xiaohan Qin , Sunan Zou , Tuo Dai , Xiaoming Shi , Huaijin Wu , Yebin Yang , Zhuo Xia
show 5 more authors
Shaofeng Zhang Lin Yao Yuliang Liu Yu Cheng Junchi Yan
This is my paper
classification cs.LG cs.CL
keywords Hyper-Connectionsresidual stream expansionsparse residual updatestemporal feature augmentationmixture-of-experts language modelslanguage model pretrainingscaling lawsmemory traffic
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper is trying to establish that a Transformer's residual stream—the channel that carries token information across layers—can be expanded well beyond the four parallel streams that current Hyper-Connections methods stop at, and that this expansion can be both useful and affordable. It diagnoses why earlier attempts saturate: each layer writes only one signal back into all streams, so extra streams become redundant, while the cost of residual mixing grows cubically with stream count. xHC addresses this with temporal feature augmentation, which enriches the write-back using multi-scale causal convolutions, and a sparse residual-stream architecture that updates only k=4 of N=16 streams while reading all of them. If correct, residual-stream expansion becomes a practical scaling axis for LLM pre-training, improving both loss and downstream performance at modest FLOP overhead.

Core claim

The paper claims that Hyper-Connections-style residual-stream expansion saturates at N=4 for two correctable reasons: each layer injects only a single write-back vector into all streams, so additional streams become increasingly redundant, and generating the dense residual mixing matrix costs O(N^3C), making larger N disproportionately expensive. xHC is proposed as the first HC-family method to achieve meaningful expansion beyond N=4 by supplying diverse write-back components—the layer output plus three causal depthwise-convolution views, orthogonalized via Gram–Schmidt—and by updating only k active streams under a router while keeping dense read access to the full N-stream state. The eviden

What carries the argument

The central object is an expanded residual state made of N=16 parallel streams. The enabling mechanism is an asymmetric access pattern: every layer reads all 16 streams through a dense pre-mapping, but only k=4 active streams, chosen by a fixed-plus-routed TopK router, receive residual mixing and write-back. Write-back is enriched by temporal feature augmentation—multi-scale causal depthwise convolutions with Gram–Schmidt orthogonalization—so active streams accumulate distinct histories instead of scaled copies of the same layer output. This combination reduces residual-mapping generation cost from O(N^3C) to O(k^3C) while preserving cross-layer information flow through dense reads.

Load-bearing premise

The paper's central comparisons assume xHC, mHC, and the vanilla baseline are trained on the same number of tokens from the same data mixture; Table 6 leaves the 'Training Tokens' row blank and the data mixture is described only qualitatively, so if xHC consumed more data or a different mix, the reported gains and compute ratios could reflect data quantity rather than architecture.

What would settle it

Train all three methods with the same, explicitly fixed token budget and data mixture at 18B scale and compare the average downstream scores; if the 4.0-point gain over mHC disappears, or if the 1.50x and 1.19x compute ratios from the scaling-law fits reverse under exactly matched data, the central claim is falsified. A cheaper check is to inspect the released training logs for exact per-run token counts and mixture proportions.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Residual-stream expansion can be treated as a genuine scaling axis: N=16 with k=4 yields lower training loss and higher downstream scores than N=4 at a few percent FLOP overhead.
  • The dense-read/sparse-write pattern, with fixed streams plus TopK routing, offers a stable way to update an expanded residual memory without disconnecting cross-layer information flow.
  • xHC-Flash shows large-N expansion can be made memory-practical: per-sublayer traffic drops from 73.5C to 40C, near mHC's 34C at N=4, with almost no validation-loss change.
  • Compute-efficiency gains generalize: fitted scaling laws indicate vanilla and mHC need roughly 1.50x and 1.19x the compute of xHC to reach the same loss.
  • The gains are not optimizer-specific: xHC improves over a Muon-trained baseline as well as over AdamW.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: The information-bottleneck diagnosis suggests that any architecture maintaining multiple persistent memory states could benefit from injecting multiple write-back components; the multi-scale causal convolution recipe is a cheap candidate to test in other memory-augmented Transformer designs.
  • Editorial inference: The asymmetric sparse-write/dense-read pattern implies routing stability depends on keeping every stream readable; a testable extension is varying the fixed-versus-routed stream ratio across layers or conditioning the router on layer-specific signals.
  • Editorial inference: If the scaling-law ratios hold beyond the fitted compute range, xHC's advantage should compound on longer training runs; a natural test is measuring loss and downstream tasks at a compute budget well outside the fitted range.
  • Editorial inference: The removal of Gram–Schmidt orthogonalization under Muon hints that the procedure is largely a conditioning crutch for AdamW; directly comparing xHC with and without Gram–Schmidt under AdamW at 18B scale would clarify whether it is essential at scale.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes xHC (Expanded Hyper-Connections), an extension of the Hyper-Connections family that scales residual-stream expansion beyond N=4. xHC combines temporal feature augmentation (causal depthwise convolutions with Gram–Schmidt orthogonalization) to enrich the write-back signal, and a sparse residual-stream architecture that updates only k=4 of N=16 streams while keeping dense reads. The authors report consistent downstream improvements on 18B and 28B MoE models over mHC and a vanilla Transformer, scaling-law fits suggesting that vanilla and mHC require 1.50x and 1.19x of xHC's compute to match its loss, and a memory-traffic-reduced variant xHC-Flash that approximately matches mHC(N=4) I/O. The paper also includes ablations supporting both design choices and a Muon-optimizer compatibility test.

Significance. If the empirical claims hold, xHC demonstrates a genuinely new scaling axis for LLM pre-training — residual-stream expansion beyond the N=4 plateau — and provides a practical path to large-N via sparse updates and memory-traffic optimization. The paper ships unusually detailed derivations for the xHC-Flash dense-read reuse (Appendix E), explicit parameter/FLOPS overhead formulas (Appendix C), and controlled ablations that isolate temporal augmentation, sparse updates, dense read, and fixed streams. These are strengths. However, the central evidence for the method's superiority depends on comparisons that are only interpretable if all methods consume identical training data; the manuscript does not report the required token budgets or data-mixture proportions, which is a load-bearing gap.

major comments (3)
  1. [§4.1, Table 6, Appendix A] Training-token budgets are not reported: Table 6 lists 'Training Tokens – – – –' and leaves Global Batch Size blank for the 18B row, while Appendix A asserts 'matched optimization recipes and data budgets' without giving token counts, batch-size × steps products, or data-mixture proportions. Since xHC adds 3–4% FLOPs per token, 'comparable training FLOPs' is ambiguous: equal token budgets give xHC more total FLOPs, while equal total FLOPs give xHC fewer tokens. Without exact budgets, the headline 4.0-point downstream gain over mHC and the 1.50x/1.19x compute-efficiency ratios could reflect unequal data rather than architecture. Please report per-run token counts and dataset proportions for every model scale and for the scaling-law suite.
  2. [§4.3, Eq. (16), Appendix B] The scaling-law comparison is fit with a single shared irreducible-loss constant E=0.72 for vanilla, mHC, and xHC, and the 1.50x/1.19x compute ratios are read from the authors' fitted curves rather than measured directly. With only four model sizes per method (Table 8), the fitted parameters in Table 9 will have nontrivial uncertainty, and the shared-E assumption directly determines the implied ratios. Please report per-run compute and token counts, fit uncertainties or confidence intervals on A and α, and a sensitivity analysis with E either free per method or varied over a plausible range.
  3. [§4.2, Table 1] All downstream numbers appear to come from a single training run and a single evaluation pass, with no error bars, standard errors, or multiple seeds. For average score differences of 4.0 (18B) and 3.1 (28B) points over 12 benchmarks, evaluation noise could be nontrivial. Please state how many seeds were used, report standard errors or per-benchmark variance, and clarify whether the table caption's 'comparable training FLOPs' means equal token budgets with xHC slightly more expensive, or equal total FLOPs with xHC seeing fewer tokens.
minor comments (5)
  1. [§3.3.1, Eq. (4)] The notation out_aug ∈ R^{S×Kr×C} is inconsistent with concatenating out and r convolutional outputs, which yields (r+1)C channels. Use S×((r+1)C) or define each component as a separate C-dimensional feature.
  2. [Table 4] The 'Total I/O' row is hard to parse: the mHC column lists both 130C (N=16) and 34C (N=4), while the xHC-Flash column lists 51C and 40C. Please restructure the table so each method/variant has its own row or clearly separated subcolumns.
  3. [§5.2, Table 5] The text says xHC-Flash-4sub 'retains most of the performance gains,' but Table 5 reports equal validation loss to full xHC (1.984 vs 1.983). Consider stating this explicitly or providing downstream-task results for the Flash variants, since validation loss alone may not fully capture task-level behavior.
  4. [Appendix A] The row-sum clamping after Sinkhorn normalization means H_res is no longer exactly doubly stochastic. The paper states this stabilizes training, but it would be useful to note whether the manifold-constraint theory in §3.1 is affected by this modification.
  5. [§5.3] End-to-end throughput numbers are reported for a specific, non-overlapped setting; the sentence 'This is higher than the 6.7% reported in mHC' is a useful caveat but should also state the exact hardware and parallelism configuration used for the measurement.

Circularity Check

0 steps flagged

No construction-level circularity: claims rest on external baselines, matched ablations, and independently fitted scaling curves.

full rationale

The paper's central claims—that xHC outperforms mHC and the vanilla baseline at 18B/28B, that N=16 with k=4 is more cost-effective than dense mHC at N=16, and that temporal augmentation and sparse updates each contribute—are evaluated against external baselines and standard benchmarks under matched recipes. No equation defines a target metric in terms of xHC's own fitted parameters; the architecture equations (Eqs. 8–15, Algorithm 1) construct the method from components whose effects are separately ablated (Table 2, Figure 5). The scaling-law ratios (1.50x, 1.19x) are read from separately fitted shifted power laws per method (Table 9), with target losses taken from actual baseline runs; this is a fitting/extrapolation practice, not an identity construction. The main weaknesses—missing training-token budgets and unspecified data mixture proportions in Table 6—are reproducibility/validity risks, not circularity. The paper cites works with overlapping authors ([30], [53]) but only for optimizer context and related-work framing; neither citation is load-bearing for the xHC derivation. Under the stated rules, no specific circular reduction can be exhibited, so the correct finding is no significant circularity.

Axiom & Free-Parameter Ledger

8 free parameters · 5 axioms · 0 invented entities

The central claims rest on hand-selected architecture hyperparameters (N, k, m, conv kernels, Sinkhorn iterations) and on fitting assumptions in the scaling law (fixed E, common curve family). No fundamentally new entities are introduced.

free parameters (8)
  • Expansion rate N = 16
    Chosen as main setting; N-sweep (Figure 1, Table 7) shows loss decreases from N=2 to N=16. N is a hyperparameter selected by performance.
  • Active streams k = 4
    Ablation Table 2: k=8 gives 1.982 vs k=4 1.983 at higher cost, k=2 gives 1.991; k=4 chosen as balance.
  • Fixed streams m = 2
    Ablation (Table 2, row 8) removing fixed streams raises loss from 1.983 to 1.986; m=2 adopted.
  • Temporal conv branches r and kernel sizes = r=3, kernels {4,8,12}
    Table 12 shows 3 branches improve over 1 (1.984 vs 1.989); kernel sizes not swept beyond this choice.
  • Irreducible loss E in scaling law = 0.72
    Set by hand as 'estimated irreducible loss' before fitting A and alpha (Eq. 16). Directly affects the compute-multiplier conclusions.
  • Shifted power-law prefactor A and exponent alpha = A=97.703, alpha=0.0919 for xHC; see Table 9
    Fitted by linear regression on log loss; used to compute 1.50x and 1.19x compute ratios.
  • Sinkhorn iterations = 20
    Hyperparameter inherited from mHC; row-sum clamping added in Appendix A for stability.
  • Gating scale init alpha = 0.01
    Initialization for mapping generators; chosen to start near static initialization.
axioms (5)
  • domain assumption The mHC formulation (Sinkhorn-constrained residual mixing) is a stable and strong baseline at N=4.
    Paper builds xHC on mHC without re-deriving its stability; cites [50].
  • domain assumption Filtered versions of the same layer output constitute additional write-back 'information' for the N streams.
    Section 3.3.1/3.2; convolutions are deterministic functions of the same output, so information-theoretic novelty is not established; ablations provide indirect support.
  • domain assumption The router's sigmoid scores plus fixed streams produce useful stream selection without disconnecting information flow.
    Sections 3.3.2 and ablations (Table 2, rows 6-11); specific TopK/fixed-stream design is empirical.
  • ad hoc to paper A common shifted power law with shared E=0.72 applies to vanilla, mHC, and xHC.
    Section 4.3 and Table 9; if irreducible loss differs by method, the compute ratios are not valid.
  • domain assumption All methods are trained on matched token budgets and data mixtures.
    Appendix A says matched recipes, but Table 6 does not report training tokens, so this premise is unverifiable from the paper.

pith-pipeline@v1.3.0-alltime-deepseek · 24423 in / 11582 out tokens · 112006 ms · 2026-08-02T01:48:07.955131+00:00 · methodology

0 comments
read the original abstract

Hyper-Connections (HC) expand the residual stream of Transformers into $N$ parallel streams, providing a form of memory scaling beyond model width and depth. Manifold-Constrained HC (mHC) stabilizes this formulation at scale. The large gains from $N{=}1$ to $N{=}4$ suggest residual-stream expansion as a promising scaling axis. However, existing HC-family methods typically stop at $N{=}4$. Our experiments reveal why: scaling mHC beyond this point yields diminishing performance gains and rapidly increasing training cost. We attribute this limitation to two bottlenecks: insufficient write-back information for an expanding number of streams and residual-mixing generation whose cost scales cubically with $N$. To address both bottlenecks, we propose xHC (Expanded Hyper-Connections), the first HC-family method to achieve meaningful expansion beyond $N{=}4$. xHC combines temporal feature augmentation for richer write-back with a sparse residual-stream architecture that updates only $k=4$ of the $N=16$ streams while retaining dense access to the full residual state. Across 18B and 28B MoE models, xHC delivers strong and consistent downstream improvements. On an 18B MoE model, xHC improves the average downstream score by 4.0 points over mHC, while adding only modest training FLOPs over the vanilla baseline. Scaling-law experiments show that the vanilla and mHC require $1.50\times$ and $1.19\times$ the compute of xHC, respectively, to reach the same loss. Practical large-$N$ training also requires controlling memory traffic from the expanded residual state. We therefore introduce xHC-Flash, which reduces the per-sublayer memory traffic from $73.5C$ to $40C$, comparable to the $34C$ required by mHC at $N{=}4$, while retaining the gains of full xHC. Together, xHC and xHC-Flash make large-$N$ residual-stream expansion effective and practical for LLM pre-training.

Figures

Figures reproduced from arXiv: 2607.14530 by Huaijin Wu, Junchi Yan, Lin Yao, Shaofeng Zhang, Sunan Zou, Tuo Dai, Xiangdong Zhang, Xiaohan Qin, Xiaoming Shi, Yebin Yang, Yu Cheng, Yuliang Liu, Zhuo Xia.

Figure 1
Figure 1. Figure 1: Expansion efficiency: loss vs. FLOPs on a 2.5B MoE model (details in [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: xHC delivers broad gains at 18B scale. (a) xHC achieves lowest training loss. (b) The loss improvement translates into a significantly higher average downstream score across benchmarks in [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Overview of xHC. (a) A standard Transformer layer maintains a single residual stream. (b) mHC expands the residual state into N=4 streams with dense residual mixing and write-back. (c) xHC scales to N=16 with only k=4 active streams: it reads all streams, applies the sublayer F (Attn/MLP), augments MLP outputs, and sparsely writes back to selected streams. Blue/orange streams denote fixed/routed active str… view at source ↗
Figure 4
Figure 4. Figure 4: Scaling-law comparison. xHC traces a consistently lower loss curve than both mHC and [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Information bot￾tleneck ablation. The gain grows with N. at matched expansion rates. In mHC, all N streams participate in dense residual mixing, so the residual-mixing overhead grows rapidly with N. In xHC, increasing N enlarges residual-memory capacity while the sparse update cost remains controlled by k [PITH_FULL_IMAGE:figures/full_fig_p010_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

56 extracted references · 30 linked inside Pith

  1. [1]

    Ainslie, J

    J. Ainslie, J. Lee-Thorp, M. De Jong, Y . Zemlyanskiy, F. Lebrón, and S. Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4895–4901, 2023

    2023

  2. [2]

    J. L. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization.arXiv preprint arXiv:1607.06450, 2016

    Pith/arXiv arXiv 2016

  3. [3]

    Bachlechner, B

    T. Bachlechner, B. P. Majumder, H. Mao, G. Cottrell, and J. McAuley. Rezero is all you need: Fast convergence at large depth. InUncertainty in artificial intelligence, pages 1352–1361. PMLR, 2021

    2021

  4. [4]

    Beltagy, M

    I. Beltagy, M. E. Peters, and A. Cohan. Longformer: The long-document transformer.arXiv preprint arXiv:2004.05150, 2020

    Pith/arXiv arXiv 2004

  5. [5]

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y . Burda, N. Joseph, G. Brockman, et al. Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374, 2021

    Pith/arXiv arXiv 2021

  6. [6]

    Child, S

    R. Child, S. Gray, A. Radford, and I. Sutskever. Generating long sequences with sparse transformers.arXiv preprint arXiv:1904.10509, 2019

    Pith/arXiv arXiv 1904

  7. [7]

    Clark, I

    P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge.arXiv preprint arXiv:1803.05457, 2018

    Pith/arXiv arXiv 2018

  8. [8]

    Cobbe, V

    K. Cobbe, V . Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    Pith/arXiv arXiv 2021

  9. [9]

    Contributors

    O. Contributors. Opencompass: A universal evaluation platform for foundation models. https: //github.com/open-compass/opencompass, 2023

    2023

  10. [10]

    D. Dai, C. Deng, C. Zhao, R. Xu, H. Gao, D. Chen, J. Li, W. Zeng, X. Yu, Y . Wu, et al. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1280–1297, 2024

    2024

  11. [11]

    L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling.arXiv preprint arXiv:2101.00027, 2020

    Pith/arXiv arXiv 2020

  12. [12]

    A. P. Gema, J. O. J. Leang, G. Hong, A. Devoto, A. C. M. Mancino, R. Saxena, X. He, Y . Zhao, X. Du, M. R. G. Madani, et al. Are we done with mmlu? InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 5069–5096, 2025

    2025

  13. [13]

    G. H. Golub and C. F. Van Loan.Matrix computations. JHU press, 2013

    2013

  14. [14]

    Grattafiori, A

    A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Pith/arXiv arXiv 2024

  15. [15]

    Gu and T

    A. Gu and T. Dao. Mamba: Linear-time sequence modeling with selective state spaces.arXiv preprint arXiv:2312.00752, 2023

    Pith/arXiv arXiv 2023

  16. [16]

    Gulati, J

    A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y . Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y . Wu, et al. Conformer: Convolution-augmented transformer for speech recognition.arXiv preprint arXiv:2005.08100, 2020

    Pith/arXiv arXiv 2005

  17. [17]

    K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770– 778, 2016. 17

    2016

  18. [18]

    Hendrycks, C

    D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300, 2020

    Pith/arXiv arXiv 2009

  19. [19]

    Henighan, J

    T. Henighan, J. Kaplan, M. Katz, M. Chen, C. Hesse, J. Jackson, H. Jun, T. B. Brown, P. Dhari- wal, S. Gray, et al. Scaling laws for autoregressive generative modeling.arXiv preprint arXiv:2010.14701, 2020

    Pith/arXiv arXiv 2010

  20. [20]

    Hoffmann, S

    J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. Casas, L. A. Hendricks, J. Welbl, A. Clark, et al. Training compute-optimal large language models.arXiv preprint arXiv:2203.15556, 10, 2022

    Pith/arXiv arXiv 2022

  21. [21]

    Huang, Z

    G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger. Densely connected convolutional networks. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 4700–4708, 2017

    2017

  22. [22]

    Huang, Y

    Y . Huang, Y . Bai, Z. Zhu, J. Zhang, J. Zhang, T. Su, J. Liu, C. Lv, Y . Zhang, Y . Fu, et al. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models.Advances in neural information processing systems, 36:62991–63010, 2023

    2023

  23. [23]

    A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. d. l. Casas, E. B. Hanna, F. Bressand, et al. Mixtral of experts.arXiv preprint arXiv:2401.04088, 2024

    Pith/arXiv arXiv 2024

  24. [24]

    Jordan, Y

    K. Jordan, Y . Jin, V . Boza, Y . Jiacheng, F. Cesista, L. Newhouse, and J. Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024.URL https://kellerjordan. github. io/posts/muon, 6(3):4, 2024

    2024

  25. [25]

    Kaplan, S

    J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Rad- ford, J. Wu, and D. Amodei. Scaling laws for neural language models.arXiv preprint arXiv:2001.08361, 2020

    Pith/arXiv arXiv 2001

  26. [26]

    Lepikhin, H

    D. Lepikhin, H. Lee, Y . Xu, D. Chen, O. Firat, Y . Huang, M. Krikun, N. Shazeer, and Z. Chen. Gshard: Scaling giant models with conditional computation and automatic sharding.arXiv preprint arXiv:2006.16668, 2020

    Pith/arXiv arXiv 2006

  27. [27]

    H. Li, Y . Zhang, F. Koto, Y . Yang, H. Zhao, Y . Gong, N. Duan, and T. Baldwin. Cmmlu: Measuring massive multitask language understanding in chinese. InFindings of the Association for Computational Linguistics: ACL 2024, pages 11260–11285, 2024

    2024

  28. [28]

    A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan, et al. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    Pith/arXiv arXiv 2024

  29. [29]

    A. Liu, A. Mei, B. Lin, B. Xue, B. Wang, B. Xu, B. Wu, B. Zhang, C. Lin, C. Dong, et al. Deepseek-v3. 2: Pushing the frontier of open large language models.arXiv preprint arXiv:2512.02556, 2025

    Pith/arXiv arXiv 2025

  30. [30]

    J. Liu, J. Su, X. Yao, Z. Jiang, G. Lai, Y . Du, Y . Qin, W. Xu, E. Lu, J. Yan, et al. Muon is scalable for llm training.arXiv preprint arXiv:2502.16982, 2025

    Pith/arXiv arXiv 2025

  31. [31]

    Z. Liu, H. Zhang, and A. Li. Beyond the birkhoff polytope: Spectral-sphere-constrained hyper-connections.arXiv preprint arXiv:2603.20896, 2026

    arXiv 2026

  32. [32]

    Loshchilov and F

    I. Loshchilov and F. Hutter. Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101, 2017

    Pith/arXiv arXiv 2017

  33. [33]

    Pagliardini, A

    M. Pagliardini, A. Mohtashami, F. Fleuret, and M. Jaggi. Denseformer: Enhancing information flow in transformers via depth weighted averaging.Advances in neural information processing systems, 37:136479–136508, 2024

    2024

  34. [34]

    N. Shazeer. Fast transformer decoding: One write-head is all you need.arXiv preprint arXiv:1911.02150, 2019

    Pith/arXiv arXiv 1911

  35. [35]

    N. Shazeer. Glu variants improve transformer.arXiv preprint arXiv:2002.05202, 2020. 18

    Pith/arXiv arXiv 2002

  36. [36]

    Shazeer, A

    N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean. Outra- geously large neural networks: The sparsely-gated mixture-of-experts layer.arXiv preprint arXiv:1701.06538, 2017

    Pith/arXiv arXiv 2017

  37. [37]

    Sinkhorn and P

    R. Sinkhorn and P. Knopp. Concerning nonnegative matrices and doubly stochastic matrices. Pacific Journal of Mathematics, 21(2):343–348, 1967

    1967

  38. [38]

    R. K. Srivastava, K. Greff, and J. Schmidhuber. Highway networks.arXiv preprint arXiv:1505.00387, 2015

    Pith/arXiv arXiv 2015

  39. [39]

    J. Su, M. Ahmed, Y . Lu, S. Pan, W. Bo, and Y . Liu. Roformer: Enhanced transformer with rotary position embedding.Neurocomputing, 568:127063, 2024

    2024

  40. [40]

    K. Sun, D. Yu, D. Yu, and C. Cardie. Investigating prior knowledge for challenging chinese machine reading comprehension.Transactions of the Association for Computational Linguistics, 8:141–155, 2020

    2020

  41. [41]

    Suzgun, N

    M. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y . Tay, H. W. Chung, A. Chowdhery, Q. Le, E. Chi, D. Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. InFindings of the Association for Computational Linguistics: ACL 2023, pages 13003–13051, 2023

    2023

  42. [42]

    Talmor, J

    A. Talmor, J. Herzig, N. Lourie, and J. Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge. InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4149–4158, 2019

    2019

  43. [43]

    K. Team, Y . Bai, Y . Bao, Y . Charles, C. Chen, G. Chen, H. Chen, H. Chen, J. Chen, N. Chen, et al. Kimi k2: Open agentic intelligence.arXiv preprint arXiv:2507.20534, 2025

    Pith/arXiv arXiv 2025

  44. [44]

    K. Team, G. Chen, Y . Zhang, J. Su, W. Xu, S. Pan, Y . Wang, Y . Wang, G. Chen, B. Yin, et al. Attention residuals.arXiv preprint arXiv:2603.15031, 2026

    Pith/arXiv arXiv 2026

  45. [45]

    Touvron, M

    H. Touvron, M. Cord, A. Sablayrolles, G. Synnaeve, and H. Jégou. Going deeper with image transformers. InProceedings of the IEEE/CVF international conference on computer vision, pages 32–42, 2021

    2021

  46. [46]

    Vaswani, N

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is all you need.Advances in neural information processing systems, 30, 2017

    2017

  47. [47]

    H. Wang, S. Ma, L. Dong, S. Huang, D. Zhang, and F. Wei. Deepnet: Scaling transformers to 1,000 layers.IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(10):6761– 6774, 2024

    2024

  48. [48]

    Y . Wang, X. Ma, G. Zhang, Y . Ni, A. Chandra, S. Guo, W. Ren, A. Arulraj, X. He, Z. Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. Advances in Neural Information Processing Systems, 37:95266–95290, 2024

    2024

  49. [49]

    F. Wu, A. Fan, A. Baevski, Y . N. Dauphin, and M. Auli. Pay less attention with lightweight and dynamic convolutions.arXiv preprint arXiv:1901.10430, 2019

    Pith/arXiv arXiv 1901

  50. [50]

    Z. Xie, Y . Wei, H. Cao, C. Zhao, C. Deng, J. Li, D. Dai, H. Gao, J. Chang, K. Yu, et al. mhc: Manifold-constrained hyper-connections.arXiv preprint arXiv:2512.24880, 2025

    Pith/arXiv arXiv 2025

  51. [51]

    A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    Pith/arXiv arXiv 2025

  52. [52]

    Yang and J

    Y . Yang and J. Gao. mhc-lite: You don’t need 20 sinkhorn-knopp iterations.arXiv preprint arXiv:2601.05732, 2026

    arXiv 2026

  53. [53]

    Y . Yang, H. Wu, F. Guo, L. Yao, X. Qin, J. Wang, D. Zhang, and J. Yan. Jtok: On token embedding as another axis of scaling law via joint token self-modulation.arXiv preprint arXiv:2602.00800, 2026. 19

    arXiv 2026

  54. [54]

    J. Yuan, H. Gao, D. Dai, J. Luo, L. Zhao, Z. Zhang, Z. Xie, Y . Wei, L. Wang, Z. Xiao, et al. Native sparse attention: Hardware-aligned and natively trainable sparse attention. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 23078–23097, 2025

    2025

  55. [55]

    D. Zhu, H. Huang, Z. Huang, Y . Zeng, Y . Mao, B. Wu, Q. Min, and X. Zhou. Hyper-connections. arXiv preprint arXiv:2409.19606, 2024

    Pith/arXiv arXiv 2024

  56. [56]

    L. Zhu, Y . Fang, B. Liao, S. Wang, T. Cheng, Z. Huang, C. Chen, L. Wei, Y . Zeng, Y . Wang, et al. Mixture-of-depths attention.arXiv preprint arXiv:2603.15619, 2026. 20 Table 6: Model configurations and training hyperparameters for the main model scales. The 2.5B setting is used for N-sweep experiments, the 10B setting for ablations, and the 18B/28B sett...

    arXiv 2026