Pith. sign in

REVIEW 2 major objections 5 minor 62 references

Dual Attention Residuals: letting one residual stream choose the other's history depths improves language model validation loss across dense and MoE Transformers.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 14:30 UTC pith:A7Y2Z4EM

load-bearing objection DAR's reciprocal cross-stream retrieval is new and carefully argued, but the loss improvements rest on single runs and need seed-level evidence before they convince. the 2 major comments →

arxiv 2607.18730 v1 pith:A7Y2Z4EM submitted 2026-07-21 cs.CL

Dual Attention Residuals

Xingda Yu , Yining Li , Xinzhang Liu , Zhihao Yang , Haowei He , Chao Wang , Yongxiang Li , Shuangyong Song This is my paper
classification cs.CL
keywords dual attention residualsresidual streamshistorical retrievalcross-stream addressingdepth-wise attentionTransformer residual connectionsgated writessparse MoE
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Dual Attention Residuals (DAR) proposes to combine two previously separate ideas in Transformer design: retrieving information from earlier depths and maintaining multiple residual streams. The central claim is that the two can interact productively if each stream's depth selection is driven by the other stream's state: DAR computes depth-attention weights over historical states using a key from the opposite stream, then aggregates values from the target stream's own history. The paper reports consistent validation-loss improvements over standard residual Transformers and Attention Residuals across dense models from 0.1B to 1B parameters and a 7B sparse-MoE model. Routing ablations indicate the gain stems from this reciprocal cross-stream addressing rather than from having an extra stream or extra value projections, and representation analyses suggest both streams remain functionally distinct. If correct, this means trajectory diversity can serve as an addressing signal, not just a content pathway.

Core claim

DAR generalizes Attention Residuals from a single residual stream to two. For each target stream j, the depth weight of each historical entry is computed as a softmax over dot products between stream j's query and the RMS-normalized state of the opposite stream (1-j), while the value aggregated under that weight is the target stream's own history entry. This decouples addressing from content: one trajectory supplies the key that selects depths, the other supplies the values to be read. Retrieved states from both streams are mixed to form the input to an unchanged Transformer branch, whose output is written back into both streams through learned gates; within a block, existing partial states

What carries the argument

The load-bearing object is the reciprocal cross-stream key-value routing rule: for each stream j and historical candidate r, the key is the RMS-normalized opposite-stream state Norm_h(C^{1-j}_r) and the value is the same stream's unprojected state C^j_r. This one-line change makes depth selection a joint choice over depth and stream roles: one trajectory's state determines where the other trajectory retrieves from, while each stream's own history determines what is aggregated. The mechanism is supported by gated branch writes (beta gates) and a constrained two-stream mixing matrix (rho) that transport partial states within blocks, with a block-form variant retrieving from completed block his

Load-bearing premise

DAR's advantage depends on the two residual streams remaining functionally distinct; if training at scale drives them toward collinearity or near-duplication, an opposite-stream key stops adding information and the method degenerates to within-stream retrieval.

What would settle it

An ablation that replaces the opposite-stream normalized keys with random fixed vectors of matched statistics should close the performance gap if the reciprocal cross-stream signal is the true source of DAR's gain; if it does not, the claimed mechanism is not the active ingredient.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If DAR's central claim holds, multi-stream residual pathways are not just parallel information highways: their states can serve as complementary addressing signals for depth-wise retrieval, suggesting new designs where streams specialize dynamically rather than by fixed roles.
  • DAR improves validation loss over standard residuals and Attention Residuals at 1B dense and 7B MoE scales, implying the benefit persists as models grow, at least within the tested range.
  • The routing ablations show that simply adding a second stream or source-specific value projections does not recover the gain, so the reciprocal cross-stream key is the active ingredient.
  • The intervention analyses indicate both streams remain functionally useful in DAR, whereas alternative two-stream designs (self-stream retrieval, fixed key-value roles) tend toward one-sided dominance or redundancy; this suggests reciprocal addressing also improves training dynamics, not just final loss.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: The reciprocal-addressing principle may generalize beyond two streams; one could test whether a cycle of streams, where stream j's key comes from stream j-1, preserves the benefit or introduces instability.
  • Editorial inference: If the advantage comes from richer addressing signals, DAR might show larger gains on tasks requiring selective long-range retrieval (e.g., long-context question answering or copying) than on next-token prediction; this is testable with existing benchmarks.
  • Editorial inference: The paper's collapse argument for branch-wise blocks suggests a general design constraint for multi-stream retrieval: each stream must have had time to diverge from the others before its state is used as a key. This could inform when and how to initialize additional streams.
  • Editorial inference: The CKA and rescue analyses are on one 0.5B model; verifying the same complementarity at 7B would be a direct stress test of the proposed mechanism.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. This paper proposes Dual Attention Residuals (DAR), a modification to the Transformer residual pathway that combines multi-stream residual transport with depth-wise historical retrieval. In DAR, for each of two residual streams, depth weights are computed from normalized states in the opposite stream (cross-stream keys) and applied to values from the target stream's own history. Retrieved states are mixed for an unchanged attention/FFN branch, and the streams are updated via gated writes and mHC-style constrained mixing. A block-form variant (Block DAR) retrieves from completed block histories at a coarser granularity. The paper reports validation-loss improvements over standard residual baselines and Attention Residuals for dense models from 0.1B to 1B parameters and a 7B sparse MoE model, routing ablations favoring the cross-stream-key design, and CKA/rescue analyses intended to show stream non-degeneracy and functional utility.

Significance. If the empirical claims hold, DAR is a clean and moderately novel contribution: it decouples depth-selection addressing from the retrieved content, and it explicitly avoids the trivial branch-wise collapse (S=1, Eqs. 19–20) that would make the two streams collinear. The mathematical formulation is coherent, the pseudocode in Figure A.1 matches the equations, and the online-softmax merge is derived exactly. The design comparisons and ablations are well structured. The main weakness is that the central empirical claim rests on single training runs with no seed variation; several decisive gaps are small, which limits the weight that can be placed on the reported ordering. The representation and intervention analyses are thought-provoking but qualitative and equally single-run.

major comments (2)
  1. [Experiments / Tables 1–2 and Table 4] All results are from single training runs with random seed 42 and fixed data order. The abstract's central claim that DAR 'consistently improves validation loss' is supported only by point estimates. Some differences are small: at 1B, DAR-Block 2.684 vs AttnRes-Block 2.704 (Δ=0.020); in the 0.5B routing ablation, DAR 2.653 vs DAR-CrossV 2.664 (Δ=0.011) and vs DAR-SelfKV 2.678 (Δ=0.025). Such gaps can be within run-to-run variation for large-model training, especially when the architecture changes optimization dynamics. Please report multiple seeds (at least 3) with confidence intervals, or otherwise justify why these differences are beyond seed-level noise, for the key comparisons: the 1B dense result, the 7B MoE result, and the Table 4 ablation ordering.
  2. [Analysis (CKA and rescue gain), Figures 5–6] The conclusion that reciprocal cross-stream selection 'preserves depth-wise diversity' and avoids functional imbalance is based on qualitative visual patterns from a single 0.5B run, using 4 validation sequences and the final 128 tokens. No quantitative summary statistic (e.g., mean off-diagonal CKA, confidence intervals on rescue-gain asymmetry) is reported, and the analyses are not repeated at other scales. This makes the mechanistic interpretation plausible but not demonstrated. Consider adding quantitative CKA summaries and rescue-gain asymmetries with variability estimates, and, if feasible, a check at 1B or 7B to show the pattern persists at scale.
minor comments (5)
  1. [Author affiliation line] The GitHub identifier in the author line is malformed ('/githubTeleai/Dual-Attention-Residuals') and should appear as a complete URL.
  2. [Eq. (13)] The notation Norm([h0_l; h1_l]) is undefined regarding the normalization axis; clarify that it is RMSNorm along the hidden dimension, as in Eq. (2).
  3. [Section 6.1 / Figures 3 and A.3] The retrieval-weight heatmaps are averaged over only eight sequences of length 4096. Please state whether these are validation sequences and report some measure of variance or at least note the small sample.
  4. [Section 4.2 / Table A.3] The comparison with mHC is not parameter-matched: mHC has substantially more parameters at every scale (e.g., 0.5B mHC 496.5M vs DAR-Full 492.2M). The abstract does not claim improvement over mHC, but the text says DAR 'also achieved lower loss than mHC'; readers may over-read this. Consider adding a parameter-matched mHC baseline or softening the claim.
  5. [Reproducibility] No plan for releasing code or checkpoints is stated. The pseudocode and configuration tables are useful, but a release would materially strengthen trust in the single-run results.

Circularity Check

0 steps flagged

No circularity: DAR is an independently specified architecture whose gains are held-out validation measurements, with no fitted quantity renamed as a prediction and no load-bearing self-citation.

full rationale

The derivation chain is not circular. DAR is defined by explicit equations (10)-(17), building on the positive-kernel retrieval of AttnRes (Eqs. 2-3) and the mHC transport M(ρ) (Eq. 16); these are adopted as prior formulations, not as outputs of the present claim. The central claim 'DAR consistently improves validation loss over standard residual Transformers and Attention Residuals' is supported by validation losses in Tables 1, 2, and 4, measured under matched training budgets and held-out data; no target result is used to fit a parameter or define an architecture. The routing ablations (Eq. 22, Table 4) and representation/intervention analyses are post hoc interpretations of the same trained runs, but they do not define the outcome; they are separate empirical checks. The S=1 degeneracy argument (Eqs. 19-20) is a mathematical observation used to justify the layer-wise block choice, not a circular reduction. The paper's single-seed design and lack of error bars are validity/robustness concerns, not circularity: they do not make the comparison equivalent to its own inputs by construction. There is no evidence of load-bearing self-citation, uniqueness-theorem import, or ansatz smuggled in via citation as if derived. The finding is therefore a non-finding: score 0.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

No new physical or ontological entities are postulated. The two-stream residual trajectory is an architectural construction whose evidence is the in-paper validation and analysis, not an out-of-paper falsifiable entity. The learned weights and block-size choices are the fitted components of the method.

free parameters (3)
  • Block size K = 2 (dense), 3 (sparse MoE)
    Chosen granularity for Block DAR history; no sweep is reported, and the central results depend on this choice.
  • Number of residual streams = 2
    Fixed by design; the central claim is specifically about reciprocal two-stream addressing. Not swept.
  • Stream-specific queries and gating projections (q^j_l, W_alpha, W_beta, W_rho with biases) = learned; counts in Tables A.3/A.5
    All are trained on the pretraining objective; they are the mechanism that produces the reported losses, so they are fitted rather than predicted.
axioms (4)
  • domain assumption Pre-Norm decoder-only Transformer with separate attention and feed-forward residual branches (Methods, Eq. 1).
    The architecture is evaluated only in this setting; DAR's design assumes branch outputs can be gated and written into two streams.
  • domain assumption Positive softmax retrieval kernel from Attention Residuals (Eq. 2-3).
    DAR builds on the exp(q^T Norm(k)) kernel for depth weights; if a different kernel changed retrieval, results could differ.
  • domain assumption mHC constrained transport M(rho) stabilizes multi-stream updates (Eq. 16-17).
    The partial-state update borrows mHC's mixing matrix; the paper does not re-derive its stability.
  • domain assumption Stream non-degeneracy is maintained (Section 'Full DAR at Layer Granularity', Eqs. 19-20; CKA analysis).
    The central mechanism requires the two streams to diverge; the paper proves the S=1 failure and empirically checks divergence for S=2, but does not guarantee it at all scales.

pith-pipeline@v1.3.0-alltime-deepseek · 14785 in / 13937 out tokens · 111562 ms · 2026-08-01T14:30:41.238846+00:00 · methodology

0 comments
read the original abstract

Recent work extends Transformer residual pathways along two complementary axes: historical retrieval selects information from earlier depths, whereas multi-stream methods maintain multiple residual trajectories. These capabilities have largely been studied in isolation, and assigning an independent retriever to each stream still prevents one trajectory from influencing depth selection in another. We propose Dual Attention Residuals (DAR), which brings multi-stream interaction into historical retrieval through reciprocal cross-stream addressing. For each target stream, DAR computes depth weights from normalized states in the opposite stream and applies them to values from the target stream's own history. The retrieved states are combined for an unchanged Transformer branch and updated through constrained gated writes; a block-form variant operates on block-level histories to control overhead. Across dense models from 0.1B to 1B parameters and a 7B sparse-MoE model, DAR consistently improves validation loss over standard residual Transformers and Attention Residuals. Routing ablations show that the gain cannot be explained by an additional stream or value projection alone. Representation and intervention analyses further show that reciprocal cross-stream selection preserves depth-wise diversity and avoids the redundancy or functional imbalance observed in alternative two-stream designs.

Figures

Figures reproduced from arXiv: 2607.18730 by Chao Wang, Haowei He, Shuangyong Song, Xingda Yu, Xinzhang Liu, Yining Li, Yongxiang Li, Zhihao Yang.

Figure 1
Figure 1. Figure 1: Block DAR, shown for branch i > 1 in block b. Opposite-stream keys retrieve self-stream values, which are mixed for an unchanged Transformer branch; its gated output combines with constrained transport of P i−1 b to form P i b . The partial path is omitted for i = 1, and P S b is appended as Hb at the block boundary. the first residual branch in block b and remains local to that block; after its S-th branc… view at source ↗
Figure 2
Figure 2. Figure 2: Training loss for the 1B dense models over 50B [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Pre-attention historical retrieval weights for sparse-MoE DAR, averaged over eight sequences of length 4096. Rows [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Training loss for the 7B sparse-MoE models on [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Cross-stream linear CKA for post-MLP residual-stream states in the 0.5B models. Rows are stream-0 layers and [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Post-MLP residual-stream rescue gain by layer in the 0.5B models. Columns indicate the retained stream; values are [PITH_FULL_IMAGE:figures/full_fig_p007_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

62 extracted references · 3 canonical work pages · 1 internal anchor

  1. [1]

    Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages =

    He, Kaiming and Zhang, Xiangyu and Ren, Shaoqing and Sun, Jian , title =. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages =. 2016 , doi =

    2016

  2. [2]

    and Kaiser, Lukasz and Polosukhin, Illia , title =

    Vaswani, Ashish and Shazeer, Noam and Parmar, Niki and Uszkoreit, Jakob and Jones, Llion and Gomez, Aidan N. and Kaiser, Lukasz and Polosukhin, Illia , title =. Advances in Neural Information Processing Systems 30 , pages =. 2017 , url =

    2017

  3. [3]

    Highway Networks , journal =

    Srivastava, Rupesh Kumar and Greff, Klaus and Schmidhuber, J. Highway Networks , journal =. 2015 , doi =

    2015

  4. [4]

    , title =

    Huang, Gao and Liu, Zhuang and van der Maaten, Laurens and Weinberger, Kilian Q. , title =. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages =. 2017 , doi =

    2017

  5. [5]

    International Conference on Learning Representations , year =

    Dehghani, Mostafa and Gouws, Stephan and Vinyals, Oriol and Uszkoreit, Jakob and Kaiser, Lukasz , title =. International Conference on Learning Representations , year =

  6. [6]

    Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 , pages =

    He, Ruining and Ravula, Anirudh and Kanagal, Bhargav and Ainslie, Joshua , title =. Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 , pages =. 2021 , publisher =. doi:10.18653/v1/2021.findings-acl.81 , url =

    doi:10.18653/v1/2021.findings-acl.81 2021

  7. [7]

    IEEE Transactions on Pattern Analysis and Machine Intelligence , volume =

    Wang, Hongyu and Ma, Shuming and Dong, Li and Huang, Shaohan and Zhang, Dongdong and Wei, Furu , title =. IEEE Transactions on Pattern Analysis and Machine Intelligence , volume =. 2024 , doi =

    2024

  8. [8]

    International Conference on Learning Representations , year =

    Zhu, Defa and Huang, Hongzhi and Huang, Zihao and Zeng, Yutao and Mao, Yunyao and Wu, Banggu and Min, Qiyang and Zhou, Xun , title =. International Conference on Learning Representations , year =

  9. [9]

    arXiv preprint arXiv:2503.14125 , year =

    Zhu, Defa and Huang, Hongzhi and Zhou, Jundong and Huang, Zihao and Zeng, Yutao and Wu, Banggu and Min, Qiyang and Zhou, Xun , title =. arXiv preprint arXiv:2503.14125 , year =. doi:10.48550/arXiv.2503.14125 , url =

    Pith/arXiv arXiv doi:10.48550/arxiv.2503.14125

  10. [10]

    International Conference on Machine Learning , year =

    Xie, Zhenda and Wei, Yixuan and Cao, Huanqi and Zhao, Chenggang and Deng, Chengqi and Li, Jiashi and Dai, Damai and Gao, Huazuo and Xu, Mingyu and Yu, Kuai and Zhao, Liang and Zhou, Shangyan and Xu, Zhean and Zhang, Zhengyan and Zeng, Wangding and Hu, Shengding and Wang, Yuqing and Yuan, Jingyang and Wang, Lean and Liang, Wenfeng , title =. International ...

  11. [11]

    2026 , doi =

    Attention Residuals , journal =. 2026 , doi =

    2026

  12. [12]

    Delta Attention Residuals

    Luo, Cheng and Cai, Zefan and Hu, Junjie , title =. arXiv preprint arXiv:2605.18855 , year =. doi:10.48550/arXiv.2605.18855 , url =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2605.18855

  13. [13]

    Advances in Neural Information Processing Systems 37 , pages =

    Pagliardini, Matteo and Mohtashami, Amirkeivan and Fleuret, Fran. Advances in Neural Information Processing Systems 37 , pages =. 2024 , doi =

    2024

  14. [14]

    Proceedings of the 42nd International Conference on Machine Learning , pages =

    Xiao, Da and Meng, Qingye and Li, Shengping and Yuan, Xingyuan , title =. Proceedings of the 42nd International Conference on Machine Learning , pages =. 2025 , url =

    2025

  15. [15]

    Proceedings of the 42nd International Conference on Machine Learning , pages =

    Heddes, Mike and Javanmard, Adel and Axiotis, Kyriakos and Fu, Gang and Bateni, Mohammadhossein and Mirrokni, Vahab , title =. Proceedings of the 42nd International Conference on Machine Learning , pages =. 2025 , url =

    2025

  16. [16]

    , title =

    Ba, Jimmy Lei and Kiros, Jamie Ryan and Hinton, Geoffrey E. , title =. arXiv preprint arXiv:1607.06450 , year =. doi:10.48550/arXiv.1607.06450 , url =

    Pith/arXiv arXiv doi:10.48550/arxiv.1607.06450

  17. [17]

    Proceedings of the 37th International Conference on Machine Learning , pages =

    Xiong, Ruibin and Yang, Yunchang and He, Di and Zheng, Kai and Zheng, Shuxin and Xing, Chen and Zhang, Huishuai and Lan, Yanyan and Wang, Liwei and Liu, Tie-Yan , title =. Proceedings of the 37th International Conference on Machine Learning , pages =. 2020 , url =

    2020

  18. [18]

    Advances in Neural Information Processing Systems 32 , pages =

    Zhang, Biao and Sennrich, Rico , title =. Advances in Neural Information Processing Systems 32 , pages =. 2019 , url =

    2019

  19. [19]

    Proceedings of the Thirty-Seventh Conference on Uncertainty in Artificial Intelligence , pages =

    Bachlechner, Thomas and Majumder, Bodhisattwa Prasad and Mao, Henry and Cottrell, Gary and McAuley, Julian , title =. Proceedings of the Thirty-Seventh Conference on Uncertainty in Artificial Intelligence , pages =. 2021 , url =

    2021

  20. [20]

    arXiv preprint arXiv:2110.09456 , year =

    Shleifer, Sam and Weston, Jason and Ott, Myle , title =. arXiv preprint arXiv:2110.09456 , year =. doi:10.48550/arXiv.2110.09456 , url =

    Pith/arXiv arXiv doi:10.48550/arxiv.2110.09456

  21. [21]

    and Chao, Lidia S

    Wang, Qiang and Li, Bei and Xiao, Tong and Zhu, Jingbo and Li, Changliang and Wong, Derek F. and Chao, Lidia S. , title =. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages =. 2019 , publisher =. doi:10.18653/v1/P19-1176 , url =

    work page doi:10.18653/v1/p19-1176 2019

  22. [22]

    Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pages =

    Bapna, Ankur and Chen, Mia Xu and Firat, Orhan and Cao, Yuan and Wu, Yonghui , title =. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pages =. 2018 , publisher =. doi:10.18653/v1/D18-1338 , url =

    work page doi:10.18653/v1/d18-1338 2018

  23. [23]

    and Salakhutdinov, Ruslan , title =

    Dai, Zihang and Yang, Zhilin and Yang, Yiming and Carbonell, Jaime and Le, Quoc V. and Salakhutdinov, Ruslan , title =. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages =. 2019 , publisher =. doi:10.18653/v1/P19-1285 , url =

    doi:10.18653/v1/p19-1285 2019

  24. [24]

    ICLR 2026 Workshop on Scientific Methods for Understanding Deep Learning , year =

    Peng, William and Rai, Josheev and Tseng, Kevin and Wang, Siwei and Wu, Sean , title =. ICLR 2026 Workshop on Scientific Methods for Understanding Deep Learning , year =

    2026

  25. [25]

    Penedo, Guilherme and Kydl. The. Advances in Neural Information Processing Systems 37 , pages =. 2024 , doi =

    2024

  26. [26]

    Proceedings of the 36th International Conference on Machine Learning , pages =

    Kornblith, Simon and Norouzi, Mohammad and Lee, Honglak and Hinton, Geoffrey , title =. Proceedings of the 36th International Conference on Machine Learning , pages =. 2019 , url =

    2019

  27. [27]

    International Conference on Learning Representations , year =

    Loshchilov, Ilya and Hutter, Frank , title =. International Conference on Learning Representations , year =

  28. [28]

    Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , pages =

    Rasley, Jeff and Rajbhandari, Samyam and Ruwase, Olatunji and He, Yuxiong , title =. Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , pages =. 2020 , publisher =. doi:10.1145/3394486.3406703 , url =

    arXiv 2020

  29. [29]

    arXiv preprint arXiv:1909.08053 , year =

    Shoeybi, Mohammad and Patwary, Mostofa and Puri, Raul and LeGresley, Patrick and Casper, Jared and Catanzaro, Bryan , title =. arXiv preprint arXiv:1909.08053 , year =. doi:10.48550/arXiv.1909.08053 , url =

    Pith/arXiv arXiv doi:10.48550/arxiv.1909.08053 1909

  30. [30]

    Tillet, Philippe and Kung, H. T. and Cox, David , title =. Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages , pages =. 2019 , publisher =. doi:10.1145/3315508.3329973 , url =

    arXiv 2019

  31. [31]

    and He, Horace and others , title =

    Ansel, Jason and Yang, Edward Z. and He, Horace and others , title =. Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 , pages =. 2024 , publisher =. doi:10.1145/3620665.3640366 , url =

    arXiv 2024

  32. [32]

    SC20: International Conference for High Performance Computing, Networking, Storage and Analysis , pages =

    Rajbhandari, Samyam and Rasley, Jeff and Ruwase, Olatunji and He, Yuxiong , title =. SC20: International Conference for High Performance Computing, Networking, Storage and Analysis , pages =. 2020 , publisher =. doi:10.1109/SC41405.2020.00024 , url =

    Pith/arXiv arXiv 2020

  33. [33]

    Neurocomputing , volume =

    Su, Jianlin and Ahmed, Murtadha and Lu, Yu and Pan, Shengfeng and Bo, Wen and Liu, Yunfeng , title =. Neurocomputing , volume =. 2024 , doi =

    2024

  34. [34]

    arXiv preprint arXiv:2002.05202 , year =

    Shazeer, Noam , title =. arXiv preprint arXiv:2002.05202 , year =. doi:10.48550/arXiv.2002.05202 , url =

    Pith/arXiv arXiv doi:10.48550/arxiv.2002.05202 2002

  35. [35]

    and Hinton, Geoffrey and Dean, Jeff , title =

    Shazeer, Noam and Mirhoseini, Azalia and Maziarz, Krzysztof and Davis, Andy and Le, Quoc V. and Hinton, Geoffrey and Dean, Jeff , title =. International Conference on Learning Representations , year =

  36. [36]

    Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages =

    Ainslie, Joshua and Lee-Thorp, James and de Jong, Michiel and Zemlyanskiy, Yury and Lebron, Federico and Sanghai, Sumit , title =. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages =. 2023 , publisher =. doi:10.18653/v1/2023.emnlp-main.298 , url =

    doi:10.18653/v1/2023.emnlp-main.298 2023

  37. [37]

    Ainslie, J.; Lee-Thorp, J.; de Jong, M.; Zemlyanskiy, Y.; Lebron, F.; and Sanghai, S. 2023. GQA : Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 4895--4901. Association for Computational Linguistics

    2023

  38. [38]

    Z.; He, H.; et al

    Ansel, J.; Yang, E. Z.; He, H.; et al. 2024. PyTorch 2 : Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, 929--947. ACM

    2024

  39. [39]

    L.; Kiros, J

    Ba, J. L.; Kiros, J. R.; and Hinton, G. E. 2016. Layer Normalization. arXiv preprint arXiv:1607.06450

    Pith/arXiv arXiv 2016

  40. [40]

    X.; Firat, O.; Cao, Y.; and Wu, Y

    Bapna, A.; Chen, M. X.; Firat, O.; Cao, Y.; and Wu, Y. 2018. Training Deeper Neural Machine Translation Models with Transparent Attention. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 3028--3033. Association for Computational Linguistics

    2018

  41. [41]

    He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016. Deep Residual Learning for Image Recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 770--778

    2016

  42. [42]

    Heddes, M.; Javanmard, A.; Axiotis, K.; Fu, G.; Bateni, M.; and Mirrokni, V. 2025. DeepCrossAttention : Supercharging Transformer Residual Connections. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, 22881--22903. PMLR

    2025

  43. [43]

    Kimi Team ; Chen, G.; Zhang, Y.; Su, J.; Xu, W.; Pan, S.; Wang, Y.; Wang, Y.; Chen, G.; Yin, B.; Chen, Y.; Yan, J.; Wei, M.; Zhang, Y.; Meng, F.; Hong, C.; Xie, X.; Liu, S.; Lu, E.; Tai, Y.; Chen, Y.; Men, X.; Guo, H.; Charles, Y.; Lu, H.; Sui, L.; Zhu, J.; Zhou, Z.; He, W.; Huang, W.; Xu, X.; Wang, Y.; Lai, G.; Du, Y.; Wu, Y.; Yang, Z.; and Zhou, X. 2026...

    Pith/arXiv arXiv 2026

  44. [44]

    Kornblith, S.; Norouzi, M.; Lee, H.; and Hinton, G. 2019. Similarity of Neural Network Representations Revisited. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, 3519--3529. PMLR

    2019

  45. [45]

    Loshchilov, I.; and Hutter, F. 2019. Decoupled Weight Decay Regularization. In International Conference on Learning Representations

    2019

  46. [46]

    Luo, C.; Cai, Z.; and Hu, J. 2026. Delta Attention Residuals. arXiv preprint arXiv:2605.18855

    Pith/arXiv arXiv 2026

  47. [47]

    Pagliardini, M.; Mohtashami, A.; Fleuret, F.; and Jaggi, M. 2024. DenseFormer : Enhancing Information Flow in Transformers via Depth Weighted Averaging. In Advances in Neural Information Processing Systems 37, 136479--136508

    2024

  48. [48]

    B.; Lozhkov, A.; Mitchell, M.; Raffel, C.; von Werra, L.; and Wolf, T

    Penedo, G.; Kydl \'i c ek, H.; Allal, L. B.; Lozhkov, A.; Mitchell, M.; Raffel, C.; von Werra, L.; and Wolf, T. 2024. The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale. In Advances in Neural Information Processing Systems 37, 30811--30849

    2024

  49. [49]

    Peng, W.; Rai, J.; Tseng, K.; Wang, S.; and Wu, S. 2026. Ablate and Rescue: A Causal Analysis of Residual Stream Hyper-Connections. In ICLR 2026 Workshop on Scientific Methods for Understanding Deep Learning

    2026

  50. [50]

    Rasley, J.; Rajbhandari, S.; Ruwase, O.; and He, Y. 2020. DeepSpeed : System Optimizations Enable Training Deep Learning Models with Over 100 Billion Parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 3505--3506. ACM

    2020

  51. [51]

    Shazeer, N. 2020. GLU Variants Improve Transformer. arXiv preprint arXiv:2002.05202

    Pith/arXiv arXiv 2020

  52. [52]

    V.; Hinton, G.; and Dean, J

    Shazeer, N.; Mirhoseini, A.; Maziarz, K.; Davis, A.; Le, Q. V.; Hinton, G.; and Dean, J. 2017. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. In International Conference on Learning Representations

    2017

  53. [53]

    Shoeybi, M.; Patwary, M.; Puri, R.; LeGresley, P.; Casper, J.; and Catanzaro, B. 2019. Megatron-LM : Training Multi-Billion Parameter Language Models Using Model Parallelism. arXiv preprint arXiv:1909.08053

    Pith/arXiv arXiv 2019

  54. [54]

    Su, J.; Ahmed, M.; Lu, Y.; Pan, S.; Bo, W.; and Liu, Y. 2024. RoFormer : Enhanced Transformer with Rotary Position Embedding. Neurocomputing, 568: 127063

    2024

  55. [55]

    T.; and Cox, D

    Tillet, P.; Kung, H. T.; and Cox, D. 2019. Triton : An Intermediate Language and Compiler for Tiled Neural Network Computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, 10--19. ACM

    2019

  56. [56]

    N.; Kaiser, L.; and Polosukhin, I

    Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, L.; and Polosukhin, I. 2017. Attention Is All You Need. In Advances in Neural Information Processing Systems 30, 5998--6008

    2017

  57. [57]

    Xiao, D.; Meng, Q.; Li, S.; and Yuan, X. 2025. MUDDFormer : Breaking Residual Bottlenecks in Transformers via Multiway Dynamic Dense Connections. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, 68440--68458. PMLR

    2025

  58. [58]

    Xie, Z.; Wei, Y.; Cao, H.; Zhao, C.; Deng, C.; Li, J.; Dai, D.; Gao, H.; Xu, M.; Yu, K.; Zhao, L.; Zhou, S.; Xu, Z.; Zhang, Z.; Zeng, W.; Hu, S.; Wang, Y.; Yuan, J.; Wang, L.; and Liang, W. 2026. mHC : Manifold-Constrained Hyper-Connections. In International Conference on Machine Learning

    2026

  59. [59]

    Xiong, R.; Yang, Y.; He, D.; Zheng, K.; Zheng, S.; Xing, C.; Zhang, H.; Lan, Y.; Wang, L.; and Liu, T.-Y. 2020. On Layer Normalization in the Transformer Architecture. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, 10524--10533. PMLR

    2020

  60. [60]

    Zhang, B.; and Sennrich, R. 2019. Root Mean Square Layer Normalization. In Advances in Neural Information Processing Systems 32, 12381--12392

    2019

  61. [61]

    Zhu, D.; Huang, H.; Huang, Z.; Zeng, Y.; Mao, Y.; Wu, B.; Min, Q.; and Zhou, X. 2025 a . Hyper-Connections. In International Conference on Learning Representations

    2025

  62. [62]

    Zhu, D.; Huang, H.; Zhou, J.; Huang, Z.; Zeng, Y.; Wu, B.; Min, Q.; and Zhou, X. 2025 b . Frac-Connections: Fractional Extension of Hyper-Connections. arXiv preprint arXiv:2503.14125

    Pith/arXiv arXiv 2025