Pith. sign in

REVIEW 2 major objections 5 minor 16 references

Depth routing in transformers works with far fewer key dimensions than the residual width, and can beat full-width Attention Residuals on both loss and FLOPs.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 17:37 UTC pith:PDKZ3EPN

load-bearing objection Clean, useful extension of AttnRes: full-d residual values with r≪d routing keys, solid FLOPs accounting, and a real Pareto win on the reported setup—single-run losses are the main soft spot, not a load-bearing collapse. the 2 major comments →

arxiv 2607.09694 v1 pith:PDKZ3EPN submitted 2026-06-19 cs.LG cs.CL

Low-Rank Attention Residuals

Jonathan Su This is my paper
classification cs.LG cs.CL
keywords attention residualslow-rank routingdepth-wise attentionresidual connectionstransformerslanguage modelsprojected keyssliced keys
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Standard residual streams always add every previous sub-layer output with weight one. Attention Residuals replace that fixed sum with learned attention over earlier outputs, but they force each full-width output to serve as both residual content and routing key. That couples two jobs that do not need the same representation and makes every depth score cost the full hidden width. This paper shows the residual value can stay full-dimensional while the routing key is cut to a much smaller rank r. A projected variant learns the low-rank key from existing output projections and records the best validation loss; a sliced variant simply reuses the last r coordinates of the value, adds no key-projection path, and still improves loss at far lower residual-side cost. Sweeps near half a billion parameters trained on ten billion tokens indicate that choosing among a few dozen depth sources is a low-dimensional problem, so low-rank keys improve both accuracy and the FLOPs tradeoff.

Core claim

Low-Rank Attention Residuals keep residual values in full dimension d while computing depth-routing scores with r-dimensional keys where r is much smaller than d. Projected LR-AttnRes, which emits a learned low-rank key from existing output projections, reaches the best reported validation loss (2.9477 for block mode with 8 blocks and r=32). Sliced LR-AttnRes, which uses the last r value coordinates as the key, nearly matches that loss (2.9480) while adding only 0.093 percent residual-side FLOPs and eliminating the auxiliary projection path. Both improve on the best standard Attention Residual result in the same setting.

What carries the argument

Low-Rank Attention Residuals (LR-AttnRes): each residual source is a pair (full-d value, r-dimensional key). Depth attention uses the low-rank key for scoring and still mixes the full residual values; the two concrete constructions are Projected LR-AttnRes (learned key rows fused into existing output projections) and Sliced LR-AttnRes (key equals the last r coordinates of the value).

Load-bearing premise

The paper treats single-run validation loss on one web corpus for a fixed half-billion-parameter model trained for ten billion tokens as a reliable enough ranking of the residual designs.

What would settle it

Retrain the best standard block Attention Residual, the best projected block model (N=8, r=32), and the best sliced block model (N=8, r=64) with several independent seeds at the same scale (or larger), and check whether the low-rank variants still beat full-width Attention Residuals on held-out loss and retain their FLOPs advantage.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes Low-Rank Attention Residuals (LR-AttnRes), which keep residual values full-dimensional while computing depth-routing scores with r-dimensional keys (r ≪ d). Projected LR-AttnRes (P-LR-AttnRes) emits learned keys by extending existing output projections (Eqs. 12–13); Sliced LR-AttnRes (S-LR-AttnRes) uses the last r coordinates of each value as the key (Eqs. 14–15), removing the auxiliary projection path. On a fixed ~0.5B PreNorm decoder trained for 10B tokens on Ultra-FineWeb, Block P-LR-AttnRes (N=8, r=32) reaches the best validation loss 2.9477 and Block S-LR-AttnRes (N=8, r=64) reaches 2.9480 at 0.0932% added FLOPs, both improving over the best standard AttnRes (2.9673). Supporting analyses include rank/block sweeps (Table 1, Fig. 1), effective source counts (Fig. 2), attention heatmaps (Fig. 3), gradient-norm trends (Table 2), and detailed FLOPs/parameter/activation accounting (Appendix A).

Significance. If the low-rank Pareto ordering holds under broader evaluation, the work cleanly isolates a useful design axis for attention residuals: routing-key width need not equal residual width. The method is well specified, the FLOPs and parameter accounting in Appendix A is careful and reproducible, and the paper releases code and models. The sliced variant is especially attractive as a near-zero-overhead improvement over standard AttnRes. The contribution is incremental relative to AttnRes and related cross-layer work, but the empirical demonstration that depth routing works with r ≪ d, together with the projected vs. sliced comparison and the source-count discussion in §4.7, is a concrete and useful addition to residual-architecture design.

major comments (2)
  1. Table 1 and §4.1–4.2: all primary claims rest on single-run validation losses for one ~0.5B recipe on Ultra-FineWeb (10B tokens). The load-bearing ordering—Block P-LR 2.9477 vs Block S-LR 2.9480 vs best standard AttnRes 2.9673—has no reported seed variance, alternative corpus, or scale check. The 0.0003 gap between the two best low-rank rows is well inside typical LLM seed noise; even the ~0.019 gap vs standard AttnRes could reorder under re-seeding. At minimum, multi-seed error bars (or a second independent run) for the key rows in Table 1 are needed before the Pareto claim in the abstract and Fig. 1 can be treated as reliable.
  2. §4.2 / Table 1: the paper reports only validation loss, not zero-shot or other downstream metrics. For an architecture paper whose central claim is improved residual routing, a small suite of standard LM evaluations (or at least perplexity on a second held-out domain) would substantially strengthen the claim that the low-rank wins are not Ultra-FineWeb-specific. This is not a correctness error, but it is load-bearing for the transferability of the design recommendation.
minor comments (5)
  1. Figure 1 caption and Table 1: clarify more explicitly in the main text (not only the caption) that projected FLOPs include the auxiliary key path while sliced/standard FLOPs are kernel-only, so the x-axis is not a pure apples-to-apples residual-kernel comparison.
  2. §3.4: the choice of the last r dimensions is stated as a convention; a one-sentence note that any fixed coordinate subset is equivalent up to a global permutation of hidden dimensions is already present, but a brief ablation (or citation to a random-subset check) would remove residual doubt.
  3. Figure 3: the two rows use separate color scales; the caption already warns about this, but a shared normalized scale or an additional panel of relative entropy would make cross-row comparison easier.
  4. Related work (§2): Delta Attention Residuals and OASIS are discussed helpfully; a short explicit statement of what is complementary vs. substitutable would help readers place LR-AttnRes relative to those concurrent lines.
  5. Typos / polish: abstract and title use both “LR-AttnRes” and “LR-ATTNRES” casing inconsistently; unify. “depth-wise” vs “depthwise” also varies.

Circularity Check

0 steps flagged

Empirical architecture paper with no circular derivation: low-rank keys are a design choice measured by held-out loss, not a tautology of the inputs.

full rationale

LR-AttnRes is proposed as an architectural change (full-d values, r-dimensional keys via projection or slicing; Eqs. 7–15) and evaluated by training from scratch on Ultra-FineWeb with held-out validation loss and FLOPs accounting (Table 1, Fig. 1, Appendix A). Nothing in the method definition forces the reported losses: the residual mixture still uses full oi, routing scores use separate low-rank keys, and superiority over baseline/AttnRes is an empirical outcome, not a fit renamed as prediction. FLOPs/parameter formulas are cost identities, not circular claims about accuracy. Neff, heatmaps, and gradient-norm analyses are post-hoc diagnostics of trained models, not self-definitional restatements of the loss. Related-work citations (AttnRes, Delta, OASIS) are external predecessors, not load-bearing self-citations that force the result. No uniqueness theorem, fitted-input-as-prediction, or renaming of a known result as a first-principles derivation appears. Single-run variance is a reliability concern, not circularity. Score 0 is appropriate.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 3 invented entities

The central empirical claim rests on standard Transformer residual/attention machinery plus design choices for key rank, block granularity, and a single training regime. No new physical entities are postulated; the invented objects are architectural mechanisms. Free parameters are the usual ML knobs plus the new rank/block axes that the sweeps explore rather than a single fitted constant that forces the result.

free parameters (4)
  • routing key rank r
    Chosen by sweep (16–512 for sliced full; 16/32/64 for projected). Best reported configs use r=32 (projected) or r=64 (sliced block); the claim that r≪d works depends on these selected operating points.
  • block count N
    Block granularity (4/8/16/full) is a discrete design choice; best results concentrate at N=8 for low-rank variants and N=16 for standard AttnRes.
  • model and training recipe (d=1024, L=24, 10B tokens, Muon/Adam LRs, Ultra-FineWeb)
    All validation comparisons are under one fixed architecture and optimizer schedule; the reported losses are not free parameters of a theory, but the ranking is conditioned on this recipe.
  • sliced key coordinate subset (last r dims)
    Convention that any fixed r-subset is equivalent up to permutation; still a design choice that could interact with learned coordinate usage.
axioms (4)
  • domain assumption PreNorm residual Transformers with attention and SwiGLU sub-layers are a valid testbed for residual-stream design.
    Invoked throughout §3–4; standard in modern LLM work.
  • domain assumption Held-out next-token validation loss on Ultra-FineWeb is a meaningful ranking metric for residual mechanisms.
    All main claims in Table 1 and Figure 1 use this metric.
  • domain assumption Depth-routing scores may use RMSNormed keys and static residual-site queries without breaking AttnRes inference properties.
    Eqs. 8–10 and §3.5; inherited from AttnRes with low-rank keys substituted.
  • ad hoc to paper Softmax attention over a small ordered set of residual sources is an appropriate inductive bias for cross-layer mixing.
    Inherited from AttnRes but load-bearing for interpreting low-rank keys as sufficient descriptors (§4.4).
invented entities (3)
  • Low-Rank Attention Residuals (LR-AttnRes) no independent evidence
    purpose: Family of residual mechanisms with full-d values and r-d depth-routing keys.
    Core proposed method; independent evidence is empirical loss/FLOPs comparisons in this paper only.
  • Projected LR-AttnRes (P-LR-AttnRes) no independent evidence
    purpose: Emit learned r-d keys from fused output projections to decouple routing from residual content.
    Accuracy-oriented variant; support is validation loss in Table 1.
  • Sliced LR-AttnRes (S-LR-AttnRes) no independent evidence
    purpose: Use last r value dimensions as keys to remove key-projection FLOPs/parameters.
    Efficiency-oriented variant; support is near-best loss at low added FLOPs.

pith-pipeline@v1.1.0-grok45 · 19838 in / 3525 out tokens · 33495 ms · 2026-07-14T17:37:07.087580+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Low-Rank Attention Residuals." pith.science (2026). https://pith.science/paper/PDKZ3EPN

@misc{pith2026260709694,
  author       = {Pith},
  title        = {Pith review of: Low-Rank Attention Residuals},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PDKZ3EPN}},
  note         = {Machine review of arXiv:2607.09694}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Attention Residuals replace the fixed residual sum with depthwise attention over previous sub-layer outputs in large language models (LLMs), but use each output as both a full-dimensional key and value. This couples routing with representation and makes depth-routing scores scale with the hidden width $d$. We propose Low-Rank Attention Residuals (LR-AttnRes), which keep full-dimensional residual values while using $r$-dimensional keys, with $r \ll d$, for routing. Projected LR-AttnRes emits learned low-rank keys from existing output projections, decoupling routing from residual content and achieving the best validation loss among the variants tested. Sliced LR-AttnRes uses the last $r$ dimensions of each value as the routing key, removing the auxiliary key-projection path and reducing residual-side FLOPs while still improving performance. Comprehensive sweeps show that depthwise routing can be effective with far fewer dimensions than the model width. We release code and models to facilitate future research.

Figures

Figures reproduced from arXiv: 2607.09694 by Jonathan Su.

Figure 1
Figure 1. Figure 1: Validation loss versus percentage of added FLOPs relative to the non-embedding Trans [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Effective number of residual sources across residual read sites. Left: comparison between [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Mean depth-wise attention heatmaps. Top row: full [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

16 extracted references · 6 linked inside Pith

  1. [1]

    Fu, Stefano Ermon, Atri Rudra, and Christopher R ´e

    Tri Dao, Daniel Y . Fu, Stefano Ermon, Atri Rudra, and Christopher R ´e. Flashattention: Fast and memory-efficient exact attention with io-awareness. In Sanmi Koyejo, S. Mo- hamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.),Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Process- ing Systems 2022,...

    2022

  2. [2]

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun

    URL http://papers.nips.cc/paper_files/paper/2022/hash/ 67d57c32e20fd0a7a302cb81d36e40d5-Abstract-Conference.html. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las V egas, NV , USA, June 27-30, 2016, pp. 770–778. IEEE Comput...

    2022

  3. [3]

    URLhttps://doi.org/10.1109/CVPR.2016.90

    doi: 10.1109/CVPR.2016.90. URLhttps://doi.org/10.1109/CVPR.2016.90. 10 Preprint Alex Henry, Prudhvi Raj Dachapally, Shubham Shantaram Pawar, and Yuxuan Chen. Query-key normalization for transformers. In Trevor Cohn, Yulan He, and Yang Liu (eds.),Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020, Findi...

    doi:10.1109/cvpr.2016.90 2016

  4. [4]

    URL https://doi.org/10.18653/v1/2020

    18653/V1/2020.FINDINGS-EMNLP.379. URL https://doi.org/10.18653/v1/2020. findings-emnlp.379. Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks. In2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pp. 2261–2269. IEEE Computer Society,

    doi:10.18653/v1/2020 2020

  5. [5]

    URLhttps://doi.org/10.1109/CVPR.2017.243

    doi: 10.1109/CVPR.2017.243. URLhttps://doi.org/10.1109/CVPR.2017.243. Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks,

    doi:10.1109/cvpr.2017.243 2017

  6. [6]

    Diederik P

    URL https: //kellerjordan.github.io/posts/muon/. Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.),3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings,

    2015

  7. [7]

    Hanze Li and Xiande Huang

    URL http: //arxiv.org/abs/1412.6980. Hanze Li and Xiande Huang. Enhancing layer attention efficiency through pruning redundant retrievals.CoRR, abs/2503.06473,

    Pith/arXiv arXiv

  8. [9]

    press/v267/menghani25a.html

    URL https://proceedings.mlr. press/v267/menghani25a.html. Matteo Pagliardini, Amirkeivan Mohtashami, Fran c ¸ois Fleuret, and Martin Jaggi. Denseformer: Enhancing information flow in transformers via depth weighted averaging. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang (eds.),Advances ...

    2024

  9. [10]

    Jianlin Su, Murtadha H

    URL http://papers.nips.cc/paper_files/paper/2024/ hash/f67449c7ab72f441d3a713b046c6818c-Abstract-Conference.html. Jianlin Su, Murtadha H. M. Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding.Neurocomputing, 568:127063,

    2024

  10. [11]

    URL https://doi.org/10.1016/j.neucom.2023

    doi: 10.1016/J.NEUCOM.2023.127063. URL https://doi.org/10.1016/j.neucom.2023. 127063. Kimi Team, Guangyu Chen, Yu Zhang, Jianlin Su, Weixin Xu, Siyuan Pan, Yaoyu Wang, Yucheng Wang, Guanduo Chen, Bohong Yin, Yutian Chen, Junjie Yan, Ming Wei, Y . Zhang, Fanqing Meng, Chao Hong, Xiaotong Xie, Shaowei Liu, Enzhe Lu, Yunpeng Tai, Yanru Chen, Xin Men, Haiqing...

    doi:10.1016/j.neucom.2023.127063 2023

  11. [12]

    11 Preprint Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N

    URLhttps://arxiv.org/abs/2603.15031. 11 Preprint Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V . N. Vishwanathan, and Roman Garnett (eds.),Advances in Neural Infor...

    Pith/arXiv arXiv 2017

  12. [13]

    Yudong Wang, Zixuan Fu, Jie Cai, Peijun Tang, Hongya Lyu, Yewei Fang, Zhi Zheng, Jie Zhou, Guoyang Zeng, Chaojun Xiao, Xu Han, and Zhiyuan Liu

    URL https://proceedings.neurips.cc/paper/2017/hash/ 3f5ee243547dee91fbd053c1c4a845aa-Abstract.html. Yudong Wang, Zixuan Fu, Jie Cai, Peijun Tang, Hongya Lyu, Yewei Fang, Zhi Zheng, Jie Zhou, Guoyang Zeng, Chaojun Xiao, Xu Han, and Zhiyuan Liu. Ultra-fineweb: Efficient data filtering and verification for high-quality LLM training data.CoRR, abs/2505.05427,

    Pith/arXiv arXiv 2017

  13. [14]

    URLhttps://doi.org/10.48550/arXiv.2505.05427

    doi: 10.48550/ ARXIV .2505.05427. URLhttps://doi.org/10.48550/arXiv.2505.05427. Da Xiao, Qingye Meng, Shengping Li, and Xingyuan Yuan. Muddformer: Breaking residual bottlenecks in transformers via multiway dynamic dense connections. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zh...

    Pith/arXiv arXiv doi:10.48550/arxiv.2505.05427 2025

  14. [15]

    URL https://proceedings.mlr.press/ v267/xiao25d.html. Zhenda Xie, Yixuan Wei, Huanqi Cao, Chenggang Zhao, Chengqi Deng, Jiashi Li, Damai Dai, Huazuo Gao, Jiang Chang, Liang Zhao, Shangyan Zhou, Zhean Xu, Zhengyan Zhang, Wangding Zeng, Shengding Hu, Yuqing Wang, Jingyang Yuan, Lean Wang, and Wenfeng Liang. mhc: Manifold-constrained hyper-connections.CoRR, ...

    Pith/arXiv arXiv

  15. [16]

    2512.24880

    doi: 10.48550/ARXIV . 2512.24880. URLhttps://doi.org/10.48550/arXiv.2512.24880. Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tie-Yan Liu. On layer normalization in the transformer architecture. InProceedings of the 37th International Conference on Machine Learning, ICML’20. JMLR.org,

    Pith/arXiv arXiv doi:10.48550/arxiv

  16. [17]

    Hyper-connections

    Defa Zhu, Hongzhi Huang, Zihao Huang, Yutao Zeng, Yunyao Mao, Banggu Wu, Qiyang Min, and Xun Zhou. Hyper-connections. InThe Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28,

    2025