Pith. sign in

REVIEW 3 major objections 4 minor 26 references

The depth read in an attention-residual Transformer is an attention, so it should be multi-head: splitting the single routing query into per-subspace heads is a zero-parameter reshape that improves validation loss at 100M, 350M, and 1B, kee

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 01:34 UTC pith:IPHQWFPB

load-bearing objection A clean zero-parameter extension of attention residuals with an honest, thorough empirical report; the multi-head-specific advantage is small on the main corpus, untested on held-out data, and unisolated at 8B. the 3 major comments →

arxiv 2607.27230 v2 pith:IPHQWFPB submitted 2026-07-22 cs.AI

Multi-Head Attention Residuals

Cheng Luo , Zefan Cai , Junjie Hu This is my paper
classification cs.AI
keywords multi-head attention residualsdepth routingresidual streamtransformer architecturepretraining losszero-parameter modificationmid-trainingvalidation loss
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper establishes that the learned retrieval a Transformer performs over its own past layer outputs—an attention residual—pays an avoidable cost: one shared query forces all feature subspaces to read depth history through the same distribution. It fixes this by reshaping that single query into H per-subspace heads, each with its own softmax over layers, at zero additional parameters or FLOPs. Across from-scratch pretraining at 100M, 350M, and 1B, the multi-head version beats both the standard residual Transformer and the single-head attention-residual baseline, with gains that grow with scale and survive on a second, web-generic corpus. The number of routing heads is a genuine design axis—validation loss is U-shaped, with a flat optimum at H=4–8—and a probe of trained queries shows the benefit tracks learned subspace disagreement. Efficiency workarounds and an identity-preserving conversion make the mechanism usable at 8B mid-training, where it adds measurable gains on math and science benchmarks.

Core claim

On the paper's own terms: attention residuals let each sublayer read a learned softmax mixture of all prior sublayer outputs instead of only the most recent state. The paper observes that this read is structurally single-head attention over depth, and that the single query is a forced compromise: every width coordinate must use the same depth distribution even when subspaces disagree. Multi-Head Attention Residuals (MHAR) reshape the one (d,) query into H queries of size d/H, run an independent softmax over depth for each, and concatenate the mixed slices. H=1 reproduces the original attention residual, so the comparison is parameter-, compute-, and memory-matched. Trained from scratch on a

What carries the argument

The central object is the multi-head routing read: instead of one learned query q in R^d scoring all N=2L+1 past sublayer outputs with a single softmax, MHAR reshapes q into an (H, d/H) tensor so each of H heads scores only its own slice of the normalized source and mixes only that slice, then concatenates. This makes the depth-read block-diagonal and is exactly a reshape: zero new parameters, negligible FLOPs, and H=1 recovers attention residuals exactly. The second piece is the identity-preserving delta conversion used for mid-training, which zero-gates the routed term so a pretrained model is untouched at step 0. The third is the fused routing kernel, which amortizes the inherently O(L^2

Load-bearing premise

The from-scratch loss figures are validation numbers from a different shuffle of the same training corpus; if the overlap, not the routing mechanism, produces the gap, the paper's central generalization claim is unsupported.

What would settle it

Train the baseline and MHAR at the same scales on the same corpus but hold out a genuinely disjoint evaluation corpus (no document overlap) and require the paired loss gap to persist; if the gap shrinks to noise when overlap is removed, multi-head routing does not improve generalization beyond the training distribution.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Attention-residual implementations should default to multi-head depth routing; at H=1 the method recovers the original exactly, so the reshape is free insurance.
  • Depth-routing head count is a real hyperparameter with a U-shaped loss curve; H=8 is a safe default at scale, and under-convergence pushes the optimum to more heads, never fewer.
  • The mechanism's gain transfers across distributions: it is the only variant that improves on web-generic data at every scale, while single-head routing can regress below a standard residual at 350M–1B.
  • Fused routing kernels reduce the throughput penalty from 0.2–0.5x to 0.55–0.88x of baseline, making depth routing practical in production training.
  • An identity-preserving conversion extends MHAR to mid-training of existing 8B checkpoints with significant gains on GSM8K and GPQA, without a loss spike.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the width-disagreement account is right, the optimum H should track the number of independent feature groups, not necessarily KV heads; a testable prediction is that architectures with more semantic subspace diversity benefit more from higher H.
  • MHAR offers an interpretability handle: each head's depth distribution is a direct readout of which earlier layers a subspace relies on, potentially enabling automatic layer-pruning or routing-analysis tools.
  • Since H=16 overshoots at convergence but helps when undertrained, adaptive or annealed head-count schedules may outperform any fixed H.
  • The mechanism decouples 'what to compute' from 'where to read' in the residual stream; combining multi-head depth routing with delta-residual source decorrelation could compound the gains, since they target two different components of the same bottleneck.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes Multi-Head Attention Residuals (MHAR), a strictly parameter-free generalization of Kimi's attention residuals: the single (d,) routing query is reshaped into H per-subspace heads, each running its own softmax over the depth history. At H=1 the method reduces exactly to attention residuals. The paper reports from-scratch training at 100M/350M/1B on an anneal corpus, claiming validation-loss improvements over a standard Transformer (-0.061/-0.149/-0.140), a U-shaped dependence on H with an optimum at H=4-8, a held-out downstream transfer check, an identity-preserving 8B mid-training conversion with +3.2 GSM8K and +3.1 GPQA, and fused Triton kernels that reduce routing overhead. The central claimed contribution is that the multi-head reshape makes depth routing robust across scales and data distributions, whereas single-head routing is unstable.

Significance. If the empirical claims hold, MHAR is a practical and conceptually clean modification: it adds no parameters over single-head attention residuals, is implementable in a few lines, and appears to improve validation loss and downstream transfer while making the routing mechanism more robust. The paper is unusually thorough in several respects: exact pseudocode for the routing kernel, paired same-seed comparisons, a multi-seed appendix, a direct probe of learned subspace disagreement, detailed systems work, and reproducibility details. The main weakness is that the distinctive multi-head-specific claim rests on a single-seed, in-distribution validation table and on a web replication that uses a different initialization protocol, while the held-out benchmarks do not report the single-head condition. The 8B result also does not isolate the multi-head split from the delta-conversion architecture. These are load-bearing empirical gaps, but they are addressable with additional experiments or explicit re-analysis, so the work is not fundamentally unsound.

major comments (3)
  1. [§3.1 (Metric) / Appendix I (Validation protocol); Table 1] The main from-scratch validation numbers are measured on a different shuffle of the same training corpus, not a held-out split; Appendix I states that this 'reduces rather than eliminates train/validation overlap.' Because MHAR changes the read path over depth, it is plausible that part of the benefit is retrieval of training-corpus-specific features. Table 2 partially rescues the baseline-vs-MHAR claim on held-out benchmarks, but it does not report the single-head condition, so the paper's distinctive multi-head-specific claim is never tested on held-out data. Please provide held-out validation for the full method set, or at least add a single-head-vs-MHAR comparison on the Table 2 benchmarks.
  2. [Table 1 vs Appendix Table 12 (seed robustness)] Table 1 is a single seed. The multi-seed replication in Table 12, on FineWeb-Edu at a shared 5e-4 LR, gives MHAR-vs-baseline deltas of -0.045/-0.090/-0.071, smaller than the anneal-corpus deltas of -0.061/-0.149/-0.140, and single-head is the highest-variance method (1B paired deltas +0.04 to +0.16). The anneal marginal gains of MHAR over single-head are only 0.001/0.028/0.006, which may be within paired-seed variability, especially because the Table 12 runs use randomly initialized routing queries whereas the main runs use zero initialization. Report multi-seed paired deltas and confidence intervals for the anneal corpus, or justify why the web replication is the appropriate basis for the multi-head-specific claim.
  3. [§3.3 (Mid-training), Table 3] The 8B experiment compares plain CPT with MHAR-delta conversion, not with a single-head attention-residual conversion. Since the identity-preserving delta conversion changes the architecture away from the pretrained Transformer before the multi-head split is applied, the reported +3.2 GSM8K and +3.1 GPQA gains do not isolate the multi-head mechanism. Add an H=1 conversion arm to the 8B comparison, or restrict the claims to 'attention-residual conversion with MHAR' rather than attributing the gain specifically to the multi-head split.
minor comments (4)
  1. [§3.2 (Compute-equivalent gain)] The phrase 'in the style of ?' appears to be a missing citation or placeholder for the power-law fitting procedure. Please complete the reference and specify the fitting details precisely.
  2. [Abstract / §2.3 / Table 4] The paper says the reshape 'adds zero parameters' and later states MHAR adds +0.02% parameters over the vanilla baseline. Clarify that zero is relative to single-head attention residuals; relative to a standard Transformer, the routing queries and norms still add a small parameter cost.
  3. [Figure 4 caption] The caption says the deviations 'replicate on disjoint evaluation text,' but the evaluation text is drawn from the same corpus under a different shuffle (Appendix I). Using the word 'disjoint' may overstate the degree of held-out-ness.
  4. [Table 2] The HellaSwag result at 100M is numerically worse for MHAR (33.0 vs 35.0). The text attributes this to noise; including a bootstrap confidence interval or per-item standard error would make this attribution more transparent.

Circularity Check

0 steps flagged

No significant circularity: central claims are empirical, the H=1 reduction is a control rather than a derivation, and the single self-citation is not load-bearing.

full rationale

The paper's central claims are empirical comparisons: multi-head routing (a parameter-free reshape of the attention-residual query) is tested from scratch at 100M–1B and in 8B mid-training against baseline/single-head/hyper-connection controls. The identity H=1 = attention residuals (Eq. 3, Appendix B) is a constructional fact, but the paper does not use it to assert the gain—single-head is the control arm. The only self-citation is [Luo et al., 2026] for the delta attention-residual conversion (§3.3); the paper itself specifies the zero-gated additive conversion and reports schedule-matched empirical results, so the citation is not load-bearing. The Appendix D probe is a post-hoc correlational explanation of observed loss differences, not a fitted input renamed as a prediction. The validation protocol's train/validation overlap (Appendix I) is a generalization-risk concern, not definitional circularity.

Axiom & Free-Parameter Ledger

5 free parameters · 6 axioms · 0 invented entities

The central claim rests on an empirical architecture comparison and a mechanistic probe; no new physical entities are introduced. The main free choices are hyperparameters (LR, H, alpha, probe slices, delta block count), all selected or fitted after seeing results.

free parameters (5)
  • Peak learning rate = 5e-4 unified; 1e-3 for web corpus; per-method sweeps
    All methods share rate in main table; appendix sweeps show ranking is not an artifact, but the exact deltas depend on LR.
  • Routing head count H = 8 adopted; optimum 4-8
    H is selected after sweeps; the U-shaped optimum is descriptive, not predictive.
  • Compute-equivalent exponent alpha = 0.089 (bracket 0.053-0.15)
    Fitted to baseline training curves to convert loss delta to FLOPs; affects compute-equivalent gain claims.
  • Probe slice count S = 4
    KL-disagreement probe uses S=4 slices of the trained query; results may vary with slice count.
  • Delta block aggregation num_blocks = 8 blocks of 4 layers
    Choice in 8B conversion determines routing source count; not swept.
axioms (6)
  • domain assumption Attention residuals (Kimi 2025) are a valid replacement for additive residuals; the forward pass of Figure 3 is inherited unchanged.
    MHAR builds directly on this formulation; if the baseline implementation is flawed, comparisons are affected.
  • domain assumption Validation under a different shuffle of the same corpus is a meaningful comparison signal.
    Appendix I: validation draws from same corpus; overlap is acknowledged.
  • domain assumption Learned subspace disagreement grows with model width and drives the single-query compromise.
    Asserted in §1 and probed in Appendix D with slices of a single trained query, not independently trained routers.
  • domain assumption Power-law scaling L = A C^{-alpha} holds for the baseline.
    Used in §3.2 to convert loss gains to compute-equivalent FLOPs; alpha is fitted.
  • domain assumption Delta attention residuals (Luo et al. 2026) preserve the pretrained model at step 0.
    8B mid-training relies on this identity-preserving conversion; based on the authors' own prior work.
  • domain assumption Zero-initialized routing queries are the correct initialization; the appendix replication used random init and predates the fix.
    The web-corpus single-head reversal may be influenced by initialization; noted in Appendix H.

pith-pipeline@v1.3.0-alltime-deepseek · 25264 in / 14877 out tokens · 144835 ms · 2026-08-04T01:34:37.623110+00:00 · methodology

0 comments
read the original abstract

Transformers propagate information across depth through a single additive residual stream: every sublayer reads only the most recent state. Attention residuals relax this by letting each sublayer attend, through a learned softmax. However, that read uses a single query shared across the entire width, so every feature subspace must read the depth history through one distribution. The cost of this forced compromise grows with how much the subspaces disagree about which layers to read, and disagreement grows with model width. We introduce Multi-Head Attention Residuals (MHAR): the routing query is reshaped into H per-subspace heads, each with its own softmax over the depth history. The read becomes block-diagonal, the reshape adds zero parameters and negligible compute, and H = 1 recovers attention residuals exactly. Trained from scratch on a deduplicated Nemotron-based anneal corpus that is quality-filtered and STEM- and code-heavy, MHAR improves validation loss over a standard Transformer at 100M, 350M, and 1B (-0.061, -0.149, and -0.140). It achieves the best result among four methods in every setting, with the gain increasing from 100M to the larger scales. The head count is a real design axis rather than a free knob: validation loss is U-shaped with respect to H, with a flat optimum at H = 4 or H = 8 across scales. We adopt H = 8 for large-scale models; over-splitting beyond this point (H = 16) consistently gives back part of the gain. A direct probe of the trained queries confirms that learned subspace disagreement is the underlying driver. Fused Triton routing kernels increase attention-residual training throughput from 0.2-0.5x to 0.55-0.88x of the baseline while maintaining near-baseline peak memory. An identity-preserving conversion using delta attention residuals supports 8B mid-training, yielding improvements of +3.2 on GSM8K and +3.1 on GPQA.

Figures

Figures reproduced from arXiv: 2607.27230 by Cheng Luo, Junjie Hu, Zefan Cai.

Figure 1
Figure 1. Figure 1: The depth read is an attention, so it should be multi-head. Left: attention residu￾als [Kimi, 2025] feed every sublayer a learned mixture of the depth history (embedding and every earlier attention/MLP output). Zoom left: one routing site is exactly single-head attention over that history—one query, one depth map shared by all d channels (Eq. 2). Zoom right: MHAR gives each of H subspaces its own map (Eq. … view at source ↗
Figure 2
Figure 2. Figure 2: What each method lets the current sublayer read across depth. Each column: forward stack (top) and the depth-read weight matrix (bottom; entry [i, c] is how strongly channel c reads depth row i; illustrative). Standard pre-norm: a one-hot read of the previous state. Attention residuals: one shared query reads all earlier sources—every column identical (rank-1). MHAR (ours): each subspace reads depth indepe… view at source ↗
Figure 3
Figure 3. Figure 3: Multi-Head Attention Residuals pseudocode. The forward pass is identical to attention residuals [Kimi, 2025]: each sublayer reads a routed combination of the prior sources and appends its raw output as a new source (replacement routing). The sole change is the route kernel—it splits the single (d,) query into self.H heads and runs H independent softmaxes over depth, adding zero parameters. Substituting the… view at source ↗
Figure 4
Figure 4. Figure 4: The H heads carry genuinely different depth-links. Each head’s token-averaged deviation w¯h − w¯ from the head-consensus routing at the first ten routing sites of the trained 1B MHAR model (dark cells: source not yet available). Deviations reach ±0.28 vs. 0.067 under a matched-norm random query, replicate on disjoint evaluation text (r=0.77), and are near-uncorrelated across heads. Compute-equivalent gain.… view at source ↗
Figure 4
Figure 4. Figure 4: The H heads carry genuinely different depth-links. Each head’s token-averaged deviation w¯h − w¯ from the head-consensus routing at the first ten routing sites of the trained 1B MHAR model (dark cells: source not yet available). Deviations reach ±0.28 vs. 0.067 under a matched-norm random query, replicate on disjoint evaluation text (r=0.77), and are near-uncorrelated across heads [PITH_FULL_IMAGE:figures… view at source ↗
Figure 5
Figure 5. Figure 5: The conversion is exactly identity and introduces no optimization shock. Train loss of the schedule-matched 8B mid-training pair (identical seed and data order). Left: the two curves coincide. Right: the per-step |∆loss| starts at ∼10−5 , rises only as the zero-initialized gate opens, and saturates 36× below step-to-step batch noise (dotted) [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 5
Figure 5. Figure 5: Head count at 1B: a flat H=4–8 optimum. (a) The KV = 8 row (tail-mean): loss falls from single-head (H=1) into a flat basin at H=4–8—the optimum, 3.132 and 3.129, essentially tied—then rises past it: H=16 (3.173) gives back roughly a third of the multi-head gain, though it stays well below H=1. (b) The full kv×H grid: multi-head routing helps at every KV, the per-KV optimum (gold stars) lands at H=4 or H=8… view at source ↗
Figure 6
Figure 6. Figure 6: The bypass wants more heads. Under-trained regime (100M, 1 × 10−3 , 5K steps): routing-head (H) × KV grid of validation loss (lower/yellow better; gold stars: per-KV optimum; red dashed line: the H= KV diagonal). The boxed KV = 1 row is the one supra-noise signal (see text). MHAR (H= KV), in contrast, improves at every scale, so multi-head’s advantage over single-head routing widens monotonically: −0.010, … view at source ↗
Figure 6
Figure 6. Figure 6: Single-head route (attention residuals). A drop-in replacement for the multi-head route of [PITH_FULL_IMAGE:figures/full_fig_p014_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Single-head route (attention residuals). A drop-in replacement for the multi-head route of [PITH_FULL_IMAGE:figures/full_fig_p014_7.png] view at source ↗
Figure 7
Figure 7. Figure 7: Validation loss of MHAR at 100M as a function of routing heads [PITH_FULL_IMAGE:figures/full_fig_p015_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Validation loss of MHAR at 100M as a function of routing heads [PITH_FULL_IMAGE:figures/full_fig_p015_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Fused routing forward. Online softmax over the depth sources with the RMSNorm fused in: each source row is read once, nothing is materialized except the [N, B, T, H] routing weights W (a factor d/H smaller than the stacked sources a reference path retains for autograd). 1 def route_bwd(V, W, dout, dV, dq_part, dg_part): # one program per pos. p 2 do = load(dout[p]) # [H, K] 3 S = zeros([H]) 4 for n in rang… view at source ↗
Figure 9
Figure 9. Figure 9: Fused routing forward. Online softmax over the depth sources with the RMSNorm fused in: each source row is read once, nothing is materialized except the [N, B, T, H] routing weights W (a factor d/H smaller than the stacked sources a reference path retains for autograd). 1 def route_bwd(V, W, dout, dV, dq_part, dg_part): # one program per pos. p 2 do = load(dout[p]) # [H, K] 3 S = zeros([H]) 4 for n in rang… view at source ↗
Figure 10
Figure 10. Figure 10: Fused routing backward. Two passes over the source rows: the first accumulates the softmax coupling term S, the second recomputes the norm from the buffer (the row is re-read anyway) and emits the closed-form key- and value-path source gradients, accumulating them across routing calls in place (safe by autograd’s reverse-order execution; asserted at runtime). Shared source buffer. Each forward pass alloca… view at source ↗
Figure 10
Figure 10. Figure 10: Fused routing backward. Two passes over the source rows: the first accumulates the softmax coupling term S, the second recomputes the norm from the buffer (the row is re-read anyway) and emits the closed-form key- and value-path source gradients, accumulating them across routing calls in place (safe by autograd’s reverse-order execution; asserted at runtime). the only activation saved for backward is the … view at source ↗
Figure 11
Figure 11. Figure 11: Training loss at 100M (EMA-smoothed): MHAR stays below the standard baseline [PITH_FULL_IMAGE:figures/full_fig_p020_11.png] view at source ↗
Figure 11
Figure 11. Figure 11: Training loss at 100M on FineWeb-Edu (EMA-smoothed): MHAR stays below the [PITH_FULL_IMAGE:figures/full_fig_p021_11.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

26 extracted references · 15 linked inside Pith

  1. [3]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168,

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168,

    Pith/arXiv arXiv

  2. [4]

    This is the loss-level counterpart of the width-isolated disagreement in Table 6 (which rises 0.235→0.281 over the same widening)

    fixed and only widening d512→d768 : single-head routing’s edge over the baselineerodes( −0.041→ −0.033) while MHAR holds (−0.050→ −0.053), so MHAR’s advantage over single-headmore than doubles( −0.009→ −0.020). This is the loss-level counterpart of the width-isolated disagreement in Table 6 (which rises 0.235→0.281 over the same widening). Single- seed; t...

    2024

  3. [5]

    Manifold-constrained hyper-connections.arXiv preprint arXiv:2512.24880,

    DeepSeek. Manifold-constrained hyper-connections.arXiv preprint arXiv:2512.24880,

    Pith/arXiv arXiv

  4. [10]

    This control is a self-contained 5×10 −4 comparison on the web corpus, so its d512 deltas differ slightly from Table 10’s tuned-rate (1×10 −3) numbers

    under a unified 5×10 −4 recipe (20K steps, sequence 2048, global batch 64, L=12, kv=4) against a matched d512 baseline trained the same way (the 100Marchitectureof Table 1; only d, the head count, and the FFN width change). This control is a self-contained 5×10 −4 comparison on the web corpus, so its d512 deltas differ slightly from Table 10’s tuned-rate ...

    2048

  5. [11]

    Gated attention for large language models: Non-linearity, sparsity, and attention-sink-free.arXiv preprint arXiv:2505.06708,

    Zihan Qiu, Zekun Wang, Bo Zheng, Zeyu Huang, Kaiyue Wen, Songlin Yang, Rui Men, Le Yu, Fei Huang, Dayiheng Liu, Jingren Zhou, and Junyang Lin. Gated attention for large language models: Non-linearity, sparsity, and attention-sink-free.arXiv preprint arXiv:2505.06708,

    Pith/arXiv arXiv

  6. [13]

    Mixture-of-depths: Dynamically allocating compute in transformer-based language models.arXiv preprint arXiv:2404.02258,

    David Raposo, Sam Ritter, Blake Richards, Timothy Lillicrap, Peter Conway Humphreys, and Adam Santoro. Mixture-of-depths: Dynamically allocating compute in transformer-based language models.arXiv preprint arXiv:2404.02258,

    Pith/arXiv arXiv

  7. [15]

    Deepnet: Scaling transformers to 1,000 layers.arXiv preprint arXiv:2203.00555,

    Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, and Furu Wei. Deepnet: Scaling transformers to 1,000 layers.arXiv preprint arXiv:2203.00555,

    Pith/arXiv arXiv

  8. [16]

    Deep delta learning.arXiv preprint arXiv:2601.00417, 2026a

    Yifan Zhang, Yifeng Liu, Mengdi Wang, and Quanquan Gu. Deep delta learning.arXiv preprint arXiv:2601.00417, 2026a. Yifan Zhang, Yifeng Liu, Mengdi Wang, and Quanquan Gu. Residual stream duality in modern transformer architectures.arXiv preprint arXiv:2603.16039, 2026b. Defa Zhu et al. Hyper-connections.arXiv preprint arXiv:2409.19606,

    Pith/arXiv arXiv

  9. [17]

    A Related Work Residual and dense connections.Residual connections [He et al., 2016] and their gated prede- cessor, highway networks [Srivastava et al., 2015], enable very deep training by providing additive identity paths; Veit et al

    2016

  10. [18]

    DenseNets [Huang et al., 2017] instead concatenate all previous feature maps, giving each layer direct access to every earlier one

    characterize residual networks as ensembles of paths of varying length. DenseNets [Huang et al., 2017] instead concatenate all previous feature maps, giving each layer direct access to every earlier one. Our sources list S is dense in this spirit, but rather than concatenating (which grows width) or summing (which is the standard residual stream), weroute...

    2017

  11. [19]

    improves quality while staying parameter- and FLOP-matched. Hyper-connections [Zhu et al., 2024] generalize residual connections into a learnable width- n set of parallel streams, and manifold-constrained hyper-connections [DeepSeek, 2025] add geometric constraints to their mixing; these enrich theconnection topologybetween layers, whereas we keep a singl...

    2024

  12. [20]

    ""Single-head depth routing = attention residuals (Kimi 2025): 3one shared query, one softmax over depth, read by all D coords. 4Equals the MHAR route at H=1

    is exactly that of attention residuals [Kimi, 2025]. Figure 6 gives the single-head kernel: one shared query produces one softmax over depth that alld coordinates read through. Dropping it in for route in theidenticalforward yields attention residuals exactly; equivalently, it is the multi-headroute of Figure 3 at self.H=1 (a single head of width d). This...

    2025

  13. [21]

    is a constant number of operations per element, with none of the data reuse that lets the attention and MLP matrix multiplies run compute-bound, so throughput is set by memory bandwidth. Second, the traffic is inherently quadratic in depth: sublayer s scores and mixes all Ns=s+1 prior states, so a training step must move P s Ns B T d=O(L2 B T d)of source ...

    2022

  14. [22]

    1def route_bwd(V, W, dout, dV, dq_part, dg_part):# one program per pos

    14for n in range(N):# tiny second loop: N*H scalars 15W[n, p] = exp(W[n, p] - m) / l# exact weights, saved for backward Figure 9:Fused routing forward.Online softmax over the depth sources with the RMSNorm fused in: each source row is read once, nothing is materialized except the [N, B, T, H]routing weights W (a factord/Hsmaller than the stacked sources a...

    2026

  15. [25]

    The two runs are identical except for the routing mechanism (same node, software, data order, and global batch)

    Figure 11: Training loss at 100M on FineWeb-Edu (EMA-smoothed): MHAR stays below the standard baseline throughout training (inset: tail zoom, steps 12k–20k). The two runs are identical except for the routing mechanism (same node, software, data order, and global batch). column reverses relative to the anneal corpus: helpful at 100M, above baseline at 350M...

    2019

  16. [26]

    Documents are streamed from the corpus shards, joined with the end-of-text token (id 151,645), and packed into contiguous sequences of length T with no padding

    with the Qwen3 byte-level BPE tokenizer (loaded from Qwen/Qwen3-0.6B). Documents are streamed from the corpus shards, joined with the end-of-text token (id 151,645), and packed into contiguous sequences of length T with no padding. The token-embedding and output projection are tied (tie_word_embeddings) with model vocabulary size 151,936. The supplementar...

    2024

  17. [2015]

    Nemotron-CC: Transforming Common Crawl into a refined long-horizon pretraining dataset.arXiv preprint arXiv:2412.02595,

    12 Dan Su, Kezhi Kong, Ying Lin, Joseph Jennings, Brandon Norick, Markus Kliegl, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. Nemotron-CC: Transforming Common Crawl into a refined long-horizon pretraining dataset.arXiv preprint arXiv:2412.02595,

    Pith/arXiv arXiv

  18. [2016]

    The fineweb datasets: Decanting the web for the finest text data at scale.arXiv preprint arXiv:2406.17557,

    Guilherme Penedo, Hynek Kydlíˇcek, et al. The fineweb datasets: Decanting the web for the finest text data at scale.arXiv preprint arXiv:2406.17557,

    Pith/arXiv arXiv

  19. [2017]

    Finepdfs

    11 Hugging Face. Finepdfs. https://huggingface.co/datasets/HuggingFaceFW/finepdfs, 2025a. Hugging Face. Stack-edu. https://huggingface.co/datasets/HuggingFaceTB/stack-edu, 2025b. Kimi. Attention residuals.arXiv preprint arXiv:2603.15031,

    Pith/arXiv arXiv

  20. [2019]

    Delta attention residuals.arXiv preprint arXiv:2605.18855,

    Cheng Luo, Zefan Cai, and Junjie Hu. Delta attention residuals.arXiv preprint arXiv:2605.18855,

    Pith/arXiv arXiv

  21. [2021]

    Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374,

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, et al. Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374,

    Pith/arXiv arXiv

  22. [2022]

    AI capabilities can be significantly improved without expensive retraining.arXiv preprint arXiv:2312.07413,

    Tom Davidson, Jean-Stan Denain, Pablo Villalobos, and Guillem Bas. AI capabilities can be significantly improved without expensive retraining.arXiv preprint arXiv:2312.07413,

    Pith/arXiv arXiv

  23. [2023]

    Program synthesis with large language models.arXiv preprint arXiv:2108.07732,

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models.arXiv preprint arXiv:2108.07732,

    Pith/arXiv arXiv

  24. [2024]

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun

    URLhttps://zenodo.org/records/10256836. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. InCVPR,

    arXiv

  25. [2025]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    Qwen. Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    Pith/arXiv arXiv

  26. [2026]

    URLhttps://microsoft.ai/pdf/mai-thinking-1.pdf. NVIDIA. Nemotron-cc-math-v1. https://huggingface.co/datasets/nvidia/ Nemotron-CC-Math-v1, 2025a. NVIDIA. Nemotron-cc-v2. https://huggingface.co/datasets/nvidia/Nemotron-CC-v2 , 2025b. NVIDIA. Nemotron-pretraining-sft-v1. https://huggingface.co/datasets/nvidia/ Nemotron-Pretraining-SFT-v1, 2025c. Matteo Pagli...

    Pith/arXiv arXiv