HAARES Half-Split Residual Basis Routing for Deep Transformers

Kehan Wang

arxiv: 2606.06564 · v2 · pith:YCZ2H6XXnew · submitted 2026-06-04 · 💻 cs.LG · cs.AI

HAARES Half-Split Residual Basis Routing for Deep Transformers

Kehan Wang This is my paper

Pith reviewed 2026-06-28 03:09 UTC · model grok-4.3

classification 💻 cs.LG cs.AI

keywords residual routingtransformer blocksdeep modelshalf-split basislanguage modelingattention updatesMLP updates

0 comments

The pith

Adding a half-split detail basis to block residual routing improves performance in 48-layer transformers.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper proposes HAARES as a lightweight addition to block-level residual routing in transformers. It keeps the cumulative block source but adds one detail basis formed by the difference between first-half and second-half residual updates inside each block. This detail is RMS-matched and updated online to give the router coarse information about the order of attention and MLP steps. Empirical results show the largest and most consistent gains in 48-layer models on OpenWebText and related benchmarks, with smaller or mixed effects at shallower depths. Ablations indicate the improvement is not explained by source duplication, random signs, or block-count changes alone.

Core claim

Block-level residual routing becomes more effective when the router also receives a half-split detail basis, defined as the RMS-matched difference between the first-half and second-half residual updates within each transformer block; this exposes ordered intra-block trajectory information at low extra cost and produces measurable accuracy gains that are most reliable in deep 48-layer models.

What carries the argument

The half-split detail basis, computed as the difference between first-half and second-half residual updates inside each block and kept RMS-matched to the source.

If this is right

Gains appear depth-dependent and are small or mixed at shallow depths but reliable at 48 layers.
The added arithmetic cost is small relative to model width and can be offset by faster convergence.
Ablations show the benefit does not arise from simply duplicating the source or using random signed vectors.
The same directional improvement appears in a larger 453M two-seed probe.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The half-split construction might generalize to other residual-stream routing schemes that currently treat each block as a single vector.
If the first-half versus second-half difference encodes genuine ordering information, similar splits could be tested at other granularities inside the block.
The method's memory overhead suggests it may be most attractive when model width is large enough to amortize the extra state.

Load-bearing premise

The difference between first-half and second-half residual updates inside a block supplies non-redundant trajectory information that a router can actually use.

What would settle it

No consistent accuracy improvement over Block AttnRes when the same 48-layer 201M models are trained across three random seeds with HAARES versus the baseline.

Figures

Figures reproduced from arXiv: 2606.06564 by Kehan Wang.

**Figure 2.** Figure 2: Training curves on OpenWebText for the 453M [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

read the original abstract

Block-level residual routing makes learned residual aggregation practical by routing over block summaries, but each summary compresses an ordered sequence of attention and MLP updates into one cumulative vector. We propose \method{}, a lightweight residual basis router that keeps the cumulative block source and adds one half-split detail basis, computed as the difference between first-half and second-half residual updates. The detail basis is RMS-matched and updated online, exposing coarse intra-block trajectory information without dense sublayer-level routing. Across OpenWebText, cross-domain character-level benchmarks, and BPE-tokenized OpenWebText, the empirical pattern is depth-dependent: gains are small or mixed at shallow depth and most reliable in 48-layer models. In the 201M 48-layer setting, \method{} improves over Block AttnRes across all three seeds, while a 453M two-seed probe shows the same direction. Ablations rule out source duplication, random signed details, fixed detail-source biases, or block-count changes alone. Cost analysis shows that the method is FLOP-light but not wall-clock-free: it adds memory and routing overhead, yet its relative arithmetic cost is amortized as width grows and earlier convergence can reduce time-to-target.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

HAARES adds a half-split detail basis to block residual routing and reports depth-dependent gains in 48-layer models, but the abstract leaves the size and robustness of those gains unclear.

read the letter

The paper's main move is to keep the existing block summary for residual routing and tack on one extra vector: the difference between the first-half and second-half residual updates inside the block, RMS-matched and updated online. This is a narrow, procedural tweak on prior block-level routers rather than a new framework, and the abstract positions it exactly that way.

It does a few things cleanly. The empirical pattern is presented as depth-dependent, with the clearest improvements showing up in 48-layer models rather than shallower stacks. In the 201M setting it beats Block AttnRes across all three seeds, and the two-seed 453M probe points the same direction. The ablations are at least aimed at the obvious objections: source duplication, random signed details, fixed biases, and block-count changes.

The soft spots are mostly about missing numbers. The abstract gives no effect sizes, no variance estimates, and no description of how the three seeds were chosen or whether they were fixed in advance. Without those, it is difficult to judge whether the half-split vector is actually supplying non-redundant trajectory information or whether the two halves are close enough that the difference is mostly noise. The cost claim is also light on wall-clock numbers.

This is for people already running 40-plus layer transformers and looking for cheap knobs on residual aggregation. It is not a foundational result, but the construction is simple enough that a referee could check the full experiments and the statistical reporting in a reasonable amount of time.

I would send it to peer review so the experimental details can be examined properly.

Referee Report

2 major / 1 minor

Summary. The paper introduces HAARES, a lightweight residual routing method for deep transformers that augments standard block-level residual aggregation (Block AttnRes) with a single half-split detail basis. This basis is defined as the RMS-matched difference between first-half and second-half residual updates within each block and is maintained online. The central empirical claim is that the added basis yields depth-dependent gains: small or mixed at shallow depths but consistent improvement over Block AttnRes in 48-layer models (201M parameters across three seeds; same direction in a 453M two-seed probe) on OpenWebText, character-level, and BPE-tokenized benchmarks. Ablations are presented to exclude source duplication, random signed details, fixed biases, and block-count changes.

Significance. If the half-split detail basis supplies non-redundant intra-block trajectory information rather than redundant or noise-dominated signal, the approach offers a low-FLOP way to expose coarse ordering information inside blocks without the cost of dense sublayer routing. The depth-dependent pattern and the set of ablations that rule out several trivial explanations are positive features; the method is also noted to amortize its arithmetic cost at larger widths.

major comments (2)

[Experimental results / abstract] Experimental results (abstract and implied § on 48-layer runs): the claim that HAARES “improves over Block AttnRes across all three seeds” reports only direction, with no effect sizes, standard deviations, or statistical tests. Without these quantities it is impossible to judge whether the observed direction is reliable or could be explained by seed-to-seed variance, directly weakening the central empirical claim.
[Method / ablations] Method definition and ablations (abstract): the half-split detail basis is motivated by the assumption that the difference between first-half and second-half residual updates encodes non-redundant trajectory information. No analysis is supplied of the correlation between the two halves, the variance of the difference vector, or its signal-to-noise ratio relative to the cumulative block source; if the halves are highly correlated or the difference is noise-dominated, the added basis supplies no new routing signal and the reported gains would not be attributable to the half-split construction.

minor comments (1)

[Cost analysis] The cost analysis states the method is “FLOP-light but not wall-clock-free”; a brief table or paragraph quantifying the added memory and routing overhead versus the baseline would improve clarity.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments, which help strengthen the presentation of our empirical claims and methodological justifications. We respond to each major comment below.

read point-by-point responses

Referee: [Experimental results / abstract] Experimental results (abstract and implied § on 48-layer runs): the claim that HAARES “improves over Block AttnRes across all three seeds” reports only direction, with no effect sizes, standard deviations, or statistical tests. Without these quantities it is impossible to judge whether the observed direction is reliable or could be explained by seed-to-seed variance, directly weakening the central empirical claim.

Authors: We agree that reporting only directional improvement without effect sizes, standard deviations, or statistical tests limits assessment of reliability relative to seed variance. The revised manuscript will add these quantities (mean differences, standard deviations across the three seeds for the 201M model, and the two-seed 453M probe) to the results section and update the abstract accordingly. revision: yes
Referee: [Method / ablations] Method definition and ablations (abstract): the half-split detail basis is motivated by the assumption that the difference between first-half and second-half residual updates encodes non-redundant trajectory information. No analysis is supplied of the correlation between the two halves, the variance of the difference vector, or its signal-to-noise ratio relative to the cumulative block source; if the halves are highly correlated or the difference is noise-dominated, the added basis supplies no new routing signal and the reported gains would not be attributable to the half-split construction.

Authors: The referee is correct that the manuscript does not include explicit correlation, variance, or SNR analysis of the half-split difference vector. While the existing ablations exclude several trivial sources of the gains, we will add the requested analysis (correlation between halves and variance/SNR of the detail basis relative to the block source) to the ablations section or appendix in the revision to directly substantiate the non-redundancy assumption. revision: yes

Circularity Check

0 steps flagged

No circularity: procedural definition with external empirical validation

full rationale

The paper defines HAARES procedurally as a residual router that retains the cumulative block source and augments it with a half-split detail basis (difference between first-half and second-half residual updates, RMS-matched and updated online). No equations or derivations are presented that reduce a claimed prediction or result to a fitted parameter or self-citation by construction. Performance claims rest on direct comparisons to the external baseline Block AttnRes across seeds, model sizes, and benchmarks, plus ablations that test against alternative constructions. This structure is self-contained against external benchmarks and contains no load-bearing self-referential steps.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 1 invented entities

The central claim rests on the procedural definition of the half-split detail basis and on the empirical observation that it improves performance; no free parameters are explicitly fitted in the abstract, but the RMS-matching step implicitly normalizes the new vector.

axioms (1)

domain assumption The difference between first-half and second-half residual updates within a block constitutes useful coarse trajectory information.
Invoked when the paper states that the detail basis 'exposes coarse intra-block trajectory information'.

invented entities (1)

half-split detail basis no independent evidence
purpose: To capture intra-block ordering information without dense sublayer routing.
New vector introduced by the method; independent evidence would be a falsifiable prediction of its effect on convergence at larger scales.

pith-pipeline@v0.9.1-grok · 5734 in / 1419 out tokens · 29818 ms · 2026-06-28T03:09:45.593456+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

8 extracted references · 3 canonical work pages · 1 internal anchor

[1]

Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , year=

Deep Residual Learning for Image Recognition , author=. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , year=
[2]

Advances in Neural Information Processing Systems , year=

Attention Is All You Need , author=. Advances in Neural Information Processing Systems , year=
[3]

Proceedings of the International Conference on Machine Learning , year=

On Layer Normalization in the Transformer Architecture , author=. Proceedings of the International Conference on Machine Learning , year=
[4]

Bachlechner et al

ReZero Is All You Need: Fast Convergence at Large Depth , author=. arXiv preprint arXiv:2003.04887 , year=

work page arXiv 2003
[5]

Proceedings of the IEEE/CVF International Conference on Computer Vision , year=

Going Deeper with Image Transformers , author=. Proceedings of the IEEE/CVF International Conference on Computer Vision , year=
[6]

Deepnet: Scaling transformers to 1,000 layers.arXiv preprint arXiv:2203.00555,

DeepNet: Scaling Transformers to 1,000 Layers , author=. arXiv preprint arXiv:2203.00555 , year=

work page arXiv
[7]

Advances in Neural Information Processing Systems , year=

Root Mean Square Layer Normalization , author=. Advances in Neural Information Processing Systems , year=
[8]

Attention Residuals

Attention Residuals , author=. arXiv preprint arXiv:2603.15031 , year=

work page internal anchor Pith review Pith/arXiv arXiv

[1] [1]

Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , year=

Deep Residual Learning for Image Recognition , author=. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , year=

[2] [2]

Advances in Neural Information Processing Systems , year=

Attention Is All You Need , author=. Advances in Neural Information Processing Systems , year=

[3] [3]

Proceedings of the International Conference on Machine Learning , year=

On Layer Normalization in the Transformer Architecture , author=. Proceedings of the International Conference on Machine Learning , year=

[4] [4]

Bachlechner et al

ReZero Is All You Need: Fast Convergence at Large Depth , author=. arXiv preprint arXiv:2003.04887 , year=

work page arXiv 2003

[5] [5]

Proceedings of the IEEE/CVF International Conference on Computer Vision , year=

Going Deeper with Image Transformers , author=. Proceedings of the IEEE/CVF International Conference on Computer Vision , year=

[6] [6]

Deepnet: Scaling transformers to 1,000 layers.arXiv preprint arXiv:2203.00555,

DeepNet: Scaling Transformers to 1,000 Layers , author=. arXiv preprint arXiv:2203.00555 , year=

work page arXiv

[7] [7]

Advances in Neural Information Processing Systems , year=

Root Mean Square Layer Normalization , author=. Advances in Neural Information Processing Systems , year=

[8] [8]

Attention Residuals

Attention Residuals , author=. arXiv preprint arXiv:2603.15031 , year=

work page internal anchor Pith review Pith/arXiv arXiv