Pith. sign in

REVIEW 3 major objections 3 minor 14 references

An affine weight update secretly moves the layer's shared boundary: in an AdamW Transformer, this hidden displacement is 134.7 times the explicit bias step and 99.4% of actual boundary motion.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 04:07 UTC pith:5RTJOOWV

load-bearing objection Honest mechanism study: clean decomposition, striking but gauge-dependent boundary ratios, and a promising diagnostic with single-seed evidence. the 3 major comments →

arxiv 2607.22927 v1 pith:5RTJOOWV submitted 2026-07-24 cs.LG

Hidden Boundary Motion in Transformer Optimization: Function-Space Orthogonalization of Affine Weight and Bias Updates

Zhang Gongyue , Sheng Yixuan , Liu donghan , Wang Zhiyong , Ren Weihong , Liu honghai This is my paper
classification cs.LG
keywords boundary motionaffine layerweight-bias decompositionAdamWTransformer optimizationfunction-space orthogonalizationcentered affine parameterizationshape-boundary split
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that weights and biases in an affine layer are not functionally separate when the layer input has nonzero mean. Any weight update ΔW produces a shared, sample-independent displacement ΔWμ that is functionally identical to a bias update; the paper calls this hidden channel boundary motion. In a four-layer Transformer trained from scratch with AdamW, this hidden channel is a median 134.7 times larger than the explicit bias step and accounts for 99.4% of realized boundary displacement. A diagnostic optimizer that splits the weight gradient into a centered shape component and a boundary component changes the training trajectory substantially in one seed, though its compensation scheme causes severe bias drift. The paper's intended takeaway is that optimizer design should target functional roles, not parameter tensor types.

Core claim

Central discovery: a responsibility inversion. With z = W(x−μ) + c where c = b + Wμ, every weight update carries a sample-independent displacement ΔWμ. In an AdamW-trained Transformer this hidden channel is a median 134.7× larger than the explicit bias step and accounts for 99.4% of realized boundary motion. Removing it from the weight gradient and optimizing it independently (SBO-AdamW) changes the trajectory sharply (+4.13 validation, +4.00 validation-selected test, best step 800 vs 3000), but the compensation prototype drifts biases badly; the paper frames this as mechanism evidence, not a finished optimizer.

What carries the argument

The load-bearing object is the shape–boundary decomposition of an affine layer relative to the empirical input mean μ. Rewriting z_i = W(x_i − μ) + c with c = b + Wμ splits any update into a centered shape displacement ΔW(x_i − μ) and a shared boundary displacement Δc; the two are exactly orthogonal in the batch output-space inner product. On the gradient side this yields g_W = (g_W − g_b μ^T) + g_b μ^T, exposing a rank-one, bias-like term inside the weight gradient. SBO-AdamW optimizes the two components with separate Adam states and compensates the weight-induced boundary displacement by setting Δb = Δc − ΔWμ. The exact orthogonality is local to the batch used to define μ, which is the sou

Load-bearing premise

The decomposition and all headline ratios center on the current minibatch mean μ; if the correct functional reference is a fixed or running center, the split of boundary motion between ΔWμ and Δb changes by construction, so the claim that weights dominate boundary motion is not a coordinate-invariant statement about AdamW.

What would settle it

Recompute the leakage ratios under AdamW using a fixed reference center (for example, the batch mean from step 0 or a running mean) instead of the current minibatch mean. If the median ratio ∥ΔWμ∥/∥Δb∥ drops from about 135 to order unity, or the realized boundary displacement is no longer dominated by ΔWμ, the claimed responsibility inversion is an artifact of the moving gauge. Separately, an amplitude-matched SBO variant that keeps boundary energy comparable to AdamW while still improving accuracy would confirm that orthogonalization, not boundary suppression, drives the reported gains.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Optimizers that treat weights and biases as independent parameter tensors are optimizing a fiction: a small explicit bias norm does not mean biases are unimportant, because the weight path may already supply the boundary translation.
  • Matrix-aware optimizers that improve the geometry of W will also change the boundary channel; functional separation requires first splitting shape and boundary subspaces, then choosing geometry within each.
  • A stable centered-affine parameterization z = W(x − μ̄) + c with c stored directly would avoid the moving-center gauge problem and the bias-coordinate drift, and is the paper's stated next step.
  • The hidden boundary channel is causally active in the studied setup: removing it from the weight update moved the best validation checkpoint from step 3000 to step 800 and improved validation-selected test accuracy by about four points.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the mechanism generalizes, separate learning-rate and weight-decay schedules for weights versus biases are implicitly tuning one shared functional coordinate; that could explain why bias-specific treatments sometimes help or hurt.
  • A testable extension: recompute the leakage ratios with a frozen reference center instead of the current minibatch mean; the theory predicts the 134.7 ratio will redistribute between ΔWμ and Δb, revealing how much of the effect is gauge-dependent.
  • The same algebra applies to embeddings and normalization parameters, which also mix shared and sample-dependent functions; measuring their leakage would show whether boundary motion is a general phenomenon beyond affine weight–bias pairs.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper studies the decomposition of affine-layer updates in Transformers. For an affine map z = Wx + b with a minibatch mean μ, it defines a shape component W(x−μ) and a boundary coordinate c = b + Wμ. Any update (ΔW, Δb) thus induces a measurable boundary displacement Δc = Δb + ΔWμ. On a four-layer Transformer trained from scratch on IMDb under AdamW, the paper reports that the bias-like gradient component g_b μ^T has a median norm 0.664 of the raw weight-gradient norm, that ∥ΔWμ∥/∥Δb∥ has median 134.7, and that ∥ΔWμ∥/∥Δb+ΔWμ∥ has median 0.9944. It then proposes a diagnostic optimizer, SBO-AdamW, that keeps independent Adam states for a centered shape gradient and the boundary gradient, and compensates the weight-induced boundary displacement in the bias coordinate. In a single-seed experiment, SBO-AdamW improves validation accuracy from 81.68% to 85.81% and validation-selected test accuracy from 78.73% to 82.73% while reaching its best validation checkpoint earlier. The paper is explicit that the prototype suppresses boundary energy and causes severe bias-coordinate drift, and it identifies a stable centered-affine parameterization as future work.

Significance. If the headline empirical claim is robust, the paper identifies an important blind spot in optimizer analysis: parameter-space norms of ΔW and Δb can dramatically misrepresent which parameter actually moves the shared affine boundary. The algebraic orthogonality of shape and boundary displacements for a fixed batch (Appendix A) is exact, and the numerical checks (relative reconstruction error 2.31e−08, boundary mapping error 2.36e−06) confirm that the implementation realizes the claimed decomposition. The authors also provide code, logs, and a careful, unusually candid limitations section. However, the headline ratios are computed with respect to the current minibatch mean, a moving coordinate system, and the SBO-AdamW performance result is confounded by boundary-amplitude suppression and independent Adam states. These issues must be addressed before the central claims can be accepted as more than a single-trajectory, gauge-dependent observation.

major comments (3)
  1. [§3.3, Eqs. (14)–(15); §6.3; Table 2] The headline ratios are gauge-dependent. The boundary displacement is defined as Δc = Δb + ΔWμ with μ the current minibatch mean. For any fixed reference center μ̄, the same update decomposes as ΔW(x−μ̄) + (Δb + ΔWμ̄). With μ̄=0, R_leak,b = 0 and the 'responsibility inversion' disappears. The paper acknowledges this dependence (§6.6, §7.3 Eq. (28), §8) but never recomputes Eqs. (14)–(15) under the fixed-reference centered parameterization it recommends. The abstract's statement that 'the observed boundary motion is almost entirely realized through the weight matrix' is therefore a statement about a moving coordinate system, not an established invariant property of AdamW. Please either recompute the ratios with a fixed (e.g., EMA or calibration-period) center, or explicitly restrict the claim to the current-minibatch convention throughout.
  2. [§6.5, Table 3; §7.4] The SBO-AdamW experiment does not isolate the orthogonalization mechanism. At step 800, E_boundary is 17.9 under AdamW but 5.21e−05 under SBO-AdamW, while shape energies also differ (8.47 vs 2.73). SBO-AdamW additionally replaces one Adam state with two independent Adam states (§4, Eqs. (18)–(21)), changing effective step sizes. Thus the accuracy difference in Table 1 could be due to boundary-amplitude suppression, changed Adam moments, or other factors, not to the functional decomposition. The paper lists amplitude matching and optimizer-state ablation as required future work but currently presents the performance gain in Contribution 3 as supporting the mechanism. An amplitude-matched control (and ideally a state-ablation) is necessary before this experiment can support the orthogonalization claim.
  3. [§5, Table 1; §6.1] All empirical results come from a single seed. The headline ratios are medians over 301 checkpoints of one training trajectory, and the SBO-AdamW improvement is a comparison of two single runs. The authors are transparent about this limitation, but for a journal-level claim about 'under AdamW' behavior, one seed cannot establish even qualitative robustness of the ratio magnitudes. At minimum, the paper should either provide additional seeds for the central ratios and the accuracy comparison, or consistently frame every conclusion as applying to the single observed trajectory rather than to AdamW generally.
minor comments (3)
  1. [§3.1, Eq. (7)] The empirical inner product is written with the sum inside the left argument; it would be clearer as (1/n)∑_i ⟨ΔW(x_i−μ), Δc⟩. Also state explicitly in §3.3 that the norms in Eqs. (13)–(15) are Euclidean norms on R^{d_out}.
  2. [§4, after Eq. (22)] Please clarify whether Δc_intended is an increment to the boundary coordinate in the current batch coordinate frame and how the boundary Adam state is carried across minibatches as μ changes. The warning about drift is clear, but the bookkeeping is not.
  3. [Figure 5] The y-axis tick labels appear garbled ('10 5' instead of 10^5). Please regenerate the figure.

Circularity Check

0 steps flagged

No significant circularity: the headline ratios are descriptive measurements of optimizer output, SBO-AdamW's guarantees are explicitly constructional, and the acknowledged gauge/amplitude confounds are identification limitations, not circular reasoning.

full rationale

The derivation chain is not circular in the load-bearing sense. The headline quantities (Eqs. 13-15) are descriptive statistics computed from AdamW checkpoint data: Rleak,b = ||ΔWμ||/||Δb|| and Rleak,c = ||ΔWμ||/||Δb+ΔWμ|| are measured ratios of actual optimizer steps, not predictions fitted from those same quantities. Equation (12), Δc_actual = Δb+ΔWμ, is an algebraic identity, and the paper does not use it as a derivation of the observed ratio; it uses it as the definition of what is being measured. The SBO-AdamW prototype does make Δb+ΔWμ = Δc_intended by construction (Eqs. 22-23), but the paper explicitly presents this as a diagnostic guarantee and validates it with mapping errors, rather than presenting it as empirical evidence. Moreover, the paper openly identifies the two main confounds: moving-center gauge drift (Section 6.6), and the fact that SBO suppresses boundary amplitude by orders of magnitude (Section 6.5), concluding that 'the performance gain is not a clean causal estimate of orthogonality alone.' There is no load-bearing self-citation: the cited related work (Muon, Bird, centering methods) is external and is not used to justify the core measurement; the paper also admits that Eq. (2) is elementary algebra with a long history, so no known result is being renamed as new. The remaining concern — that the shape/boundary split depends on the choice of reference center μ and is therefore gauge-dependent — is a real identification limitation, explicitly acknowledged in Sections 4, 6.6, 7.3, and 8, but it is not circularity: the numerical ratios are not forced by a fitted parameter, by an imported theorem, or by the definitions alone in a way that would make the empirical claim vacuous. Given the hard rule that non-findings are allowed and expected, the appropriate score is 0.

Axiom & Free-Parameter Ledger

0 free parameters · 5 axioms · 2 invented entities

The decomposition is algebraically elementary and no fitted constants underwrite the measured ratios. The main hidden premise is that the minibatch mean is the correct functional reference, making all boundary measurements gauge-dependent. AdamW behavior is assumed from the implementation, and orthogonality is local to the batch. No genuinely new physical or independent entities are introduced.

axioms (5)
  • domain assumption The minibatch mean μ is the correct reference center for the shape/boundary split.
    Sections 3.1 and 6.6; all ratios and energies are defined relative to this μ. If a fixed or running center were used instead, the measured hidden boundary channel would change.
  • standard math AdamW with zero weight decay is numerically equivalent to Adam for this experiment, and the optimizer-generated ΔW and Δb follow the standard coordinate-wise update rule.
    Section 4; the ratios Rleak,b and Rleak,c are computed from these optimizer steps, so the standard Adam update rule is assumed.
  • domain assumption The layer inputs have nonzero mean at the logged checkpoints.
    If μ were zero, hidden boundary motion would vanish by definition; the reported ratios themselves imply μ ≠ 0, but no independent measurement is given.
  • standard math Exact empirical orthogonality of shape and boundary displacements holds only for the observations used to define μ.
    Appendix A, Eqs. (30)–(32); the paper explicitly notes this exactness is local to the minibatch (Section 8).
  • standard math Backpropagation produces the mini-batch gradients of Eq. (9) and the loss gradients δ_i = ∂L/∂z_i.
    Section 3.2; the decomposition relies on the standard backprop equations for affine layers.
invented entities (2)
  • Boundary coordinate c = b + Wμ no independent evidence
    purpose: To separate sample-dependent shape from shared translation in an affine function and define boundary energy.
    A reparameterization variable, not a new physical object; it is defined by algebra and exists for any affine layer. No independent falsifiable prediction.
  • Hidden boundary channel ΔWμ no independent evidence
    purpose: Quantifies the bias-like displacement produced by weight updates during optimization.
    A defined observable from existing parameters and data; the paper provides no independent empirical handle outside its own measurements.

pith-pipeline@v1.3.0-alltime-deepseek · 8315 in / 14681 out tokens · 147394 ms · 2026-08-01T04:07:54.387571+00:00 · methodology

0 comments
read the original abstract

Weights and biases are normally optimized as separate parameter tensors, yet they do not represent separate functions when the input to an affine layer has nonzero mean. For an affine map $z=Wx+b$ with input mean $\mu$, a weight update contains a sample-independent displacement $\Delta W\mu$ that is functionally indistinguishable from a bias update. We call this hidden contribution \emph{boundary motion} and decompose each update into a centered, sample-varying \emph{shape} component and a shared \emph{boundary} component. On a four-layer Transformer trained from scratch on IMDb, the bias-like term $g_b\mu^\top$ has a median norm equal to 0.664 of the raw weight-gradient norm across affine layers and training checkpoints. More strikingly, the median ratio $\norm{\Delta W\mu}/\norm{\Delta b}$ is 134.7, while $\norm{\Delta W\mu}/\norm{\Delta b+\Delta W\mu}$ is 0.994. Thus, under AdamW, the observed boundary motion is almost entirely realized through the weight matrix rather than the explicit bias. We implement a diagnostic optimizer, Shape--Boundary Orthogonal AdamW (SBO-AdamW), that optimizes $g_W-g_b\mu^\top$ and $g_b$ with independent Adam states and compensates the weight-induced boundary displacement. In a single-seed experiment, SBO-AdamW raises validation accuracy from 81.68\% to 85.81\% and validation-selected test accuracy from 78.73\% to 82.73\%, with the best validation checkpoint occurring at step 800 instead of step 3000. However, the moving-batch-center compensation produces severe bias-coordinate drift and strongly reduces boundary energy. The present evidence therefore supports hidden boundary motion as an important optimization mechanism, but it does not yet establish a final general-purpose optimizer. A stable centered-affine parameterization is identified as the required next step.

Figures

Figures reproduced from arXiv: 2607.22927 by Liu Donghan, Liu Honghai, Ren Weihong, Sheng Yixuan, Wang Zhiyong, Zhang Gongyue.

Figure 1
Figure 1. Figure 1: Validation and test accuracy over training. SBO-AdamW fits substantially faster but begins to overfit [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Median bias-like gradient magnitude by affine module under AdamW. The ratio is large in most modules [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Median hidden boundary leakage under AdamW. Every affine module has [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Global hidden-boundary ratio over training. In the SBO run, the ratio measures the size of the [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Summed boundary function energy. The current SBO prototype strongly suppresses realized boundary [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Affine bias parameter norms. The compensation-based SBO implementation develops severe gauge drift, [PITH_FULL_IMAGE:figures/full_fig_p010_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

14 extracted references · 9 linked inside Pith

  1. [3]

    Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies , pages=

    Learning Word Vectors for Sentiment Analysis , author=. Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies , pages=

  2. [6]

    Neural Computation , volume=

    Natural Gradient Works Efficiently in Learning , author=. Neural Computation , volume=. 1998 , doi=

    1998

  3. [13]

    Natural gradient works efficiently in learning

    Shun-ichi Amari. Natural gradient works efficiently in learning. Neural Computation, 10(2):251--276, 1998

    1998

  4. [14]

    Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. arXiv:1607.06450, 2016

    Pith/arXiv arXiv 2016

  5. [15]

    The affine divergence: Aligning activation updates beyond normalisation

    George Bird. The affine divergence: Aligning activation updates beyond normalisation. arXiv:2512.22247v2, 2026

    arXiv 2026

  6. [16]

    Natural neural networks

    Guillaume Desjardins, Karen Simonyan, Razvan Pascanu, and Koray Kavukcuoglu. Natural neural networks. arXiv:1507.00210, 2015

    Pith/arXiv arXiv 2015

  7. [17]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv:1412.6980, 2014

    Pith/arXiv arXiv 2014

  8. [18]

    Linearly constrained weights: Reducing activation shift for faster training of neural networks

    Takuro Kutsuna. Linearly constrained weights: Reducing activation shift for faster training of neural networks. arXiv:2403.13833, 2024

    Pith/arXiv arXiv 2024

  9. [19]

    Muon is scalable for LLM training

    Jingyuan Liu, Jianlin Su, Xingcheng Yao, Zhejun Jiang, Guokun Lai, Yulun Du, Yidao Qin, Weixin Xu, Enzhe Lu, Junjie Yan, et al. Muon is scalable for LLM training. arXiv:2502.16982, 2025

    Pith/arXiv arXiv 2025

  10. [20]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv:1711.05101, 2017

    Pith/arXiv arXiv 2017

  11. [21]

    Maas, Raymond E

    Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of ACL-HLT, pages 142--150, 2011

    2011

  12. [22]

    Optimizing neural networks with Kronecker-factored approximate curvature

    James Martens and Roger Grosse. Optimizing neural networks with Kronecker-factored approximate curvature. arXiv:1503.05671, 2015

    Pith/arXiv arXiv 2015

  13. [23]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. arXiv:1706.03762, 2017

    Pith/arXiv arXiv 2017

  14. [24]

    Gradient centralization: A new optimization technique for deep neural networks

    Hongwei Yong, Jianqiang Huang, Xiansheng Hua, and Lei Zhang. Gradient centralization: A new optimization technique for deep neural networks. arXiv:2004.01461, 2020

    Pith/arXiv arXiv 2004