Pith. sign in

REVIEW 2 major objections 4 minor 35 references

Transformative or Conservative? Conservation laws for ResNets and Transformers

T0 review · 2 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Conservation laws of deep residual networks and Transformers reduce exactly to conservation laws of their building blocks in isolation; a single attention layer conserves exactly the functions of $QQ^\top-KK^\top$ and $VV^\top-OO^\top$.

desk verdict Conservation laws for attention and block reduction are new and mostly solid, but the Transformer half of Theorem 4.6 rests on an unproved open-map claim for attention layers. read the letter →

arxiv 2506.06194 v1 pith:77TAVBO2 submitted 2025-06-06 cs.LG

Sibylle Marcotte , Rémi Gribonval , Gabriel Peyré This is my paper
classification cs.LG MSC 68T0734C14
keywords conservationlawsgradientflowResNetsTransformersattentionlayersimplicitbiasstochasticdescentweightdecay
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks which smooth functions of a network's weights remain constant during gradient-flow training, and shows that for ResNets and Transformers the answer decomposes block by block. The central result is that a function depending only on one block's parameters is conserved in the whole deep network if and only if it is conserved when that block is trained in isolation (Theorem 4.6). For a single attention layer the paper gives a complete answer: every conservation law is a function of $QQ^\top-KK^\top$ and $VV^\top-OO^\top$ under a full-rank condition. It also proves that adding a skip connection leaves conservation laws unchanged, and that these continuous-time invariants survive discrete SGD training approximately, with expected error scaling like the squared step size.

What carries the argument

The argument runs through the vector-field space $W^{g,\ell}_\theta = \mathrm{span}\{\partial_\theta g(\theta,x)^\top w\}$ of gradients produced by all datasets and loss directions: a function $h$ is a conservation law exactly when $\nabla h$ is pointwise orthogonal to $W^{g,\ell}_\theta$, and the number of independent laws is the parameter count minus the dimension of the Lie algebra generated by these fields via the Frobenius theorem. Deep networks are reduced to shallow ones by the reparametrization $g(\theta,x)=f(\phi(\theta),x)$, which turns the gradient span into $\partial\phi(\theta)^\top$, and by Lemma 4.4, which shows that when each residual block is an open map (sends open sets to open sets) with nonzero weight rows, the smooth-input set $X_\theta$ is all of $\mathbb{R}^m$. A structure theorem removes weight decay and time dependence from the problem, and Proposition 5.1 transfers the continuous results to SGD by a Taylor expansion around the exact orthogonality that defines a conservation law.

What would settle it

Take a two-block residual network and choose the first block's parameters so that the Jacobian $I + U_l\,\mathrm{diag}(D)\,V_l$ is singular for some activation pattern $D$ (violating the open-map condition of Lemma 4.4), then train by gradient flow and check numerically whether the isolated block's law $\|u_{1,j}\|^2 - \|v_{j,1}\|^2$ continues to hold in the full network; an extra drift that vanishes for the isolated block would show the genericity assumption is doing real work. A cheaper check targets the paper's own boundary case: with a single token ($N=1$) the query and key matrices $Q$ and $K$ are themselves conserved, so the 'all laws are functions of $QQ^\top-KK^\top$' statement is known to fail outside $N\ge 2$.

Watch

Extended reading notes

Core claim

The paper establishes that conservation laws do not mix across residual blocks under generic regularity conditions: a differentiable function of block $l$'s parameters is conserved for the full deep network exactly when it is conserved for the isolated block, both for convolutive ResNet blocks and for Transformer blocks (attention or MLP), and the same holds for a final softmax classification layer. Complete descriptions follow for the elementary blocks: $c_1$ independent laws $\sum_k \|u_{k,j}\|^2 - \sum_i \|v_{j,i}\|^2$ for a multi-channel convolutive ReLU layer; all laws of one attention layer as functions of $QQ^\top-KK^\top$ and $VV^\top-OO^\top$; the column sums of the weight matrix for a softmax layer. The skip connection is shown to be inert, and for blocks overlapping a skip connection there are no conservation laws at all. Finally, the same quadratic invariants are approximately conserved under SGD, with an expected drift bounded by $\frac{C_hC_L}{2}\sum_i \tau_i^2$.

Load-bearing premise

The load-bearing premise is Lemma 4.4's requirement that every residual block be an open map (one that sends open sets to open sets) and that its weight rows be nonzero; the open-map condition is asserted to be generic but is only sketched via a Jacobian-invertibility argument, and if it failed on a whole set of parameters the density identity $X_\theta = \mathbb{R}^m$ that identifies global with block conservation laws would break down.

Editorial extensions

If this is right

  • Block-level invariants of a deep convolutive ResNet are exactly the per-channel differences $\sum_k \|u_{k,j}\|^2 - \sum_i \|v_{j,i}\|^2$, and of a Transformer's single-head attention block exactly the functions of $QQ^\top-KK^\top$ and $VV^\top-OO^\top$, so these quantities cannot change during gradient-flow training.
  • Attention training is confined to orbits of simultaneous rotations: the query-key pair $(Q,K)$ and value-output pair $(V,O)$ can only move in ways that keep their Gram differences fixed.
  • Functions of parameters belonging to two consecutive blocks that straddle a residual connection have no conservation laws, so cross-block invariants are absent exactly where the residual structure couples the blocks.
  • Under SGD the expected drift of any quadratic conservation law is at most $\frac{C_hC_L}{2}\sum_i \tau_i^2$; with a decaying step size the bound stays finite, and the paper's ResNet-18 and Transformer experiments confirm the $\tau^2$ scaling.
  • The softmax classification layer enforces exactly $m$ conservation laws, one per input dimension: the column sums $\sum_i \theta_{i,j}$ of its weight matrix.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One consequence the paper leaves implicit is a design principle: to find the block-level invariants of any new residual architecture it suffices to analyze each block in isolation, provided one can verify the open-map condition; normalization layers, which the paper omits, act nonlinearly on the block weights and are the natural first place where genuinely new laws could appear.
  • The $O(\tau^2)$ SGD bound also offers a practical diagnostic the authors do not pursue: measuring the empirical drift of a proved invariant versus a random quadratic function separates the 'nearly continuous' part of a training run from the regime where discretization artifacts dominate.
  • The completeness of the attention laws suggests that causal masking or relative position biases, which alter the softmax structure, will change the conserved quantities; the paper's masking experiments show the Gram-difference laws persist, but a formal characterization for masked attention remains open.
  • Read together with Theorem 4.7, the results hint that one function of skip connections is to remove trajectory constraints: an invariant that spans a skip connection cannot exist, so residuals free the parameters to explore directions that a plain stacked network would forbid.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper studies conservation laws of gradient flow (and related discrete dynamics) for residual networks and Transformers. It first proves a structure theorem relating weight-decay and non-weight-decay conservation laws, then gives complete characterizations for shallow blocks: ReLU/linear networks with convolutions, single attention layers, and softmax classification layers. The central new result is Theorem 4.6, which asserts that conservation laws of a deep residual network depending only on one residual block's parameters coincide with conservation laws of that block in isolation, under an open-map and nonzero-row condition on all blocks. The paper also proves an absence result for blocks overlapping skip connections and a bound on approximate conservation under SGD, with numerical experiments on ResNet-18 and a Transformer.

Significance. If the main theorems are correct, this is a substantial advance: complete conservation-law characterizations for attention layers and convolutive residual blocks were missing, and the block-isolation reduction is a useful tool for studying implicit bias. The appendices contain detailed proofs, and the SGD bound (27) with O(τ^2) scaling is validated numerically on both ResNet and Transformer experiments with released code. The main results are new applications and extensions of the authors' prior Lie-algebra framework; I find no circularity. However, the current proof rests on Lemma 4.4, whose statement and genericity justification are not correct as written, so the central reduction is not yet established for the claimed scope.

major comments (2)
  1. [§K, Lemma 4.4] The lemma states that X_θ = R^m, but the proof only establishes that the complement of X_θ has empty interior, i.e. that X_θ is dense. Equality is false in general: for a single residual ReLU block g(θ,x)=x+Uσ(Vx) with m≥2, a row V_j≠0, and U_j≠0, any nonzero x with V_j·x=0 is not in X_θ because the map V ↦ U_jσ(V_j·x) is not differentiable at V_j, yet such x are not excluded by the Baire argument. Appendix N Part A uses the equality X_θ=R^m at several points (e.g. "as X_θ=R^n by Lemma 4.4"); the proof would work with density alone, but as written the lemma is false and its uses are not legitimate.
  2. [§K, Lemma 4.4, Item 1; §L] The open-map assumption is not established for attention blocks, and the genericity discussion is invalid even for the ReLU residual blocks it targets. Section L argues that if all matrices I_m+U_l D V_l are invertible then the block is open; this implication is false: for g(x)=x+uσ(vx) with uv=-2, both one-sided Jacobians are invertible but the map is -|x|, which is not open, and the set of such parameters has nonempty interior. The section also does not treat the attention map F(X)=X+softmax(XQ^TKX^T)XV^TO at all. Since Appendix N's proof of Theorem 4.6 relies on Lemma 4.4 to obtain X_θ=R^m (or density) for Transformers containing attention blocks, the Transformer half of Theorem 4.6 is currently unsupported. The authors should either prove openness (or a sufficient weaker surrogate) for generic attention parameters, or explicitly restrict Theorem 4.6 to the ReLU ResNet case.
minor comments (4)
  1. [§L] The section heading refers to "genericity of Item 2" but the argument concerns Item 1 (the open-map condition); additionally, the displayed Jacobian omits the identity term: it should read I_m + U_l D_l(x) V_l, as in the finite set defined just below.
  2. [After Theorem 4.6] The sentence identifying block conservation laws with those "described in Corollary 3.9" should carry the full-rank hypotheses of Corollary 3.9. At degenerate attention parameters such as V=O=0 (used in Lemma M.1 to realize the identity map), the isolated attention layer has many more conservation laws than the functions of QQ^T−KK^T and VV^T−OO^T.
  3. [Example 4.8] The claim that there are exactly three conservation laws on Ω fails at parameters with u=v=t=0 (and s arbitrary), where the vector field χ_1 vanishes and the Lie algebra dimension is not 1; the example should restrict to an open subset where u, v, and t are all nonzero and of the same sign.
  4. [Throughout] There are several typos and copy-paste artifacts, e.g. "taylored" in the introduction, "traing" in the introduction, and "Withvswithout" in the heading of Section 2.1.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the paper extends a prior peer-reviewed Lie-algebra framework with new proofs; the central reduction theorem is not a renamed fit or a definitional identity.

full rationale

The derivation chain is internally supported and does not reduce its predictions to fitted inputs or to self-citations of the target claims. The conservation-law framework (Proposition 2.3, Proposition 2.9, Theorem 2.11) is recalled from the authors' prior work, but Theorem 2.11 is re-proved in Appendix C, and the framework is used as a mathematical tool rather than as a substitute for the new claims. The new results (Theorem 3.6, Corollary 3.9, Proposition 3.11, Theorem 4.6, Theorem 4.7) are derived from explicit computations of tangent spaces, Lie algebras, and density arguments. In particular, Theorem 4.6 is not circular: it uses Lemma M.1 to choose surrounding blocks equal to the identity, Lemma 4.4 to obtain X_theta = R^m, and a closure argument to identify the block-level span with the isolated-block span; it does not rename a fitted quantity as a prediction. No parameter in the paper is fit to a subset of data and then reported as a prediction. The main genuine weakness is that Lemma 4.4's open-map condition is justified in Section L only for a 2-layer ReLU residual block, while attention blocks are only covered by the bare assumption; the section heading even refers to 'Item 2' while the argument concerns 'Item 1'. That is a correctness or completeness gap, not circularity: an unproved hypothesis is not an input that is being repackaged as an output. Self-citations are frequent, but they point to peer-reviewed prior results and are not invoked as an unverified uniqueness theorem to force the paper's conclusions. Therefore the circularity burden is low.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

No fitted constants appear in the theory; the contribution is the derivation of invariant functions. The deduction relies on the authors' previously published Lie-algebra framework and on generic smoothness and boundedness assumptions.

assumptions (5)
  • domain assumption Assumption 2.4: V_ell(z) is independent of z for the loss ell.
    Needed for Corollary 2.5 and all characterization theorems; holds for MSE, KL, and cross-entropy losses.
  • standard math Assumption 2.6: local reparameterization g = f composed with phi exists.
    Always possible with phi = id; used to compute W via Jacobians.
  • domain assumption Lemma 4.4 Item 1: each residual block g^l_{theta_l} is an open map; Item 2: all rows nonzero.
    Required for X_theta = R^m and the density argument in Theorem 4.6. Item 2 is a lower-dimensional exclusion, but Item 1 is only discussed heuristically in Section L.
  • domain assumption Corollary 3.9 full-rank condition: horizontally concatenated matrices [Q,K] and [V,O] have full rank.
    Completeness of attention conservation laws is established only on neighborhoods of such parameters.
  • domain assumption Proposition 5.1 uniform gradient bound: E ||nabla L_{Z_k}(theta_k)||^2 <= C_L.
    Acknowledged not generally true for deep networks; needed for the approximate SGD conservation bound.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Transformative or Conservative? Conservation laws for ResNets and Transformers." pith.science (2026). https://pith.science/paper/77TAVBO2

@misc{pith2026250606194,
  author       = {Pith},
  title        = {Pith review of: Transformative or Conservative? Conservation laws for ResNets and Transformers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/77TAVBO2}},
  note         = {Machine review of arXiv:2506.06194}
}
read the original abstract

While conservation laws in gradient flow training dynamics are well understood for (mostly shallow) ReLU and linear networks, their study remains largely unexplored for more practical architectures. This paper bridges this gap by deriving and analyzing conservation laws for modern architectures, with a focus on convolutional ResNets and Transformer networks. For this, we first show that basic building blocks such as ReLU (or linear) shallow networks, with or without convolution, have easily expressed conservation laws, and no more than the known ones. In the case of a single attention layer, we also completely describe all conservation laws, and we show that residual blocks have the same conservation laws as the same block without a skip connection. We then introduce the notion of conservation laws that depend only on a subset of parameters (corresponding e.g. to a pair of consecutive layers, to a residual block, or to an attention layer). We demonstrate that the characterization of such laws can be reduced to the analysis of the corresponding building block in isolation. Finally, we examine how these newly discovered conservation principles, initially established in the continuous gradient flow regime, persist under discrete optimization dynamics, particularly in the context of Stochastic Gradient Descent (SGD).

Figures

Figures reproduced from arXiv: 2506.06194 by the authors.

Figure 1
Figure 1. Tracking a conserved function during ResNet-18 training on CIFAR-10. lines show the theoretical slopes Cτ 2 derived from (27), confirming that the function is approximately conserved and that the slope coefficient maintains proportionality with τ 2 . Our code is available at our GitHub repository. In another experiment, we train a transformer model on the IMDb sentiment analysis dataset (Maas et al., 2011) using SGD… view at source ↗
Figure 2
Figure 2. Block overlapping a residual connection 29 [PITH_FULL_IMAGE:figures/full_fig_p029_2.png] view at source ↗
Figure 3
Figure 3. Block sharing a residual connection P. Proof of Theorem 4.7 : no block conservation laws for blocks overlapping a residual connexion Theorem 4.7. Consider a layer index 1 ≤ l ≤ q − 1 and Θ defined as in Lemma 4.4 with the exception that for each θl+1 ∈ Θl+1, we further require that the rows of V l+1 are pairwise non-colinear. If nv = p then any conservation law of g with respect to the Euclidean loss that only depen… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Tracking a conserved function and the loss during ResNet-18 training on CIFAR-10. For each learning rate between 1e-3 and 5e-3, we train 10 models for 50 steps using SGD without momentum or weight decay, with 10 different random seeds. For each configuration, we record…
Figure 5
Figure 5. Figure 5: Tracking a conserved function and the loss during a Transformer training on IMDb dataset. For each learning rate between 1e-3 and 5e-3, we train 8 models for 50 steps using SGD without momentum or weight decay, with 8 different random seeds. For each configuration, we …

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

35 extracted references · 24 canonical work pages

  1. [1]

    On the optimization of deep networks: Implicit acceleration by overparameterization

    Arora, S., Cohen, N., and Hazan, E. On the optimization of deep networks: Implicit acceleration by overparameterization. In International Conference on Machine Learning, pp.\ 244--253. PMLR, 2018

    work page 2018

  2. [2]

    A convergence analysis of gradient descent for deep linear neural networks

    Arora, S., Cohen, N., Golowich, N., and Hu, W. A convergence analysis of gradient descent for deep linear neural networks. In International Conference on Learning Representations, 2019

    work page 2019

  3. [3]

    Learning theory from first principles

    Bach, F. Learning theory from first principles. MIT press, 2024

    work page 2024

  4. [4]

    Learning deep linear neural networks: Riemannian gradient flows and convergence to global minimizers

    Bah, B., Rauhut, H., Terstiege, U., and Westdickenberg, M. Learning deep linear neural networks: Riemannian gradient flows and convergence to global minimizers. Information and Inference: A Journal of the IMA, 11 0 (1): 0 307--353, 2022

    work page 2022

  5. [5]

    Neural machine translation by jointly learning to align and translate

    Bahdanau, D., Cho, K., and Bengio, Y. Neural machine translation by jointly learning to align and translate. In International Conference on Learning Representations (ICLR), 2015

    work page 2015

  6. [6]

    and Hennig, P

    Balles, L. and Hennig, P. Dissecting adam: The sign, magnitude and variance of stochastic gradients. In International Conference on Machine Learning, pp.\ 404--413. PMLR, 2018

    work page 2018

  7. [7]

    and Bianchi, P

    Barakat, A. and Bianchi, P. Convergence and dynamical behavior of the adam algorithm for nonconvex stochastic optimization. SIAM Journal on Optimization, 31 0 (1): 0 244--274, 2021

    work page 2021

  8. [8]

    D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 0 1877--1901, 2020

    1901

Show all 35 references
  1. [9]

    T., Rubanova, Y., Bettencourt, J., and Duvenaud, D

    Chen, R. T., Rubanova, Y., Bettencourt, J., and Duvenaud, D. K. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018

    2018

  2. [10]

    and Bach, F

    Chizat, L. and Bach, F. Implicit bias of gradient descent for wide two-layer neural networks trained with the logistic loss. In Conf. on Learning Theory, pp.\ 1305--1338. PMLR, 2020

    2020

  3. [11]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021

    2021

  4. [12]

    S., Hu, W., and Lee, J

    Du, S. S., Hu, W., and Lee, J. D. Algorithmic regularization in learning deep homogeneous models: Layers are automatically balanced. Advances in Neural Information Processing Systems, 31, 2018

    2018

  5. [13]

    and Gower, R

    Garrigos, G. and Gower, R. M. Handbook of convergence theorems for (stochastic) gradient methods. arXiv preprint arXiv:2301.11235, 2023

    2023 arXiv

  6. [14]

    Deep residual learning for image recognition

    He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 770--778, 2016

    2016

  7. [15]

    Nonlinear system control

    Isidori, A. Nonlinear system control. New York: Springer Verlag, 61: 0 225--236, 1995

    1995

  8. [16]

    and Telgarsky, M

    Ji, Z. and Telgarsky, M. Gradient descent aligns the layers of deep linear networks. In International Conference on Learning Representations, 2019 a

    2019

  9. [17]

    and Telgarsky, M

    Ji, Z. and Telgarsky, M. The implicit bias of gradient descent on nonseparable data. In Conference on learning theory, pp.\ 1772--1798. PMLR, 2019 b

    2019

  10. [18]

    Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  11. [19]

    Learning multiple layers of features from tiny images

    Krizhevsky, A. Learning multiple layers of features from tiny images. 2009

    2009

  12. [20]

    L., Daly, R

    Maas, A. L., Daly, R. E., Pham, P. T., Huang, D., Ng, A. Y., and Potts, C. Learning word vectors for sentiment analysis. In Lin, D., Matsumoto, Y., and Mihalcea, R. (eds.), Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language ...

    2011

  13. [21]

    Abide by the law and follow the flow: Conservation laws for gradient flows

    Marcotte, S., Gribonval, R., and Peyr \'e , G. Abide by the law and follow the flow: Conservation laws for gradient flows. Advances in neural information processing systems, 36, 2023

    2023

  14. [22]

    Keep the momentum: Conservation laws beyond euclidean gradient flows

    Marcotte, S., Gribonval, R., and Peyr \'e , G. Keep the momentum: Conservation laws beyond euclidean gradient flows. In 41st International Conference on Machine Learning, 2024

    2024

  15. [23]

    E., and Biau, G

    Marion, P., Wu, Y.-H., Sander, M. E., and Biau, G. Implicit regularization of deep residual networks towards neural odes. arXiv preprint arXiv:2309.01213, 2023

    2023 arXiv

  16. [24]

    On the explicit role of initialization on the convergence and implicit bias of overparametrized linear networks

    Min, H., Tarmoun, S., Vidal, R., and Mallada, E. On the explicit role of initialization on the convergence and implicit bias of overparametrized linear networks. In International Conference on Machine Learning, pp.\ 7760--7768. PMLR, 2021

    2021

  17. [25]

    Newns, W. F. Functional dependence. The American Mathematical Monthly, 74 0 (8): 0 911--920, 1967. ISSN 00029890, 19300972

    1967

  18. [26]

    Saul, L. K. Weight-balancing fixes and flows for deep learning. Transactions on Machine Learning Research, 2023

    2023

  19. [27]

    M., McClelland, J

    Saxe, A. M., McClelland, J. L., and Ganguli, S. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprint arXiv:1312.6120, 2013

    2013 arXiv

  20. [28]

    S., Gunasekar, S., and Srebro, N

    Soudry, D., Hoffer, E., Nacson, M. S., Gunasekar, S., and Srebro, N. The implicit bias of gradient descent on separable data. The Journal of Machine Learning Research, 19 0 (1): 0 2822--2878, 2018

    2018

  21. [29]

    Equi-normalization of neural networks

    Stock, P., Graham, B., Gribonval, R., and Jégou, H. Equi-normalization of neural networks. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=r1gEqiC9FX

    2019

  22. [30]

    D., and Vidal, R

    Tarmoun, S., Franca, G., Haeffele, B. D., and Vidal, R. Understanding the dynamics of gradient flow in overparameterized linear models. In International Conference on Machine Learning, pp.\ 10153--10161. PMLR, 2021

    2021

  23. [31]

    Implicit bias and fast convergence rates for self-attention

    Vasudeva, B., Deora, P., and Thrampoulidis, C. Implicit bias and fast convergence rates for self-attention. arXiv preprint arXiv:2402.05738, 2024

    2024 arXiv

  24. [32]

    Attention is all you need

    Vaswani, A. Attention is all you need. Advances in Neural Information Processing Systems, 2017

    2017

  25. [33]

    Yang, L., Zhang, J., Shenouda, J., Papailiopoulos, D., Lee, K., and Nowak, R. D. A better way to decay: Proximal gradient training algorithms for neural nets. In OPT 2022: Optimization for Machine Learning (NeurIPS 2022 Workshop), 2022. URL https://openreview.net/forum?id=4y1xh8jClhC

    2022

  26. [34]

    Making convolutional networks shift-invariant again

    Zhang, R. Making convolutional networks shift-invariant again. In International conference on machine learning, pp.\ 7324--7334. PMLR, 2019

    2019

  27. [35]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.