Pith. sign in

REVIEW 4 major objections 6 minor 21 references

Mask-Based Priors Are More Persistent than Query-Key Initializations

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Mask-based priors fix Transformer extrapolation where QK init fails

desk verdict Clean, large-effect evidence that where a structural prior lives in a Transformer determines whether it survives training; the why is underdetermined. read the letter →

arxiv 2608.00418 v1 pith:WIOUJTLD submitted 2026-08-01 cs.AI

Mingze Ma , Hemanth Saratchandran , Cameron Gordon , Simon Lucey This is my paper · ORCID
classification cs.AI MSC 68T07
keywords attentionmasksstructuredinitializationinductivebiasgeneralizationontheunseenBooleanextrapolationminimum-degreeinterpolatorlow-dataarithmetictransformers
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Transformers often fit the observed domain on Boolean extrapolation tasks yet generalize according to a simpler minimum-degree interpolator rather than the true target function. This paper asks whether that systematically wrong extrapolation can be corrected by injecting the target's interaction structure into attention, and answers that it depends on where the prior lives. Encoded indirectly through query-key projections, the same structural prior is rapidly overwritten during training and the wrong extrapolation rule persists. Encoded directly as a learnable additive attention-logit bias, the prior survives optimization and produces near-perfect extrapolation on the Boolean tasks studied. The same mechanism improves low-data arithmetic and stays competitive on vision and language benchmarks, suggesting attention masks are a simple substrate for persistent inductive bias.

What carries the argument

The carrying object is a learnable additive attention mask $M\in\mathbb{R}^{N\times N}$, introduced into attention as $\mathrm{softmax}(XW_QW_K^\top X^\top/\sqrt{d}+\log\sigma(M))$, where $\sigma$ is the sigmoid. The mask is a finite, learnable attention-logit bias initialized from task-level interaction structure: entries for desired token interactions are opened and undesired ones closed, and because $\log\sigma(M)\leq 0$ the bias can only down-weight interactions, with softmax renormalizing afterward. Unlike causal or padding masks, it is not a hard architectural constraint but a persistent, adaptable inductive bias. The argument turns on where the prior sits: structure encoded through $QK^\top$ is entangled with query and key feature scales, so attention gradients that grow with query and key norms can overwrite it, while the additive-mask parameterization keeps the structural bias outside the content-dependent similarity score, with gradient $\partial p_i/\partial m_j=p_i(\delta_{ij}-p_j)\sigma(-m_j)$ bounded independently of feature norms. This decoupling is what lets the prior persist through optimization and change the extrapolation rule.

What would settle it

Run 2Parity with a deliberately misspecified mask, placing the same 2Parity pattern on a non-interacting token pair such as x1–x2 instead of x0–x1 while keeping all training settings fixed. If extrapolation accuracy stays near 100 percent, the persistence mechanism rather than the task-aligned pattern is doing the work; if it collapses to the roughly 75 percent minimum-degree level, the oracle interaction prior is the active ingredient.

Watch

Extended reading notes

Core claim

On Boolean Generalization-on-the-Unseen tasks, the central discovery is that parameterization, not just pattern, determines whether an injected structural prior changes what a Transformer learns. Fitting the same interaction pattern into the query and key projections so that $\mathrm{softmax}(QK^\top/\sqrt{d})$ approximates the desired attention structure matches that structure at initialization, but by the end of training the pattern is washed out: 2Parity accuracy stays at 75 percent, matching the minimum-degree interpolator, and Cyclic3 and 3-Bit Majority stay near 87 percent. Encoding the identical pattern as a finite, learnable additive mask, $\mathrm{softmax}(XW_QW_K^\top X^\top/\sqrt{d}+\log\sigma(M))$, changes the learned rule: 2Parity and 3-Bit Majority reach 100 percent accuracy, Cyclic3 reaches 99.98 percent, and losses drop by orders of magnitude. A gradient analysis explains why: derivatives of attention probabilities with respect to query and key scale with the query and key norms, so QK-encoded structure is easily distorted, whereas the mask gradient is bounded by $p_i\sigma(-m_j)\leq p_i$ and decouples structural bias from content similarity. The same mask mechanism lifts low-data arithmetic sequence accuracy from 5.01 to 69.49 percent on 3-by-3 multiplication at a 3 percent training fraction and from 1.19 to 77.49 percent on addition at 0.4 percent.

Load-bearing premise

The load-bearing premise is the oracle prior assumption: for every Boolean task the interaction mask is hand-designed from the exact ground-truth function decomposition, including its behavior on the unseen domain, so if a meaningful interaction prior is not known in advance the near-perfect extrapolation and arithmetic gains are not guaranteed to survive.

Editorial extensions

If this is right

  • On the four Boolean GOTU tasks, mask initialization reaches near-perfect extrapolation, with 100 percent accuracy on 2Parity and 3-Bit Majority and 99.98 percent on Cyclic3, while vanilla and both QK-initialized baselines remain at the minimum-degree plateau.
  • QK-based structured initialization can reproduce the desired attention pattern at initialization but does not reliably change the learned extrapolation rule because the prior is overwritten during training, as shown for both SVD-based and directly optimized QK priors.
  • Mask-based priors substantially improve low-data arithmetic: on 3-by-3 digit multiplication at a 3 percent training fraction, sequence accuracy rises from 27.82 percent for the best structured baseline to 69.49 percent, and on addition at 0.4 percent from 3.39 to 77.49 percent.
  • On CIFAR10, CIFAR100, and ImageNet100 with a ViT-T model, mask initialization achieves the best top-1 accuracy among default, mimetic, and impulse initialization, indicating the persistence mechanism transfers to practical vision.
  • Because gradients of attention probabilities with respect to mask parameters are bounded by $p_i$ and independent of query and key norms, the injected structure updates more slowly and remains identifiable in the attention logits after training, as the paper visualizes for Head 0 at Layer 0.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: the paper's mechanism predicts a testable scaling pattern — the more a prior can be stated as a sparse interaction graph, the larger the expected gain from mask initialization over QK initialization; on dense, content-dependent tasks the advantage should shrink, which matches the modest NLP improvements reported.
  • Inference: the gradient argument implies that normalizing query-key feature scales, for example with spectral norm or LayerNorm on the QK logits, should make QK-initialized priors more persistent; if such an intervention closes the gap with mask initialization, the gradient-entanglement explanation would be confirmed rather than some other difference between parameterizations.
  • Inference: since the paper's own ablation shows an all-zero mask matches vanilla performance, the practical value of the method depends on obtaining interaction priors from data or pretrained models; learning masks from data is the natural next step that the paper leaves open.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes to initialize a learnable additive attention-logit bias (a 'mask') from task structure, arguing that this parameterization preserves the injected inductive bias through training, whereas structured initialization that encodes the same prior through query-key projections is overwritten. On Boolean generalization-on-the-unseen (GOTU) tasks, the proposed mask initialization yields near-perfect test accuracy (e.g., 100% on 2Parity vs 75% for baseline and QK-based methods), improves low-data arithmetic, and matches or slightly improves vision and language benchmarks. The authors provide a gradient-level argument (Theorem 1 and Corollary 1), ablations (all-zero mask, fixed mask), and visualizations of attention patterns.

Significance. If the central claim were fully established, the work would offer a simple and practically useful mechanism for encoding persistent inductive bias in Transformers, complementing QK-based structured initialization. The paper's framing of the GOTU setting as a test of inductive bias is appropriate, and the all-zero mask ablation (Table 4) is a useful control showing that the gains are not merely from extra parameters. However, the reported evidence is currently limited by the oracle construction of Boolean masks, single-seed runs, and a theoretical argument that does not directly establish 'slower updates'; the paper's own limitations passage acknowledges the oracle assumption.

major comments (4)
  1. [§4.1, Eq. (4), Table 7, Section E] The Boolean masks are initialized with open=10 and close=-10 through logσ(M), and are constructed from the exact ground-truth function decomposition of the target function, including its behavior on the unseen domain (Section E). Because logσ(10) is approximately 0 and logσ(-10) is approximately -10, closed positions receive essentially zero probability after softmax; consequently, the gradient ∂p_i/∂m_j in Eq. (4) is vanishingly small for open entries (σ(-10)≈4.5e-5) and for closed entries (p_j≈0). The mask is therefore effectively frozen at the oracle pattern, so the near-perfect extrapolation in Table 1 reflects a correctly specified hard constraint baked in at initialization, not a learnable prior that survives optimization. The comparison with QK-based init is also confounded because QK methods must represent the pattern through a low-rank product and are not given a near-hard, exactly-oracle constraint of the same effective scale. To support the persistence claim, the paper needs experiments with imperfect, coarsely specified, or lower-contrast masks, and measurements of how quickly each parameterization deviates from the initial attention pattern.
  2. [§6.1, Theorem 1, Corollary 1] The gradient bounds in Eqs. (5)–(7) do not establish that mask-based priors update slower than QK-based priors. The bounds compare per-parameter gradient norms in different parameter spaces: the QK bounds scale with B_k and B_q, which can be small or large depending on normalization, while the mask bound is at most 1 and is largest (≈1) precisely for closed positions when p_i is large; the smallness of the open-mask gradient comes from the specific choice m=10, an initialization-scale effect rather than a property of the additive parameterization. The statement that 'mask-based priors will update slower' is a dynamical claim that requires measuring the actual drift of attention patterns over training (e.g., Frobenius distance from the initial pattern as a function of step) for both parameterizations at matched effective initialization. Please either add such measurements or soften the conclusion.
  3. [Tables 7–11 and All Boolean/Arithmetic Results] All experiments are reported from a single seed (seed 0 for Boolean and most vision tasks, seed 42 for ImageNet100, and no seed specified for arithmetic in Table 8). The differences between methods, especially the small gaps on cyclic tasks (87.45 vs 87.48) and the near-chance 75% accuracy on 2Parity, cannot be assessed for significance without multiple independent runs. At minimum, the central Boolean and arithmetic comparisons should be repeated over at least three seeds and reported as mean ± standard deviation.
  4. [Table 3 vs Table 5] Mask Init on ImageNet100 is reported as 85.92 in Table 3 but as 85.82 (+2.44) in Table 5; the implied improvement over Default (83.38) is 2.54 under the Table 3 value, not 2.44. Please correct the inconsistency and verify the underlying run.
minor comments (6)
  1. [§4.1] Please clarify whether the reported open/close initialization values (e.g., 10 and -10) are the raw mask parameters M or the effective logσ(M) values, since the text describes M as being transformed by logσ.
  2. [§2.1] The related-work paragraph contains a malformed citation ('Wang et al., Zhong and Andreas, 2024'); the citation and reference list should be cleaned up.
  3. [Appendix G.2] The arithmetic setup table omits the seed and some optimizer hyperparameters (e.g., AdamW betas), which should be added for reproducibility.
  4. [§5.3] The vision benchmarks in Table 3 report no variance; adding at least two additional seeds would strengthen the claim that Mask Init 'consistently' outperforms baselines.
  5. [Figure 2] The bottom-left subfigure label appears garbled ('(a) Addition10010010'); please correct the caption and the subfigure labels.
  6. [Appendix F, Table 6] The TinyStories test perplexity difference (10.48 vs 10.49) is within noise; the text should avoid claiming a 'consistent' improvement based on this difference alone.

Circularity Check

1 steps flagged · score 4.0 of 10

Near-perfect Boolean extrapolation is baked in by oracle masks; the core QK-vs-mask comparison remains empirical.

  1. self definitional [Section E ('Task-specific Mask Design'); Table 1; Section G.1 (mask init open=10, close=-10)]
    "For 2Parity, the target depends only on the pairwise interaction x0x1. Following Principle (1) and Principle (2), we allow attention only between x0 and x1, while suppressing all other interactions. This explicitly selects the correct computation path and prevents spurious correlations."

    The independent variable (the mask) is constructed from the exact ground-truth function decomposition, including the interaction structure on the unseen domain. The reported near-perfect extrapolation (Table 1: Mask Init 100.00 on 2Parity, 99.98 on Cyclic3, 100.00 on 3-Bit Majority) is therefore not a test of whether the mask parameterization preserves a prior through optimization; it is the oracle prior itself, inserted verbatim, that fixes the correct rule. Because log-sigma saturates at the chosen open=10/close=-10 values, the mask is effectively frozen, so the 'persistence' result is by construction rather than learned behavior.

full rationale

The paper is mostly a clean empirical study: the theorem and corollary are standard softmax calculus, and the vision and NLP results are benchmarked against external settings. The QK-initialization-versus-mask-initialization comparison is an experimental observation rather than a citation-derived claim, so most of the paper is not circular. However, the headline Boolean GOTU evidence is partially circular by construction. The masks are hand-designed from the exact ground-truth function decomposition in Section E, including the unseen interaction graph, and the open=10/close=-10 log-sigma values effectively freeze those oracle patterns. The near-perfect extrapolation in Table 1 is therefore largely an encoding of the answer, not a demonstration that the additive-mask parameterization is what makes priors persist. The paper's own Limitations paragraph concedes that the approach is 'in effect an oracle assumption.' Because the persistence claim itself also rests on this oracle-based Boolean comparison, the central demonstration carries a moderate, partial circularity, though the parameterization comparison is independently meaningful.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central empirical result rests on hand-chosen mask magnitudes, a hand-designed interaction prior for every task, and the MDI framework from Abbe et al. No new physical or architectural entities are introduced; the learnable mask is a standard attention-logit bias.

free parameters (3)
  • mask open logit value = 10 for Boolean and vision; 5 for arithmetic
    Hand-chosen positive initialization for allowed interactions in log sigma(M); sets how strongly the prior is encoded and is used directly in the central comparisons.
  • mask closed logit value = -10 for Boolean and vision; -5 for arithmetic
    Hand-chosen negative initialization suppressing unwanted interactions; the magnitude is a free choice that affects extrapolation results.
  • NLP relative-distance scale s = 10
    Used for TinyStories and WikiText-2 masks as log sigma(s M); selected by hand, affects the language results only.
assumptions (3)
  • domain assumption A meaningful task-level interaction prior is available before training and can be encoded as a mask.
    Stated in Limitations as an 'oracle assumption'. For Boolean tasks the mask patterns in Section E are built from the exact ground-truth function decomposition, so this axiom carries the extrapolation result.
  • standard math Transformer attention on Boolean GOTU problems follows the minimum-degree interpolator bias described in Abbe et al.
    The paper adopts the MDI construction in Section 3 as the definition of the failure mode and designs masks to suppress MDI pairwise terms.
  • domain assumption With the correct interaction support exposed by attention, a Transformer can compute the target Boolean and arithmetic functions.
    Section E reasons that a two-layer MLP can express XOR and that attention's role is selecting variables; the method does not add computation, only interaction structure.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Mask-Based Priors Are More Persistent than Query-Key Initializations." pith.science (2026). https://pith.science/paper/WIOUJTLD

@misc{pith2026260800418,
  author       = {Pith},
  title        = {Pith review of: Mask-Based Priors Are More Persistent than Query-Key Initializations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WIOUJTLD}},
  note         = {Machine review of arXiv:2608.00418}
}
read the original abstract

Transformers do not merely lack data on some Boolean extrapolation tasks; they generalize in a systematically wrong way. Recent work on generalization on the unseen has shown that, despite fitting the observed domain, Transformers often extrapolate according to a simpler minimum-degree interpolator rather than the true target function. These Boolean tasks are not practical applications, but controlled stress tests for understanding Transformer inductive bias. We ask whether this failure mode can be corrected by injecting explicit structural priors into attention. Existing structured-initialization methods alter Transformer inductive bias indirectly, by choosing query and key projections whose similarity scores approximate a desired attention pattern. However, we find that when applied to Boolean extrapolation, these QK-based priors can be rapidly overwritten during training and fail to change the learned extrapolation rule. We propose a simpler alternative: initialize the additive attention mask directly. Unlike standard hard masks used for causality or locality attention, our mask is a finite, learnable attention-logit bias initialized from task-level interaction structure. This separates the structural prior from content-dependent attention scores, allowing it to persist throughout optimization. On Boolean reasoning tasks, mask-based initialization achieves near-perfect extrapolation where vanilla and QK-initialized Transformers remain trapped by the default inductive bias. The same mechanism also improves low-data arithmetic performance and remains competitive on vision and language benchmarks. These results show that attention masks can serve not only as architectural constraints, but as a simple substrate for encoding persistent inductive bias in Transformers.

Figures

Figures reproduced from arXiv: 2608.00418 by the authors.

Figure 1
Figure 1. Where the prior lives determines whether it changes extrapolation. Boolean GOTU tasks expose a failure mode in which Transformers fit the observed domain but extrapolate according to the wrong rule, often a simpler minimum-degree interpolator. We compare two ways of injecting the same structural prior into attention. QK-based initialization encodes the prior indirectly through content-dependent query-key projections… view at source ↗
Figure 2
Figure 2. Visualization of task-specific mask design. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Test performance on Boolean reasoning tasks under an extrapolation setting. The left panel [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: Test sequence accuracy under varying training fractions for multiplication (left) and addition [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Comparison of attention patterns before and after training across initialization methods [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Initial mask logits for different tasks. Each row corresponds to a task and each column [PITH_FULL_IMAGE:figures/full_fig_p019_6.png]
Figure 7
Figure 7. Figure 7: Training curves on TinyStories (top) and WikiText-2 (bottom). [PITH_FULL_IMAGE:figures/full_fig_p020_7.png]
Figure 8
Figure 8. Figure 8: Relative mask bias before and after training on TinyStories and WikiText. The lower [PITH_FULL_IMAGE:figures/full_fig_p021_8.png]
Figure 9
Figure 9. Figure 9: Structured initialization for vision Transformers. Convolutional impulse filters (above) [PITH_FULL_IMAGE:figures/full_fig_p026_9.png]
Figure 10
Figure 10. Figure 10: Learned mask (after training) for CIFAR10, CIFAR100, and ImageNet100 across different [PITH_FULL_IMAGE:figures/full_fig_p027_10.png]
Figure 11
Figure 11. Figure 11: Attention maps for the first validation sample on CIFAR10. [PITH_FULL_IMAGE:figures/full_fig_p028_11.png]
Figure 12
Figure 12. Figure 12: Attention maps for the first validation sample on CIFAR100. [PITH_FULL_IMAGE:figures/full_fig_p029_12.png]
Figure 13
Figure 13. Figure 13: Attention maps for the first validation sample on ImageNet100. [PITH_FULL_IMAGE:figures/full_fig_p030_13.png]
Figure 14
Figure 14. Figure 14: Training curves on CIFAR100 (left) and ImageNet100 (right) under identical training [PITH_FULL_IMAGE:figures/full_fig_p031_14.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 5 canonical work pages

  1. [1]

    It remains to prove the gradient inequality

    Therefore, ∥softmax(A)∥F≥1. It remains to prove the gradient inequality. Let P= softmax(A), H(A,M) =σ(M)◦P. For fixedM, the derivative ofHwith respect toAsatisfies ∂Hij ∂Aik =σ(M ij)∂Pij ∂Aik , and derivatives across different rows are zero, as in the standard row-wise softmax. Since 0≤σ(M ij)≤1, we have entrywise ∂Hij ∂Aik ≤ ∂Pij ∂Aik . Therefore, summin...

    work page 2024

  2. [5]

    Structured initialization for vision transformers.arXiv preprint arXiv:2505.19985,

    Jianqiao Zheng, Xueqian Li, Hemanth Saratchandran, and Simon Lucey. Structured initialization for vision transformers.arXiv preprint arXiv:2505.19985,

    work page arXiv

  3. [8]

    Table 8: Arithmetic task setup. Item Value Task Addition / Multiplication (3-digit) Dataset size10 6 (all combinations) Model Transformer Layers / Heads 6 / 4 Model dim 256 MLP dim 256 Head dim 64 Positional encoding learnable Loss Cross-entropy Optimizer AdamW Learning rate1×10 −3 Weight decay1×10 −3 Batch size 256 Epochs 50 Scheduler cosine Mask init op...

    work page 2000

  4. [9]

    On the minimal degree bias in generalization on the unseen for non-boolean functions

    Denys Pushkin, Raphaël Berthier, and Emmanuel Abbe. On the minimal degree bias in generalization on the unseen for non-boolean functions. InForty-first International Conference on Machine Learning. Orit Davidovich and Zohar Ringel. Algorithmic capture, computational complexity, and inductive bias of infinite transformers.arXiv preprint arXiv:2603.11161,

    arXiv

  5. [11]

    Xiaohuan Pei, Tao Huang, YanXiang Ma, and Chang Xu

    URL https://proceedings.mlr.press/v235/yin24a.html. Xiaohuan Pei, Tao Huang, YanXiang Ma, and Chang Xu. Rethinking causal mask attention for vision-language inference.arXiv preprint arXiv:2505.18605,

    arXiv

  6. [13]

    Generating long sequences with sparse transformers.arXiv preprint arXiv:1904.10509,

    Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers.arXiv preprint arXiv:1904.10509,

    arXiv 1904

  7. [14]

    k-means mask transformer

    Qihang Yu, Huiyu Wang, Siyuan Qiao, Maxwell Collins, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chieh Chen. k-means mask transformer. In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors,Computer Vision – ECCV 2022, pages 288–307, Cham,

    work page 2022

  8. [15]

    ISBN 978-3-031-19818-2

    Springer Nature Switzerland. ISBN 978-3-031-19818-2. Jonathan Ho, Nal Kalchbrenner, Dirk Weissenborn, and Tim Salimans. Axial attention in multidi- mensional transformers.arXiv preprint arXiv:1912.12180,

    arXiv 1912

Show all 21 references
  1. [16]

    Alex Krizhevsky

    doi: 10.1007/s11263-015-0816-y. Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009a. Hemanth Saratchandran, Jianqiao Zheng, Yiping Ji, Wenbo Zhang, and Simon Lucey. Rethinking attention: Polynomial alternatives to softmax in transfor...

  2. [17]

    Tinystories: How small can language models be and still speak coherent english?arXiv preprint arXiv:2305.07759,

    Ronen Eldan and Yuanzhi Li. Tinystories: How small can language models be and still speak coherent english?arXiv preprint arXiv:2305.07759,

    arXiv

  3. [18]

    Pointer sentinel mixture models.arXiv preprint arXiv:1609.07843,

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models.arXiv preprint arXiv:1609.07843,

    arXiv

  4. [2009]

    doi: 10.1109/CVPR.2009.5206848. 13 A Ablation Study A.1 Does the Gain Come From Additional Mask Parameters? We ablate structured mask initialization using an all-zero initialized learnable mask, corresponding to a neutral bias over interactions. As shown in Table 4, it matches...

    2009

  5. [2010]

    URLhttps://proceedings.mlr.press/v9/glorot10a.html

    PMLR. URLhttps://proceedings.mlr.press/v9/glorot10a.html. Zhiqiu Xu, Yanjie Chen, Kirill Vishniakov, Yida Yin, Zhiqiang Shen, Trevor Darrell, Lingjie Liu, and Zhuang Liu. Initializing models with larger ones.arXiv preprint arXiv:2311.18823,

    arXiv

  6. [2015]

    Cutting the skip: Training residual-free transformers.arXiv preprint arXiv:2510.00345,

    Yiping Ji, James Martens, Jianqiao Zheng, Ziqin Zhou, Peyman Moghadam, Xinyu Zhang, Hemanth Saratchandran, and Simon Lucey. Cutting the skip: Training residual-free transformers.arXiv preprint arXiv:2510.00345,

  7. [2019]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

    arXiv

  8. [2020]

    Longformer: The long-document transformer

    Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150,

    2004 arXiv

  9. [2022]

    On weight initialization in deep neural networks.arXiv preprint arXiv:1704.08863,

    Siddharth Krishna Kumar. On weight initialization in deep neural networks.arXiv preprint arXiv:1704.08863,

    arXiv

  10. [2023]

    Scaling smart: Accelerating large language model pre-training with small model initialization.arXiv preprint arXiv:2409.12903,

    11 Mohammad Samragh, Iman Mirzadeh, Keivan Alizadeh Vahid, Fartash Faghri, Minsik Cho, Moin Nabi, Devang Naik, and Mehrdad Farajtabar. Scaling smart: Accelerating large language model pre-training with small model initialization.arXiv preprint arXiv:2409.12903,

    arXiv

  11. [2024]

    Weight subcloning: direct initialization of transformers using larger pretrained ones.arXiv preprint arXiv:2312.09299,

    Mohammad Samragh, Mehrdad Farajtabar, Sachin Mehta, Raviteja Vemulapalli, Fartash Faghri, Devang Naik, Oncel Tuzel, and Mohammad Rastegari. Weight subcloning: direct initialization of transformers using larger pretrained ones.arXiv preprint arXiv:2312.09299,

    arXiv

  12. [2025]

    An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:2010.11929,

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:...

    2010 arXiv

  13. [2026]

    Jianqiao Zheng, Xueqian Li, and Simon Lucey

    URL https://arxiv.org/abs/2511.13945. Jianqiao Zheng, Xueqian Li, and Simon Lucey. Structured initialization for attention in vision transformers.arXiv preprint arXiv:2404.01139,

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.