Pith. sign in

REVIEW 2 major objections 7 minor 68 references

This paper claims that standard transformers are rarely a local optimum in architecture space, and that replacing GeLU and softmax with splines optimized on held-out data yields large gains on algorithmic tasks and smaller but consistent ga

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 17:27 UTC pith:562ZKPUS

load-bearing objection Solid empirical study showing transformers can be improved by learned non-linearities, but the cross-task 'sacrifice universality' claim is not fully supported because the transfer comparisons use baseline-tuned budgets. the 2 major comments →

arxiv 2607.17624 v1 pith:562ZKPUS submitted 2026-07-20 cs.LG

Can Transformers Really Do It All? On the Compatibility of Inductive Biases Across Tasks

Damien Teney , Liangze Jiang , Hemanth Saratchandran , Simon Lucey This is my paper
classification cs.LG
keywords inductive biastransformerslearned activation functionsspline parametrizationtask compatibilitylength generalizationlanguage modelingalgorithmic reasoning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper asks whether the transformer design that dominates modern AI is actually the best design for any given task. It answers no: by replacing the two main nonlinearities—the GeLU in MLP blocks and the softmax in attention—with piecewise-linear splines learned on held-out data, the authors find architectures that learn much faster, generalize better, and vary less across random seeds on algorithmic tasks such as addition, copying, and modular arithmetic. The same procedure gives small but consistent improvements on language and code modeling, and those improvements transfer across datasets and domains. The central message is that standard transformers are rarely a local optimum in architecture space: simple alternatives can do much better, but the best designs are often task-specific, especially for algorithmic reasoning.

Core claim

The paper's central claim, stated on its own terms, is that the inductive biases of a standard transformer are rarely optimal for any single task. Replacing the GeLU nonlinearity and the softmax attention kernel with linear splines that are optimized on a held-out split, then frozen and reused to train fresh models, produces architectures that dramatically accelerate convergence, improve in- and out-of-distribution generalization, and stabilize seed-to-seed behavior on algorithmic toy tasks. These gains are highly task-specific: an architecture optimized for one algorithmic skill often underperforms even the baseline on another. For natural language and code, the same method yields smaller b

What carries the argument

The mechanism that carries the argument is a family of learnable nonlinearities: the elementwise MLP activation (normally GeLU) and the attention similarity kernel (normally softmax) are each replaced by a 1D linear spline whose knot values are learned in stage I on a held-out split of the target dataset. Stage I uses two losses—one for weights on training data, one for the spline parameters on held-out data—and trains several models in parallel sharing the splines, so the splines capture a bias that survives re-training rather than a co-adaptation to particular weights. In stage II the splines are frozen and models are trained from scratch, which is what turns an optimization procedure into

Load-bearing premise

The load-bearing premise is that the two-stage procedure isolates a task's inductive bias rather than an artifact of optimization: if the splines merely smoothed the loss landscape or acted as a regularizer, the dramatic gains on algorithmic tasks would not prove a better inductive bias and the cross-task incompatibility conclusion would collapse.

What would settle it

Train stage II models with the learned spline replaced by a random spline of matched smoothness and the same training recipe; if the matched random spline reproduces the convergence and length-generalization gains, then the specific learned shapes are not the source of the improvement and the paper's compatibility story fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the central claim is right, the search for better LLMs should not stop at scaling: simple architectural tweaks to MLP nonlinearities can yield faster learning and better data efficiency at fixed parameter counts.
  • Because algorithmic skills demand highly task-specific designs, attempts to give LLMs robust arithmetic and reasoning may need to introduce explicit specialized modules or multi-task architecture optimization rather than one universal transformer.
  • The observed transfer of optimized non-linearities across English and code datasets implies that a single improved nonlinearity could serve multilingual and multimodal language modeling without retuning.
  • Better length generalization on COPY from spline-optimized linear-bias positional encodings suggests that part of the length-generalization problem is architectural, not just positional-encoding based.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves open whether the same spline optimization can satisfy several algorithmic tasks at once; a multi-task version of stage I would be a direct test of whether the observed task-specificity is fundamental or a consequence of optimizing one task at a time.
  • If the gains on algorithmic tasks truly come from inductive bias rather than optimization smoothing, then distilling the learned splines into cheap closed-form activations could make specialized models practical, since the paper finds high-degree polynomial approximations nearly as good.
  • The asymmetry—MLP nonlinearities matter while softmax is hard to beat—suggests future architecture search should concentrate on the MLP block, where the flexibility budget is largest.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 7 minor

Summary. The paper proposes a two-stage method to optimize the non-linear components of a transformer (GeLU MLP activations and softmax attention) by replacing them with learned linear splines. In stage I, the spline parameters are optimized on a held-out split while model weights train on the main split, using multiple parallel models to reduce co-adaptation. In stage II, the learned splines are frozen and models are retrained from scratch on the same or other datasets. The method is applied to eight algorithmic toy tasks and seven code/language modeling datasets. The main findings are: (i) on algorithmic tasks, optimized architectures dramatically improve learning speed, generalization, and seed stability, but the benefits are highly task-specific (strong diagonal in the cross-task transfer matrix); (ii) on code/language modeling, the improvements are small but consistent, and the optimized architectures transfer much better across datasets and domains; (iii) these results are used to argue that standard transformers are rarely a local optimum in the space of architectures, but that better task-specific architectures sacrifice universality.

Significance. If the findings hold, the paper makes a meaningful empirical contribution to the study of inductive biases in transformers. Its strengths include the use of multiple seeds (6 for algorithmic, 3 for language), a two-stage procedure that avoids a naive co-adaptation of weights and non-linearities, a tuned baseline, and a comparison against several existing alternative activations and attention variants. The paper also ships code and includes a careful discussion of limitations. The central empirical claims are reproducible in principle and the results on algorithmic tasks are striking. However, the interpretation of the cross-task transfer results as evidence of inductive-bias incompatibility, and the claim of consistent language-model improvements, rest on assumptions about hyperparameter transfer and statistical significance that are not fully demonstrated.

major comments (2)
  1. [§3.2, Fig. 5, Appendix B] The claim that optimized architectures 'sacrifice universality' rests on the diagonal-dominant pattern in Figure 5, but the comparison is made after a fixed number of steps and with hyperparameters tuned only for the baseline transformer (Appendix B, Table 4). The optimized splines have very different shapes and scales (Fig. 6), so the baseline-tuned learning rate and step budget may be strongly suboptimal for them. The statement in §3.1 that 'we get no further improvements by re-tuning them' was established only for same-task stage II training, not for cross-task transfers. Without per-architecture LR re-tuning or a sensitivity analysis for the off-diagonal entries, the negative off-diagonal values could reflect slower convergence or LR mismatch rather than a fundamental incompatibility of inductive biases. This directly weakens the headline conclusion in the abstract and Section 6.
  2. [§4, Fig. 9, Table 6, Table 7] The language-modeling improvements are small (e.g. 0.1–0.4 percentage points in accuracy, differences of a few hundredths in perplexity), but the paper does not report confidence intervals, error bars, or seed-wise values for most of these results. Figure 9 presents a single scalar per cell, and Table 6 in Appendix E reports averages over 3 seeds with no variance. Without an indication of variability, the claims that the improvements are 'consistent' and that the architectures 'transfer much better' across language datasets cannot be evaluated statistically. This is load-bearing for the conclusion that standard transformers are 'not a local optimum' for language modeling, because the differences are close to the noise level one might expect from 3 seeds.
minor comments (7)
  1. [Abstract] Typo: 'GeLUs,;softmax' should be 'GeLUs and softmax'.
  2. [Figure 6] The optimized MLP non-linearities are presented without axis labels or units. Adding x/y axis labels (e.g. input activation vs. output) would make the figure more informative.
  3. [Figure 9 caption] The caption says 'relative difference in perplexity with the baseline in %, lower is better', but the sign convention is not self-evident. It would be helpful to state explicitly that negative values indicate improvement over the baseline.
  4. [§3.1] The sentence 'we get no further improvements by re-tuning them' is ambiguous. It should explicitly say that this was tested for same-task stage II training only, not for cross-task transfers, especially given the concern in the major comment.
  5. [Appendix B, Table 4] The number of parallel models M differs between algorithmic tasks (M=8) and language modeling (M=3). A brief justification of this choice would help readers assess the comparability of the two experimental settings.
  6. [Appendix F, Listing 2] Typo: 'TorchsSscript' should be 'TorchScript'.
  7. [Table 3] The abbreviations 'Tr. perplexity' and 'Val. acc.' are not defined in the caption. They appear to be training perplexity and validation token accuracy; please spell them out.

Circularity Check

0 steps flagged

No significant circularity: the optimized architectures are evaluated on disjoint held-out/OOD data after freezing the splines, so the central empirical claims do not reduce to their inputs.

full rationale

This is an empirical study rather than a formal derivation, and I found no step where a claimed prediction or first-principles result is equivalent to its inputs by construction. The paper's two-stage protocol (Section 2) optimizes spline nonlinearities on a held-out subset (or an OOD length split) in stage I, then freezes them and retrains all weights from scratch in stage II before measuring performance on disjoint test sets. Reported improvements are therefore not fitted values renamed as predictions; the same-task gains, cross-task transfer matrices, and length-generalization results are all measured after retraining with frozen architectures. The spline parametrization is chosen as a deliberately unbiased search space, and the paper compares against existing activations and attention variants (Table 3) and strong external baselines such as NanoGPT Speedrun on FINEWEB (Appendix E), so the central claims do not rest on a self-citation chain. The self-citations to Teney et al. (2024, 2025) are used for background on simplicity bias and for a minor design choice (linear vs. higher-order splines) identified as cheaper in prior work; they are not load-bearing for the main empirical findings. The cross-task compatibility results use fixed step budgets and baseline-tuned hyperparameters (Appendix B, Table 4), which is a legitimate potential confound about optimization artifact versus true inductive-bias incompatibility, but it is a threat to external validity, not a definitional circularity: the off-diagonal entries are computed from actual retraining runs, not imposed by construction. The paper's own limitations section acknowledges the restricted search space and small scale, further indicating that the authors do not claim a derivation from first principles. No circular step meets the evidentiary bar of exhibiting Eq. X = Eq. Y by construction or a fitted parameter renamed as a prediction.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

The central empirical result relies on hand-chosen hyperparameters of the spline search space and on domain assumptions about the tasks and the two-stage training isolating inductive bias. No new physical or conceptual entities are posited.

free parameters (5)
  • Spline range [a,b] = [-20, 20]
    Chosen by hand; limits the input domain of the learned activation functions, affecting which functions can be represented.
  • Number of spline control points n_c = 122 (spacing 1/3 or 1/9)
    Chosen by hand; controls the resolution of the piecewise-linear approximation; 122 points in [-20,20].
  • Held-out fraction α = 0.2
    Fraction of training data used exclusively for architecture optimization in stage I; chosen by hand to balance weight vs. architecture fitting.
  • Number of parallel models M = 8 (algorithmic), 3 (language)
    Number of models sharing the optimized splines during stage I; larger M reduces variance but increases compute.
  • Polynomial degree for approximation = n=18
    Used for efficient stage-II evaluation; chosen as a trade-off between approximation fidelity and speed.
axioms (5)
  • domain assumption The algorithmic toy tasks (Table 1) and language/code datasets (Table 2) are representative proxies for the capabilities the paper discusses (reasoning, fluency, code understanding).
    The conclusions about inductive biases and task compatibility rely on the chosen tasks spanning distinct skill types; Section 3 and 4.
  • domain assumption Optimizing splines on a held-out split and freezing them during stage II isolates the inductive bias of the architecture from the particular optimization trajectory.
    The two-loss training with held-out data (Section 2) is assumed to prevent co-adaptation of weights and non-linearities; this is the core methodological assumption.
  • domain assumption The improvements measured in stage II with frozen splines transfer across random seeds and are not due to the spline providing a gradient or optimization advantage rather than a representational inductive bias.
    The paper relies on multi-model training (M>1) and re-training from scratch to support this; this is not proven, only empirically supported.
  • domain assumption The test sets are strictly disjoint from training and held-out splits, and comparisons against baseline use the same hyperparameters and training budget.
    Fairness of the comparison depends on this; described in Appendix B.
  • standard math Piecewise-linear splines can approximate any univariate function on a bounded interval to arbitrary accuracy, and this parametrization has no strong prior towards specific function classes.
    The claim that splines are a neutral search space relies on this property; stated in Section 2.

pith-pipeline@v1.3.0-alltime-deepseek · 24125 in / 14400 out tokens · 120857 ms · 2026-08-01T17:27:16.241611+00:00 · methodology

0 comments
read the original abstract

Transformers are remarkably versatile and their design is largely consistent across a variety of applications. But are they optimal for any given task or dataset? The answer may be key for pushing AI beyond merely scaling current designs. *Method.* We present a method to optimize a transformer architecture for a given dataset, which we use as a tool to study optimal task-specific inductive biases. This method replaces the most important non-linearities (GeLUs,;softmax) with functions learned on held-out data. We then train the resulting architectures on other datasets, as a way to evaluate the compatibility between pairs of tasks. Findings. On algorithmic toy tasks, we identify new architectures with dramatic improvements in learning speed, in- and out-of-distribution generalization, and stability across seeds. The new designs prove very task-specific however, and indicate that these tasks require inductive biases very different from those of standard transformers. On code and language modeling datasets, we also find architectures with consistent, yet smaller improvements. These designs transfer much better across datasets and domains (English & computer code). Implications. Our results show that standard transformers are rarely a local optimum in the space of architectures. Simple alternatives can perform much better but sacrifice universality. This suggests that there may be room for improved architectures that better support multiple capabilities simultaneously, such as fluency and robust reasoning.

Figures

Figures reproduced from arXiv: 2607.17624 by Damien Teney, Hemanth Saratchandran, Liangze Jiang, Simon Lucey.

Figure 1
Figure 1. Figure 1: Our approach to discover better task-specific inductive biases. (Left) We replace the non-linearities in a transformer (softmax, GeLUs) with splines optimized from scratch for one cho￾sen dataset. (Right) The new resulting architectures (with frozen splines as non-linearities) allow us to train models with dramatically better convergence, generalization, and stability across seeds, on algorithmic tasks and… view at source ↗
Figure 2
Figure 2. Figure 2: Training curves (test accuracy vs. training step, one curve per random seed) of models [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Length generalization on the COPY task. The baseline completely fails on unseen lengths (≫10). Alibi positional encodings (Press et al., 2021) help. Optimizing the Alibi archi￾tecture with our method further improves the accuracy and extends the benefits to longer sequences. Performance with smaller models. We train models of different widths for each task. Results in [PITH_FULL_IMAGE:figures/full_fig_p00… view at source ↗
Figure 4
Figure 4. Figure 4: Test accuracy of models of different widths ( [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Compatibility of architectures across algorithmic tasks. We plot the absolute difference in [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: MLP non-linearities optimized for each algorithmic task. [PITH_FULL_IMAGE:figures/full_fig_p006_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: (Left) Absolute improvements in token prediction accuracy (%) of the best optimized architectures on TINYSTORIES compared to our baseline transformer. The accuracy is consistently slightly better at different model sizes. (Right) Visualization of MLP non-linearities optimized from scratch (results on the left) or from a GeLU initialization (GeLU + Ours in [PITH_FULL_IMAGE:figures/full_fig_p007_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Perplexity on code and natural language (lower is better; numbers on bars correspond to [PITH_FULL_IMAGE:figures/full_fig_p008_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: show that the variations across architectures are very small. This contrasts with the results on algorithmic tasks ( [PITH_FULL_IMAGE:figures/full_fig_p008_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Training curves (test accuracy vs. training steps, one curve per seed) of models trained [PITH_FULL_IMAGE:figures/full_fig_p020_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Compatibility of architectures across algorithmic tasks (difference in test accuracy with [PITH_FULL_IMAGE:figures/full_fig_p021_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: MLP non-linearities optimized for TINYSTORIES and versions modified to enforce sym￾metry. Almost all of these perform worse than the original ones, whose fine details therefore matter. Multi-model training. We compare in [PITH_FULL_IMAGE:figures/full_fig_p022_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Absolute improvements in training perplexity on character-level S [PITH_FULL_IMAGE:figures/full_fig_p023_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Same as Figure 13 with subword-level tokenization. [PITH_FULL_IMAGE:figures/full_fig_p023_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Training curves on language datasets with baseline ( [PITH_FULL_IMAGE:figures/full_fig_p023_15.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

68 extracted references · 46 linked inside Pith

  1. [1]

    out- put

    Algorithm 1Proposed method (stage I) to optimize a transformer architecture for a specific task. Input: Training dataD={s i}n i=1 as token sequencess∈S. Baseline architectureTinstantiable as next-token prediction modelT θ :S→Sof weightsθ. L(·,·): Loss function.α: Fraction of held-out data.M: Number of parallel models. Method: Define a new architecture ˆT ...

    2019

  2. [3]

    •Adaptive softmax: Veli ˇckovi´c et al. (2024). •NormSoftmax: Jiang et al. (2023). •Polynomial attention P1:(Q ⊤K)/√seqLength: Saratchandran et al. (2024b). •Polynomial attention P3:(Q ⊤K)3/√seqLength: Saratchandran et al. (2024b). •GLU: Shazeer (2020). •ReLU 2: So et al. (2021b). •Sinc: Saratchandran et al. (2024a). •Gaussian: Saragadam et al. (2023). 22...

    2024

  3. [4]

    Simplicity bias in transformers and their ability to learn sparse boolean functions.arXiv:2211.12316,

    Satwik Bhattamishra, Arkil Patel, Varun Kanade, and Phil Blunsom. Simplicity bias in transformers and their ability to learn sparse boolean functions.arXiv:2211.12316,

    Pith/arXiv arXiv

  4. [5]

    cleaning

    We show the effect when optimizing the non-linearities in MLP or attention layers, or both. Optimizing the non-linearities in the attention proves to be really challeng- ing, and the best results are usually obtained by optimizing only the MLPs. Architectures optimized for specific tasks Memorize AddMod Add AddReversed Haystack Parentheses Memorize AddMod...

    2026

  5. [6]

    Microsoft CodeGPT (available on HuggingFace), 2024.https://huggingface

    CodeGPT. Microsoft CodeGPT (available on HuggingFace), 2024.https://huggingface. co/microsoft/CodeGPT-small-py. Giacomo De Palma, Bobak Kiani, and Seth Lloyd. Random deep neural networks are biased towards simple functions.NeurIPS,

    2024

  6. [7]

    Input–output maps are strongly biased towards simple outputs.Nature communications,

    10 Published as a conference paper at ICLR 2026 Kamaludin Dingle, Chico Q Camargo, and Ard A Louis. Input–output maps are strongly biased towards simple outputs.Nature communications,

    2026

  7. [8]

    Tinystories: How small can language models be and still speak coherent english?arXiv:2305.07759,

    Ronen Eldan and Yuanzhi Li. Tinystories: How small can language models be and still speak coherent english?arXiv:2305.07759,

    Pith/arXiv arXiv

  8. [9]

    The no free lunch theorem, kolmogorov complexity, and the role of inductive biases in machine learning

    Micah Goldblum, Marc Finzi, Keefer Rowan, and Andrew Gordon Wilson. The no free lunch theorem, kolmogorov complexity, and the role of inductive biases in machine learning. arXiv:2304.05366,

    Pith/arXiv arXiv

  9. [11]

    Scaling down deep learning with MNIST-1D.arXiv preprint arXiv:2011.14439,

    Sam Greydanus and Dmitry Kobak. Scaling down deep learning with MNIST-1D.arXiv preprint arXiv:2011.14439,

    Pith/arXiv arXiv 2011

  10. [12]

    Why are sensitive functions hard for transformers? arXiv:2402.09963,

    Michael Hahn and Mark Rofin. Why are sensitive functions hard for transformers? arXiv:2402.09963,

    Pith/arXiv arXiv

  11. [16]

    Asnn: Learning to suggest neural architectures from performance distributions

    Jinwook Hong. Asnn: Learning to suggest neural architectures from performance distributions. arXiv:2507.20164,

    Pith/arXiv arXiv

  12. [17]

    Curvature tuning: Provable training-free model steering from a single parameter.arXiv:2502.07783,

    11 Published as a conference paper at ICLR 2026 Leyang Hu, Matteo Gamba, and Randall Balestriero. Curvature tuning: Provable training-free model steering from a single parameter.arXiv:2502.07783,

    arXiv 2026

  13. [18]

    Code- searchnet challenge: Evaluating the state of semantic code search.arXiv:1909.09436,

    Hamel Husel, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. Code- searchnet challenge: Evaluating the state of semantic code search.arXiv:1909.09436,

    Pith/arXiv arXiv 1909

  14. [19]

    Repeat after me: Trans- formers are better than state space models at copying.arXiv:2402.01032,

    Samy Jelassi, David Brandfonbrener, Sham M Kakade, and Eran Malach. Repeat after me: Trans- formers are better than state space models at copying.arXiv:2402.01032,

    Pith/arXiv arXiv

  15. [20]

    Entropy-guided attention for private llms

    Nandan Kumar Jha and Brandon Reagen. Entropy-guided attention for private llms. arXiv:2501.03489,

    Pith/arXiv arXiv

  16. [21]

    The unreasonable effectiveness of recurrent neural networks.http:// karpathy.github.io/2015/05/21/rnn-effectiveness/,

    Andrej Karpathy. The unreasonable effectiveness of recurrent neural networks.http:// karpathy.github.io/2015/05/21/rnn-effectiveness/,

    2015

  17. [22]

    Regularization for deep learning: A taxonomy

    Jan Kukaˇcka, Vladimir Golkov, and Daniel Cremers. Regularization for deep learning: A taxonomy. arXiv:1710.10686,

    Pith/arXiv arXiv

  18. [25]

    CodeXGlue: A machine learning bench- mark dataset for code understanding and generation.arXiv:2102.04664,

    12 Published as a conference paper at ICLR 2026 Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et al. CodeXGlue: A machine learning bench- mark dataset for code understanding and generation.arXiv:2102.04664,

    Pith/arXiv arXiv 2026

  19. [26]

    Llms on the line: Data determines loss-to-loss scaling laws.arXiv:2502.12120,

    Prasanna Mayilvahanan, Thadd¨aus Wiedemer, Sayak Mallick, Matthias Bethge, and Wieland Bren- del. Llms on the line: Data determines loss-to-loss scaling laws.arXiv:2502.12120,

    Pith/arXiv arXiv

  20. [27]

    Neural networks are a priori biased towards boolean functions with low entropy

    Chris Mingard, Joar Skalse, Guillermo Valle-P´erez, David Mart´ınez-Rubio, Vladimir Mikulik, and Ard A Louis. Neural networks are a priori biased towards boolean functions with low entropy. arXiv:1909.11522,

    Pith/arXiv arXiv 1909

  21. [28]

    Do deep neural networks have an inbuilt occam’s razor?arXiv:2304.06670,

    Chris Mingard, Henry Rees, Guillermo Valle-P ´erez, and Ard A Louis. Do deep neural networks have an inbuilt occam’s razor?arXiv:2304.06670,

    Pith/arXiv arXiv

  22. [29]

    Relu strikes back: Exploiting activation sparsity in large language models.arXiv:2310.04564,

    Iman Mirzadeh, Keivan Alizadeh, Sachin Mehta, Carlo C Del Mundo, Oncel Tuzel, Golnoosh Samei, Mohammad Rastegari, and Mehrdad Farajtabar. Relu strikes back: Exploiting activation sparsity in large language models.arXiv:2310.04564,

    Pith/arXiv arXiv

  23. [30]

    Training transformers with enforced lipschitz constants.arXiv:2507.13338,

    Laker Newhouse, R Preston Hess, Franz Cesista, Andrii Zahorodnii, Jeremy Bernstein, and Phillip Isola. Training transformers with enforced lipschitz constants.arXiv:2507.13338,

    Pith/arXiv arXiv

  24. [31]

    Arithmetic without algo- rithms: Language models solve math with a bag of heuristics.arXiv:2410.21272,

    Yaniv Nikankin, Anja Reusch, Aaron Mueller, and Yonatan Belinkov. Arithmetic without algo- rithms: Language models solve math with a bag of heuristics.arXiv:2410.21272,

    Pith/arXiv arXiv

  25. [32]

    Optimizers qualitatively alter solu- tions and we should leverage this.arXiv preprint arXiv:2507.12224,

    Razvan Pascanu, Clare Lyle, Ionut-Vlad Modoranu, Naima Elosegui Borras, Dan Alistarh, Petar Velickovic, Sarath Chandar, Soham De, and James Martens. Optimizers qualitatively alter solu- tions and we should leverage this.arXiv preprint arXiv:2507.12224,

    Pith/arXiv arXiv

  26. [33]

    How does code pretraining affect language model task performance?arXiv:2409.04556,

    Jackson Petty, Sjoerd van Steenkiste, and Tal Linzen. How does code pretraining affect language model task performance?arXiv:2409.04556,

    Pith/arXiv arXiv

  27. [34]

    Train short, test long: Attention with linear biases enables input length extrapolation.arXiv preprint arXiv:2108.12409,

    13 Published as a conference paper at ICLR 2026 Ofir Press, Noah A Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation.arXiv preprint arXiv:2108.12409,

    Pith/arXiv arXiv 2026

  28. [35]

    Swish: a self-gated activation function

    Prajit Ramachandran, Barret Zoph, and Quoc V Le. Swish: a self-gated activation function. arXiv:1710.05941,

    Pith/arXiv arXiv

  29. [36]

    A sampling theory perspective on activations for implicit neural representations

    Hemanth Saratchandran, Sameera Ramasinghe, Violetta Shevchenko, Alexander Long, and Si- mon Lucey. A sampling theory perspective on activations for implicit neural representations. arXiv:2402.05427, 2024a. Hemanth Saratchandran, Jianqiao Zheng, Yiping Ji, Wenbo Zhang, and Simon Lucey. Rethinking softmax: Self-attention with polynomial activations.arXiv:24...

    Pith/arXiv arXiv

  30. [37]

    Glu variants improve transformer.arXiv:2002.05202,

    Noam Shazeer. Glu variants improve transformer.arXiv:2002.05202,

    Pith/arXiv arXiv 2002

  31. [38]

    Transformers pretrained on procedural data contain modular structures for algorithmic reasoning.arXiv preprint arXiv:2505.22308,

    Zachary Shinnick, Liangze Jiang, Hemanth Saratchandran, Anton van den Hengel, and Damien Teney. Transformers pretrained on procedural data contain modular structures for algorithmic reasoning.arXiv preprint arXiv:2505.22308,

    Pith/arXiv arXiv

  32. [39]

    Searching for efficient transformers for language modeling.NeurIPS, 34, 2021a

    David So, Wojciech Ma´nke, Hanxiao Liu, Zihang Dai, Noam Shazeer, and Quoc V Le. Searching for efficient transformers for language modeling.NeurIPS, 34, 2021a. David R So, Wojciech Manke, Hanxiao Liu, Zihang Dai, Noam Shazeer, and Quoc V Le. Primer: Searching for efficient transformers for language modeling, 2022.arXiv:2109.08668, 2021b. Leon Ren´e S¨utfe...

    Pith/arXiv arXiv 2022

  33. [40]

    On the learning dynamics of deep neural networks.arXiv:1809.06848,

    Remi Tachet, Mohammad Pezeshki, Samira Shabanian, Aaron Courville, and Yoshua Bengio. On the learning dynamics of deep neural networks.arXiv:1809.06848,

    Pith/arXiv arXiv

  34. [41]

    Your attention matters: to improve model robustness to noise and spurious correlations.arXiv:2507.20453,

    Camilo Tamayo-Rousseau, Yunjia Zhao, Yiqun Zhang, and Randall Balestriero. Your attention matters: to improve model robustness to noise and spurious correlations.arXiv:2507.20453,

    Pith/arXiv arXiv

  35. [42]

    Scaling laws vs model architectures: How does inductive bias influence scaling?arXiv:2207.10551,

    Yi Tay, Mostafa Dehghani, Samira Abnar, Hyung Won Chung, William Fedus, Jinfeng Rao, Sharan Narang, Vinh Q Tran, Dani Yogatama, and Donald Metzler. Scaling laws vs model architectures: How does inductive bias influence scaling?arXiv:2207.10551,

    Pith/arXiv arXiv

  36. [43]

    Evading the simplic- ity bias: Training a diverse set of models discovers solutions with superior ood generalization

    14 Published as a conference paper at ICLR 2026 Damien Teney, Ehsan Abbasnejad, Simon Lucey, and Anton van den Hengel. Evading the simplic- ity bias: Training a diverse set of models discovers solutions with superior ood generalization. arXiv:2105.05612,

    Pith/arXiv arXiv 2026

  37. [44]

    Deep learning generalizes because the parameter-function map is biased towards simple functions.arXiv:1805.08522,

    Guillermo Valle-Perez, Chico Q Camargo, and Ard A Louis. Deep learning generalizes because the parameter-function map is biased towards simple functions.arXiv:1805.08522,

    Pith/arXiv arXiv

  38. [45]

    Simplicity bias of transformers to learn low sensitivity functions.arXiv:2403.06925,

    Bhavya Vasudeva, Deqing Fu, Tianyi Zhou, Elliott Kau, Youqi Huang, and Vatsal Sharan. Simplicity bias of transformers to learn low sensitivity functions.arXiv:2403.06925,

    Pith/arXiv arXiv

  39. [46]

    softmax is not enough (for sharp out-of-distribution).arXiv:2410.01104,

    Petar Veliˇckovi´c, Christos Perivolaropoulos, Federico Barbero, and Razvan Pascanu. softmax is not enough (for sharp out-of-distribution).arXiv:2410.01104,

    Pith/arXiv arXiv

  40. [47]

    Neural architecture search: Insights from 1000 papers

    Colin White, Mahmoud Safari, Rhea Sukthanker, Binxin Ru, Thomas Elsken, Arber Zela, De- badeepta Dey, and Frank Hutter. Neural architecture search: Insights from 1000 papers. arXiv:2301.08727,

    Pith/arXiv arXiv

  41. [49]

    Initial- ization is critical to whether transformer fits composite function by inference or memorizing

    Zhongwang Zhang, Pengxiao Lin, Zhiwei Wang, Yaoyu Zhang, and Zhi-Qin John Xu. Initial- ization is critical to whether transformer fits composite function by inference or memorizing. arXiv:2405.05409,

    Pith/arXiv arXiv

  42. [50]

    Algorithmic capabilities of random transformers

    Ziqian Zhong and Jacob Andreas. Algorithmic capabilities of random transformers. arXiv:2410.04368,

    Pith/arXiv arXiv

  43. [51]

    What algorithms can transformers learn? a study in length generalization

    Hattie Zhou, Arwen Bradley, Etai Littwin, Noam Razin, Omid Saremi, Josh Susskind, Samy Bengio, and Preetum Nakkiran. What algorithms can transformers learn? a study in length generalization. arXiv:2310.16028,

    Pith/arXiv arXiv

  44. [52]

    Trans- formers can achieve length generalization but not robustly.arXiv preprint arXiv:2402.09371,

    Yongchao Zhou, Uri Alon, Xinyun Chen, Xuezhi Wang, Rishabh Agarwal, and Denny Zhou. Trans- formers can achieve length generalization but not robustly.arXiv preprint arXiv:2402.09371,

    Pith/arXiv arXiv

  45. [53]

    15 Published as a conference paper at ICLR 2026 A ADDITIONALRELATEDWORK Inductive biases in deep learningare due to choices of architecture (Goyal & Bengio,

    2026

  46. [54]

    and of the learning algorithm (optimizer, objective, regularizers Kuka ˇcka et al. 2017). We focus on the former. The simplicity bias has been studied from both aspects. Most explanations attribute it to loss functions (Pezeshki et al.,

    2017

  47. [55]

    and gradient descent (Arora et al., 2019; Hermann & Lampinen, 2020; Lyu et al., 2021; Tachet et al., 2018). But work on untrained networks shows that it can be explained with architectures alone (De Palma et al., 2019; Goldblum et al., 2023; Mingard et al., 2019; Teney et al., 2024; Valle-Perez et al., 2018). Teney et al. (2024) showed that the choice of ...

    2019

  48. [56]

    Simplicity bias in transformers.The hypothesis of a simplicity bias in NNs has also been studied specifically in transformers

    is a related but different effect related to training dynamics: NNs approximate low-frequency components of the target function earlier during training with SGD. Simplicity bias in transformers.The hypothesis of a simplicity bias in NNs has also been studied specifically in transformers. Hahn et al. (2021) shows that common models in NLP are biased to lea...

    2021

  49. [57]

    The research community has slowly converged towards smooth handcrafted variants of ReLUs such as GeLUs (Dubey et al., 2022; Hendrycks & Gimpel, 2016; Ramachandran et al., 2017)

    which may be just as important. The research community has slowly converged towards smooth handcrafted variants of ReLUs such as GeLUs (Dubey et al., 2022; Hendrycks & Gimpel, 2016; Ramachandran et al., 2017). Some works proposedlearning activation functionsusing extra parameters optimized alongside the weights of the network (Alexandridis et al., 2025; A...

    2022

  50. [58]

    Length generalizationrefers to the ability of a model to generalize to sequences longer than seen during training, especially for algorithmic tasks (e.g

    or from a narrow set of parametric functions (Alexandridis et al., 2025). Length generalizationrefers to the ability of a model to generalize to sequences longer than seen during training, especially for algorithmic tasks (e.g. arithmetic operations on numbers with more digits). This remains a challenge despite extensive work on positional encodings, whic...

    2025

  51. [60]

    and even responsible for failure cases such asshortcut learning(Geirhos et al., 2020; Puli et al., 2023; Teney et al.,

    2020

  52. [61]

    or the amplification of biases and performance disparities (Bell & Sagun, 2023). Even the underlying principle supporting the simplicity bias, known asOccam’s razor, has long been debated in the philosophical literature because it lacks a justification from first principles (Mingard et al., 2023, Appendix A). A prominent argument for simplicity is rooted ...

    2023

  53. [62]

    a bias in the distribution of target functions must be towards low complexity

    with results stating essentially that “a bias in the distribution of target functions must be towards low complexity”. However, this only means that simplicity is a good prior on average, but not necessarily the best choice for any task or dataset. Studies in linguistics and cognitive science have also examined the question of learnability. This includes ...

    2023

  54. [66]

    The best configuration is to optimize the MLP non- linearities while keeping the original softmax attention (second panels from the left). 128 256 512 1 2 4 -.02 -.02 -.04 -.05 -.05 -.11 -.06 -.09 -.16 Better Baseline Worse 128 256 512 1 2 4 -.02 -.02 -.04 -.03 -.05 -.14 -.05 -.10 -.25 Better Baseline Worse 128 256 512 1 2 4 +.03 +.02 -.02 +.02 +.02 -.03 ...

    2000

  55. [67]

    4.023.82 3.733.713.69 Training time (sec) Number of layers2 4 8 10 12 Linear1,440 1,920 2,940 3,540 19,680 GeLU1,440 1,9203,090 20,580 34,020 ReLU 1,500 1,980 3,120 13,080 28,020 Ours: linear spline 1,500 2,070 8,520 26,700 81,720 Ours: polynomial approx. (n= 18)1,4402,040 3,180 14,070 29,100 Number of layers2 4 6 8 10 12 Validation loss 3.63.73.83.944.14...

    2026

  56. [68]

    Note that this function was learned for theNanoGPT speedruncodebase which has unusually large internal activations, and is unlikely to directly work well with other architectures

    It uses Horner’s method with hard-coded coefficients and is compiled withTorchScriptfor efficiency. Note that this function was learned for theNanoGPT speedruncodebase which has unusually large internal activations, and is unlikely to directly work well with other architectures. Importance ofhigh degreepolynomials.We tried reducing the maximum degree of t...

    2026

  57. [1980]

    Meanwhile, neural networks have nevertheless proved widely successful

    and that no learning algorithm is universally useful, as per one of the no-free lunch theorems (Wolpert, 2002)). Meanwhile, neural networks have nevertheless proved widely successful. The broad applicability of transformers, in particular, suggests that their inductive bias has a broad relevance to real-world data (Goldblum et al., 2023). Thesimplicity bi...

    2002

  58. [2002]

    Frequency principle: Fourier analysis sheds light on deep neural networks.arXiv:1901.06523,

    Zhi-Qin John Xu, Yaoyu Zhang, Tao Luo, Yanyang Xiao, and Zheng Ma. Frequency principle: Fourier analysis sheds light on deep neural networks.arXiv:1901.06523,

    Pith/arXiv arXiv 1901

  59. [2016]

    What shapes feature representations? exploring datasets, architectures, and training.arXiv:2006.12433,

    Katherine L Hermann and Andrew K Lampinen. What shapes feature representations? exploring datasets, architectures, and training.arXiv:2006.12433,

    Pith/arXiv arXiv 2006

  60. [2017]

    Teaching arithmetic to small transformers.arXiv:2307.03381,

    Nayoung Lee, Kartik Sreenivasan, Jason D Lee, Kangwook Lee, and Dimitris Papailiopoulos. Teaching arithmetic to small transformers.arXiv:2307.03381,

    Pith/arXiv arXiv

  61. [2018]

    Kan: Kolmogorov-arnold networks.arXiv:2404.19756,

    Ziming Liu, Yixuan Wang, Sachin Vaidya, Fabian Ruehle, James Halverson, Marin Solja ˇci´c, Thomas Y Hou, and Max Tegmark. Kan: Kolmogorov-arnold networks.arXiv:2404.19756,

    Pith/arXiv arXiv

  62. [2019]

    To code, or not to code? exploring impact of code in pre-training.arXiv:2408.10914,

    Viraat Aryabumi, Yixuan Su, Raymond Ma, Adrien Morisot, Ivan Zhang, Acyr Locatelli, Marzieh Fadaee, Ahmet ¨Ust¨un, and Sara Hooker. To code, or not to code? exploring impact of code in pre-training.arXiv:2408.10914,

    Pith/arXiv arXiv

  63. [2020]

    Ex- trapolation by association: Length generalization transfer in transformers.arXiv:2506.09251,

    Ziyang Cai, Nayoung Lee, Avi Schwarzschild, Samet Oymak, and Dimitris Papailiopoulos. Ex- trapolation by association: Length generalization transfer in transformers.arXiv:2506.09251,

    Pith/arXiv arXiv

  64. [2021]

    Tropical attention: Neural algorith- mic reasoning for combinatorial algorithms.arXiv:2505.17190,

    Baran Hashemi, Kurt Pasque, Chris Teska, and Ruriko Yoshida. Tropical attention: Neural algorith- mic reasoning for combinatorial algorithms.arXiv:2505.17190,

    arXiv

  65. [2022]

    Learning activation functions: A new paradigm of understanding neural networks.arXiv:1906.09529,

    Mohit Goyal, Rajan Goyal, and Brejesh Lall. Learning activation functions: A new paradigm of understanding neural networks.arXiv:1906.09529,

    Pith/arXiv arXiv 1906

  66. [2023]

    Simplicity bias leads to amplified performance disparities

    Samuel James Bell and Levent Sagun. Simplicity bias leads to amplified performance disparities. InProceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency, pp. 355–369,

    2023

  67. [2024]

    Scaling MLPs: A tale of inductive bias.arXiv:2306.13575,

    Gregor Bachmann, Sotiris Anagnostidis, and Thomas Hofmann. Scaling MLPs: A tale of inductive bias.arXiv:2306.13575,

    Pith/arXiv arXiv

  68. [2025]

    Gaussian error linear units (GeLUs).arXiv:1606.08415,

    Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (GeLUs).arXiv:1606.08415,

    Pith/arXiv arXiv