Pith. sign in

REVIEW 2 major objections 5 minor 1 cited by

Minimalist Softmax Attention Provably Learns Constrained Boolean Functions

T0 review · 2 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read One softmax attention head, with no feed-forward layer, provably learns an unknown k-bit AND/OR in a single gradient step when teacher forcing exposes the relevant bits; without such hints, the paper claims, no efficient learner can…

desk verdict Teacher-forcing upper bound is a genuine KS25-style contribution; the lower bound as stated is false because constant predictors zero out the min-over-coordinates loss. read the letter →

arxiv 2505.19531 v1 pith:WC7H7GRP submitted 2025-05-26 cs.LG cs.AIstat.ML

Jerry Yao-Chieh Hu , Xiwen Zhang , Maojiang Su , Zhao Song , Han Liu This is my paper
classification cs.LGcs.AIstat.ML MSC 68Q3268T07
keywords softmaxattentionteacherforcingBooleanfunctionssupportrecoverysupervisiongapchain-of-thoughtsamplecomplexitygradientdescent
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper studies the simplest possible attention learner, a single softmax head with no feed-forward network, on the task of identifying which $k$ of $d$ input bits determine an unknown AND or OR function, where $k=\Theta(d)$. It claims that under teacher forcing, a training loss that reveals intermediate pairwise products of the relevant bits, one gradient step from zero initialization aligns the attention weights with the true support and drives the error to $O(d^{-\epsilon/8})$ using $n=\Omega(d^{\epsilon})$ samples. It further claims that without that hint, no differentiable end-to-end learner can recover the support even with $e^{\Theta(d)}$ samples, so the error stays bounded away from zero. If both halves hold, the paper shows that expressive capacity is not the bottleneck for this Boolean task; the supervision signal is what flips the problem from tractable to effectively impossible. The result matters as a minimal theoretical benchmark for when chain-of-thought-style hints are necessary.

What carries the argument

The load-bearing object is the reparametrized single-head softmax attention column map $\mathrm{Att}_W(X)=X\,\mathrm{softmax}(W)$ with $W\in\mathbb{R}^{d\times t}$, the value matrix fixed to the identity, and $t=k/2$. Teacher forcing reorganizes the target AND/OR into $t$ pairwise products, and the squared-error surrogate $L(W)=\frac{1}{2n}\|\mathrm{Att}_W(X)-E\|_F^2$ over those intermediates produces the gradient identity $\frac{\partial L}{\partial w_{j,m}}\big|_{W=0}=-\frac{1}{8d}\mathbf{1}\{p[j]=m\}+O(d^{-1-\epsilon/4})$, which is the mechanism that makes one step lock onto the support. The lower-bound machinery is the combinatorial averaging over $\mathcal{B}=\binom{[d]}{k}$: a Chernoff argument shows that with $n=e^{\Theta(d)}$ samples all labels are, with high probability, constant for AND, so the data leave only a fraction $e^{-\Theta(d)}$ of subsets consistent, and the loss is averaged against the $k/d$ versus $(d-k)/d$ split of coordinate entries.

What would settle it

Test Theorem 4.2 literally with the stated $\min_{j\in[d]}|(v_b-f)_j|$ metric: for any $b$ with $0<k<d$, the differentiable constant predictors $f\equiv 0$ and $f\equiv 1$ both give $\min_j|(v_b-f)_j|=0$ for every sample, so the claimed expectation $\min\{k/d,1-k/d\}$ is violated. The decisive check is to compare the theorem's metric with the fixed-coordinate expectation $\mathbb{E}[|(v_b-f)_j|]$ that the appendix actually bounds; only the latter can be nonzero uniformly over $b$.

Watch

Extended reading notes

Core claim

The paper's central discovery is a supervision gap inside a single softmax-attention layer, stripped of all other machinery. For an unknown $k$-bit AND/OR over $d$ inputs with $k=\Theta(d)$, the target is encoded by the indicator vector $v_b$ of the relevant subset. When the loss is replaced by a teacher-forced surrogate that compares the attention output against pairwise products $x_{d+i}=x_{j_{2i-1}}x_{j_{2i}}$ of the hidden relevant bits, the gradient at $W^{(0)}=0_{d\times t}$ has a sharp component $-1/(8d)$ on exactly the relevant pairs and negligible components elsewhere; one update with learning rate $\Theta(d^{1+\epsilon/8})$ makes the softmax put roughly half its mass on each relevant bit and $\exp(-\Theta(d))$ on everything else, yielding $\|2\cdot\mathrm{Softmax}(W^{(1)})\mathbf{1}_t - v_b\|_\infty = O(d^{-\epsilon/8})$ with high probability. The paper argues the converse: without the hint, any differentiable parameterized model given only input-output pairs must search over all $\binom{d}{k}$ subsets, and the stated lower bound asserts its expected minimum-coordinate error never falls below $\min\{k/d,1-k/d\}$ even with $e^{\Theta(d)}$ samples.

Load-bearing premise

The load-bearing premise of the hardness half is that error is measured by the minimum over output coordinates of the entrywise deviation from the support vector; under that metric the stated lower bound cannot hold for $0<k<d$, because a constant all-zero or all-one prediction attains zero minimum-coordinate error for every target, and the appendix's proof actually bounds a fixed-coordinate expectation instead.

Editorial extensions

If this is right

  • Under teacher forcing, a one-layer softmax attention head without any feed-forward network is provably sufficient for $k$-bit AND/OR with $k=\Theta(d)$; depth and FFNs are not needed for representational or optimization reasons.
  • One gradient update with $n=\Omega(d^{\epsilon})$ samples achieves error $O(d^{-\epsilon/8})$ on the support indicator, so an auxiliary loss that exposes intermediate products can replace multi-step chain-of-thought for this Boolean family.
  • If the lower-bound claim is taken as the paper states it, end-to-end training on raw input-output pairs cannot identify the relevant bits in polynomial time even with exponentially many examples, locating the bottleneck in the absence of intermediate supervision rather than in architecture.
  • The same one-step teacher-forced guarantee extends to $p$-noisy AND/OR functions, and a version of the upper bound applies to a local two-variable majority problem.
  • The pair of theorems gives a minimal testbed for curriculum and auxiliary-loss design: supplying the pairwise hint flips the task from intractable to one-step learnable.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The one-step success of AND/OR suggests a general recipe: any Boolean concept decomposable into low-degree intermediate products over an unknown support should be one-step learnable by the same teacher-forced gradient, with the concentration lemma determining which interaction orders survive.
  • If the hardness result is restated with per-coordinate or average error instead of the minimum-coordinate metric, the appendix's averaging argument still supports the intended dichotomy; what the proof genuinely shows is that no fixed coordinate can be pinned down by the data.
  • A minimal empirical check would train a single softmax head on $n=d^{\epsilon}$ planted-AND samples with an auxiliary loss on pairwise intermediates and inspect the attention weights after one step; the theorem predicts each relevant pair splits about half of the attention mass and all other positions vanish.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper studies whether a single-head softmax attention layer of the form X ↦ X softmax(W), W ∈ R^{d×t}, can learn an unknown k-bit AND/OR function over d Boolean inputs, with k = Θ(d). Theorem 4.1 claims that under teacher forcing — training on the intermediate products E = (x_{d+1}, …, x_{d+t}) — one gradient step from zero initialization with n = Ω(d^ε) samples recovers the support vector v_b up to sup-norm error O(d^{-ε/8}). Theorem 4.2 claims that under standard end-to-end training, any differentiable parameterized model fails even with n = e^{Θ(d)} samples, in the sense that E[min_j |(v_b - f_θ(x,y))_j|] ≥ min{k/d, 1-k/d} - e^{-Θ(d)}. Together these are advertised as a fundamental supervision gap for a minimalist attention architecture.

Significance. If Theorem 4.1 and a correctly stated hardness result could both be established, the paper would provide a clean and useful dichotomy: a constructive one-step teacher-forced upper bound with an explicit gradient calculation, and an information-theoretic lower bound for unassisted training. The proof of Theorem 4.1 is largely explicit and checkable, and the fixed-coordinate argument in Appendix F is a legitimate ambiguity-based lower-bound strategy. These are real strengths. However, the central negative theorem is false as stated, and both the positive and negative results are proved only for the AND variant, not for the OR variant that the paper also claims to cover. The advertised supervision gap is therefore not established in the form claimed.

major comments (2)
  1. [Theorem 4.2 and Appendix F] Theorem 4.2 is false as stated. For any b with 0 < k < d, the constant model f ≡ 0 satisfies min_j |(v_b - f(x,y))_j| = min_j |(v_b)_j| = 0, because every such v_b has both a zero and a one coordinate; hence the left-hand side of the claimed inequality is 0. The claimed right-hand side is min{k/d, 1-k/d} - e^{-Θ(d)}, which is Θ(1) for k = Θ(d). Thus the theorem's bound cannot hold for the model class it explicitly allows. The proof in Appendix F establishes a lower bound on E[|(v_b - f_θ)_j|] for a fixed coordinate j; since min_j z_j is never larger than any individual coordinate, a fixed-coordinate lower bound cannot be converted into the stated min_j lower bound. This invalidates the advertised hardness half of the supervision-gap claim as written.
  2. [Definition 3.2, Section 4.1, Appendix E and F] Both main results are proven only for the AND variant, not for the OR variant that Definition 3.2 and the abstract claim. The teacher-forcing matrix E in Theorem 4.1 and Appendix E is built from x_{d+i,l} = x_{j_{2i-1},l} · x_{j_{2i},l}, which decomposes the AND target ∏_{j∈b} x_j. The OR target 1 - ∏_{j∈b}(1-x_j) has no analogous product-of-input-columns decomposition, and the architecture X softmax(W) has no bias or constant column with which to represent 1-x_j. No separate construction or gradient analysis is supplied for OR. Appendix F is likewise AND-specific: its high-probability event is that all labels y_i are 0, which holds with probability about 1 - n/2^k only for AND; for OR, y_i = 0 has probability 2^{-k}. The claimed coverage of OR is therefore unsupported.
minor comments (5)
  1. [Appendix F, first paragraph of the proof] The phrase 'there is probability 1 - exp(Θ(d))' should read '1 - exp(-Θ(d))'; the stated high-probability event cannot hold with probability greater than 1.
  2. [Lemma D.2] Lemma D.2 is stated for max over r ∈ [2], but the proof and the use in Appendix E for triple interactions require r = 3; the union-bound display also contains an exponent mismatch, since Hoeffding with κ = 4√(log(d/p)/n) does not lead to the displayed factor e^{-4 log(d/p)}.
  3. [Appendix E, term A2 after Eq. (3)] The lower-order algebra for (1/(nd))A2 should give 1/(8d) + 1/(4d^2), not 1/(8d^2) + 1/(8d); the leading-order gradient is unaffected.
  4. [Theorem 4.1 proof, Step 2 and Eq. (5)] The non-relevant softmax entries are described as exp(-Θ(d)), but with η = Θ(d^{1+ε/8}) the weight gap is Θ(d^{ε/8}), so the exponential decay is exp(-Θ(d^{ε/8})); the final O(d^{-ε/8}) bound is unchanged.
  5. [Claim 4.3 and Section 1] The discussion calls the problem 'computationally intractable', but Appendix F establishes an information-theoretic sample-complexity lower bound for any function of the samples, with no run-time restriction; the computational phrasing should either be proved or replaced by a statement about sample complexity.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the teacher-forcing upper bound is an explicit strong-supervision construction and the lower-bound proof is self-contained; the min-over-coordinates flaw is a correctness gap, not a circular reduction.

full rationale

The paper's derivation chain does not reduce to its own inputs by construction or by an authors-only citation chain. In Theorem 4.1, the surrogate loss L(W) = (1/2n)||Att_W(X) - E||_F^2 is deliberately defined using teacher-forcing intermediate products E, where each column of E is x_{d+i} = x_{j_{2i-1}} x_{j_{2i}} for the true subset b. This is an explicitly declared external supervision signal, not a hidden reuse of the target v_b; the gradient computation is self-contained, and the paper's Section B openly acknowledges that 'the positive result requires teacher-forcing signals that expose the hidden subset.' The lower-bound proof in Appendix F is also self-contained: it shows that with exponentially many all-zero labels, many candidate subsets remain consistent, so any fixed output f must incur a per-coordinate error. The theorem statement's min_j loss is not what the proof establishes — the proof bounds E[|(v_b - f_{θ(A)})_j|] for a fixed j — and this is a genuine correctness gap (a constant predictor can make min_j |(v_b - f)_j| = 0), but that is not a circularity pattern. Self-citations such as [HSK+25], [CLL+24], [LLS+24], and [LLS+25] appear only in motivation or related-work contexts and are not load-bearing for Theorems 4.1 or 4.2; the reused technical lemma, Lemma D.1, is cited from the external ICLR 2025 work [KS25] and is applied transparently. No fitted parameter is renamed as a prediction, and no uniqueness theorem is imported from the authors' own prior work. Accordingly, the paper's central claims, whatever their other mathematical weaknesses, are not circular.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The upper bound rests on a very strong teacher-forcing assumption (the pairwise products of the true relevant bits) plus standard concentration tools; the lower bound rests on an uninformative-data argument whose metric (min over coordinates) is flawed. No new physical entities are introduced.

free parameters (4)
  • epsilon = arbitrary constant > 0
    Trades sample size n = Ω(d^ε) against error O(d^{-ε/8}) and learning rate η = Θ(d^{1+ε/8}); chosen by hand for the bound.
  • learning rate eta = Θ(d^{1+ε/8})
    Set to amplify the relevant-column gradient Θ(d^{-1}) into weight separation Θ(d^{ε/8}).
  • concentration threshold kappa = Θ(d^{-ε/4})
    Set through n = Ω(d^ε) and p = exp(-d^{ε/2}) via Lemma D.2; controls the O(d^{-1-ε/4}) gradient error.
  • noisy threshold phi = 0.5 d^{ε/8}
    Post-training threshold on W^{(1)} in Theorem G.2 to map weights to a support indicator; chosen ad hoc.
assumptions (5)
  • domain assumption Inputs are i.i.d. uniform over {0,1}^d and the hidden subset b is uniform over all k-subsets.
    Definition 3.2; used in the concentration lemma and lower-bound averaging.
  • domain assumption k = Θ(d) and even, with t = k/2 pairs; the teacher pairs the relevant bits arbitrarily.
    Theorem 4.1 setup; requires even k to form pairwise intermediate products.
  • ad hoc to paper The teacher provides intermediate products x_{j_{2i-1}} ⊙ x_{j_{2i}} for all i ∈ [t] (the matrix E).
    This is the teacher-forcing signal; it directly encodes the hidden subset b. Section B admits this supervision is seldom available in practice.
  • domain assumption An O(d^{-1-ε/4})-approximate gradient oracle is available for the one-step update.
    Used in Theorem 4.1 to allow gradient noise while maintaining separation.
  • domain assumption In Theorem 4.2, f_θ can be any differentiable parameterized model, with no constraint that its output be an indicator-like vector.
    This unconstrained output space allows constant predictors, which breaks the min_j lower bound as stated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Minimalist Softmax Attention Provably Learns Constrained Boolean Functions." pith.science (2026). https://pith.science/paper/WC7H7GRP

@misc{pith2026250519531,
  author       = {Pith},
  title        = {Pith review of: Minimalist Softmax Attention Provably Learns Constrained Boolean Functions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WC7H7GRP}},
  note         = {Machine review of arXiv:2505.19531}
}
abstract

We study the computational limits of learning $k$-bit Boolean functions (specifically, $\mathrm{AND}$, $\mathrm{OR}$, and their noisy variants), using a minimalist single-head softmax-attention mechanism, where $k=\Theta(d)$ relevant bits are selected from $d$ inputs. We show that these simple $\mathrm{AND}$ and $\mathrm{OR}$ functions are unsolvable with a single-head softmax-attention mechanism alone. However, with teacher forcing, the same minimalist attention is capable of solving them. These findings offer two key insights: Architecturally, solving these Boolean tasks requires only minimalist attention, without deep Transformer blocks or FFNs. Methodologically, one gradient descent update with supervision suffices and replaces the multi-step Chain-of-Thought (CoT) reasoning scheme of [Kim and Suzuki, ICLR 2025] for solving Boolean problems. Together, the bounds expose a fundamental gap between what this minimal architecture achieves under ideal supervision and what is provably impossible under standard training.

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Transformers with RL or SFT Provably Learn Sparse Boolean Functions, But Differently

    cs.LG 2025-11 conditional novelty 6.0 of 10

    Under hand-designed masks and task-specific activations, RL fine-tuning learns a k-sparse Boolean reasoning chain in one gradient update while SFT learns it one CoT step per update.

Reference graph

Works this paper leans on

28 extracted references · 6 canonical work pages · cited by 1 Pith paper

  1. [2]

    Fast rope attention: Combining the polynomial method and fast fourier transform.arXiv preprint arXiv:2505.11892,

    [AS25] Josh Alman and Zhao Song. Fast rope attention: Combining the polynomial method and fast fourier transform.arXiv preprint arXiv:2505.11892,

    arXiv

  2. [4]

    Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901,

    [BMR+20] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901,

    1901

  3. [6]

    Circuit Complexity Bounds for RoPE-based Transformer Architecture

    [CLL+24] Bo Chen, Xiaoyu Li, Yingyu Liang, Jiangxuan Long, Zhenmei Shi, and Zhao Song. Circuit complexity bounds for rope-based transformer architecture.arXiv preprint arXiv:2411.07602,

    work page Pith review arXiv

  4. [7]

    Provable Failure of Language Models in Learning Majority Boolean Logic via Gradient Descent

    [CSSZ25] Bo Chen, Zhenmei Shi, Zhao Song, and Jiahao Zhang. Provable failure of lan- guage models in learning majority boolean logic via gradient descent.arXiv preprint arXiv:2504.04702,

    work page Pith review arXiv

  5. [8]

    Bert: Pre- training of deep bidirectional transformers for language understanding.arXiv preprint arXiv:1810.04805,

    [DCLT18] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre- training of deep bidirectional transformers for language understanding.arXiv preprint arXiv:1810.04805,

    arXiv

  6. [10]

    T2vphysbench: A first-principles benchmark for physical consistency in text-to-video generation.arXiv preprint arXiv:2505.00337,

    [GHS+25a] Xuyang Guo, Jiayan Huo, Zhenmei Shi, Zhao Song, Jiahao Zhang, and Jiale Zhao. T2vphysbench: A first-principles benchmark for physical consistency in text-to-video generation.arXiv preprint arXiv:2505.00337,

    arXiv

  7. [11]

    Subquadratic Algorithms and Hardness for Attention with Any Temperature

    [GHS+25b] Shreya Gupta, Boyang Huang, Barna Saha, Yinzhan Xu, and Christopher Ye. Sub- quadratic algorithms and hardness for attention with any temperature. InarXiv preprint arXiv:2505.14840,

    work page Pith review arXiv

  8. [12]

    In-context convergence of transformers

    [HCL23] Yu Huang, Yuan Cheng, and Yingbin Liang. In-context convergence of transformers. arXiv preprint arXiv:2310.05249,

    arXiv

Show all 28 references
  1. [16]

    Transformers learn nonlinear features in context: Noncon- vex mean-field dynamics on the attention landscape.arXiv preprint arXiv:2402.01258,

    [KS24b] Juno Kim and Taiji Suzuki. Transformers learn nonlinear features in context: Noncon- vex mean-field dynamics on the attention landscape.arXiv preprint arXiv:2402.01258,

    arXiv

  2. [17]

    Theoretical constraints on the expressive power of rope-based tensor attention transformers.arXiv preprint arXiv:2412.18040,

    [LLS+24] Xiaoyu Li, Yingyu Liang, Zhenmei Shi, Zhao Song, and Mingda Wan. Theoretical constraints on the expressive power of rope-based tensor attention transformers.arXiv preprint arXiv:2412.18040,

    arXiv

  3. [18]

    On the computational capability of graph neural networks: A circuit complexity bound perspective.arXiv preprint arXiv:2501.06444,

    29 [LLS+25] Xiaoyu Li, Yingyu Liang, Zhenmei Shi, Zhao Song, Wei Wang, and Jiahao Zhang. On the computational capability of graph neural networks: A circuit complexity bound perspective.arXiv preprint arXiv:2501.06444,

    arXiv

  4. [19]

    One step of gradient descent is provably the optimal in-context learner with one layer of linear self-attention

    [MHM23] Arvind Mahankali, Tatsunori B Hashimoto, and Tengyu Ma. One step of gradient descent is provably the optimal in-context learner with one layer of linear self-attention. arXiv preprint arXiv:2307.03576,

    arXiv

  5. [20]

    Evalu- ating numeracy of language models as a natural language inference task

    [MSCV25] Rahmad Mahendra, Damiano Spina, Lawrence Cavedon, and Karin Verspoor. Evalu- ating numeracy of language models as a natural language inference task. InFindings of the Association for Computational Linguistics: NAACL 2025, pages 8336–8361,

    2025

  6. [22]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288,

    [TMS+23] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yas- mine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhos- ale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288,

    arXiv

  7. [23]

    Simplicity bias of transformers to learn low sensitivity functions.arXiv preprint arXiv:2403.06925,

    [VFZ+24] Bhavya Vasudeva, Deqing Fu, Tianyi Zhou, Elliott Kau, Youqi Huang, and Vatsal Sha- ran. Simplicity bias of transformers to learn low sensitivity functions.arXiv preprint arXiv:2403.06925,

    arXiv

  8. [24]

    Sub-task decomposition enables learn- ing in sequence to sequence tasks.arXiv preprint arXiv:2204.02892,

    30 [WLS22] Noam Wies, Yoav Levine, and Amnon Shashua. Sub-task decomposition enables learn- ing in sequence to sequence tasks.arXiv preprint arXiv:2204.02892,

    arXiv

  9. [25]

    From sparse de- pendence to sparse attention: unveiling how chain-of-thought enhances transformer sample efficiency.arXiv preprint arXiv:2410.05459,

    [WZLZ24] Kaiyue Wen, Huaqing Zhang, Hongzhou Lin, and Jingzhao Zhang. From sparse de- pendence to sparse attention: unveiling how chain-of-thought enhances transformer sample efficiency.arXiv preprint arXiv:2410.05459,

    arXiv

  10. [26]

    Dnabert-2: Efficient foundation model and benchmark for multi-species genome.arXiv preprint arXiv:2306.15006,

    [ZJL+23] Zhihan Zhou, Yanrong Ji, Weijian Li, Pratik Dutta, Ramana Davuluri, and Han Liu. Dnabert-2: Efficient foundation model and benchmark for multi-species genome.arXiv preprint arXiv:2306.15006,

    arXiv

  11. [27]

    Genomeocean: An efficient genome foundation model trained on large-scale metagenomic assemblies

    [ZRK+25] Zhihan Zhou, Robert Riley, Satria Kautsar, Weimin Wu, Rob Egan, Steven Hofmeyr, Shira Goldhaber-Gordon, Mutian Yu, Harrison Ho, Fengchen Liu, et al. Genomeocean: An efficient genome foundation model trained on large-scale metagenomic assemblies. bioRxiv, pages 2025–01,

    2025

  12. [28]

    Dnabert-s: Learning species-aware dna embedding with genome foundation models.arXiv preprint arXiv:2402.08777,

    [ZWH+24] Zhihan Zhou, Winmin Wu, Harrison Ho, Jiayi Wang, Lizhen Shi, Ramana V Davuluri, Zhong Wang, and Han Liu. Dnabert-s: Learning species-aware dna embedding with genome foundation models.arXiv preprint arXiv:2402.08777,

    arXiv

  13. [1867]

    Transformers in uniform tc0.arXiv preprint arXiv:2409.13629,

    [Chi24] David Chiang. Transformers in uniform tc0.arXiv preprint arXiv:2409.13629,

    arXiv

  14. [1963]

    Why are sensitive functions hard for transformers? arXiv preprint arXiv:2402.09963,

    [HR24] Michael Hahn and Mark Rofin. Why are sensitive functions hard for transformers? arXiv preprint arXiv:2402.09963,

    arXiv

  15. [1993]

    Can you count to nine? a human evaluation benchmark for counting limits in modern text-to-video models.arXiv preprint arXiv:2504.04051,

    [GHH+25] Xuyang Guo, Zekai Huang, Jiayan Huo, Yingyu Liang, Zhenmei Shi, Zhao Song, and Jiahao Zhang. Can you count to nine? a human evaluation benchmark for counting limits in modern text-to-video models.arXiv preprint arXiv:2504.04051,

    arXiv

  16. [2021]

    Are transformers with one layer self-attention using low-rank weight matrices universal approximators?arXiv preprint arXiv:2307.14023,

    [KS23] Tokio Kajitsuka and Issei Sato. Are transformers with one layer self-attention using low-rank weight matrices universal approximators?arXiv preprint arXiv:2307.14023,

    arXiv

  17. [2022]

    Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971,

    [TLI+23] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ ee Lacroix, Baptiste Rozi` ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971,

    arXiv

  18. [2023]

    Optimal memorization capacity of transformers.arXiv preprint arXiv:2409.17677,

    [KS24a] Tokio Kajitsuka and Issei Sato. Optimal memorization capacity of transformers.arXiv preprint arXiv:2409.17677,

    arXiv

  19. [2024]

    How to capture higher-order correlations? generaliz- ing matrix softmax attention to kronecker computation

    [AS24b] Josh Alman and Zhao Song. How to capture higher-order correlations? generaliz- ing matrix softmax attention to kronecker computation. InICLR. arXiv preprint arXiv:2310.04064,

    arXiv

  20. [2025]

    Sparks of artificial general intelligence: Early experiments with gpt-4.arXiv preprint arXiv:2303.12712,

    [BCE+24] S´ ebastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4.arXiv preprint arXiv:2303.12712,

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.