Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

Solving Empirical Bayes via Transformers

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A small transformer can solve Poisson empirical Bayes by adapting to an unseen prior in context, the paper claims.

desk verdict Real result, overstated abstract: the paper's own tables show T24r losing to NPMLE on BookCorpusOpen and on in-family neural priors at large n, so the blanket 'outperforms NPMLE' claim needs qualification before this is citable as claimed. read the letter →

arxiv 2502.09844 v2 pith:IS3K73OG submitted 2025-02-14 cs.LG stat.ML

Anzo Teh , Mark Jabbour , Yury Polyanskiy This is my paper · ORCID
classification cs.LGstat.ML MSC 62C1268T07
keywords empiricalBayesPoissonmeanstransformersin-contextlearningnonparametricmaximumlikelihoodregretlengthgeneralizationamortizedinference
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a small transformer, trained only on synthetic data, can solve the empirical-Bayes problem of estimating many Poisson means from a single batch of counts. It reports that this frozen model beats the classical nonparametric maximum-likelihood estimator (NPMLE) on validation error while running about 100 times faster. It also proves an existence result: a sufficiently wide transformer can achieve arbitrarily small regret relative to an oracle that knows the unknown prior, once the batch is large enough. If the practical claim holds, cheap amortized empirical-Bayes inference on count data becomes available without retraining per dataset.

What carries the argument

At the statistical core is the Bayes identity $\hat\theta_\pi(x)=(x+1)f_\pi(x+1)/f_\pi(x)$, which expresses the optimal estimate as a ratio of the mixture density at $x+1$ and $x$. The paper's machinery is an encoder-only transformer without positional encoding, trained with MSE loss on synthetic batches where each batch's $\theta_i$ come from a single prior, drawn from a mixture of two prior families: neural-network-transported uniform priors and a Dirichlet process. Because the Bayes estimator is permutation-invariant, the architecture lets full attention pool the entire batch of counts to infer the latent prior, and the randomized scale $\theta_{\max}$ at training time is meant to make the estimator robust to unknown magnitude. The theoretical machinery is a universal-approximation argument: the paper constructs attention weights that implement the clipped Robbins estimator and shows that NPMLE can also be approximated, yielding Corollary 4.3 that a wide transformer has minimax regret below any $\epsilon$ once $n$ is large enough.

What would settle it

Take a prior family that is deliberately far from the neural and Dirichlet training distributions, for example a mixture of a sharp atom at 0 and a heavy tail above 500, generate Poisson counts from it, and check whether the frozen transformer's average regret remains below NPMLE's as the batch size grows; if NPMLE wins at large $n$, the zero-shot generalization claim is falsified.

Watch

Extended reading notes

Core claim

In Poisson empirical Bayes, one observes $X_i \sim \mathrm{Poisson}(\theta_i)$ with $\theta_i$ drawn i.i.d. from an unknown prior $\pi$, and the goal is to estimate $\theta_i$ with low mean-squared error. The paper's central discovery is that a transformer pre-trained on synthetic $(X,\theta)$ pairs performs in-context empirical Bayes: given a new sequence of counts, it adapts to the unseen prior and produces estimates close to the Bayes oracle, without any fine-tuning or labels at test time. Concretely, the 24-layer model T24r, with about 100k parameters, achieves lower average regret than NPMLE on worst-case and multinomial priors and is about 100x faster, while a linear-attention variant L24r is competitive and even faster. The paper argues via linear probes that the model's internal representation is not merely a Robbins ratio or an NPMLE density estimate, but something closer to the Bayes posterior mean itself.

Load-bearing premise

The load-bearing premise is that real-world priors over Poisson means resemble the synthetic prior family used in pretraining, so that the frozen transformer's zero-shot in-context adaptation continues to work when the prior changes.

Editorial extensions

If this is right

  • If the central claim is correct, a single frozen small transformer can serve as a plug-in empirical-Bayes estimator for count data, avoiding per-dataset optimization and offering a roughly 100-fold speedup over NPMLE.
  • The observed length generalization implies that the transformer can be trained on one sequence length and deployed on longer sequences, with regret continuing to decrease up to four times the training length.
  • The linear-attention variant shows that a cheaper architecture retains most of the accuracy gain, suggesting that the computational advantage can be pushed further on long sequences.
  • The theoretical result implies that transformers are expressive enough to match the oracle Bayes estimator to arbitrary precision at large sample sizes, so the practical gains are not fundamentally blocked by architecture capacity.
  • On real sports and word-frequency datasets, the transformers generally improve over MLE and often over NPMLE in RMSE, with the exceptions and variability reported in the paper's tables and plots.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable consequence not pursued in the paper: the same pretraining recipe could be applied to the normal-means empirical-Bayes problem, and the comparison to NPMLE would reveal whether the advantage is specific to Poisson counts or generalizes across likelihoods.
  • The sensitivity to the training prior family suggests that practical gains may shrink on priors far from the training mixture; a benchmark suite of adversarial priors would measure that frontier.
  • The linear-probe finding that the transformer does not track Robbins or NPMLE features hints that amortized estimators may implement a new algorithmic primitive, which could guide the design of hand-crafted estimators.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes to solve Poisson empirical Bayes (EB) mean estimation with a small encoder-only transformer pre-trained on synthetic data. The model observes an exchangeable sequence of Poisson counts and outputs estimates of the underlying means, with no exposure to labeled data at inference time. The authors provide expressibility theorems (transformers can approximate clipped Robbins and NPMLE estimators), a minimax-regret corollary for sufficiently wide transformers, synthetic experiments on neural, Dirichlet, worst-case, and multinomial priors, and real-data experiments on NHL, MLB, and BookCorpusOpen datasets, together with linear-probe analyses of the learned representations. The central practical claim is that a frozen, pre-trained transformer 'outperforms NPMLE both in runtime and validation loss', with the runtime advantage quantified as roughly 100x.

Significance. If the stated claim were fully supported, this would be a significant demonstration that a single frozen transformer can serve as a fast, general-purpose EB estimator on new count datasets, with practical value given NPMLE's computational cost. The paper's strengths include a clean problem formulation, reproducible code, explicit acknowledgment of limitations, and a genuinely interesting runtime advantage for the transformer (Fig. 3). The theoretical expressibility results, while existence-style, are a useful first step. However, the headline practical claim is contradicted by the paper's own controlled comparisons: T24r loses to NPMLE on BookCorpusOpen (Table 1) and on in-family neural priors at large n (Fig. 2c and Tables 4–5). The contribution therefore needs to be reframed as a qualified, regime-dependent advantage rather than a blanket improvement over NPMLE.

major comments (4)
  1. [Abstract and Section 1] The abstract and Section 1 claim that small transformers 'outperform the best classical algorithm (NPMLE) both in runtime and validation loss.' The runtime part is credible, but the validation-loss part is contradicted by the paper's own results: Table 1 shows that on BookCorpusOpen the RMSE improvement over MLE is 9.43 ± 0.12% for T24r versus 10.82 ± 0.11% for NPMLE (and Table 7 reports T24r significantly worse with p > 1 − 1e-100). The claim should be qualified, for example as 'often outperforms NPMLE in validation loss, with documented exceptions on neural priors at large n and on BookCorpusOpen.'
  2. [Section 5.1, Fig. 2c and Tables 4–5] The synthetic experiments on neural prior-on-priors, which are inside the training family, show NPMLE outperforming both transformer variants at long sequence lengths. The caption to Fig. 2c states 'NPMLE generalizes better at longer sequence lengths' and reports that at n = 4096 NPMLE beats the best transformer (regret 0.104 vs 0.153). Table 4 reports P[Regret(T24r) > Regret(NPMLE)] > 1 − 1e-100 at Neural-2048, and Table 5 reports the same for L24r at Neural-1024 and Neural-2048. This is not a minor exception: it shows the claimed advantage is not only OOD-dependent but also fails on an in-family prior as n grows. The paper should explicitly discuss this regime-dependence and adjust the 'outperform NPMLE' narrative.
  3. [Corollary 4.3 and its proof (Appendix B.1)] Corollary 4.3 asserts the existence of a single transformer Γ with O(ϵ−1) parameters such that, for all n ≥ N, the worst-case regret over π ∈ P([0, θmax]) is at most ϵ. The proof invokes Theorem 4.2 to approximate NPMLE (or Robbins) uniformly, but Theorem 4.2 is stated for a fixed clipping level d and does not explicitly address whether the same Γ achieves the stated uniform approximation error simultaneously for every sequence length n ≥ N. The proof also moves between 'the Robbins estimator' and 'NPMLE' without specifying which estimator Γ approximates in the regret decomposition. Please clarify that the same construction works for all n in the claimed range, and state precisely which classical estimator is being approximated.
  4. [Section 3.3 and Appendix A.2] The training protocol mixes neural priors with a Dirichlet process prior, and the appendix states: 'Note that Dirichlet process implies that our data is not generated i.i.d. for each batch, so the Bayes estimator has to be estimated differently. We omit the calculation of this Bayes estimator.' This is a missing piece of the training description: for DP batches, the posterior mean E[θ_i|X] is not the standard i.i.d. EB estimator in Eq. (1), and the regret definition in Section 1 assumes i.i.d. coordinates. The paper should explain why training on DP batches with the plain MSE target still serves the stated goal of learning an i.i.d. EB estimator, or acknowledge that the training objective mixes two different targets.
minor comments (5)
  1. [Section 1] Typo: 'disecting' should be 'dissecting.'
  2. [Section 3.1] The sentence 'Given that π is unknown, an estimatorπ can only instead approximate ˆθπ' contains a typo and is unclear; it should read something like 'an estimator can only approximate ˆθπ.'
  3. [Tables 4, 5, 7, 8, 11, 12] The p-value notation 'P[Regret(T24r) > Regret(Classical)]' is inverted relative to the usual presentation: a p-value near 1 means the transformer is worse. Because the tables report values like '> 1 - 1e-100', it is easy to misread them as strong evidence of improvement. Please state the direction explicitly in the table captions or use a two-sided test.
  4. [Section 5.1, Fig. 2c] The text says 'At n = 4096 (not shown) NPMLE beats the best performing T24f (regret 0.104 vs 0.153)' but the figure and surrounding text only show data up to n = 2048; this off-plot result is important enough to be displayed or at least included in a table.
  5. [Appendix A.2] In the Dirichlet process description, the symbol α is reused for the concentration parameter and for the scaling in θ = θbase · θmax; the notation is understandable in context but should be disambiguated for clarity.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: held-out evaluation and independent minimax bounds; abstract overclaim is a correctness issue, not circularity.

full rationale

The derivation chain is not circular. Empirically, the transformer is trained on synthetic (X, theta) pairs, model selection is done on a validation split of the training prior families, and the reported results are on held-out synthetic priors (worst-case, multinomial), held-out draws from neural priors, and real datasets; no parameter is fitted to the test data, so the claimed predictions are not forced by construction. The theoretical existence result (Corollary 4.3) is supported by in-paper approximation theorems (Theorems 4.1 and 4.2) plus published minimax regret bounds for Robbins and NPMLE from [PW21] and [JPW22]. Those bounds concern classical estimators and do not assume the transformer conclusion; they are external mathematical facts, even though the authors overlap with the present paper. The use of [JPW22, Theorem 1] inside the proof of Theorem 4.2 is a technical uniqueness/continuity lemma about NPMLE, not an assumption that transformers already achieve the target regret. The abstract's blanket statement that transformers outperform NPMLE in validation loss is contradicted by the paper's own benchmarks (Fig. 2c and Table 7 show T24r losing on neural priors at large n and on BookCorpusOpen), but that is an overclaim or qualification problem, not a circular reduction: no equation or fitted parameter is identical to the predicted quantity. The linear-probe discussion is interpretive and does not define the transformer's output in terms of the probed features.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The final estimator has no free parameters fitted to test data. The training procedure uses several hand-chosen hyperparameters that shape the prior family, and these are listed as free parameters. The theoretical results rely on standard minimax bounds and universal approximation theorems from prior work, which are listed as axioms. No new entities are introduced.

free parameters (4)
  • Dirichlet process concentration alpha = 50
    Chosen by hand for training data generation (Appendix A.2); controls how clustered the theta samples are and shapes the training prior family.
  • theta_max sampling mixture weights = 3/4 Unif([0,200]) + 1/8 Exp(50) + 1/8 Cauchy(50,10), capped at 500
    Hand-chosen training distribution to make the transformer robust to unknown scale (Section 3.3).
  • Prior-class mixture weight = 1/2 each class
    Each batch draws theta from one of the two prior classes with probability 1/2 (Appendix A.2).
  • Training hyperparameters (LR, decay, epochs) = 0.02, 0.9 per 300 epochs, 50k epochs
    Standard Adam settings; not fitted to test data but part of the training recipe.
assumptions (5)
  • domain assumption X_i ~ Poi(theta_i) with theta_i iid from unknown prior pi
    The entire EB task is defined by this model (Section 1).
  • domain assumption Real-data counts follow X ~ Poi(n_X theta), Y ~ Poi(n_Y theta) with a shared theta per item
    Used to transfer synthetic-trained estimators to sports and text datasets (Section 6).
  • domain assumption Prior support is contained in [0, theta_max] with theta_max fixed at evaluation
    The minimax regret bound in Corollary 4.3 and the worst-case prior construction require compact support (Appendix A.1).
  • standard math Universal approximation of transformers [FdHP24, Theorem 1] and uniqueness of NPMLE solution [JPW22, Theorem 1]
    Theorem 4.2 depends on these external results for existence of a transformer approximating NPMLE (Appendix B.1).
  • standard math Minimax regret bounds for Robbins and NPMLE from [PW21, Theorem 2] and [JPW22, Theorem 3]
    Corollary 4.3 combines these bounds with transformer approximation to get vanishing regret (Appendix B.1).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Solving Empirical Bayes via Transformers." pith.science (2026). https://pith.science/paper/IS3K73OG

@misc{pith2026250209844,
  author       = {Pith},
  title        = {Pith review of: Solving Empirical Bayes via Transformers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IS3K73OG}},
  note         = {Machine review of arXiv:2502.09844}
}
abstract

This work applies modern AI tools (transformers) to solving one of the oldest statistical problems: Poisson means under empirical Bayes (Poisson-EB) setting. In Poisson-EB a high-dimensional mean vector $\theta$ (with iid coordinates sampled from an unknown prior $\pi$) is estimated on the basis of $X=\mathrm{Poisson}(\theta)$. A transformer model is pre-trained on a set of synthetically generated pairs $(X,\theta)$ and learns to do in-context learning (ICL) by adapting to unknown $\pi$. Theoretically, we show that a sufficiently wide transformer can achieve vanishing regret with respect to an oracle estimator who knows $\pi$ as dimension grows to infinity. Practically, we discover that already very small models (100k parameters) are able to outperform the best classical algorithm (non-parametric maximum likelihood, or NPMLE) both in runtime and validation loss, which we compute on out-of-distribution synthetic data as well as real-world datasets (NHL hockey, MLB baseball, BookCorpusOpen). Finally, by using linear probes, we confirm that the transformer's EB estimator appears to internally work differently from either NPMLE or Robbins' estimators.

Figures

Figures reproduced from arXiv: 2502.09844 by the authors.

Figure 1
Figure 1. (a), (b): R2 score of linear probe result against N(x), fπˆ(x) and x for T24r. We see that while x itself is easily recoverable from any layer, “knowledge” about the former two quantities appears to either decrease (in (a)) or plateau (in (b)) with depth. (c) In the multinomial prior case, T24r does not seem to use any information on the atom weight PMFπ(θ). 5 Synthetic experiments We now evaluate our trained transf… view at source ↗
Figure 2
Figure 2. Average regret of NPMLE vs transformers on various priors in [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Average time (in seconds) per batch vs sequence length, showing that the inference time of T24r is comparable with that of ERM monotone, and 100x faster than NPMLE. Also shown is L24r which scales better at n = 2048. The inference time of T24r in terms of sequence length n, number of layers L, and embedding dimension is O(Ln2d). In contrast, the time complexity of NPMLE (the most competitive classical algorithm) is … view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Violin plots of RMSE ratio of ERM-monotone (blue), NPMLE (orange), T24r (green), and L24r [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Discussion on Worst Prior θbase is then produced by sampling from a mixture of 4 priors in M. Dirichlet process. Let the base distribution be defined as H0 ≜ Unif([0, 1]). Within each batch, elements θbase,1, · · · , θbase,n are generated as follows: θbase,j = ( θbase,…
Figure 6
Figure 6. Figure 6: Violin plots of MAE ratio of ERM-monotone (blue), NPMLE (orange), T24r (green) and L24r [PITH_FULL_IMAGE:figures/full_fig_p028_6.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Universal priors: solving empirical Bayes via Bayesian inference and pretraining

    stat.ML 2026-02 conditional novelty 8.0 of 10

    A simple random prior-on-prior lets pretrained transformers achieve near-optimal empirical Bayes regret uniformly over all test priors, and length generalization matches α-posterior inference.

  2. Merging of Bayes and quasi-Bayes empirical Bayes procedures for Poisson compound decisions

    stat.ME 2026-07 unverdicted novelty 6.5 of 10

    Proves frequentist merging of Bayesian (Dirichlet process) and quasi-Bayesian (Newton's algorithm) empirical Bayes estimators for Poisson compound decisions via concentration rates on marginal PMFs and excess risks, w...

Reference graph

Works this paper leans on

36 extracted references · 6 canonical work pages · cited by 2 Pith papers

  1. [1]

    Define: WQ = Idk, WV = diag(1, 1, · · ·, 1, 0), WK

    Define: D = max{100, d2 k}. Define: WQ = Idk, WV = diag(1, 1, · · ·, 1, 0), WK. for i = 1 to d + 1do for j = 1 to d + 1do if i = j then Wk[i, j] = D else if j = i + 1then Wk[i, j] = D + √ d + 1 logi else Wk[i, j] = 0 end if end for end for Define: AttnLayer = Attn(WQ, WK, WV). Define: Z = AttnLayer(Y, Y, Y). Z ′ = ReLU(Y + Z − 1). Z1 = rowsum(Z ′). return...

    work page 2000

  2. [2]

    What learning algorithm is in-context learning? Investigations with linear models.arXiv preprint arXiv:2211.15661,

    [ASA+22] Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algorithm is in-context learning? Investigations with linear models.arXiv preprint arXiv:2211.15661,

    arXiv

  3. [4]

    (except in a handful of cases)

    From thep-value we conclude that the transformers outperform other baselines by a significant margin on various experiments. (except in a handful of cases). 23 Table 3: Plackett-Luce coefficients of estimators’ regrets on synthetic experiments. The coefficient of MLE is set to 0 throughout. Experiments GS Robbins ERM NPMLE T24r L24r Neural-128 -0.004 -3.3...

    work page

  4. [8]

    [ETST01] Bradley Efron, Robert Tibshirani, John D Storey, and Virginia Tusher

    Accessed: 2024-10-25. [ETST01] Bradley Efron, Robert Tibshirani, John D Storey, and Virginia Tusher. Empirical bayes analysis of a microarray experiment.Journal of the American statistical association, 96(456):1151–1160,

    work page 2024

  5. [10]

    Mamba: Linear-time sequence modeling with selective state spaces

    [GD23] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752,

    arXiv

  6. [12]

    How do transformers learn in-context beyond simple functions? A case study on learning with representations

    12 [GHM+23] Tianyu Guo, Wei Hu, Song Mei, Huan Wang, Caiming Xiong, Silvio Savarese, and Yu Bai. How do transformers learn in-context beyond simple functions? A case study on learning with representations. arXiv preprint arXiv:2310.10616,

    arXiv

  7. [14]

    Optimal empirical Bayes estimation for the Poisson model via minimum-distance methods.arXiv preprint arXiv:2209.01328,

    [JPW22] Soham Jana, Yury Polyanskiy, and Yihong Wu. Optimal empirical Bayes estimation for the Poisson model via minimum-distance methods.arXiv preprint arXiv:2209.01328,

    arXiv

  8. [16]

    Adam: A method for stochastic optimization

    [Kin14] Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980,

    arXiv

Show all 36 references
  1. [18]

    A mean field approach to empirical Bayes estimation in high-dimensional linear regression.arXiv preprint arXiv:2309.16843,

    [MSS23] Sumit Mukherjee, Bodhisattva Sen, and Subhabrata Sen. A mean field approach to empirical Bayes estimation in high-dimensional linear regression.arXiv preprint arXiv:2309.16843,

    arXiv

  2. [19]

    Transformers are expressive, but are they expressive enough for regression?arXiv preprint arXiv:2402.15478,

    [NKB24] Swaroop Nath, Harshad Khadilkar, and Pushpak Bhattacharyya. Transformers are expressive, but are they expressive enough for regression?arXiv preprint arXiv:2402.15478,

    arXiv

  3. [20]

    In-context learning through the bayesian prism

    [PAG23] Madhur Panwar, Kabir Ahuja, and Navin Goyal. In-context learning through the bayesian prism. arXiv preprint arXiv:2306.04891,

    arXiv

  4. [21]

    Nonparametric MLE for gaussian location mixtures: certified computation and generic behavior.arXiv preprint arXiv:2503.20193,

    [PS25] Yury Polyanskiy and Mark Sellke. Nonparametric MLE for gaussian location mixtures: certified computation and generic behavior.arXiv preprint arXiv:2503.20193,

    arXiv

  5. [22]

    Sharp regret bounds for empirical bayes and compound decision problems

    [PW21] Yury Polyanskiy and Yihong Wu. Sharp regret bounds for empirical bayes and compound decision problems. arXiv preprint arXiv:2109.03943,

    arXiv

  6. [23]

    14 [Rob51] Herbert Robbins

    Accessed: 2024-10-25. 14 [Rob51] Herbert Robbins. Asymptotically subminimax solutions of compound statistical decision prob- lems. InProceedings of the second Berkeley symposium on mathematical statistics and probability, pages 131–149. University of California Press,

    2024

  7. [26]

    Length generalization of causal transformers without position encoding.arXiv preprint arXiv:2404.12224,

    [WJW+24] Jie Wang, Tao Ji, Yuanbin Wu, Hang Yan, Tao Gui, Qi Zhang, Xuanjing Huang, and Xiaoling Wang. Length generalization of causal transformers without position encoding.arXiv preprint arXiv:2404.12224,

    arXiv

  8. [27]

    An explanation of in-context learning as implicit bayesian inference.arXiv preprint arXiv:2111.02080,

    [XRLM21] Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference.arXiv preprint arXiv:2111.02080,

    arXiv

  9. [29]

    Gated delta networks: Improving mamba2 with delta rule.arXiv preprint arXiv:2412.06464,

    [YKH24] Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated delta networks: Improving mamba2 with delta rule.arXiv preprint arXiv:2412.06464,

    arXiv

  10. [30]

    Gated linear attention transformers with hardware-efficient training.arXiv preprint arXiv:2312.06635,

    [YWS+23] Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with hardware-efficient training.arXiv preprint arXiv:2312.06635,

    arXiv

  11. [31]

    Parallelizing linear transformers with the delta rule over sequence length.arXiv preprint arXiv:2406.06484,

    [YWZ+24] Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen, and Yoon Kim. Parallelizing linear transformers with the delta rule over sequence length.arXiv preprint arXiv:2406.06484,

    arXiv

  12. [32]

    Transformers can achieve length generalization but not robustly.arXiv preprint arXiv:2402.09371,

    [ZAC+24] Yongchao Zhou, Uri Alon, Xinyun Chen, Xuezhi Wang, Rishabh Agarwal, and Denny Zhou. Transformers can achieve length generalization but not robustly.arXiv preprint arXiv:2402.09371,

    arXiv

  13. [33]

    Trained transformers learn linear models in-context

    [ZFB23] Ruiqi Zhang, Spencer Frei, and Peter L Bartlett. Trained transformers learn linear models in-context. arXiv preprint arXiv:2306.09927,

    arXiv

  14. [34]

    Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books.arXiv preprint arXiv:1506.06724,

    15 [ZKZ+15] Yukun Zhu, Ryan Kiros, Richard Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books.arXiv preprint arXiv:1506.06724,

    arXiv

  15. [1943]

    de Hoop, and Gabriel Peyré

    [FdHP24] Takashi Furuya, Maarten V. de Hoop, and Gabriel Peyré. Transformers are Universal In-context Learners. arXiv preprint arXiv:2408.01367,

    arXiv

  16. [1956]

    BERT rediscovers the classical NLP pipeline

    [TDP19] Ian Tenney, Das Dipanjan, and Ellie Pavlick. BERT rediscovers the classical NLP pipeline. arXiv preprint arXiv:1905.05950,

    1905 arXiv

  17. [1996]

    Bayes’ power for explaining in-context learning generalizations

    [MHH24] Samuel Müller, Noah Hollmann, and Frank Hutter. Bayes’ power for explaining in-context learning generalizations. arXiv preprint arXiv:2410.01565,

    arXiv

  18. [2000]

    [HS83] JC van Houwelingen and Th Stijnen

    Accessed: 2024-09-30. [HS83] JC van Houwelingen and Th Stijnen. Monotone empirical bayes estimators for the continuous one-parameter exponential family.Statistica Neerlandica, 37(1):29–43,

    2024

  19. [2009]

    Empirical bayes for the reluctant frequentist.arXiv preprint arXiv:2404.03422,

    [KG24] Roger Koenker and Jiaying Gu. Empirical bayes for the reluctant frequentist.arXiv preprint arXiv:2404.03422,

    arXiv

  20. [2013]

    Language models are few-shot learners

    11 [BMR+20] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901,

    1901

  21. [2018]

    arXiv preprint arXiv:1610.01644,

    arXiv

  22. [2019]

    Transform- ers as support vector machines.arXiv preprint arXiv:2308.16898,

    [TLTO23] Davoud Ataee Tarzanagh, Yingcong Li, Christos Thrampoulidis, and Samet Oymak. Transform- ers as support vector machines.arXiv preprint arXiv:2308.16898,

    arXiv

  23. [2020]

    Understanding in-context learning in transformers and llms by learning to learn discrete functions

    [BPBK23] Satwik Bhattamishra, Arkil Patel, Phil Blunsom, and Varun Kanade. Understanding in-context learning in transformers and llms by learning to learn discrete functions. arXiv preprint arXiv:2310.03016,

    arXiv

  24. [2021]

    Are transformers universal approximators of sequence-to-sequence functions?arXiv preprint arXiv:1912.10077,

    [YBR+19] Chulhee Yun, Srinadh Bhojanapalli, Ankit Singh Rawat, Sashank J Reddi, and Sanjiv Kumar. Are transformers universal approximators of sequence-to-sequence functions?arXiv preprint arXiv:1912.10077,

    1912 arXiv

  25. [2022]

    Why can GPT learn in-context? Language models implicitly perform gradient descent as meta-optimizers

    [DSD+22] Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Shuming Ma, Zhifang Sui, and Furu Wei. Why can GPT learn in-context? Language models implicitly perform gradient descent as meta-optimizers. arXiv preprint arXiv:2212.10559,

    arXiv

  26. [2023]

    All-in-one simulation-based inference.arXiv preprint arXiv:2404.09636,

    [GDW+24] Manuel Gloeckler, Michael Deistler, Christian Weilbach, Frank Wood, and Jakob H Macke. All-in-one simulation-based inference.arXiv preprint arXiv:2404.09636,

    arXiv

  27. [2024]

    [DLD+22] Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Tianyu Liu, et al

    Accessed: 2024-11-09. [DLD+22] Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Tianyu Liu, et al. A survey on in-context learning.arXiv preprint arXiv:2301.00234,

    2024 arXiv

  28. [2025]

    Transformers are ssms: Generalized models and efficient algorithms through structured state space duality.arXiv preprint arXiv:2405.21060,

    [DG24] Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality.arXiv preprint arXiv:2405.21060,

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.