Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

Towards Theoretical Understanding of Transformer Test-Time Computing: Investigation on In-Context Linear Regression

T0 review · 4 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that sampling-based test-time computing, modeled as noisy gradient descent with a linear noise term plus ensembling, provably avoids the label-noise overfitting that vanilla multi-step gradient-descent chain-of-thought suff

desk verdict The binary majority-vote result is worth reading, but the linear-NFT theorem in the continuous case has a sign inconsistency between the algorithm and the proof that is fatal as written. read the letter →

arxiv 2508.07571 v2 pith:LF2DV2RA submitted 2025-08-11 cs.LG cs.AI

Xingwu Chen , Miao Lu , Beining Wu , Difan Zou This is my paper
classification cs.LGcs.AI
keywords test-timecomputingin-contextlearninglinearregressionnoisygradientdescentchain-of-thoughtmajorityvotingensemblesamplingtransformers
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to close the gap between sampling-based test-time computing in large language models and theoretical transformer analysis by studying in-context linear regression. It models each decoding step as a transformer performing noisy gradient descent on the regression coefficient, then derives excess-risk bounds for aggregated sampled reasoning paths. Its central theoretical finding is that a particular 'linear' noise transformation, combined with ensembling many paths, keeps prediction error bounded in high dimension, whereas deterministic multi-step gradient descent overfits to label noise and worsens as the reasoning length grows. In the sparse binary-coefficient case, it shows that majority vote over sampled paths recovers the truth with high probability from a single in-context example, while greedy decoding cannot. A sympathetic reader would care because this offers a principled reason why sampling-based inference techniques such as ensembling and majority voting help, and it supplies concrete scaling predictions — though the paper assumes the required noisy-GD transformer exists rather than training it.

What carries the argument

The load-bearing mechanism is the sampling-based auto-regressive inference mechanism (Definition 3.1): at each reasoning step the one-layer transformer computes a gradient update and a sampling algorithm turns it into the next token embedding, so a chain of reasoning steps is a random walk $w_{\ell+1}\sim p(\cdot\mid w_\ell-\frac{\eta}{n}X^\top(Xw_\ell-y))$. The proof then decomposes the ensemble risk into bias, variance, and fluctuation. For the linear noise function $\varphi_\xi(w)=\xi\xi^\top w$, a regularized covariance matrix $G$ absorbs the label noise and prevents it from accumulating along the reasoning path; the fluctuation term is controlled by second-moment iteration and concentra

What would settle it

Train a one-layer linear-attention transformer from scratch on in-context linear regression with label noise (for example n=36, d=72, $\sigma_\epsilon^2=1$), then compare deterministic chain-of-thought risk with an ensemble of $N$ sampled paths using linear noise at long reasoning lengths; the theory predicts ensemble risk stays near $(\eta d)^{-r/(r+1)}$ while greedy GD risk grows with $t$. If the ensemble risk tracks the greedy curve instead, the central claim is refuted. In the binary case, Monte Carlo simulation of the exact sampling with n=k=1, $\sigma_\epsilon=0$, d=10 and large N,t shou

Watch

Extended reading notes

Core claim

The paper's central object is a one-layer linear-attention transformer with residual connections whose output token, after sampling, implements $w_{\ell+1}\sim p(\cdot \mid w_\ell-\eta n^{-1}X^\top(Xw_\ell-y))$ — a noisy gradient-descent step. It proves, for a Gaussian prior on the coefficient with population covariance $H$ and polynomially decaying eigenvalues $\lambda_i=i^{-(r+1)}$, that the risk of vanilla multi-step GD is $\lesssim \omega^2(t\eta)^{-r/(r+1)}+\sigma_\epsilon^2 n^{-1}(t\eta)^{1/(r+1)}$, so increasing reasoning length amplifies label noise. Ensembling $N$ independent noisy paths with additive noise leaves the same leading risk, but with the linear noise $\varphi_\xi(w)=\xi\

Load-bearing premise

The load-bearing premise is that a transformer genuinely runs the exact noisy-gradient-descent step with the specified noise models — the paper assumes such a transformer exists (Section 3.1) rather than training one — so if real trained transformers or LLM decoding do not follow this recurrence, the risk bounds do not transfer.

Editorial extensions

If this is right

  • With linear noise, ensembling $N$ sampled reasoning paths drives the sampling fluctuation down as $\varsigma_n/N$, so the ensemble's risk is governed by the bias-variance floor rather than by the chain-of-thought length $t$.
  • In the high-dimensional regime $n\asymp\eta d$, vanilla multi-step GD risk grows with effective reasoning length as $\tilde{t}^{1/(r+1)}(\eta d)^{-r/(r+1)}$, while the linear-noise ensemble stays at $(\eta d)^{-r/(r+1)}$ once $N$ is at least roughly $d^{(2r+1)/(r+1)}$; sample budget substitutes for reasoning depth.
  • Constant additive noise does not produce this improvement — its ensemble shares GD's leading risk — so the structure of the sampling noise, not randomness alone, is what prevents overfitting.
  • With a single noiseless in-context example and a binary sparse coefficient, majority vote has success probability at least $1-2^{-(d-1)}$, while greedy decoding succeeds with probability at most $1/(2d-1)+2/d$; sampling, not extra deterministic steps, is what escapes failure.
  • The fitted exponential forms $\mathrm{Acc}(T,N)\approx\alpha_T-\beta_T e^{-\Delta_T^2 N/2}$ and $\Delta_T\approx\gamma-\kappa e^{-\mu T}$ yield a low-cost-to-high prediction procedure that the paper validates on GSM8K and a MATH subset, suggesting budget extrapolation from cheap runs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implication the authors leave implicit: if real decoding noise is state-dependent rather than additive, sampling should be paired with noise scaled to the current candidate; additive temperature-style noise alone would not dodge overfitting in high dimension.
  • A direct test of the paper's main assumption (Section 3.1 states the noisy-GD transformer is assumed, not trained) would be to train a one-layer linear-attention transformer on noisy in-context regression and compare ensemble-versus-greedy risk curves.
  • The binary result suggests a general principle: deterministic decoding can get trapped in cycles, while sampling makes all states reachable, so majority voting acts like randomized restarts; this may generalize to discrete reasoning tasks where greedy search fails.
  • The low-cost-to-high extrapolation was validated on two math datasets; a natural stress test is whether the fitted exponent $\mu$ transfers across model sizes and question types, which the paper leaves open.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes a theoretical framework for transformer test-time computing by injecting randomness into the decoding process. It studies in-context linear regression with continuous coefficients (Gaussian prior) and binary sparse coefficients. For the continuous case, a one-layer linear attention transformer is shown to implement noisy gradient descent; two noise-transformation functions are considered, and the paper claims that ensembling with the linear noise transformation avoids the label-noise overfitting suffered by vanilla multi-step GD (Theorem 4.2). For the binary case, the paper analyzes majority voting versus greedy decoding and claims that majority voting recovers the true coefficient with high probability even with a single in-context example, while greedy decoding fails (Theorem 5.3). The paper also fits exponential curves to low-cost inference data in an attempt to predict high-cost LLM accuracy (Section 6.2).

Significance. The paper makes a useful step by explicitly modeling stochastic sampling during transformer inference and by decomposing the risk of ensembled reasoning paths into bias, variance, and fluctuation terms. The appendices contain detailed derivations, and the core risk bounds in Section 4 are derived from explicit assumptions rather than obtained by fitting. If the main theorems were correct as stated, the paper would provide concrete, falsifiable predictions about when test-time sampling helps. However, two load-bearing issues prevent acceptance in the current form: the linear-NFT analysis is for a different noise sign than the defined algorithm, and Theorem 5.3 overclaims the required sample size N. The paper also explicitly assumes, without training, the existence of a transformer satisfying Proposition 3.2 (Section 3.1), which limits the scope of the claims.

major comments (4)
  1. [§3.2, Example 3.5 and Appendix D.3.2] Algorithm 1 line 4 defines w ← ew + φξ(ew); for linear NFT, Example 3.5 sets φξ(w) := ξξ^T w, so the next weight is (I + ξ_{ℓ+1}ξ_{ℓ+1}^T)(GD-update). The proof for Example 3.5, however, analyzes the recurrence w_{t+1} = (I − ξ_{t+1}ξ_{t+1}^T)(I − ηΣ)(w_t + ηX^T y/n), as seen in the definition of Δ in Lemma D.2 and in the fluctuation iteration in Lemma D.6. The entire analysis relies on E[I − ξξ^T] = (1 − σ²)I and on Lemma D.7 for the minus-sign operator. With the plus sign of the main text, E[I + ξξ^T] = (1 + σ²)I and the fluctuation covariance in Lemma D.6 would grow rather than contract, so the bounds in Lemmas D.2–D.6, and hence Theorem 4.2 for linear NFT, do not follow for the algorithm as defined. This is a load-bearing internal inconsistency, not a harmless typo; the definitions must be aligned with the proof or the additive-noise case must be proved separately.
  2. [Theorem 5.3 and Appendix E.3 (Lemma E.5)] The theorem states that majority vote succeeds for any sampling number N ≥ 1, but the proof in Lemma E.5 explicitly requires 'sufficient large sampling number N' and invokes Proposition 5.1. For N=1, majority vote is a single draw, and the claimed probability 1 − 1/(2^{d−1}) is not derivable from the argument, which only establishes P(w_T = w*) > 1/2 for sufficiently large T. The statement needs an explicit lower bound on N (e.g., N ≳ |W| log(1/δ)/Δ_T²) or a weaker probability bound. This affects the central comparison with greedy decoding in the abstract and Section 5.
  3. [Appendix A.1 and Lemma D.6 / Theorem 4.2] The continuous-regression experiments are run with σ²=4 (Figure 1 settings) and σ²∈{1,4} (Figure 2 captions) at d=72, while Lemma D.6 assumes σ² < 1/(d+1) ≈ 0.0137 and Theorem 4.2 takes σ² ≍ d^{-1}. The experiments therefore operate outside the provable regime. Consequently, the statement in Section 6.1 that the results 'confirm Corollary 4.2' is not supported; the tested noise level is not covered by the theory. Please rerun the simulations within the provable regime or explicitly label them as exploratory and outside the theorem's assumptions.
  4. [§6.2, Eq (6.1)–(6.3), Algorithm 3] The low-cost-to-high prediction rests on an assumed exponential form Δ_T ≈ γ − κ e^{−μT} and fits μ, γ, κ to the low-cost data. Equations (6.2) and (6.3) then reuse the same μ to predict high-cost accuracy. The paper itself labels the transfer as conjectural ('we further claim two conjectures'), and Figure 3 does not provide an out-of-sample assessment of the fitted curves. The abstract and conclusion should not claim that real-world LLM performance is predicted by the theoretical framework unless a held-out evaluation is provided, with the fitted low-cost data kept separate from the predicted high-cost data.
minor comments (4)
  1. [§3.1, paragraph after Definition 3.3] The paper explicitly assumes the existence of a transformer satisfying Proposition 3.2 without training one. This is an important limitation that should appear not only in the body but also in the abstract: the theoretical results are for a hand-constructed model and are not directly about trained transformers.
  2. [Theorem 4.2] The statement says 'taking the noise variance σ² ≍ d^{-1}', but the proof requires the stronger condition σ² < 1/(d+1). The constants in the ≍ notation should be made explicit so that readers can see that the theorem's regime satisfies the proof's condition.
  3. [Figure 2 captions] The caption text is garbled in places (e.g., '2 = 1, 2 = 0.5' instead of σ² = 1, σ_ϵ² = 0.5). The axes and parameter labels should be corrected for readability.
  4. [Appendix D.3.2, Lemma D.4] Lemma D.4 says 'according to the definition of Bias' but the displayed quantity is E_{w*}[Bias]; the wording should be adjusted. There are several similar small wording issues in the appendix where expectations are written without the subscript.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity in the core derivations; however, the main linear-NFT theorem is proved for a subtractive noise recurrence that differs from Algorithm 1 as written, and Section 6.2 is an explicitly fitted extrapolation rather than a first-principles prediction.

full rationale

The paper's core derivation chain is self-contained rather than circular. Proposition 3.2 constructs explicit transformer weights (Appendix B.1) and verifies the noisy-GD update directly; it does not rely on a fitted parameter or on the conclusion being proved. The continuous-coefficient risk bounds (Theorem 4.2 / Theorem C.2) are derived from Assumption D.1 and the stated recurrences via standard bias/variance/fluctuation decompositions, and the discrete majority-vote results (Theorems 5.1-5.3) are proved from Hoeffding and Markov-type arguments on the defined sampling process. The cited prior work of Huang et al. (2025) supplies the GD construction, but the paper proves the needed proposition with explicit matrices, and the self-citations (e.g., Chen & Zou 2024) are background only. Thus no claim reduces to its own input by construction. Two non-circular caveats should be flagged. First, Section 3.1 states: 'we assume the existence of a transformer satisfying Proposition 3.2 without explicitly training such one from scratch, which is left as an interesting future work.' This makes Sections 4-5 conditional on an existence assumption, but it is an openly stated limitation rather than a circular input. Second, the proof of the linear-NFT bound does not match the algorithm as defined in the main text. Algorithm 1 (line 4) sets w <- ew + phi_xi(ew) and Example 3.5 sets phi_xi(w) := xi xi^T w, so the sampled update is (I + xi xi^T) times the GD update. The proof in Appendix D.3.2 (Lemma D.2 and the recurrences around Eq. D.13-D.16) instead analyzes (I - xi xi^T)(I - eta Sigma)(w + eta X^T y/n), using E[I - xi xi^T] = (1-sigma^2)I as a contraction. With the defined plus sign the expectation is (1+sigma^2)I, an expansion, so the contraction-based fluctuation bound and the stated Theorem 4.2 bound for linear NFT are not established for Algorithm 1 as written. This is a serious correctness gap, but it is an internal inconsistency rather than a circular reduction. Section 6.2's Eqs. (6.1)-(6.3) are explicitly introduced as approximations ('we assume', 'we further claim two conjectures'), and Algorithm 3 fits their parameters to low-cost data before extrapolating to higher cost. The extrapolated high-cost points are not the same data used to fit the parameters, so this is ordinary empirical curve-fitting/forecasting, not a fitted input disguised as a first-principles prediction. For these reasons, the circularity score is 0.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

The ledger lists the design choices and assumptions the central results depend on. The fitted curve parameters are used only in the LLM-prediction section; the risk bounds themselves depend on the domain assumptions and the noise model.

free parameters (5)
  • sigma^2 (noise variance for continuous sampling) = sigma^2 ~ 1/d in Theorem 4.2; sigma^2=4 in Figure 1 experiments (d=72)
    Introduced by Algorithm 1; the linear-NFT benefit requires sigma^2<1/(d+1) and sigma^2~1/d (Lemma D.6), but the experiments use sigma^2=4, outside the theoretical regime.
  • mu (decay exponent in Delta_T curve) = fitted on GSM8K/MATH data (Algorithm 3)
    Appears in Eq (6.1); not derived from theory. Also used to form predictions at high compute.
  • gamma, kappa (saturation parameters of Delta_T) = fitted on data
    Eq (6.1); fitted from low-cost data in Algorithm 3.
  • alpha_N, beta_N (accuracy curve parameters) = fitted per N
    Eq (6.3) and Algorithm 3 use these to extrapolate Acc(T,N).
  • eta (gradient descent step size) = eta=1e-3 in experiments; theory requires eta << Tr(H)^{-1}
    Standard hyperparameter; not fitted to the target result but required to satisfy the step-size assumption for the risk bounds.
assumptions (6)
  • domain assumption Design vectors are sub-Gaussian with identity covariance up to H (Assumption D.1)
    Used in all proofs in Section 4 and Appendix D.
  • domain assumption True coefficient w* ~ N(0, omega^2 I_d) and label noise is i.i.d. with zero mean and variance sigma_epsilon^2 (Assumption D.1)
    Prior model underpinning the in-context linear regression task.
  • domain assumption For the binary case, x ~ N(0, I_d) and epsilon_i ~ N(0, sigma_epsilon^2) (Section 5)
    Used for concentration bounds in Lemma E.1.
  • ad hoc to paper The sampling noise model (Gaussian noise in Algorithm 1, discrete sampling in Algorithm 2) faithfully simulates LLM decoding distribution
    No evidence that actual LLM decoding follows these specific noise models; this is the paper's modeling bridge to real inference.
  • ad hoc to paper Delta_T approx gamma - kappa e^{-mu T} (Eq 6.1)
    Functional form assumed, not derived; used for the low-cost-to-high-cost prediction.
  • domain assumption Population covariance H has polynomially decaying eigenvalues lambda_i = i^{-(r+1)} (Section 4)
    Gives closed-form rates; standard in this literature.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards Theoretical Understanding of Transformer Test-Time Computing: Investigation on In-Context Linear Regression." pith.science (2026). https://pith.science/paper/LF2DV2RA

@misc{pith2026250807571,
  author       = {Pith},
  title        = {Pith review of: Towards Theoretical Understanding of Transformer Test-Time Computing: Investigation on In-Context Linear Regression},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LF2DV2RA}},
  note         = {Machine review of arXiv:2508.07571}
}
read the original abstract

Using more test-time computation during language model inference, such as generating more intermediate thoughts or sampling multiple candidate answers, has proven effective in significantly improving model performance. This paper takes an initial step toward bridging the gap between practical language model inference and theoretical transformer analysis by incorporating randomness and sampling. We focus on in-context linear regression with continuous/binary coefficients, where our framework simulates language model decoding through noise injection and binary coefficient sampling. Through this framework, we provide detailed analyses of widely adopted inference techniques. Supported by empirical results, our theoretical framework and analysis demonstrate the potential for offering new insights into understanding inference behaviors in real-world language models.

Figures

Figures reproduced from arXiv: 2508.07571 by the authors.

Figure 1
Figure 1. Comparison between real-world LLM’s inference (above) and our designed sampling [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Numerical experiments on in-context linear regression with continuous coefficients ( [PITH_FULL_IMAGE:figures/full_fig_p013_2.png] view at source ↗
Figure 3
Figure 3. Utilizing data with low computational costs to forecast results for high computational [PITH_FULL_IMAGE:figures/full_fig_p014_3.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Transformers Efficiently Perform In-Context Logistic Regression via Normalized Gradient Descent

    cs.LG 2026-05 conditional novelty 7.0 of 10

    Multi-layer transformers can implement in-context logistic regression by performing normalized gradient descent steps layer by layer, obtained via supervised training of a single attention layer followed by recurrent ...

Reference graph

Works this paper leans on

66 extracted references · 34 canonical work pages · cited by 1 Pith paper

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type url volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.all := #1 'mid.sentence := ...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  4. [4]

    , Adler, S

    Achiam, J. , Adler, S. , Agarwal, S. , Ahmad, L. , Akkaya, I. , Aleman, F. L. , Almeida, D. , Altenschmidt, J. , Altman, S. , Anadkat, S. et al. (2023). Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  5. [5]

    , Ali, A

    Agarwal, N. , Ali, A. , Bala, M. , Balaji, Y. , Barker, E. , Cai, T. , Chattopadhyay, P. , Chen, Y. , Cui, Y. , Ding, Y. et al. (2025). Cosmos world foundation model platform for physical ai. arXiv preprint arXiv:2501.03575

    arXiv 2025

  6. [6]

    , Cheng, X

    Ahn, K. , Cheng, X. , Daneshmand, H. and Sra, S. (2023 a ). Transformers learn to implement preconditioned gradient descent for in-context learning. Advances in Neural Information Processing Systems 36 45614--45650

    work page 2023

  7. [7]

    , Cheng, X

    Ahn, K. , Cheng, X. , Song, M. , Yun, C. , Jadbabaie, A. and Sra, S. (2023 b ). Linear attention is (maybe) all you need (to understand transformer optimization). arXiv preprint arXiv:2310.01082

    arXiv 2023

  8. [8]

    , Schuurmans, D

    Aky \"u rek, E. , Schuurmans, D. , Andreas, J. , Ma, T. and Zhou, D. (2022). What learning algorithm is in-context learning? investigations with linear models. In The Eleventh International Conference on Learning Representations

    work page 2022

Show all 66 references
  1. [9]

    , Chen, F

    Bai, Y. , Chen, F. , Wang, H. , Xiong, C. and Mei, S. (2024). Transformers as statisticians: Provable in-context learning with in-context algorithm selection. Advances in neural information processing systems 36

    2024

  2. [10]

    Bartlett, P. L. , Long, P. M. , Lugosi, G. and Tsigler, A. (2020). Benign overfitting in linear regression. Proceedings of the National Academy of Sciences 117 30063--30070

    2020

  3. [11]

    , Mann, B

    Brown, T. , Mann, B. , Ryder, N. , Subbiah, M. , Kaplan, J. D. , Dhariwal, P. , Neelakantan, A. , Shyam, P. , Sastry, G. , Askell, A. et al. (2020). Language models are few-shot learners. Advances in neural information processing systems 33 1877--1901

    2020

  4. [12]

    , Sheen, H

    Chen, S. , Sheen, H. , Wang, T. and Yang, Z. (2024 a ). Training dynamics of multi-head softmax attention for in-context learning: Emergence, convergence, and optimality. arXiv preprint arXiv:2402.19442

    2024 arXiv

  5. [13]

    , Zhao, L

    Chen, X. , Zhao, L. and Zou, D. (2024 b ). How transformers utilize multi-head attention in in-context learning? a case study on sparse linear regression. arXiv preprint arXiv:2408.04532

    2024 arXiv

  6. [14]

    and Zou, D

    Chen, X. and Zou, D. (2024). What can transformer learn with varying depth? case studies on sequence learning tasks. arXiv preprint arXiv:2404.01601

    2024 arXiv

  7. [15]

    , Kosaraju, V

    Cobbe, K. , Kosaraju, V. , Bavarian, M. , Chen, M. , Jun, H. , Kaiser, L. , Plappert, M. , Tworek, J. , Hilton, J. , Nakano, R. et al. (2021). Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

    2021 arXiv

  8. [16]

    , Xiong, W

    Dong, H. , Xiong, W. , Goyal, D. , Zhang, Y. , Chow, W. , Pan, R. , Diao, S. , Zhang, J. , Shum, K. and Zhang, T. (2023). Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint arXiv:2304.06767

    2023 arXiv

  9. [17]

    , Jauhri, A

    Dubey, A. , Jauhri, A. , Pandey, A. , Kadian, A. , Al-Dahle, A. , Letman, A. , Mathur, A. , Schelten, A. , Yang, A. , Fan, A. et al. (2024). The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  10. [18]

    , Chen, T.-Q

    Fu, D. , Chen, T.-Q. , Jia, R. and Sharan, V. (2023). Transformers learn higher-order optimization methods for in-context learning: A study with linear models. arXiv preprint arXiv:2310.17086

    2023 arXiv

  11. [19]

    , Tsipras, D

    Garg, S. , Tsipras, D. , Liang, P. and Valiant, G. (2023). What Can Transformers Learn In-Context ? A Case Study of Simple Function Classes

    2023

  12. [20]

    , Yang, L

    Giannou, A. , Yang, L. , Wang, T. , Papailiopoulos, D. and Lee, J. D. (2024). How Well Can Transformers Emulate In-context Newton 's Method ?

    2024

  13. [21]

    , Yang, D

    Guo, D. , Yang, D. , Zhang, H. , Song, J. , Zhang, R. , Xu, R. , Zhu, Q. , Ma, S. , Wang, P. , Bi, X. et al. (2025). Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948

    2025 arXiv

  14. [22]

    Guo, T. , Hu, W. , Mei, S. , Wang, H. , Xiong, C. , Savarese, S. and Bai, Y. (2023). How Do Transformers Learn In-Context Beyond Simple Functions ? A Case Study on Learning with Representations

    2023

  15. [23]

    , Burns, C

    Hendrycks, D. , Burns, C. , Kadavath, S. , Arora, A. , Basart, S. , Tang, E. , Song, D. and Steinhardt, J. (2021). Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874

    2021 arXiv

  16. [24]

    , Zhang, F

    Hu, X. , Zhang, F. , Chen, S. and Yang, Z. (2024). Unveiling the statistical foundations of chain-of-thought prompting methods. arXiv preprint arXiv:2408.14511

    2024 arXiv

  17. [25]

    , Wang, Z

    Huang, J. , Wang, Z. and Lee, J. D. (2025). Transformers learn to implement multi-step gradient descent with chain of thought. In The Thirteenth International Conference on Learning Representations

    2025

  18. [26]

    , Cheng, Y

    Huang, Y. , Cheng, Y. and Liang, Y. (2023). In- Context Convergence of Transformers

    2023

  19. [27]

    , McCandlish, S

    Kaplan, J. , McCandlish, S. , Henighan, T. , Brown, T. B. , Chess, B. , Child, R. , Gray, S. , Radford, A. , Wu, J. and Amodei, D. (2020). Scaling laws for neural language models. arXiv preprint arXiv:2001.08361

    2020 arXiv

  20. [28]

    , Kim, M

    Kim, J. , Kim, M. and Mozafari, B. (2023). Provable memorization capacity of transformers. In The Eleventh International Conference on Learning Representations

    2023

  21. [29]

    and Suzuki, T

    Kim, J. and Suzuki, T. (2024). Transformers provably solve parity efficiently with chain of thought. arXiv preprint arXiv:2410.08633

    2024 arXiv

  22. [30]

    Kojima, T. , Gu, S. S. , Reid, M. , Matsuo, Y. and Iwasawa, Y. (2022). Large language models are zero-shot reasoners. Advances in neural information processing systems 35 22199--22213

    2022

  23. [31]

    and Lounici, K

    Koltchinskii, V. and Lounici, K. (2017). Concentration inequalities and moment bounds for sample covariance operators. Bernoulli 110--133

    2017

  24. [32]

    , Zhuang, V

    Kumar, A. , Zhuang, V. , Agarwal, R. , Su, Y. , Co-Reyes, J. D. , Singh, A. , Baumli, K. , Iqbal, S. , Bishop, C. , Roelofs, R. et al. (2024). Training language models to self-correct via reinforcement learning. arXiv preprint arXiv:2409.12917

    2024 arXiv

  25. [33]

    , Zhang, C

    Li, T. , Zhang, C. , Chen, X. , Cao, Y. and Zou, D. (2025). On the robustness of transformers against context hijacking for linear classification. arXiv preprint arXiv:2502.15609

    2025 arXiv

  26. [34]

    Liu, F. et al. (2020). Learning to summarize from human feedback. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics

    2020

  27. [35]

    , Zhou, R

    Liu, R. , Zhou, R. , Shen, C. and Yang, J. (2024). On the learn-to-optimize capabilities of transformers in in-context sparse recovery. arXiv preprint arXiv:2410.13981

    2024

  28. [36]

    , Tandon, N

    Madaan, A. , Tandon, N. , Gupta, P. , Hallinan, S. , Gao, L. , Wiegreffe, S. , Alon, U. , Dziri, N. , Prabhumoye, S. , Yang, Y. et al. (2023). Self-refine: iterative refinement with self-feedback (2023). arXiv preprint arXiv:2303.17651

    2023 arXiv

  29. [37]

    , Liao, R

    Mahdavi, S. , Liao, R. and Thrampoulidis, C. (2023). Memorization Capacity of Multi-Head Attention in Transformers

    2023

  30. [38]

    Malach, E. (2023). Auto- Regressive Next-Token Predictors are Universal Learners

    2023

  31. [39]

    , Hilton, J

    Nakano, R. , Hilton, J. , Balaji, S. , Wu, J. , Ouyang, L. , Kim, C. , Hesse, C. , Jain, S. , Kosaraju, V. , Saunders, W. et al. (2021). Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332

    2021 arXiv

  32. [40]

    Learning to reason with llms

    OpenAI (2024). Learning to reason with llms. https://openai.com/index/learning-to-reason-with-llms/

    2024

  33. [41]

    and Xie, S

    Peebles, W. and Xie, S. (2023). Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision

    2023

  34. [42]

    , Yeh, C

    Saunders, W. , Yeh, C. , Wu, J. , Bills, S. , Ouyang, L. , Ward, J. and Leike, J. (2022). Self-critiquing models for assisting human evaluators. arXiv preprint arXiv:2206.05802

    2022 arXiv

  35. [43]

    , Lee, J

    Snell, C. , Lee, J. , Xu, K. and Kumar, A. (2024 a ). Scaling llm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314

    2024 arXiv

  36. [44]

    , Lee, J

    Snell, C. , Lee, J. , Xu, K. and Kumar, A. (2024 b ). Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters

    2024

  37. [45]

    , Ouyang, L

    Stiennon, N. , Ouyang, L. , Wu, J. , Ziegler, D. , Lowe, R. , Voss, C. , Radford, A. , Amodei, D. and Christiano, P. F. (2020). Learning to summarize with human feedback. Advances in Neural Information Processing Systems 33 3008--3021

    2020

  38. [46]

    and Suzuki, T

    Takakura, S. and Suzuki, T. (2023). Approximation and Estimation Ability of Transformers for Sequence-to-Sequence Functions with Infinite Dimensional Input

    2023

  39. [47]

    and Bartlett, P

    Tsigler, A. and Bartlett, P. L. (2023). Benign overfitting in ridge regression. Journal of Machine Learning Research 24 1--76

    2023

  40. [48]

    Vaswani, A. (2017). Attention is all you need. Advances in Neural Information Processing Systems

    2017

  41. [49]

    Vershynin, R. (2018). High-dimensional probability: An introduction with applications in data science, vol. 47. Cambridge university press

    2018

  42. [50]

    , Niklasson, E

    von Oswald , J. , Niklasson, E. , Randazzo, E. , Sacramento, J. , Mordvintsev, A. , Zhmoginov, A. and Vladymyrov, M. (2023). Transformers learn in-context by gradient descent

    2023

  43. [51]

    , Niklasson, E

    Von Oswald, J. , Niklasson, E. , Randazzo, E. , Sacramento, J. , Mordvintsev, A. , Zhmoginov, A. and Vladymyrov, M. (2023). Transformers learn in-context by gradient descent. In International Conference on Machine Learning. PMLR

    2023

  44. [52]

    , Wei, J

    Wang, X. , Wei, J. , Schuurmans, D. , Le, Q. , Chi, E. , Narang, S. , Chowdhery, A. and Zhou, D. (2022). Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171

    2022 arXiv

  45. [53]

    , Wei, J

    Wang, X. , Wei, J. , Schuurmans, D. , Le, Q. , Chi, E. , Narang, S. , Chowdhery, A. and Zhou, D. (2023). Self- Consistency Improves Chain of Thought Reasoning in Language Models

    2023

  46. [54]

    , Wang, X

    Wei, J. , Wang, X. , Schuurmans, D. , Bosma, M. , Xia, F. , Chi, E. , Le, Q. V. , Zhou, D. et al. (2022). Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems 35 24824--24837

    2022

  47. [55]

    , Zhang, H

    Wen, K. , Zhang, H. , Lin, H. and Zhang, J. (2024). From sparse dependence to sparse attention: Unveiling how chain-of-thought enhances transformer sample efficiency. arXiv preprint arXiv:2410.05459

    2024 arXiv

  48. [56]

    , Zou, D

    Wu, J. , Zou, D. , Chen, Z. , Braverman, V. , Gu, Q. and Bartlett, P. L. (2023). How many pretraining tasks are needed for in-context learning of linear regression? arXiv preprint arXiv:2310.08391

    2023 arXiv

  49. [57]

    , Sun, Z

    Wu, Y. , Sun, Z. , Li, S. , Welleck, S. and Yang, Y. (2024). Inference Scaling Laws : An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Models

    2024

  50. [58]

    , Huang, Y

    Yang, T. , Huang, Y. , Liang, Y. and Chi, Y. (2024). In- Context Learning with Representations : Contextual Generalization of Trained Transformers

    2024

  51. [59]

    Yao, S. , Yu, D. , Zhao, J. , Shafran, I. , Griffiths, T. , Cao, Y. and Narasimhan, K. (2024). Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems 36

    2024

  52. [60]

    , Zhuang, H

    Yue, Z. , Zhuang, H. , Bai, A. , Hui, K. , Jagerman, R. , Zeng, H. , Qin, Z. , Wang, D. , Wang, X. and Bendersky, M. (2024). Inference scaling for long-context retrieval augmented generation. arXiv preprint arXiv:2410.04343

    2024 arXiv

  53. [61]

    , Meng, X

    Zhang, C. , Meng, X. and Cao, Y. (2025). Transformer learns optimal variable selection in group-sparse classification. arXiv preprint arXiv:2504.08638

    2025 arXiv

  54. [62]

    and Chen, C

    Zhang, H. and Chen, C. (2024). Test-time compute scaling laws. https://github.com/hughbzhang/o1_inference_scaling_laws

    2024

  55. [63]

    , Frei, S

    Zhang, R. , Frei, S. and Bartlett, P. L. (2023). Trained transformers learn linear models in-context. arXiv preprint arXiv:2306.09927

    2023 arXiv

  56. [64]

    , Yin, L

    Zheng, L. , Yin, L. , Xie, Z. , Sun, C. , Huang, J. , Yu, C. H. , Cao, S. , Kozyrakis, C. , Stoica, I. , Gonzalez, J. E. et al. (2024). Sglang: Efficient execution of structured language model programs. arXiv preprint arXiv:2312.07104

    2024 arXiv

  57. [65]

    , Yan, K

    Zhou, A. , Yan, K. , Shlapentokh-Rothman, M. , Wang, H. and Wang, Y.-X. (2023). Language agent tree search unifies reasoning acting and planning in language models. arXiv preprint arXiv:2310.04406

    2023 arXiv

  58. [66]

    Zou, D. , Wu, J. , Braverman, V. , Gu, Q. and Kakade, S. (2022). Risk bounds of multi-pass sgd for least squares in the interpolation regime. Advances in Neural Information Processing Systems 35 12909--12920

    2022

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.