Pith. sign in

REVIEW 5 major objections 6 minor 54 references

Brewing Knowledge in Context: Distillation Perspectives on In-Context Learning

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that in-context learning is an implicit knowledge-distillation process, where prompt demonstrations initialize a reference model inside a single attention pass and a provable bound ties the distilled weights' bias to the…

desk verdict A neat KD gloss on an existing equivalence, but the headline MMD theorem is internally contradicted by the paper's own appendix. read the letter →

arxiv 2506.11516 v1 pith:VKIBJWX4 submitted 2025-06-13 cs.LG cs.CL

Chengye Li , Haiyun Liu , Yuanxi Li This is my paper
classification cs.LGcs.CL MSC 68T0768Q32
keywords in-contextlearningknowledgedistillationsoftmaxattentiongradientdescentequivalencemaximummeandiscrepancyRademachercomplexitygeneralizationboundpromptshift
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that in-context learning is secretly knowledge distillation: during a forward pass, the demonstrations initialize a small reference model inside the attention computation, and the following tokens adjust it by one gradient step. The central construction is an exact weight equivalence: the reference model's initial weight equals one distillation step from the pretrained LLM (Theorem 3.1), so ICL inherits KD's generalization story. The authors prove a Rademacher-complexity generalization bound and show the distilled weight's bias grows at most linearly with the MMD between the demonstration distribution and the target distribution (Theorem 4.1), with a monotone risk-gap consequence (Theorem 4.2). If correct, prompt quality stops being a qualitative intuition and becomes a measurable quantity.

What carries the argument

The load-bearing object is the implicit reference model $f(x) = W\phi(x)$ together with the softmax-kernel feature map $\phi$ satisfying $K_{\mathrm{softmax}}(x, y) = \exp(x^\top y) = \phi(x)^\top \phi(y)$. The paper reassigns the two terms of the softmax attention output: demonstrations supply the initial weight $W_0 = \frac{1}{D'} W^V X_D \phi(W^K X_D)^\top$ and preceding queries act as the gradient term $-\eta \frac{\partial L}{\partial W}$. The normalization constant $D'$ (the partition function of softmax) becomes the inverse learning rate. This identification converts attention into a single KD step (Theorem 3.1) and makes MMD in the RKHS induced by $x\phi(W^K x)^\top$ the natural measure of prompt-task mismatch (Theorem 4.1).

What would settle it

Run ICL on a real LLM with prompts drawn from distributions at known MMD distances from the target distribution (varying domain shift), measure task accuracy, and test whether performance degrades linearly with MMD as Theorem 4.2 predicts; a single dataset where ordered demonstrations of the same distribution give systematically different accuracy would falsify the distribution-only bound.

Watch

Extended reading notes

Core claim

In-context learning with softmax attention is formally a one-step knowledge-distillation process. Treating softmax attention as one gradient-descent step on a reference model $f(x) = W\phi(x)$ (Lemma 2.2), the paper swaps the roles of demonstration and query tokens (Corollary 1): demonstrations set the initial weight $W_0$ and queries provide the gradient. Theorem 3.1 then shows $W_0$ equals $W^*$ after one gradient step of an $\ell_2$ distillation loss from the pretrained teacher $f_T(x) = W^V x$, with equality $2\eta^*/N = 1/D'$. The same perspective yields a generalization bound (Theorem 3.2) and, crucially, a prompt-shift theorem (Theorem 4.1): the expected distilled weight deviates from the task-optimal weight by at most $\eta M_V M_x M_\phi \operatorname{MMD}(\mathcal{D}, \mathcal{Q})$, a linear bound in the maximum mean discrepancy between prompt distribution $\mathcal{Q}$ and target distribution $\mathcal{D}$. Theorem 4.2 converts this into a monotone risk gap: a smaller MMD means weakly smaller KD risk on the target domain. Thus, prompt quality is quantified, and attention's normalization constant plays the role of the learning rate.

Load-bearing premise

The whole construction assumes softmax attention in a real transformer is exactly one gradient-descent step on a single linear reference model, and that demonstrations are random draws from a fixed prompt distribution; if attention executes a different number of steps, or if demonstration selection and ordering matter, the distillation bounds and the MMD bound do not transfer.

Editorial extensions

If this is right

  • If Theorem 3.1 holds, the first prompt pass literally distills the pretrained value space into demonstration keys, so in-context learning inherits the generalization story of knowledge distillation.
  • Theorem 3.2 implies that more demonstrations shrink the Rademacher term, formally accounting for the empirical benefit of longer prompts.
  • Theorems 4.1 and 4.2 make maximum mean discrepancy between the demonstration set and the target query distribution a principled objective for prompt retrieval, ranking, and construction.
  • The explicit constants in the bounds identify embedding normalization, temperature scaling, and learning-rate clipping as levers that reduce sensitivity to prompt shift.
  • A smaller MMD between prompt distribution and target distribution provably yields weakly smaller distillation risk, explaining why noisy or off-domain demonstrations hurt.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The bound points directly to a practical estimator that the paper does not build: compute the MMD between a candidate demonstration set and unlabeled target examples, and use it to rank prompts before inference.
  • If attention implements several layers of the same mechanism, real LLMs may be performing iterative distillation; the paper's single-layer derivation would then extend layer by layer with a shrinking effective learning rate.
  • The KD framing implies a safety corollary: prompts that raise MMD against the intended task may distill harmful or off-target teacher knowledge, a testable link between prompt quality and harmful generation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes viewing in-context learning (ICL) as an implicit form of knowledge distillation (KD). It builds on a softmax-attention-as-gradient-descent equivalence (Lemma 2.2), swaps the roles of demonstration and query tokens so that demonstrations initialize a reference model (Corollary 1, Theorem 3.1), derives a Rademacher-complexity generalization bound for the resulting distillation loss (Theorem 3.2), and proves an MMD-based bound on the deviation of the one-step distilled weights from an optimal reference weight (Theorem 4.1), together with a prompt-shift risk gap (Theorem 4.2). The central claims are that the bias of the distilled weights grows linearly with the MMD between prompt and target distributions and that better prompts, measured by smaller MMD, provably improve ICL performance.

Significance. If the results were correct, the paper would provide a quantitative prompt-quality criterion and a unified view of gradient-based and distributional analyses of ICL. The Rademacher bound is standard and the paper is transparent in Appendix G about the unwhitened bound, which is a point in its favor. However, the central theorems contain algebraic errors and a definitional circularity: the promised MMD-only bound is contradicted by the paper's own Appendix G, and the KD interpretation is largely a relabeling of the existing one-step-gradient-descent equivalence rather than an independently established mechanism. Because the headline quantitative claims fail, the paper does not currently provide a reliable theoretical foundation for the proposed ICL-KD perspective.

major comments (5)
  1. [§2.2, Eq. (2) and Appendix C, Eq. (17)] The claimed kernel factorization exp(xᵀy) = exp(‖x‖² + ‖y‖²) exp(−‖x−y‖²) is algebraically false; the right-hand side equals exp(2xᵀy), not exp(xᵀy). This factorization is used to justify the existence of the feature map φ in Lemma 2.2 and hence to identify the softmax-attention output with one gradient step on a reference model. The derivation should either use the correct identity, exp(xᵀy) = exp(0.5‖x‖² + 0.5‖y‖²) exp(−0.5‖x−y‖²), or cite a valid softmax-kernel feature map.
  2. [§3.2, Eq. (11) and Corollary 1] Theorem 3.1 is an equality by choice of the free learning rate η*. Setting 2η*/N = 1/D′ makes W* equal to W0 by construction, since no constraint is imposed on η*. This does not establish a distillation mechanism; it only rewrites a sum of outer products as one gradient step from zero. In addition, Corollary 1 swaps the roles of demonstrations and queries relative to Lemma 2.2 without justification: Lemma 2.2 identifies the query-token term as the initialization and the demonstration term as the gradient, while Theorem 3.1 reverses these roles. The swap is a definitional choice, not a derived property.
  3. [§4, Theorem 4.1 vs. Appendix G, Theorem G.1] The claimed bound ∥ΔW∥_F ≤ η M_V M_x M_φ MMD(D,Q) is false for the W⋆ defined in Eq. (14). The main-text proof obtains it only after 'whitening W⋆', but whitening the feature map changes the optimization problem and the definition of W⋆. The unwhitened bound in Theorem G.1 is ∥ΔW∥_F ≤ M_V M_x M_φ(∥ηI − Σ_φ^{-1}∥_2 + ∥Σ_φ^{-1}∥_2 MMD(D,Q)). The additive term is independent of MMD: taking Σ_φ = 2I, η = 1, and Q = D with E_Q[xφ(W_Kx)ᵀ] ≠ 0 gives MMD(D,Q) = 0 but ΔW = (1/2)W_V E_Q[xφᵀ] ≠ 0. Thus the headline statement that the bias grows linearly with MMD is internally inconsistent with the paper's own appendix.
  4. [§4, Theorem 4.2, Step 3 of the proof] The proof bounds ∥A_b − A_g∥_F = ∥E_{Q_b}[xφ(W_Kx)ᵀ] − E_{Q_g}[xφ(W_Kx)ᵀ]∥_F by M_T M_φ (MMD_b − MMD_g). This does not follow from Theorem 4.1 or from the triangle inequality; the reverse triangle inequality gives a lower bound, not an upper bound, on the distance between Q_g and Q_b in terms of |MMD_b − MMD_g|. Consequently the prompt-shift risk gap bound in Eq. (15) is not established.
  5. [§3.3, Theorem 3.2 and Eq. (13)] The generalization bound controls the expected squared distance between the teacher output W_V x and the student output W φ(W_K x) over the prompt distribution, not any downstream prediction error on queries. The abstract and introduction present this as a generalization bound for ICL, but the loss is a distillation loss on demonstration tokens. The bound therefore does not support claims about ICL performance. Additionally, the i.i.d. assumption on demonstrations is not representative of ICL practice, and the paper's own Related Work (Appendix A) cites DMND24 and SMK24 as reporting empirical failures of the one-step-gradient-descent equivalence for real LLMs without explaining how the present framework is robust to those failures.
minor comments (6)
  1. [§2.2, Eq. (2)] The correct identity is exp(xᵀy) = exp(0.5‖x‖² + 0.5‖y‖²) exp(−0.5‖x−y‖²); the missing 1/2 factors should be corrected wherever the factorization appears.
  2. [Appendix F, Step 5] The stated choice of t in the McDiarmid step does not make the right-hand side equal δ/2; the exponent evaluates to −(9/4) log(2/δ). The bound remains valid with a smaller probability, but the constants should be rechecked.
  3. [Appendix D] The proof of Lemma 3.1 contains undefined notation and typos, including 'F −2' and 'Um−1(h)', which make the proof hard to follow.
  4. [§3.2, Eq. (12)] The symbol D′ is used both as a scalar constant and as the softmax partition function, while D is earlier introduced as a diagonal matrix; the notation should be harmonized.
  5. [§4, Theorem 4.2] The main-text bound assumes 2ηM_φ² < 1 without stating this condition in the theorem statement; Appendix H gives the exact bound and should be reflected in the theorem statement.
  6. [Introduction and references] There are formatting errors in the references, such as '[ MLZH22' in the introduction, and several references are missing page or venue details.

Circularity Check

2 steps flagged · score 8.0 of 10

Core claims reduce to definitions: Theorem 3.1's KD equivalence is forced by choosing η*, and Theorem 4.1's MMD bound restates the definition of MMD with the feature map A(x)=vec(xφ(W_Kx)^T).

  1. self definitional [Section 3.2, Theorem 3.1 proof, Eqs. (10)-(11)]
    "In Eq. (6), W0 of the reference model is 1/D′ W V XDϕ(W KXD)⊤. Obviously, when let 2η∗/N = 1/D′ we can have: W ∗ = 2η∗/N ΣNi=1 W V xiϕ(W Kxi)⊤ = 2η∗/N W V XDϕ(W KxD)⊤ = W0 (11)"

    The theorem claims to show that the reference-model initialization W0 is 'formally equivalent to a knowledge distillation process.' But W0 was already defined in Eq. (6) as (1/D') W_V X_D φ(W_K X_D)^T, and the proof constructs a KD loss whose one-step gradient from zero is (2/N) Σ W_V x_i φ(W_K x_i)^T, then sets the learning rate η* = N/(2D') to make the distilled weight exactly equal to W0. The equality is enforced by tuning η*, not derived; the 'KD equivalence' is a restatement of the definition of W0.

  2. self definitional [Section 4, Theorem 4.1 statement and proof, Step 3]
    "the maximum-mean-discrepancy between D and Q is MMD(D, Q) = sup∥f∥H≤1 |Ex∼D[f(x)] − Ex∼Q[f(x)]|, where H is the RKHS induced by the kernel⟨x ϕ(W Kx)⊤, x′ ϕ(W Kx′)⊤⟩ ... Then vectorize A(x) = vec(x ϕ(W Kx)⊤) ... by the definition of MMD in the stated RKHS, ∥EQ[A] − ED[A]∥2 ≤ MxMϕ MMD(D, Q)."

    With kernel k(x,x') = ⟨A(x), A(x')⟩, the RKHS consists of functions f_v(x)=⟨v,A(x)⟩ and, by the definition of MMD, MMD(D,Q) = ∥E_D[A] − E_Q[A]∥_2. The theorem's ΔW is (after whitening) η W_V(E_Q[A] − E_D[A]), so the claimed bound ∥ΔW∥_F ≤ η M_V M_x M_φ MMD is just the elementary inequality ∥η W_V(E_QA−E_DA)∥_F ≤ η M_V ∥E_QA−E_DA∥_F with an extra M_x M_φ factor. The 'bias grows linearly with MMD' conclusion is therefore the definition of MMD restated in new variables, not an independently derived bound.

full rationale

The Rademacher generalization bound (Theorem 3.2) is a standard, largely self-contained application of Talagrand contraction and McDiarmid, and is not circular. The circularity is concentrated in the two advertised contributions. First, Theorem 3.1's 'weight initialization as knowledge distillation' is manufactured by the free choice of the learning rate η* = N/(2D'), so the equality W* = W0 is an identity imposed by construction rather than a derived equivalence. Second, Theorem 4.1 defines the MMD in the RKHS whose feature map is exactly A(x)=vec(xφ(W_Kx)^T), which is the very moment entering ΔW; the theorem's linear-in-MMD bound is the definition of MMD (mean-embedding distance) with constants, so the headline quantitative claim is definitional. The paper's own Appendix G also reveals a further problem: the clean MMD-only main-text bound is obtained only after 'whitening W⋆,' while the unwhitened Theorem G.1 contains an additive ∥ηI−Σ_φ^{-1}∥ term that does not vanish when MMD=0, so the main-text bound is not even the valid consequence for the stated W⋆. Since the two central claims reduce by construction or by definition, the appropriate circularity score is high.

Assumptions & free parameters 1 free parameters · 6 assumptions · 2 invented entities

The paper rests on the imported ICL=one-step-GD equivalence, an ad hoc teacher/student identification, an i.i.d. assumption on demonstrations, and an MMD kernel chosen so the bias bound is definitional. The learning rate η is a free constant that absorbs the softmax normalization. No experiments or code corroborate the interpretation.

free parameters (1)
  • KD learning rate η (Theorem 3.1) = 2η*/N = 1/D' (Eq. 11)
    The equivalence between the one-step KD weight W* and the attention-defined W0 holds only for this relationship; the learning rate is a free constant chosen to absorb the softmax normalization. In Theorem 4.1, η>0 is arbitrary and no value is specified.
assumptions (6)
  • domain assumption Softmax attention output equals one gradient descent step on reference model f(x)=Wφ(x) (Lemma 2.2 from RL24).
    This imported equivalence is the foundation for the ICL-KD interpretation and both main bounds. The proof in the paper contains an incorrect kernel factorization (Eq. 2).
  • ad hoc to paper The pretrained LLM's knowledge can be represented by teacher f_T(x)=W_V x and student f_S(x)=Wφ(W_Kx) with fixed W_K, so minimizing squared L2 distance over prompt tokens is the relevant distillation process.
    Theorem 3.1 defines the teacher and student to make W0 equal to a gradient step; no independent evidence ties this choice to the pretrained model's behavior on the actual ICL task.
  • domain assumption Demonstrations are i.i.d. samples from a prompt distribution Q.
    Used in Theorem 3.2 and Theorem 4.1 condition (iv); real prompts are ordered, selected, and correlated, so the i.i.d. assumption is not satisfied in practice.
  • domain assumption Second-moment matrix Σ_φ = E_D[φ(W_Kx)φ(W_Kx)^T] is invertible.
    Needed for the closed form of W* (Theorem 4.1, condition iii); not guaranteed for arbitrary key weights and prompts.
  • ad hoc to paper The softmax denominator D' can be treated as a scalar and absorbed into the learning rate or D'.
    In Lemma 2.2 and Theorem 3.1, the normalization 1/D' is folded into η or treated as a constant, but D' depends on the query x', which is not covered by the i.i.d. demonstration assumption.
  • standard math Standard Rademacher complexity, Talagrand contraction, McDiarmid, and Mercer theorem.
    Used without proof beyond citations in Lemmas 3.1, 3.2, and the proof of Theorem 3.2.
invented entities (2)
  • Implicit reference model f(x)=Wφ(x)
    purpose: Object that supposedly receives distilled knowledge from the prompt and is used to predict query outputs.
    The model is defined by the attention equation it is intended to reproduce; no falsifiable prediction outside the paper's own construction is provided.
  • Implicit knowledge distillation process at inference
    purpose: Interpretive mechanism explaining how demonstrations affect ICL.
    No new measurable quantity (e.g., a distillation loss observable in activations) is identified; it is a relabeling of the one-step gradient view.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Brewing Knowledge in Context: Distillation Perspectives on In-Context Learning." pith.science (2026). https://pith.science/paper/VKIBJWX4

@misc{pith2026250611516,
  author       = {Pith},
  title        = {Pith review of: Brewing Knowledge in Context: Distillation Perspectives on In-Context Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VKIBJWX4}},
  note         = {Machine review of arXiv:2506.11516}
}
read the original abstract

In-context learning (ICL) allows large language models (LLMs) to solve novel tasks without weight updates. Despite its empirical success, the mechanism behind ICL remains poorly understood, limiting our ability to interpret, improve, and reliably apply it. In this paper, we propose a new theoretical perspective that interprets ICL as an implicit form of knowledge distillation (KD), where prompt demonstrations guide the model to form a task-specific reference model during inference. Under this view, we derive a Rademacher complexity-based generalization bound and prove that the bias of the distilled weights grows linearly with the Maximum Mean Discrepancy (MMD) between the prompt and target distributions. This theoretical framework explains several empirical phenomena and unifies prior gradient-based and distributional analyses. To the best of our knowledge, this is the first to formalize inference-time attention as a distillation process, which provides theoretical insights for future prompt engineering and automated demonstration selection.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

54 extracted references · 38 canonical work pages

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 , 2023

    arXiv 2023

  2. [2]

    What learning algorithm is in-context learning? investigations with linear models

    Ekin Aky \"u rek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algorithm is in-context learning? investigations with linear models. In The Eleventh International Conference on Learning Representations , 2023

    work page 2023

  3. [3]

    Transformers as statisticians: Provable in-context learning with in-context algorithm selection

    Yu Bai, Fan Chen, Huan Wang, Caiming Xiong, and Song Mei. Transformers as statisticians: Provable in-context learning with in-context algorithm selection. Advances in neural information processing systems , 36:57125--57211, 2023

    work page 2023

  4. [4]

    On the opportunities and risks of foundation models

    Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 , 2021

    arXiv 2021

  5. [5]

    Dick, Hidenori Tanaka, and Tomer Ullman

    Eric J Bigelow, Ekdeep Singh Lubana, Robert P. Dick, Hidenori Tanaka, and Tomer Ullman. In-context learning dynamics with random binary sequences. In The Twelfth International Conference on Learning Representations , 2024

    work page 2024

  6. [6]

    Rademacher and gaussian complexities: Risk bounds and structural results

    Peter L Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research , 3(Nov):463--482, 2002

    work page 2002

  7. [7]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877--1901, 2020

    1901

  8. [8]

    Palm: Scaling language modeling with pathways

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research , 24(240):1--113, 2023

    2023

Show all 54 references
  1. [9]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human langu...

    2019

  2. [10]

    A survey on in-context learning

    Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Baobao Chang, et al. A survey on in-context learning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages 1107--1128, 2024

    2024

  3. [11]

    In-context learning and gradient descent revisited

    Gilad Deutch, Nadav Magar, Tomer Natan, and Guy Dar. In-context learning and gradient descent revisited. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , ...

    2024

  4. [12]

    Why can gpt learn in-context? language models secretly perform gradient descent as meta-optimizers

    Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Shuming Ma, Zhifang Sui, and Furu Wei. Why can gpt learn in-context? language models secretly perform gradient descent as meta-optimizers. In Findings of the Association for Computational Linguistics: ACL 2023 , pages 4005--4019, 2023

    2023

  5. [13]

    A mathematical framework for transformer circuits

    Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dari...

    2021

  6. [14]

    The evolution of statistical induction heads: In-context learning markov chains

    Ezra Edelman, Nikolaos Tsilivis, Benjamin Edelman, Eran Malach, and Surbhi Goel. The evolution of statistical induction heads: In-context learning markov chains. Advances in Neural Information Processing Systems , 37:64273--64311, 2024

    2024

  7. [15]

    Transformers learn to achieve second-order convergence rates for in-context linear regression

    Deqing Fu, Tian-qi Chen, Robin Jia, and Vatsal Sharan. Transformers learn to achieve second-order convergence rates for in-context linear regression. Advances in Neural Information Processing Systems , 37:98675--98716, 2024

    2024

  8. [16]

    How do transformers learn in-context beyond simple functions? a case study on learning with representations

    Tianyu Guo, Wei Hu, Song Mei, Huan Wang, Caiming Xiong, Silvio Savarese, and Yu Bai. How do transformers learn in-context beyond simple functions? a case study on learning with representations. In The Twelfth International Conference on Learning Representations , 2024

    2024

  9. [17]

    What can transformers learn in-context? a case study of simple function classes

    Shivam Garg, Dimitris Tsipras, Percy S Liang, and Gregory Valiant. What can transformers learn in-context? a case study of simple function classes. Advances in Neural Information Processing Systems , 35:30583--30598, 2022

    2022

  10. [18]

    A theory of emergent in-context learning as implicit structure induction

    Michael Hahn and Navin Goyal. A theory of emergent in-context learning as implicit structure induction. arXiv preprint arXiv:2303.07971 , 2023

    2023 arXiv

  11. [19]

    Distilling the knowledge in a neural network

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 , 2015

    2015 arXiv

  12. [20]

    Lee, Qi Lei, and Benjamin Van Roy

    Hong Jun Jeon, Jason D. Lee, Qi Lei, and Benjamin Van Roy. An information-theoretic analysis of in-context learning. In Forty-first International Conference on Machine Learning , 2024

    2024

  13. [21]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations , 2019

    2019

  14. [22]

    Transformers as algorithms: Generalization and stability in in-context learning

    Yingcong Li, Muhammed Emrullah Ildiz, Dimitris Papailiopoulos, and Samet Oymak. Transformers as algorithms: Generalization and stability in in-context learning. In International conference on machine learning , pages 19565--19594. PMLR, 2023

    2023

  15. [23]

    The closeness of in-context learning and weight shifting for softmax regression

    Shuai Li, Zhao Song, Yu Xia, Tong Yu, and Tianyi Zhou. The closeness of in-context learning and weight shifting for softmax regression. Advances in Neural Information Processing Systems , 37:62584--62616, 2024

    2024

  16. [24]

    Probability in Banach Spaces: isoperimetry and processes

    Michel Ledoux and Michel Talagrand. Probability in Banach Spaces: isoperimetry and processes . Springer Science & Business Media, 2013

    2013

  17. [25]

    Effects of prompt length on domain-specific tasks for large language models

    Qibang Liu, Wenzhe Wang, and Jeffrey Willard. Effects of prompt length on domain-specific tasks for large language models. arXiv preprint arXiv:2502.14255 , 2025

    2025 arXiv

  18. [26]

    Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing

    Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM computing surveys , 55(9):1--35, 2023

    2023

  19. [27]

    On the method of bounded differences

    Colin McDiarmid et al. On the method of bounded differences. Surveys in combinatorics , 141(1):148--188, 1989

    1989

  20. [28]

    James Mercer. Xvi. functions of positive and negative type, and their connection the theory of integral equations. Philosophical transactions of the royal society of London. Series A, containing papers of a mathematical or physical character , 209(441-458):415--446, 1909

    1909

  21. [29]

    Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pag...

    2022

  22. [30]

    Metaicl: Learning to learn in context

    Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. Metaicl: Learning to learn in context. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 2791--2809, 2022

    2022

  23. [31]

    In-context learning and induction heads

    Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. arXiv preprint arXiv:2209.11895 , 2022

    2022 arXiv

  24. [32]

    In-context learning through the bayesian prism

    Madhur Panwar, Kabir Ahuja, and Navin Goyal. In-context learning through the bayesian prism. In The Twelfth International Conference on Learning Representations , 2024

    2024

  25. [33]

    Improving systematic generalization of linear transformer using normalization layers and orthogonality loss function

    Taewon Park and Hyun-Chul Kim. Improving systematic generalization of linear transformer using normalization layers and orthogonality loss function. Mathematics , 12(21):3390, 2024

    2024

  26. [34]

    Fitnets: Hints for thin deep nets

    Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-...

    2015

  27. [35]

    Towards understanding how transformers learn in-context through a representation learning lens

    Ruifeng Ren and Yong Liu. Towards understanding how transformers learn in-context through a representation learning lens. Advances in Neural Information Processing Systems , 37:892--933, 2024

    2024

  28. [36]

    Prompt programming for large language models: Beyond the few-shot paradigm

    Laria Reynolds and Kyle McDonell. Prompt programming for large language models: Beyond the few-shot paradigm. In Extended abstracts of the 2021 CHI conference on human factors in computing systems , pages 1--7, 2021

    2021

  29. [37]

    Improving language understanding by generative pre-training

    Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018

    2018

  30. [38]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research , 21(140):1--67, 2020

    2020

  31. [39]

    Language models are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019

    2019

  32. [40]

    Schema-learning and rebinding as mechanisms of in-context learning and emergence

    Sivaramakrishnan Swaminathan, Antoine Dedieu, Rajkumar Vasudeva Raju, Murray Shanahan, Miguel Lazaro-Gredilla, and Dileep George. Schema-learning and rebinding as mechanisms of in-context learning and emergence. Advances in Neural Information Processing Systems , 36:28785--28804, 2023

    2023

  33. [41]

    Position: Do pretrained transformers learn in-context by gradient descent? In Forty-first International Conference on Machine Learning , 2024

    Lingfeng Shen, Aayush Mishra, and Daniel Khashabi. Position: Do pretrained transformers learn in-context by gradient descent? In Forty-first International Conference on Machine Learning , 2024

    2024

  34. [42]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023

    2023 arXiv

  35. [43]

    Function vectors in large language models

    Eric Todd, Millicent Li, Arnab Sen Sharma, Aaron Mueller, Byron C Wallace, and David Bau. Function vectors in large language models. In The Twelfth International Conference on Learning Representations , 2024

    2024

  36. [44]

    Transformers learn in-context by gradient descent

    Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, Jo \ a o Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. In International Conference on Machine Learning , pages 35151--35174. PMLR, 2023

    2023

  37. [45]

    Uncovering mesa-optimization algorithms in transformers

    Johannes Von Oswald, Maximilian Schlegel, Alexander Meulemans, Seijin Kobayashi, Eyvind Niklasson, Nicolas Zucchet, Nino Scherrer, Nolan Miller, Mark Sandler, Max Vladymyrov, et al. Uncovering mesa-optimization algorithms in transformers. arXiv preprint arXiv:2309.05858 , 2023

    2023 arXiv

  38. [46]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017

    2017

  39. [47]

    The learnability of in-context learning

    Noam Wies, Yoav Levine, and Amnon Shashua. The learnability of in-context learning. In Thirty-seventh Conference on Neural Information Processing Systems , 2023

    2023

  40. [48]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems , 35:24824--24837, 2022

    2022

  41. [49]

    Large language models are implicitly topic models: Explaining and finding good demonstrations for in-context learning

    Xinyi Wang, Wanrong Zhu, Michael Saxon, Mark Steyvers, and William Yang Wang. Large language models are implicitly topic models: Explaining and finding good demonstrations for in-context learning. In Workshop on Efficient Systems for Foundation Models @ ICML2023 , 2023

    2023

  42. [50]

    An explanation of in-context learning as implicit bayesian inference

    Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. In International Conference on Learning Representations , 2022

    2022

  43. [51]

    Can we edit factual knowledge by in-context learning? In The 2023 Conference on Empirical Methods in Natural Language Processing , 2023

    Ce Zheng, Lei Li, Qingxiu Dong, Yuxuan Fan, Zhiyong Wu, Jingjing Xu, and Baobao Chang. Can we edit factual knowledge by in-context learning? In The 2023 Conference on Empirical Methods in Natural Language Processing , 2023

    2023

  44. [52]

    The mystery of in-context learning: A comprehensive survey on interpretation and analysis

    Yuxiang Zhou, Jiazheng Li, Yanzheng Xiang, Hanqi Yan, Lin Gui, and Yulan He. The mystery of in-context learning: A comprehensive survey on interpretation and analysis. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages 14365--1437...

    2024

  45. [53]

    Root mean square layer normalization

    Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in Neural Information Processing Systems , 32, 2019

    2019

  46. [54]

    What and how does in-context learning learn? bayesian model averaging, parameterization, and generalization

    Yufeng Zhang, Fengzhuo Zhang, Zhuoran Yang, and Zhaoran Wang. What and how does in-context learning learn? bayesian model averaging, parameterization, and generalization. In The 28th International Conference on Artificial Intelligence and Statistics , 2025

    2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.