Pith. sign in

REVIEW 4 major objections 6 minor 36 references

Understanding Task Vectors in In-Context Learning: Emergence, Functionality, and Limitations

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper proposes that an injected task vector is a single synthetic demonstration formed as a linear combination of the original ones, so task-vector inference is equivalent to one-shot ICL and can only realize rank-one coefficient…

desk verdict A serious paper with a testable rank-one limitation prediction that deserves review, despite the toy-to-LLM transfer gap. read the letter →

arxiv 2506.09048 v1 pith:B3ROC4YS submitted 2025-06-10 cs.LG

Yuxin Dong , Jiachen Jiang , Zhihui Zhu , Xia Ning This is my paper
classification cs.LG MSC 68T07
keywords taskvectorsin-contextlearninglinear-attentiontransformerslinearcombinationconjecturerank-onelimitationbijectiontaskssaliencyanalysismulti-vectorinjection
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish why task vectors work in in-context learning and where they break. The central claim, called the Linear Combination Conjecture, is that a task vector extracted from a prompt is simply a single synthetic demonstration obtained as a linear combination of the original demonstrations' hidden states. In linear-attention transformers trained on triplet prompts, the authors prove that such vectors emerge at the arrow tokens through a weighted summation mechanism, and they show that injecting the vector is equivalent to running one-shot ICL with that synthetic example. A direct consequence is that task vectors can only represent rank-one coefficient matrices, and the authors confirm on real LLMs that task vectors fail on bijection tasks where the mapping and its inverse are combined, while plain ICL often still works. If the conjecture holds, task-vector methods inherit the expressiveness of a single example and can be improved by injecting several vectors, which the paper demonstrates.

What carries the argument

The load-bearing object is a linear self-attention transformer with triplet-formatted prompts, where each demonstration is written as $x_i \to y_i$ with an arrow token between them. The key mechanism is the decomposition of the positional mixing matrix into three components: embedding concatenation, self magnification, and task-vector formation $\Lambda_4 \otimes \Lambda_5$, where the rows of $\Lambda_4$ weight demonstrations to form $z_{\mathrm{tv}}^i = [\alpha_1 X\beta_i;\, \alpha_2 Y\beta_i]$ at the arrow tokens. This structure carries the argument because it lets the authors read off the effective algorithm the transformer learns: first concatenate each $(x_i, y_i)$ pair, then run a preconditioned gradient-descent step on the coefficient matrix. It also yields the rank-one form of the meta-predictor after task-vector injection, and hence the bijection impossibility result.

What would settle it

Find a single bijection task that is a permutation with cycles longer than 2, so it is neither the identity nor negation, and run task-vector injection on a real LLM: the conjecture predicts near-random performance, so high accuracy on such a task would falsify the universality of the rank-one limitation.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that task vectors are not arbitrary summaries: under the linear-attention model studied, the hidden state at each arrow token after the first layer is exactly a linear combination of the demonstrations, produced by the $\Lambda_4 \otimes \Lambda_5$ component of the positional mixing matrix. Because the last row of the optimal weight matrix $\Lambda_4$ vanishes and the first $n$ rows are orthonormal, the formed vectors are diverse summaries rather than duplicates. When such a vector is injected into a zero-shot prompt, the prompt reduces to a single-token one-shot demonstration, and the meta-predictor is a rank-one matrix $W' = (Y\beta)(X\beta)^\top$. The paper proves that a rank-one matrix can satisfy $y = Wx$ and $x = Wy$ only when $x = y$ or $x = -y$, so task vectors cannot solve general bijection tasks; saliency maps and parameter visualizations in real LLMs show the same aggregation pattern, and experiments confirm the predicted failure and the value of injecting multiple task vectors.

Load-bearing premise

The argument depends on real language models having a hidden-state layout where one half of the representation encodes inputs and the other half encodes outputs, so a task vector has the form $[X\beta; Y\beta]$; if that split does not hold in practice, the predicted failure on bijection tasks could be explained by something else.

Editorial extensions

If this is right

  • Task-vector injection is equivalent to one-shot ICL with a synthetic demonstration, so any task that cannot be learned from one example will in principle resist task vectors.
  • Task vectors are limited to rank-one coefficient matrices, so combining a mapping with its inverse in a bijection task is out of reach, matching experiments on Llama and Pythia models.
  • Because plain ICL keeps working on bijection tasks, the failure is specific to the compression task vectors perform, not to the underlying transformer's in-context capability.
  • Injecting multiple task vectors into few-shot prompts distributes information across arrow tokens and consistently improves accuracy, especially on bijection tasks.
  • In linear models, task vectors emerge as redundancy against token dropout, explaining why they form even when they do not directly reduce the in-context risk.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the conjecture transfers to nonlinear transformers, then any task whose minimal effective mapping has rank greater than one should degrade under single-vector injection; a natural test is composing two independent tasks, such as uppercase transformation followed by a letter shift, in one two-way prompt.
  • The rank-one limitation suggests an inverse probing lens: measuring which tasks task vectors can solve reveals the rank of the internal linearized meta-predictor at the injection layer of a real LLM.
  • The paper's extension to inseparable covariates covers pairwise demonstrations, so whether the triplet task-vector mechanism survives when input and output subspaces are fully mixed in real hidden states remains an open empirical question that could be tested by perturbing the hidden-state partition.
  • One could push the multi-vector idea further: if each injected vector is one synthetic demonstration, then injecting $k$ vectors should behave like $k$-shot ICL, predicting a quantitative accuracy curve that could be compared across models.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes the Linear Combination Conjecture: a task vector injected into a zero-shot ICL prompt acts as a single in-context demonstration formed by a linear combination of the original demonstrations' hidden states. The authors study linear-attention transformers trained on random linear regression with pairwise and triplet prompt formats, characterize critical points of the ICL risk under structured parameterizations, and derive that an injected task vector should produce a rank-one coefficient matrix. They then predict that task vectors cannot solve general bijection tasks, report experiments on Llama and Pythia models consistent with that prediction, and propose a multi-vector injection strategy that yields small but consistent accuracy gains.

Significance. If the conjecture is correct, it offers a unifying mechanistic explanation of task vectors in ICL, connecting them to 1-shot ICL and providing a falsifiable limitation (rank-one expressiveness). The theoretical analysis is carried out under explicit restrictions (eq. (6), one-hot positions, an extra EOS token), and the proofs in Appendix B are detailed and appear internally coherent. The bijection-task design is a clean, concrete falsification test, and the synthetic experiments in Section 6.1 usefully demonstrate that a linear transformer can reproduce the task-vector mechanism. The paper also ships proofs and code in the supplementary materials. However, the transfer from the 2d toy model to real LLMs rests on an asserted hidden-state partition (Appendix D.2), and several claims are stated more strongly than the evidence supports. The central idea is valuable, but the current evidence is partial.

major comments (4)
  1. [Section 4, Eq. (10) and Appendix D.2] The rank-one prediction W' = Yβ(Xβ)^T presupposes that the injected task vector can be written as z_tv = [Xβ; Yβ] in a hidden-state space cleanly split into an input half and an output half. Section 5 explicitly concedes that xi and yi are not generally linearly separable, and the proposed fix in Theorem 5 is proved only for pairwise demonstrations, not for the triplet format in which task vectors are actually formed. Appendix D.2 then simply asserts that 'practical LLMs adopt a similar partition in the hidden state space' without direct evidence. Because the bijection experiments in Table 1 are the main empirical validation of the conjecture, this unverified geometric assumption is load-bearing: the observed failures could be caused by extraction/injection artifacts on overlapping domains rather than by the rank-one limitation. The authors should either provide direct evidence for the partition (e.g., probing or intervention experiments in LLM hidden states) or substantially soften the claim that the bijection failures confirm the rank-one mechanism.
  2. [Section 3.2, Theorem 2] The 'Task Vector Formation' component Λ4 ⊗ Λ5 is included in the admissible parameter set SP by definition, and Theorem 2 establishes existence of critical points within this pre-specified structure. It does not show that gradient-based training dynamics converge to this structure. The phrase 'task vectors naturally emerge' therefore overstates what the theorem proves; the result demonstrates consistency with the ansatz, not emergence from unconstrained optimization. Figure 2b provides supporting evidence, but the theoretical claim should be phrased as a critical-point characterization rather than an emergence result unless convergence is shown.
  3. [Section 4, paragraph after Eq. (10)] The statement that the rank-one limitation 'naturally extends to multi-layer transformers' is not supported. The derivation uses the optimal single-layer formula (4), and linear attention layers that implement GD++ can stack to produce more complex updates; it is not obvious that an L-layer model operating on a rank-one synthetic demonstration must yield a rank-one effective coefficient matrix. Since all LLM experiments use deep models, this extension is load-bearing for the central prediction. The authors should either prove the multi-layer claim under the linear-attention model or test it in their synthetic setup before transferring it to LLMs.
  4. [Table 1] The empirical contrast between ICL and task vectors on bijection tasks is less clean than the text suggests. On several bijection tasks the ICL baseline itself is near chance (e.g., 0.54 for English-French, 0.52 for Present-Past, 0.55 for Present-Past Perfect), while task vectors score near 0.35-0.47. Thus the claim that 'ICL preserves performance in many cases' is overstated, and the conclusion that task vectors 'consistently fail' while ICL succeeds is not supported on all tasks. The data may still be consistent with the rank-one limitation, but the failure pattern also admits alternative explanations, especially given the unverified partition assumption. A more careful statistical comparison and a task-by-task discussion are needed.
minor comments (6)
  1. [Abstract and Section 3] The abstract says task vectors 'naturally emerge' from loss-landscape analysis, but Theorems 1-2 characterize critical points under a restricted parameterization rather than convergence of training. Please adjust the wording to 'arise as critical points' or add a convergence result.
  2. [Section 5, Proposition 7] The proposition is labeled informal in the text but has a full proof in Appendix B.7; it should be presented as a formal proposition with its assumptions clearly stated in the main text, since the construction relies on doubling the hidden dimension and using two heads.
  3. [Section 6.1] The synthetic training details state that Al, Bl, Cl are restricted to SI during training, while the theorems allow Cl in SΣ; for Σ=I the sets coincide, but this should be stated explicitly to avoid confusion.
  4. [Table 1] The table formatting for the Copy task leaves the X-to-Y and Y-to-X cells empty; please add a note explaining that the domain and codomain coincide for this task.
  5. [Section 4, bijection task definition] The term 'bijection tasks' is used for tasks formed by combining a map with its inverse. Please define the domain and codomain precisely for each example, since the proposition uses vectors x, y in R^d whereas the empirical tasks use discrete tokens.
  6. [Appendix D.3] The limitations paragraph is honest but short; it could explicitly mention the unverified hidden-state partition and the pairwise-only Theorem 5 as limitations, since these are the main gaps for the LLM transfer.

Circularity Check

1 steps flagged · score 4.0 of 10

Emergence theorem partly encodes the linear-combination ansatz in SP; the rank-one limitation is tested independently, so circularity is moderate.

  1. self definitional [Section 3.2, Theorem 2 (definition of SP) and the following 'Task Vector Formation' paragraph]
    "SP = {diag(In ⊗ Λ1, Λ2) + In+1 ⊗ Λ3 + Λ4 ⊗ Λ5 | ... } ... Task Vector Formation: The third component, Λ4 ⊗ Λ5, performs a weighted summation across all demonstrations in the prompt. This operation is central to the emergence of task vectors. ... the first self-attention layer then outputs n + 1 linear combinations of the demonstrations as the hidden states for the arrow tokens, expressed as z^i_tv = [α1 Xβ_i; α2 Yβ_i]."

    The 'emergence' theorem does not derive the task-vector formation term from optimization over a general parameter class: the set SP over which inf ∥∇L∥² = 0 is proved is defined to contain the component Λ4 ⊗ Λ5. Therefore the hidden states at arrow tokens being linear combinations of demonstrations is true by construction, and reading them off as support for the Linear Combination Conjecture restates the ansatz in eq. (6)/SP. Proposition 3 does optimize the weights within that fixed structure, and the rank-one/bijection predictions are tested externally, so the circularity is partial rather than total.

full rationale

The paper's main falsifiable content is not circular: the rank-one limitation W' = Yβ(Xβ)^T follows algebraically from the conjecture and eq. (10), and it is then tested on real LLMs with bijection tasks, giving independent grounding. The principal circular element is the theoretical 'emergence' result in Section 3.2: Theorem 2's parameter subspace SP is defined to include the Λ4 ⊗ Λ5 weighted-summation term, so the theorem establishes compatibility with the linear-combination conjecture rather than deriving that structure from scratch. The transfer to practical LLMs additionally rests on an asserted 2d input/output partition (Appendix D.2) rather than evidence, and the inseparable-covariate extension in Theorem 5 covers only pairwise prompts, leaving the triplet transfer weaker; this is a missing-support limitation rather than a definitional circularity, but it compounds the burden. No load-bearing self-citation is present: the cited prior work (e.g., [1], [24]) is independent of the authors. Overall, the central claim still has independent empirical content, so the score is moderate rather than high.

Assumptions & free parameters 0 free parameters · 8 assumptions · 0 invented entities

The theoretical results rely on a restricted linear-attention model with one-hot positional encodings, an extra EOS token, and a 2d-dimensional hidden-state partition. These are realistic only in a toy setting; the transfer to real LLMs is an analogy rather than a derivation. No new physical entities are introduced.

assumptions (8)
  • domain assumption Linear self-attention layers with skip connections (eq. 1) model transformer computation.
    The entire theory is built on linear attention; real transformers use softmax and nonlinear MLPs.
  • ad hoc to paper Parameterization V_l=diag(A_l,B_l), Q_l=diag(C_l,0,D_l) in eq. (6).
    Imposed for tractability; the paper does not prove the unconstrained loss landscape has critical points of this form.
  • ad hoc to paper One-hot position encodings P=I_dp appended to tokens.
    Real position encodings are learned or sinusoidal and typically added, not concatenated as one-hot vectors.
  • ad hoc to paper An extra zero ('EOS') token is appended to prompts in Theorems 1, 2, and 5.
    All main proofs rely on this token; removal is only justified by informal Proposition 7.
  • domain assumption Gaussian data distributions: P_x=N(0,Sigma), P_w=N(0,Sigma^{-1}).
    Standard in the in-context linear regression literature, but specific to the synthetic setting.
  • ad hoc to paper xi and yi embeddings are linearly separable and hidden space is 2d-dimensional.
    Section 5 acknowledges the assumption is unrealistic; Theorem 5 fixes it only for pairwise, not triplet prompts; D.2 assumes the 2d partition for real LLMs.
  • ad hoc to paper C1=0 in Proposition 3 for optimal task vector weights.
    A restrictive condition for the 2-layer analysis; not shown to hold in unconstrained training.
  • domain assumption Token-wise dropout model with Bernoulli variables in Proposition 6.
    A modeling choice to explain task vector emergence under perturbation; dropout is not always applied during LLM pretraining.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Understanding Task Vectors in In-Context Learning: Emergence, Functionality, and Limitations." pith.science (2026). https://pith.science/paper/B3ROC4YS

@misc{pith2026250609048,
  author       = {Pith},
  title        = {Pith review of: Understanding Task Vectors in In-Context Learning: Emergence, Functionality, and Limitations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/B3ROC4YS}},
  note         = {Machine review of arXiv:2506.09048}
}
read the original abstract

Task vectors offer a compelling mechanism for accelerating inference in in-context learning (ICL) by distilling task-specific information into a single, reusable representation. Despite their empirical success, the underlying principles governing their emergence and functionality remain unclear. This work proposes the Linear Combination Conjecture, positing that task vectors act as single in-context demonstrations formed through linear combinations of the original ones. We provide both theoretical and empirical support for this conjecture. First, we show that task vectors naturally emerge in linear transformers trained on triplet-formatted prompts through loss landscape analysis. Next, we predict the failure of task vectors on representing high-rank mappings and confirm this on practical LLMs. Our findings are further validated through saliency analyses and parameter visualization, suggesting an enhancement of task vectors by injecting multiple ones into few-shot prompts. Together, our results advance the understanding of task vectors and shed light on the mechanisms underlying ICL in transformer-based models.

Figures

Figures reproduced from arXiv: 2506.09048 by the authors.

Figure 1
Figure 1. Overview of task vector and our main conjecture. (a) Task vector emerges during ICL as a [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Visualization of learned Dl weights. (a) Pairwise demonstrations yield a block-diagonal structure aligned with Theorem 1. (b) Triplet demonstrations yield a richer structure aligned with Theorem 2. (c) The learned matrix Λ4 has nearly orthonormal rows as suggested by Proposition 3. The following theorem suggests that certain critical points of the in-context risk effectively solve the regression problem by first con… view at source ↗
Figure 3
Figure 3. Visualization of saliency matrices as bipartite graphs between layer [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: (a, b) Comparison of the best ICL risk achieved using single (S), pairwise (P), and triplet [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

36 extracted references · 28 canonical work pages

  1. [1]

    Transformers learn to imple- ment preconditioned gradient descent for in-context learning

    Kwangjun Ahn, Xiang Cheng, Hadi Daneshmand, and Suvrit Sra. Transformers learn to imple- ment preconditioned gradient descent for in-context learning. Advances in Neural Information Processing Systems, 36:45614–45650, 2023

    work page 2023

  2. [2]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020

    1901

  3. [3]

    Data distributional properties drive emer- gent in-context learning in transformers

    Stephanie Chan, Adam Santoro, Andrew Lampinen, Jane Wang, Aaditya Singh, Pierre Richemond, James McClelland, and Felix Hill. Data distributional properties drive emer- gent in-context learning in transformers. Advances in neural information processing systems, 35:18878–18891, 2022

    work page 2022

  4. [4]

    Why can gpt learn in-context? language models secretly perform gradient descent as meta-optimizers

    Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Shuming Ma, Zhifang Sui, and Furu Wei. Why can gpt learn in-context? language models secretly perform gradient descent as meta-optimizers. In Findings of the Association for Computational Linguistics: ACL 2023, pages 4005–4019, 2023

    2023

  5. [5]

    In-context learning and gradient descent revisited

    Gilad Deutch, Nadav Magar, Tomer Natan, and Guy Dar. In-context learning and gradient descent revisited. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 1017–1028, 2024

    work page 2024

  6. [6]

    What can transformers learn in-context? a case study of simple function classes

    Shivam Garg, Dimitris Tsipras, Percy S Liang, and Gregory Valiant. What can transformers learn in-context? a case study of simple function classes. Advances in Neural Information Processing Systems, 35:30583–30598, 2022

    work page 2022

  7. [7]

    In-context learning creates task vectors

    Roee Hendel, Mor Geva, and Amir Globerson. In-context learning creates task vectors. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 9318–9333, 2023

    2023

  8. [8]

    Finding visual task vectors

    Alberto Hojel, Yutong Bai, Trevor Darrell, Amir Globerson, and Amir Bar. Finding visual task vectors. In European Conference on Computer Vision, pages 257–273. Springer, 2024

    work page 2024

Show all 36 references
  1. [9]

    Multimodal task vectors enable many-shot multimodal in-context learning

    Brandon Huang, Chancharik Mitra, Leonid Karlinsky, Assaf Arbelle, Trevor Darrell, and Roei Herzig. Multimodal task vectors enable many-shot multimodal in-context learning. Advances in Neural Information Processing Systems, 37:22124–22153, 2024

    2024

  2. [10]

    Transformers are rnns: Fast autoregressive transformers with linear attention

    Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In International conference on machine learning, pages 5156–5165. PMLR, 2020

    2020

  3. [11]

    The impact of positional encoding on length generalization in transformers

    Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, and Siva Reddy. The impact of positional encoding on length generalization in transformers. Advances in Neural Information Processing Systems, 36:24892–24928, 2023

    2023

  4. [12]

    In-context learning state vector with inner and momentum optimization

    Dongfang Li, Xinshuo Hu, Zetian Sun, Baotian Hu, Min Zhang, et al. In-context learning state vector with inner and momentum optimization. Advances in Neural Information Processing Systems, 37:7797–7820, 2024

    2024

  5. [13]

    In-context vectors: Making in context learn- ing more effective and controllable through latent space steering

    Sheng Liu, Haotian Ye, Lei Xing, and James Y Zou. In-context vectors: Making in context learn- ing more effective and controllable through latent space steering. In International Conference on Machine Learning, pages 32287–32307. PMLR, 2024

    2024

  6. [14]

    Mahankali, Tatsunori Hashimoto, and Tengyu Ma

    Arvind V . Mahankali, Tatsunori Hashimoto, and Tengyu Ma. One step of gradient descent is provably the optimal in-context learner with one layer of linear self-attention. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview. net/for...

    2024

  7. [15]

    Position: Do pretrained transformers learn in-context by gradient descent? In Proceedings of the 41st International Conference on Machine Learning, pages 44712–44740

    Lingfeng Shen, Aayush Mishra, and Daniel Khashabi. Position: Do pretrained transformers learn in-context by gradient descent? In Proceedings of the 41st International Conference on Machine Learning, pages 44712–44740. PMLR, 2024. 10

    2024

  8. [16]

    Function vectors in large language models

    Eric Todd, Millicent Li, Arnab Sen Sharma, Aaron Mueller, Byron C Wallace, and David Bau. Function vectors in large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=AwyxtyMwaG

    2024

  9. [17]

    Transformers learn in-context by gradient descent

    Johannes V on Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. In International Conference on Machine Learning , pages 35151–35174. PMLR, 2023

    2023

  10. [18]

    Label words are anchors: An information flow perspective for understanding in-context learning

    Lean Wang, Lei Li, Damai Dai, Deli Chen, Hao Zhou, Fandong Meng, Jie Zhou, and Xu Sun. Label words are anchors: An information flow perspective for understanding in-context learning. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pag...

    2023

  11. [19]

    On the role of unstructured training data in transformers’ in-context learning capabilities

    Kevin Christian Wibisono and Yixin Wang. On the role of unstructured training data in transformers’ in-context learning capabilities. In NeurIPS 2023 Workshop on Mathematics of Modern Machine Learning, 2023

    2023

  12. [20]

    How many pretraining tasks are needed for in-context learning of linear regression? In The Twelfth International Conference on Learning Representations , 2024

    Jingfeng Wu, Difan Zou, Zixiang Chen, Vladimir Braverman, Quanquan Gu, and Peter Bartlett. How many pretraining tasks are needed for in-context learning of linear regression? In The Twelfth International Conference on Learning Representations , 2024. URL https:// openreview.ne...

    2024

  13. [21]

    An explanation of in-context learning as implicit bayesian inference

    Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=RdJVFCHjUMI

    2022

  14. [22]

    Theoret- ical understanding of in-context learning in shallow transformers with unstructured data

    Yue Xing, Xiaofeng Lin, Chenheng Xu, Namjoon Suh, Qifan Song, and Guang Cheng. Theoret- ical understanding of in-context learning in shallow transformers with unstructured data. arXiv preprint arXiv:2402.00743, 2024

    2024 arXiv

  15. [23]

    Everything everywhere all at once: Llms can in-context learn multiple tasks in superposition

    Zheyang Xiong, Ziyang Cai, John Cooper, Albert Ge, Vasilis Papageorgiou, Zack Sifakis, Ange- liki Giannou, Ziqian Lin, Liu Yang, Saurabh Agarwal, et al. Everything everywhere all at once: Llms can in-context learn multiple tasks in superposition. arXiv preprint arXiv:2410.05603, 2024

    arXiv

  16. [24]

    Task vectors in in-context learning: Emergence, formation, and benefit

    Liu Yang, Ziqian Lin, Kangwook Lee, Dimitris Papailiopoulos, and Robert Nowak. Task vectors in in-context learning: Emergence, formation, and benefit. arXiv preprint arXiv:2501.09240, 2025

    2025 arXiv

  17. [25]

    Trained transformers learn linear models in-context

    Ruiqi Zhang, Spencer Frei, and Peter L Bartlett. Trained transformers learn linear models in-context. Journal of Machine Learning Research, 25(49):1–55, 2024

    2024

  18. [26]

    Position information emerges in causal transformers without positional encodings via similarity of nearby embeddings

    Chunsheng Zuo, Pavel Guerzhoy, and Michael Guerzhoy. Position information emerges in causal transformers without positional encodings via similarity of nearby embeddings. In Proceedings of the 31st International Conference on Computational Linguistics, pages 9418–9430, 2025. 1...

    2025

  19. [27]

    We first show that for any l ∈ [1, L], the following equations hold: Xl(X0 × ← −UΣ) = UΣXl, (16) d dt Xl(X0 × ← −UΣ, Ai + ← −tR) t=0 = UΣ d dt Xl(Ai + ← −tU −1 Σ RUΣ) t=0

    Equation (15) holds for Ai. We first show that for any l ∈ [1, L], the following equations hold: Xl(X0 × ← −UΣ) = UΣXl, (16) d dt Xl(X0 × ← −UΣ, Ai + ← −tR) t=0 = UΣ d dt Xl(Ai + ← −tU −1 Σ RUΣ) t=0 . (17) It is straightforward to verify that eq. (16) holds for l = 0. Now supp...

  20. [28]

    From the recursive expressions in eq

    Equation (15) holds for Bi. From the recursive expressions in eq. (14), we can conclude that the values of Xl do not depend on Bi. Therefore, we naturally have Xl(Bi + ← −tR) = Xl. (20) Next, we would like to show that for any l ∈ [1, L], EW W ⊤ d dt Yl(Bi + ← −tR) t=0 = Σ−1 d...

  21. [29]

    Similar to the Ai case, we will first prove that for any l ∈ [1, L], d dt Xl(X0 × ← −UΣ, Ci + ← −tR) t=0 = UΣ d dt Xl(Ci + ← −tU ⊤ Σ RUΣ) t=0

    Equation (15) holds for Ci. Similar to the Ai case, we will first prove that for any l ∈ [1, L], d dt Xl(X0 × ← −UΣ, Ci + ← −tR) t=0 = UΣ d dt Xl(Ci + ← −tU ⊤ Σ RUΣ) t=0 . (22) The equation above holds trivially for l < i. For the case l = i, we have d dt Xl(X0 × ← −UΣ, Ci + ←...

  22. [30]

    Let Up ∈ Rn×n be a uniformly sampled permutation matrix, i.e., a binary matrix that has exactly one1 entry in each row and column with all other entries0

    Equation (15) holds for Di. Let Up ∈ Rn×n be a uniformly sampled permutation matrix, i.e., a binary matrix that has exactly one1 entry in each row and column with all other entries0. Let U◦ = diag(Up ⊗ I2, I2) ∈ R(2n+2)×(2n+2). 18 One can verify that by multiplying X0U◦, it is...

  23. [31]

    21 One can easily verify that eqs

    Equation (36) holds for Ai. 21 One can easily verify that eqs. (16) and (17) still hold. Furthermore, eqs. (18) and (19) hold for both Fl and Gl. With these observations, we can then verify d dt L(Ai + ← −tR) t=0 = 2 EX0,U⊥ tr (I − M )F ⊤ L (X × ← −UΣ) d dt FL(X × ← −UΣ, Ai + ...

  24. [32]

    From the definition of Fl and Gl, we can verify that d dt Yl(Bi + ← −tR) t=0 = R(Fi−1 + W Gi−1)M (X ⊤ i−1CiXi−1 + Di) lY j=i+1 I + bjM (X ⊤ j−1CjXj−1 + Dj)

    Equation (36) holds for Bi. From the definition of Fl and Gl, we can verify that d dt Yl(Bi + ← −tR) t=0 = R(Fi−1 + W Gi−1)M (X ⊤ i−1CiXi−1 + Di) lY j=i+1 I + bjM (X ⊤ j−1CjXj−1 + Dj) . Define F i l = Fi−1 + BiFi−1M (X ⊤ i−1CiXi−1 + Di) lY j=i+1 I + bjM (X ⊤ j−1CjXj−1 + Dj) , ...

  25. [33]

    Similarly, one can verify that eqs

    Equation (36) holds for Ci, Di. Similarly, one can verify that eqs. (22) and (23) still hold (also eqs. (24) to (27)), and finish the proof by following the same reductions as Theorem 1 with Fl and Gl. B.5 Proof of Proposition 3 Proof. Let Al = alId, Bl = blId, Cl = clId and D...

  26. [34]

    Similarly, we have Y1 = Y0 + b1Y0M diag(In ⊗ D1 1, D2

    + In+1 ⊗ D3 1 + D4 1 ⊗ D5 1 = [ (1 + a1dx x)x1 a1tx Pn+1 i=1 di,1xi a1dy xx1 · · · (1 + a1dx x)xn a1tx Pn+1 i=1 di,nxi a1dy xxn (1 + a1dx x)xtest a1tx Pn+1 i=1 di,n+1xi a1dy xxtest ] . Similarly, we have Y1 = Y0 + b1Y0M diag(In ⊗ D1 1, D2

  27. [35]

    By the definition of linear attention, we can show that TF(Z0; {Vl, Ql}2 l=1) = (Y2)3n+3 = b2Y1M c2X ⊤ 1 (X1)3n+3 + (D2)3n+3 = b2c2a1dy x 3n+2X i=1 (Y1)i(X1)⊤ i ! xtest

    + In+1 ⊗ D3 1 + D4 1 ⊗ D5 1 = [ b1dx yy1 b1ty Pn i=1 di,1yi (1 + b1dy y)y1 · · · b1dx yyn b1ty Pn i=1 di,nyi (1 + b1dy y)yn 0 b1ty Pn i=1 di,n+1yi 0 ] . By the definition of linear attention, we can show that TF(Z0; {Vl, Ql}2 l=1) = (Y2)3n+3 = b2Y1M c2X ⊤ 1 (X1)3n+3 + (D2)3n+3...

  28. [1000]

    For training efficiency and stability, we restrict theAl, Bl, and Cl matrices to SI during training, and initialize Dl ∈ Rdp×dp with i.i.d

    We additionally apply ℓ1 regularization to simplify the found solutions. For training efficiency and stability, we restrict theAl, Bl, and Cl matrices to SI during training, and initialize Dl ∈ Rdp×dp with i.i.d. Gaussian matrices. For each case, we train 40 models with differ...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.