Pith. sign in

REVIEW 4 major objections 5 minor 24 references

A prompt chunk can be condensed into a reusable 'thought vector' and 'thought matrix'; applying them to feed-forward weights reproduces the instruction, explaining both activation steering and low-rank editing from first principles.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 10:41 UTC pith:EB4TWCOQ

load-bearing objection A useful empirical kernel wrapped in a theoretical story the math does not yet support; the low-rank editing derivation hinges on an isotropic-activation assumption that real transformers fail. the 4 major comments →

arxiv 2510.08734 v3 pith:EB4TWCOQ submitted 2025-10-09 cs.LG

Transmuting prompts into weights

Hanna Mazzawi , Benoit Dherin , Michael Munn , Adrian Goldwaser , Michael Wunder , Javier Gonzalvo This is my paper
classification cs.LG
keywords thought vectorsthought matricesactivation steeringmodel editinglow-rank updatesin-context learningtransformer interpretabilityprompt compression
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that a prompt's effect on a transformer is internally realized as implicit, token-dependent weight updates, and that these transient updates can be distilled into a single reusable thought patch. The thought vector is the mean of per-token attention differences; the thought matrix is a sum of rank-one outer products. Applied to selected layers of a 1B Gemma model, such a patch makes the model follow instructions like 'Sum the numbers' or 'Translate to French' without the instruction text at inference time. The paper's central claim is that this gives a first-principles explanation for why two empirical model-control families work: contrastive activation averaging and low-rank matrix editing are both approximations of the same least-squares update law.

Core claim

The central claim is that a prompt chunk I can be condensed into a token-independent thought patch—a thought vector δ(I) and thought matrix Δ(I)—that reproduces the chunk's effect when added to feed-forward bias and weights. δ(I) is the mean of per-token attention differences δ_i = A(I,x_i) − A(x_i). Δ(I) is the least-squares fit of one matrix to all token patches, whose exact solution (Σ_i δ_i a_i^T)Z^{−1}, with Z=Σ_i a_i a_i^T, collapses to λΣ_i δ_i a_i^T if the attention outputs are spherically distributed. Building on the single-block token-patch theorem, the paper extends the construction to deep transformers by applying patches recursively block by block, and argues that contrastive ac

What carries the argument

The carrying object is the thought patch, built from per-token patches. For a context chunk I, each token has a patch (δ_x, Δ_x) that would exactly replicate the block's output with I present. The key identity is the least-squares fit: the optimal single-matrix update is (Σ_i δ_i a_i^T)Z^{−1}, and under the spherical-distribution assumption Z≈nσ²I, so the update reduces to the rank-one sum λΣ_i δ_i a_i^T. This reduction is the bridge to practice: the mean vector explains contrastive averaging of steering vectors, and the rank-one sum explains low-rank model edits.

Load-bearing premise

The load-bearing premise is the spherical-distribution assumption of Section 3.4 (Lemma B.7): attention outputs a_i are taken to be so directionally uniform that Z = Σ_i a_i a_i^T is effectively σ²nI; if real attention activations are too structured for this to hold, the simplified thought-matrix formula λΣδ_i a_i^T is no longer the least-squares optimum, and the deep-stack extension also relies on the two-block sketch of Section 2.2 rather than a proof for arbitrary depth.

What would settle it

Compute Z = Σ_i a_i a_i^T from attention outputs at the edited layers on a diverse prompt corpus and measure how far its eigenvalue distribution is from a multiple of identity; a large condition number or strong directional concentration would violate the spherical assumption that produces the simplified formula. A direct behavioral test: apply the exact least-squares patch (Σδ_i a_i^T)Z^{−1} and the simplified λΣδ_i a_i^T to the same task and check whether the exact patch recovers measurably more of the instruction's effect wherever the simplified patch degrades.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Activation steering is not arbitrary: averaging contrastive activations is the least-squares-optimal token-independent vector for a prompt's effect.
  • Low-rank model editing is native to the architecture because the thought matrix decomposes into rank-one outer products δ_i a_i^T, the same structure used by rank-one editing methods.
  • Vector-only steering is incomplete: a full prompt effect has a matrix component as well, giving a theoretical reason vector-only interventions sometimes underperform direct prompting.
  • Any prompt chunk, not just task prefixes, can in principle be compiled into durable weights, letting prompts act as editable model state.
  • The quality of the compressed patch should improve as the attention-vector sample becomes larger and more isotropic, since that is exactly the regime where the spherical approximation to Z holds.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A diagnostic the paper does not report: computing Z's eigenvalue spread at the edited layers over a diverse corpus should predict where the simplified formula will work, giving a cheap pre-check before any edit.
  • The deep-stack extension is shown only with a two-block sketch; a reader could test it by comparing full-context outputs against patched outputs layer by layer on a deep model.
  • Because the thought matrix is a sum over demonstration tokens, its effective rank is limited by the diversity of attention directions captured, suggesting a direct scaling test between prompt diversity and edit capacity.
  • The framework suggests a natural composition test: if multiple instruction patches are truly additive, two thought patches applied sequentially should implement both instructions at once, which the paper leaves unexamined.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper extends the single-block token-patch theorem of Dherin et al. (2025) to deep transformers and introduces token-independent "thought patches": a thought vector δ(I) = (1/n)Σδ_i and a thought matrix Δ(I) = λΣδ_i a_i^T, where δ_i are per-token activation differences and a_i are attention outputs. The authors claim these constructs provide a first-principles explanation for activation steering (averaging contrastive activations) and low-rank model editing (ROME-style updates), and they support the claim with experiments on arithmetic and machine translation using Gemma 3 1B, editing layers 10–20 via Algorithm 1.

Significance. If the theoretical derivation were sound, the paper would unify two important families of empirical LLM control methods and provide a principled way to convert prompts into weight edits. The paper makes a clear and falsifiable algorithmic proposal, and the experiments, though small, give some evidence that rank-one weight updates can partially reproduce instruction following. However, the central theoretical claim—that the derived update law is the least-squares solution of the stated minimization problem—is not established. The step from Theorem 3.1's exact minimizer to the practical formula rests on an unsupported isotropy assumption that fails in the paper's own setting. The paper also explicitly acknowledges a performance gap and hyperparameter sensitivity, which limits the strength of the empirical support. With a corrected or substantially weakened theoretical claim, the work could still be a useful contribution, but as it stands the claimed first-principles derivation is not convincing.

major comments (4)
  1. [§3.4, Lemma B.7, Eq. (8)] The transition from the exact minimizer M=(Σδ_i a_i^T)Z^{-1} in Theorem 3.1 to the practical formula Δ(I)=λΣδ_i a_i^T is the core of the paper's claimed unification. The only justification is Lemma B.7, which requires the attention vectors a_i to be i.i.d. draws from a spherical distribution and n large enough for (1/n)Σa_i a_i^T≈σ²I. Both requirements are impossible in the paper's own setting: transformer activations are highly structured and correlated, and even under the i.i.d. spherical assumption, a single prompt has n≈10–100 tokens while d≈2000+, so Z has rank at most n and cannot converge to σ²I in spectral norm. Lemma B.7's law-of-large-numbers argument requires n≫d, which never holds for a single prompt. Consequently Δ(I)=λΣδ_i a_i^T is not the solution of optimization problem (7); the derivation in §3.4 collapses and Section 3.5's 'theoretical justification' for ROME-style low-
  2. [§3.2] The paper proposes to approximate the output error by 'minimizing the error for the vector and matrix components independently,' but never defines the actual end-to-end objective. The thought matrix is obtained by minimizing Σ_i∥M a_i − Δ_i a_i∥² (Eq. 7), whereas the quantity that matters is the difference between the outputs of the full block, f_W g_θ((W+Δ)a_i+b)+... . No argument is given that the component-wise least-squares solution bounds or approximates the activation-output error. Since the feedforward map g_θ is nonlinear, the two objectives can be very different. This is a load-bearing gap in the derivation of δ(I) and Δ(I); the paper should state the true objective and prove an approximation bound, or weaken the claim that the thought patch is a least-squares approximation of the prompt's computational effect.
  3. [§2.2, Eqs. (5)–(6)] The extension from the single-block theorem of [1] to deep transformers is presented only as a two-block calculation. The key claim 'the principle extends to any model depth' is not proved; no induction argument is given, and the notation conflates activations under the full context C^(0) with activations in the patched, context-free path. In particular, Eq. (5) defines the patch for block i using activations from the previous block 'in the presence of the full context,' but the recursive substitution on the following lines applies patches sequentially to the patched context-free path. It is not clear that the patches computed with full-context activations remain correct after earlier layers have been patched. Since Algorithm 1 applies thought patches to layers 10–20 independently, this composition issue is load-bearing; a precise multi-layer theorem with hypotheses is needed.
  4. [§4, Tables 1–3, Algorithm 1] The empirical evaluation is too weak to support the central claims. Accuracy is reported on batches of only 10 randomly chosen inputs, with no variance, no multiple seeds, and no statistical test. The hyperparameters c1 and c2 are tuned per task (c1=0.015 for addition, 0.03 for translation, c2=0.0 vs 0.2), and the text acknowledges strong sensitivity to c1 and a performance gap versus direct prompting. Because λ in the theoretical formula is not predicted by the theory and is tuned, the experiments cannot validate the derivation of Δ=λΣδ_i a_i^T. At best they demonstrate that a rank-one update with tuned scale can partially mimic an instruction; this is consistent with the prior heuristic literature but does not address the theoretical gap in §3.4. A more substantial evaluation would be needed to support the 'viability' conclusion.
minor comments (5)
  1. [Algorithm 1, line 22] The line 'δb_l ← ∆δb_l/s' appears to be a typo; likely 'δb_l ← δb_l/s'.
  2. [Algorithm 1 vs §4.1] Algorithm 1 line 16 uses δ_i a_i^T without normalization, while §4.1 says 'we also normalize the rank-one updates by the norm of the attention vector, summing δ_i a_i^T/∥a_i∥².' The pseudocode and text disagree.
  3. [Appendix B.2, Lemma B.2 proof] In the proof, 'span(y_1,...,y_n)=R^n' should read R^d; the notation is inconsistent.
  4. [Appendix A.2, Eq. (22)] The expansion (I+ε^{-1}Z)^{-1} ε^{-1} ≃ (I−ε^{-1}Z)ε^{-1} requires ∥ε^{-1}Z∥<1; the condition is not stated. Without this condition, the approximation is not justified.
  5. [§4.1, text] The sentence 'Gemma 3.0 of size 1B' is ungrammatical and should be 'Gemma 3, 1B size.' Also, the selection of layers 10–20 is not explained or justified.

Circularity Check

2 steps flagged

Thought-matrix 'derivation' replaces Z^-1 with a tunable λI and then claims to explain ROME; the thought-vector 'explanation' is the chosen squared-error loss.

specific steps
  1. self definitional [Section 3.3 and Section 3.5]
    "we can find the optimal thought vector by minimizing the squared error against all the individual token vectors (δi) derived from that collection. The solution to this minimization problem is simply the mean of all the token vectors: δ(I) := 1/n Σ δi ... Therefore, the common heuristic of averaging contrastive activations [4] is not arbitrary, but is in fact the correct least-squares approximation for a single, token-independent vector that captures the prompt's instructional content."

    The paper's 'theoretical explanation' of activation averaging is that the mean of the per-token differences δ_i minimizes a squared-error loss. But the loss was defined by the authors in exactly the form whose minimizer is the mean. Since the empirical steering heuristic is also the mean of activation differences, the derived thought vector is the heuristic restated as the minimizer of an objective chosen to produce it. The explanation is therefore built into the definition of the objective; no independent reason from transformer mechanics is given for why squared error is the fundamental loss.

  2. fitted input called prediction [Section 3.4 (Theorem 3.1, Eq. (8), final approximation), Appendix A.1/A.2, Algorithm 1, Section 3.5]
    "Now the inverse of Z is computationally difficult to calculate in general. In Appendix A.1, we show that a practical simplification arises if we assume that vectors ai are spherically distributed for instance. In this case, Z is proportional to the identity matrix, which motivates our final approximation: Δ(I) := λ Σ δ_i a_i^T. Here, λ is a tunable hyperparameter."

    Theorem 3.1's exact minimizer is (Σδ_i a_i^T)Z^{-1}. The paper replaces Z^{-1} by a scalar λI using Lemma B.7, which requires i.i.d. spherical attention vectors with n large enough that (1/n)Σ a_i a_i^T ≈ σ²I. Real attention outputs are structured and n≈10-100 while d≈1000, so Z has rank at most n and this approximation is unsupported; the same rank-one-sum form is then claimed to explain ROME-style low-rank editing. λ is not predicted but tuned as c1/c2 in Algorithm 1, so the central 'theoretical justification' of low-rank edits reduces to a fitted ansatz rather than a first-principles derivation.

full rationale

The derivation chain starts from the single-block token-patch theorem of [1], which is a separate parameter-free theorem by overlapping authors; reliance on it is a self-citation but not by itself a circular step under the stated rules. The first genuine circularity is in the thought vector: the squared-error objective is selected so that its closed-form solution is the mean of the δ_i's, and the paper then presents the empirical averaging heuristic as 'the correct least-squares approximation.' That makes the explanation equivalent to the definition of the objective. The more serious issue is the thought matrix: the exact solution (Σδ_i a_i^T)Z^{-1} is replaced by λΣδ_i a_i^T on the basis of a spherical-distribution law-of-large-numbers approximation that is invalid in the paper's own regime (n≪d, structured activations). The scalar λ is a tunable hyperparameter, tuned per task as c1/c2 in Algorithm 1, and the claimed low-rank structure that is said to explain ROME is exactly the ansatz inserted at this step. Thus the central first-principles result for matrix editing reduces to a fitted heuristic; the two-block stacking sketch is an unproven generalization but not a circular step. Overall score 6: partial circularity, with the thought-matrix prediction reducing by construction.

Axiom & Free-Parameter Ledger

5 free parameters · 6 axioms · 2 invented entities

The central derivation rests on the prior exact-replication theorem of the same authors (ref [1]), a spherical-activation assumption, a component-wise loss simplification, and a fitted scale λ. The free parameters are all tuned per task, and no code or data is shipped, so the 'derivation' is not independently checkable.

free parameters (5)
  • thought matrix scale λ (c1 in Algorithm 1) = c1=0.015 (addition), c1=0.03 (translation); effective c1 0–0.03 across steps
    Section 3.4 replaces Z^{-1} with scalar λ and calls λ 'a tunable hyperparameter'; Algorithm 1 multiplies the accumulated rank-one updates by c1, tuned per task.
  • thought vector coefficient c2 = c2=0.0 (arithmetic), c2=0.2 (translation)
    Controls the bias-update strength from the mean token vector; tuned per task (§4.2–4.3).
  • implicit schedule constant 300 = 300
    Section 4.1: 'we divide by a large, fixed constant (300)' instead of step count, which grows the effective c1 over tokens; chosen by hand.
  • selected layer range 10–20 = layers 10–19 of Gemma 3 1B
    Algorithm 1 targets layers 10–20 only, whereas the theory states patches must be applied to every block; the subset is a design choice not derived.
  • spherical variance σ² of attention vectors = not estimated; folded into λ
    Lemma B.7 gives Z ≈ nσ²I, but σ² is never measured; the 1/(nσ²) scale is absorbed into the fitted λ.
axioms (6)
  • domain assumption Single-block token-patch theorem of [1]: removing a chunk can be exactly compensated by per-token vector/matrix weight updates.
    Stated and used throughout §2; not proved here, and ref [1] is by the same five authors.
  • domain assumption Transformer block has the simplified form T(C,x)=f_W g_θ(W A(C,x)+b)+b̃+A(C,x).
    Assumed in §2.1; real Gemma blocks include norms/residuals, so the theorem's exactness in practice is taken on faith.
  • ad hoc to paper Attention vectors a_i are spherically distributed.
    Section 3.4 uses this to conclude Z∝I; Lemma B.7 is only a limit result for spherical samples, with no evidence it holds for real activations.
  • ad hoc to paper Minimizing vector and matrix component errors independently approximates minimizing the activation-output error.
    Section 3.2 states this simplification without justification.
  • domain assumption Token patches compose recursively across stacked blocks, yielding exact replication of the deep stack.
    Section 2.2 provides a two-block chain of equalities and asserts the principle extends to any depth; no full proof is given.
  • ad hoc to paper The O(λ²) correction in the Z^{-1} expansion is negligible.
    Appendix A.2/Remark 3.2 derive Δ=λΣδ_i a_i^T − λ²Σ⟨a_i,a_j⟩δ_i a_j^T + ...; the quadratic term is dropped without showing it is small.
invented entities (2)
  • thought vector δ(I) no independent evidence
    purpose: A token-independent bias update that durably encodes an instruction; claimed theoretical analogue of steering vectors (§3.3).
    Defined as the mean of token patches; no falsifiable handle outside the paper's own construction. Its existence is demonstrated only by the paper's behavior-change experiments.
  • thought matrix Δ(I) no independent evidence
    purpose: A token-independent low-rank weight update encoding an instruction; claimed theoretical analogue of matrix-editing methods like ROME (§3.4).
    Defined via λΣδ_i a_i^T with fitted λ; no external prediction beyond the paper's own experiments; its rank-one structure matches existing methods by construction.

pith-pipeline@v1.3.0-alltime-deepseek · 14256 in / 17181 out tokens · 143189 ms · 2026-08-04T10:41:58.034794+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Transmuting prompts into weights." pith.science (2026). https://pith.science/paper/EB4TWCOQ

@misc{pith2026251008734,
  author       = {Pith},
  title        = {Pith review of: Transmuting prompts into weights},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EB4TWCOQ}},
  note         = {Machine review of arXiv:2510.08734}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

A growing body of research has demonstrated that the behavior of large language models can be effectively controlled at inference time by directly modifying their internal states, either through vector additions to their activations or through updates to their weight matrices. These techniques, while powerful, are often guided by empirical heuristics, such as deriving ``steering vectors'' from the average activations of contrastive prompts. Building on the foundational work of Dherin et al. (2025), who discovered that a prompt's influence mathematically maps to token-dependent implicit weight updates and introduced the initial concept of a static thought patch for prompt compression, we elevate this framework into a robust algorithm for direct model editing. We derive a principled method for condensing this transient information into token-independent thought vectors and thought matrices. These constructs provide a theoretical explanation for existing vector-and-matrix-based model editing techniques and offer a direct, computationally-grounded method for transmuting textual input into reusable weight updates for complex architectures and new knowledge injection.

Figures

Figures reproduced from arXiv: 2510.08734 by Adrian Goldwaser, Benoit Dherin, Hanna Mazzawi, Javier Gonzalvo, Michael Munn, Michael Wunder.

Figure 1
Figure 1. Figure 1: Summation on the left, and multiplication on the right. Accuracy (left Y-axis) given a step [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Accuracy when applying ∆Ws and δb during the various steps. Vanilla Gemma with instructions achieves 0.72 accuracy based on the same Gemini model evaluator. 8 [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

24 extracted references · 4 linked inside Pith

  1. [1]

    Learning without training: The implicit dynamics of in-context learning.arXiv preprint arXiv:2507.16003, July 2025

    Benoit Dherin, Michael Munn, Hanna Mazzawi, Michael Wunder, and Javier Gonzalvo. Learning without training: The implicit dynamics of in-context learning.arXiv preprint arXiv:2507.16003, July 2025

    Pith/arXiv arXiv 2025

  2. [2]

    Nishant Subramani, Nivedita Suresh, and Matthew E. Peters. Extracting latent steering vectors from pretrained language models. InFindings of the Association for Computational Linguistics: ACL 2022, pages 566–581, 2022

    2022

  3. [3]

    Inference- time intervention: Eliciting truthful answers from a language model.arXiv preprint arXiv:2306.03341, 2023

    Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference- time intervention: Eliciting truthful answers from a language model.arXiv preprint arXiv:2306.03341, 2023. NeurIPS 2023 spotlight

    Pith/arXiv arXiv 2023

  4. [4]

    Steering language models with activation engineering, 2025

    Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J Vazquez, Ulisse Mini, and Monte MacDiarmid. Steering language models with activation engineering, 2025

    2025

  5. [5]

    Function vectors in large language models

    Eric Todd, Millicent Li, Arnab Sen Sharma, Aaron Mueller, Byron C Wallace, and David Bau. Function vectors in large language models. InThe Twelfth International Conference on Learning Representations, 2024

    2024

  6. [6]

    Locating and editing factual associations in GPT

    Kevin Meng, David Bau, Alex J Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors,Advances in Neural Information Processing Systems, 2022

    2022

  7. [7]

    Fast model editing at scale

    Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. Fast model editing at scale. InInternational Conference on Learning Representations, 2022

    2022

  8. [8]

    Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, J

    Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, J. Zico Kolter, and Dan Hendrycks. Representation engineering: A top-down approach to a...

    2023

  9. [9]

    A unified understanding and evaluation of steering methods.ArXiv, 2025

    Shawn Im and Yixuan Li. A unified understanding and evaluation of steering methods.ArXiv, 2025

    2025

  10. [10]

    In-context learning creates task vectors

    Roee Hendel, Mor Geva, and Amir Globerson. In-context learning creates task vectors. InThe 2023 Conference on Empirical Methods in Natural Language Processing, 2023

    2023

  11. [11]

    Analysing the generalisation and reliability of steering vectors

    Daniel Chee Hian Tan, David Chanin, Aengus Lynch, Brooks Paige, Dimitrios Kanoulas, Adrià Garriga-Alonso, and Robert Kirk. Analysing the generalisation and reliability of steering vectors. InAdvances in Neural Information Processing Systems, volume 37, 2024

    2024

  12. [12]

    Towards reliable evaluation of behavior steering interventions in llms.ArXiv, abs/2410.17245, 2024

    Itamar Pres, Laura Ruis, Ekdeep Singh Lubana, and David Krueger. Towards reliable evaluation of behavior steering interventions in llms.ArXiv, abs/2410.17245, 2024

    Pith/arXiv arXiv 2024

  13. [13]

    Comparing bottom-up and top-down steering approaches on in-context learning tasks.ArXiv, 2024

    Madeline Brumley, Joe Kwon, David Krueger, Dmitrii Krasheninnikov, and Usman Anwar. Comparing bottom-up and top-down steering approaches on in-context learning tasks.ArXiv, 2024

    2024

  14. [14]

    Task vectors in in-context learning: Emergence, formation, and benefit, 2025

    Liu Yang, Ziqian Lin, Kangwook Lee, Dimitris Papailiopoulos, and Robert Nowak. Task vectors in in-context learning: Emergence, formation, and benefit, 2025

    2025

  15. [15]

    Transformer feed-forward layers are key-value memories, 2021

    Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories, 2021

    2021

  16. [16]

    Editing factual knowledge in language models, 2021

    Nicola De Cao, Wilker Aziz, and Ivan Titov. Editing factual knowledge in language models, 2021

    2021

  17. [17]

    Assessing the brittleness of safety alignment via pruning and low-rank modifications

    Boyi Wei, Kaixuan Huang, Yangsibo Huang, Tinghao Xie, Xiangyu Qi, Mengzhou Xia, Prateek Mittal, Mengdi Wang, and Peter Henderson. Assessing the brittleness of safety alignment via pruning and low-rank modifications. InICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models, 2024. 11

    2024

  18. [18]

    Model editing as a robust and denoised variant of DPO: A case study on toxicity

    Rheeya Uppaal, Apratim Dey, Yiting He, Yiqiao Zhong, and Junjie Hu. Model editing as a robust and denoised variant of DPO: A case study on toxicity. InNeurips Safe Generative AI Workshop 2024, 2024

    2024

  19. [19]

    Editing models with task arithmetic

    Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Ha- jishirzi, and Ali Farhadi. Editing models with task arithmetic. InThe Eleventh International Conference on Learning Representations, 2023

    2023

  20. [20]

    What can transformers learn in-context? a case study of simple function classes

    Shivam Garg, Dimitris Tsipras, Percy Liang, and Gregory Valiant. What can transformers learn in-context? a case study of simple function classes. InNeurIPS, 2022

    2022

  21. [21]

    Gemma 3 Technical Report.arXiv preprint arXiv:2503.19786, 2025

    Gemma Team and other authors listed in the paper. Gemma 3 Technical Report.arXiv preprint arXiv:2503.19786, 2025. 12 A Thought Matrix Estimation Theorem In Section 3.4, we introduced the thought matrix representing the thought expressed in a chunk I of a prompt as the matrix ∆(I) that that minimizes the errors ∥∆(I)a i −∆ iai∥2 for all completions [I, x1,...

    Pith/arXiv arXiv 2025

  22. [22]

    , yn is a basis of the space (which implies that n=d ), then the inverse takes the form Z= X i ωiωT i , where the vectors ωi are the rows of Y −1

    When y1, . . . , yn is a basis of the space (which implies that n=d ), then the inverse takes the form Z= X i ωiωT i , where the vectors ωi are the rows of Y −1. This means that Z −1 = (Y T )−1Y −1 and that the ωi are the co-vector basis associated with the basis y1, . . . , yn (i.e. ωT i yi =δ ij where δij is the Kronecker delta. (See Lemma B.3.)

  23. [23]

    , yn is an orthonormal basis (n=d) of the space Z −1 =I

    Wheny 1, . . . , yn is an orthonormal basis (n=d) of the space Z −1 =I. (See Lemma B.4.)

  24. [24]

    , yn are vectors independently sampled from a spherical distribution, then for nlarge enough Z −1 = 1 σ2n I, whereσ 2 is the distribution variance

    When y1, . . . , yn are vectors independently sampled from a spherical distribution, then for nlarge enough Z −1 = 1 σ2n I, whereσ 2 is the distribution variance. (See Lemma B.7.) 14 A.2 Getting an approximation ofMwhenZis not invertible We are seeking operators M such that when applied to the yi’s they give back the δi’s as closely as possible. That is, ...