Pith. sign in

REVIEW 3 major objections 4 minor 52 references

Adaptive Task Vectors for Large Language Models

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that a frozen large language model can be steered per query by injecting a task vector generated from a small language model, and that this adaptive vector is as expressive as LoRA under matched rank and more expressive…

desk verdict Useful empirical method with an unsupported theoretical wrapper: the ATV-to-LoRA equivalence proof has an input-dependent LoRA factor and a rank argument that does not establish the claim. read the letter →

arxiv 2506.03426 v1 pith:EAXJUERD submitted 2025-06-03 cs.LG cs.CL

Joonseong Kang , Soojeong Lee , Subeen Park , Sumin Park , Taero Kim , Jihee Kim , Ryunyi Lee , Kyungwoo Song This is my paper
classification cs.LGcs.CL
keywords adaptivetaskvectorsin-contextlearningfrozenlanguagemodelsparameter-efficientadaptationLoRAprefix-tuningvectorinjectionfew-shotgeneralization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Adaptive Task Vectors (ATV) is a method for making a frozen large language model adapt to each individual query without writing demonstrations into the prompt. A small language model reads the query, produces a compact vector, and a linear layer expands it into layer-wise steering signals that are added to the hidden states of the large model. The authors argue this is not merely a heuristic: with matched rank budgets it realizes the same class of low-rank additive updates as LoRA, and under a linear attention approximation it contains every attention behavior Prefix-Tuning can produce. Empirically, ATV matches or beats 16-shot in-context learning on average across twenty in-domain tasks and five unseen tasks while using the same token count as zero-shot prompts. If the claims hold, frozen LLMs can be adapted to new tasks cheaply and per input, without prompt engineering or weight updates.

What carries the argument

The central object is the adaptive task vector $v_{\mathrm{ATV}} = f_\theta(v_{\mathrm{small}})$, a bank of layer-wise steering vectors produced from the small model's last-token hidden state and added to the frozen large model's last-token hidden states as $\tilde h^l = h^l + \lambda v^l_{\mathrm{ATV}}$. It carries the argument because it is a rank-$d_s$ additive perturbation: the proof of Theorem 1 rewrites it as a LoRA update with down-projection $M = x^{+\top}(\lambda v_{\mathrm{small}})$ and up-projection $A_\ell$, and the proof of Theorem 2 decomposes linear attention into eight terms, showing ATV contains Prefix-Tuning's two terms plus six query/key/value interaction terms.

What would settle it

The equivalence claim would be falsified by finding an input-dependent ATV update that no static LoRA pair of rank $r=d_s$ can reproduce on a given query set; a concrete check is to train both on the same data and then test whether one fixed $(W_{\mathrm{down}}, W_{\mathrm{up}})$ matches ATV's per-query hidden-state perturbations. The Prefix-Tuning containment would be falsified by a softmax attention pattern that prefix-tuning can produce but ATV cannot, since the proof currently relies on the linear approximation $QK^\top V$.

Watch

Extended reading notes

Core claim

The paper's central claim is that an input-conditioned additive perturbation of the last-token hidden states, computed as $\tilde h^l = h^l + \lambda f_\theta(v_{\mathrm{small}})^l$, can replace both in-context demonstrations and fixed task vectors. The theoretical argument has two parts: under the rank match $r = d_s$, every ATV update can be rewritten as a LoRA update with down-projection $M = x^{+\top}(\lambda v_{\mathrm{small}})$ and vice versa, so the two are expressively equivalent; and under the linear attention approximation $\mathrm{Attn}(Q,K,V) \approx QK^\top V$, the attention output of Prefix-Tuning is contained in that of ATV, with six additional cross terms giving ATV extra query, key, and value degrees of freedom. The empirical claim is that this per-query steering outperforms fixed task-vector methods, 16-shot ICL, and LoRA on twenty in-domain tasks and five unseen tasks, with the token cost of zero-shot prompting.

Load-bearing premise

The load-bearing premise is that attention can be treated as the linear map $QK^\top V$ and that a matrix built from the current input can be treated as a fixed LoRA weight; if either simplification is not faithful, the formal expressivity claims are not established.

Editorial extensions

If this is right

  • On both LLaMA3-8B and Mistral-7B, ATV reaches the highest average accuracy among all compared methods while using roughly 108 tokens per query, versus roughly 1,900 to 2,700 tokens for 16-shot prompting.
  • Any behavior that Prefix-Tuning can produce on the frozen attention is also producible by ATV, so ATV inherits prefix-tuning's uses while adding query-side modulation that prefix-tuning cannot express.
  • ATV generalizes to tasks it never trained on, because the steering signal is generated from the query rather than retrieved from a fixed capability library.
  • The comparison with LoRA (in-domain 62.1 versus 56.0; unseen 63.4 versus 52.0 on LLaMA3) is consistent with the claim that input-dependent updates add an adaptability that static low-rank factors lack.
  • Because the large model stays frozen, ATV can be swapped by retraining only the small generator and the linear expansion, making it a lightweight route to per-task or per-user adaptation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable consequence the paper leaves implicit: ATV's margin over fixed task vectors should grow as within-task queries become more heterogeneous; on a homogeneous task, a single static vector should recover most of ATV's gain.
  • The same hidden-state injection mechanism could steer properties other than task accuracy, such as tone, style, or refusal behavior, since it modifies internal activations rather than the answer distribution directly.
  • The Prefix-Tuning containment is proven only under a linear attention approximation; a useful stress test is whether the advantage survives on inputs where softmax attention is sharply peaked and the linear model is least faithful.
  • The capacity ablation (137M versus 1.6B generator) hints that the expansion layer, not the encoder, may be the bottleneck; an even smaller encoder could make the per-query overhead nearly negligible.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes Adaptive Task Vectors (ATV), a framework in which a small language model (GPT-2 in the experiments) produces a query-specific vector that is linearly expanded and additively injected into the hidden states of a frozen large language model at every layer. The authors claim that ATV generalizes better than fixed task-vector methods and ICL, and they provide two formal results: Theorem 1 states that ATV is expressively equivalent to LoRA under matched rank budgets, and Theorem 2 states that ATV is more expressive than Prefix-Tuning under a linear attention approximation. The empirical sections report strong average accuracy on 20 in-domain tasks and 5 unseen tasks for LLaMA3-8B and Mistral-7B, along with ablations on generator capacity, injection depth, and a comparison to LoRA.

Significance. If the formal claims were sound, ATV would provide a principled, token-free way to make frozen LLMs input-adaptive, with a clear representational comparison against two influential parameter-efficient methods. The empirical study is a genuine strength: it includes held-out unseen tasks, multiple backbones, test-time prompt-template variation, and a public code release, and the layer-wise and t-SNE analyses are informative. However, the theoretical backbone is a central advertised contribution, and the two theorems as stated are not established; the LoRA comparison in Table 5 also does not test the theorem's equal-rank condition. The empirical results may survive a corrected theory, but the paper's current formal contribution cannot be accepted.

major comments (3)
  1. [Section 3.3 / Appendix A.1.3, Theorem 1] The proof of the ATV-to-LoRA direction is not a valid LoRA construction. The theorem requires static LoRA factors, but Step 3 sets W_down := M := x^{+T}(λ v_small), which depends on the current input activation x; the paper itself concedes in Section 3.3 that this matrix 'unlike LoRA, varies with the current activation x.' A LoRA configuration with input-dependent W_down is not the object defined in the theorem or in standard LoRA. In addition, Step 1's rank bound is incorrect: Δh_ATV = λ v_small A^ℓ is a 1×ds row vector times a ds×dℓ matrix, so rank(Δh_ATV) ≤ 1, not ≤ ds. Consequently ATV cannot realize a generic rank-ds LoRA update, and the claimed equivalence under equal rank budgets is not merely unproven; as stated it is false. This invalidates the sentence in Section 3.3 that 'ATV inherits LoRA's expressiveness.'
  2. [Appendix A.2, Theorem 2] The containment proof for F(Attn_prefix) ⊆ F(Attn_ATV) is incomplete and rests on an unjustified identification. The terms P'_k and P'_v are defined as (e_m · (v_l_ATV)^T)W_k and (e_m · (v_l_ATV)^T)W_v, so their rows are all identical and both are determined by the same vector v_l_ATV through fixed projections. General prefix matrices P_k, P_v ∈ R^{p×dl} with p > 1 and rank greater than one cannot be represented in this form. The derivation in A.2.3 expands the ATV attention into eight terms and asserts that T2 exactly matches the prefix term and that T3–T8 are 'not representable in PT,' but no construction maps an arbitrary prefix-tuning configuration (P_k, P_v) to an ATV vector v_l_ATV that produces the same output. Therefore the claimed containment is unsupported, and the statement that ATV 'can emulate PT exactly' does not follow from the arguments given.
  3. [Table 5 / Appendix B.2] The empirical LoRA comparison does not test the theoretical claim. Theorem 1 assumes a LoRA rank r = ds = 768, but the LoRA configuration in Appendix B.2 uses rank r = 8 with scaling α = 32. With ds = 768, ATV can move within a much larger subspace than a rank-8 LoRA update, and the trainable parameter counts are also very different (the ATV generator plus expansion is about 237M parameters, while rank-8 LoRA on Q and V projections is far smaller). The caption's statement that the results are 'consistent with our theoretical analysis' is therefore not justified, because the experiment does not instantiate the equal-rank condition under which equivalence was claimed.
minor comments (4)
  1. [Section 4.2 / Table 1] The paper emphasizes token efficiency, but token count alone does not capture inference cost: ATV requires an additional forward pass through GPT-2 and a large linear expansion per query. The efficiency claim should be qualified with wall-clock time or FLOPs comparisons.
  2. [Section 4.5 / Table 4] Table 4 reports an ELICIT 'All Layers' accuracy of 30.9, whereas Table 1 reports an ELICIT average of 43.5. The text explains that ELICIT normally uses per-task optimal injection layers, but the relationship between the two tables should be stated explicitly to avoid the appearance of inconsistency.
  3. [Appendix A.1.4] The discussion of the ds > r case concludes that equivalence holds 'only up to rank-r projection,' which conflicts with the unconditional phrasing of Theorem 1. The theorem statement and the discussion should be reconciled.
  4. [Appendix A.2.3] The derivation contains typographical errors, including an unmatched parenthesis in the linearized prefix attention expression. Please proofread the appendix carefully.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the theoretical proof gaps are correctness issues, not circular dependencies.

full rationale

ATV's central empirical claims are not circular: the small generator and expansion module are trained on in-domain tasks with cross-entropy on the frozen LLM's output, and evaluation is reported on held-out unseen tasks and on template variants not used in training; no parameter is fitted to the evaluation metric. The theoretical sections contain the only by-construction flavor, but it is a proof-validity problem rather than circularity. In Theorem 1 (Appendix A.1.3, Step 3), the ATV-to-LoRA direction chooses M := x^{+T}(\lambda v_small) so that x^T M = \lambda v_small, making the equality \Delta h_LoRA = \Delta h_ATV an algebraic identity for that M; however, this M depends on the runtime query, contradicting the theorem's own static-W_down requirement, and Step 1's rank bound is ≤1, not ≤ds. Those are internal mathematical flaws that can be checked directly, not a case where a prediction reduces to its fitted input. Theorem 2 also relies on the external linear-attention approximation of [28] and its prefix-containment step is under-specified; again this is an independent correctness concern. There are no load-bearing self-citations, no fitted parameter renamed as a prediction, and no uniqueness argument imported from the authors' prior work. The derivation chain, though formally flawed, is not circular.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The core method rests on a handful of design choices: a fixed scaling constant, a supervised training budget, last-token-only injection, and the linear attention approximation. No exotic entities are introduced. The free parameters are modest, but the theoretical containment proof depends on an approximation whose validity is not justified for real attention.

free parameters (2)
  • ATV injection scaling λ = 0.001
    Chosen by hand and fixed across all experiments; it controls the strength of every layerwise steering injection and is not derived from first principles.
  • Training examples per task = 90
    Hand-picked to match ELICIT's setup; in-domain results depend on this supervised training budget, which is not shared by zero-shot and ICL baselines.
assumptions (4)
  • domain assumption The linear attention approximation Attn(Q,K,V) ≈ QK^T V from Dai et al. [28] is sufficient to compare expressivity of ATV and Prefix-Tuning.
    Used in Appendix A.2.1; the softmax nonlinearity is dropped, which can change the set of representable attention outputs.
  • domain assumption Injecting into the last-token hidden state at every layer is sufficient to steer the model; token-wise modulation is not needed.
    Used in Eq. (5)-(6); the paper does not compare against injecting at all token positions.
  • domain assumption The small model's last-layer hidden state vsmall captures enough task information from the query alone, without demonstrations.
    Core ATV design premise; no information-theoretic or empirical analysis of what is lost by dropping demonstrations.
  • standard math The activation row x^T is never zero, so the pseudoinverse identity x^T x^{+⊤} = 1 is valid.
    Invoked in Appendix A.1.3 to construct the LoRA down-projection; stated as benign in practice but still a non-vacuous condition.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Adaptive Task Vectors for Large Language Models." pith.science (2026). https://pith.science/paper/EAXJUERD

@misc{pith2026250603426,
  author       = {Pith},
  title        = {Pith review of: Adaptive Task Vectors for Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EAXJUERD}},
  note         = {Machine review of arXiv:2506.03426}
}
read the original abstract

In-Context Learning (ICL) enables Large Language Models (LLMs) to perform tasks without parameter updates by conditioning on a few demonstrations provided in the prompt. Despite its success, ICL suffers from several limitations, including sensitivity to demonstration order, context length constraints, and computational inefficiency. To address these challenges, task vector-based approaches compress task information into a single vector. However, these methods typically construct task vectors from fixed sets of demonstrations and reuse them across input queries, without conditioning on the specific input. This limitation can lead models to struggle with effective adaptation when the input query is not well aligned with the underlying demonstrations, consequently degrading their generalization performance on unseen tasks. To overcome this limitation, we propose Adaptive Task Vectors (ATV), a simple and effective framework that dynamically generates task vectors conditioned on each input query. ATV employs a small language model to generate task vectors, which are then transformed to match the target LLM's architecture and applied to guide its output generation. In contrast to ICL and previous vector-based approaches, which rely on fixed demonstration sets and their corresponding vectors, ATV dynamically generates task vectors tailored to each specific input query and task. Consequently, ATV demonstrates strong performance and generalization capabilities, even for unseen tasks. Furthermore, we provide a theoretical analysis indicating that ATV is expressively equivalent to LoRA under equal rank budgets and more expressive than Prefix-Tuning, thereby offering formal support for its representational advantage.

Figures

Figures reproduced from arXiv: 2506.03426 by the authors.

Figure 1
Figure 1. Comparison between task vector methods: a) Prior work uses a fixed task vector for all [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of the Adaptive Task Vector (ATV) framework. Top: During training, the small [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Layer-wise analysis of vector injec￾tion magnitudes. Left: ATV concentrates vec￾tor strength in lower layers, while Right: ELICIT shows a monotonic increase toward top layers. These patterns align with each method’s layer￾specific performance impact. We divide the transformer into bottom, mid￾dle, and top thirds, and evaluate each method by restricting injection to a single region. As shown in [PITH_FULL_IMAGE:figu… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: t-SNE visualization of task vector distributions for two BBH tasks. Each dot represents a query-specific task vector generated by ATV, while crosses denote the fixed task vectors used by ELICIT. We observe that vectors from similar queries tend to be grouped together, …

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

52 extracted references · 18 canonical work pages

  1. [1]

    Language models are few-shot learners,

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., “Language models are few-shot learners,” Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020

    1901

  2. [2]

    A survey on in-context learning,

    Q. Dong, L. Li, D. Dai, C. Zheng, J. Ma, R. Li, H. Xia, J. Xu, Z. Wu, T. Liu et al., “A survey on in-context learning,” arXiv preprint arXiv:2301.00234, 2022

    arXiv 2022

  3. [3]

    What makes good in-context examples for gpt-3?

    J. Liu, D. Shen, Y . Zhang, B. Dolan, L. Carin, and W. Chen, “What makes good in-context examples for gpt-3?” arXiv preprint arXiv:2101.06804, 2021

    arXiv 2021

  4. [4]

    Revisiting demonstration selection strategies in in-context learning,

    K. Peng, L. Ding, Y . Yuan, X. Liu, M. Zhang, Y . Ouyang, and D. Tao, “Revisiting demonstration selection strategies in in-context learning,” arXiv preprint arXiv:2401.12087, 2024

    arXiv 2024

  5. [5]

    Large language models are implicitly topic models: Explaining and finding good demonstrations for in-context learning,

    X. Wang, W. Zhu, and W. Y . Wang, “Large language models are implicitly topic models: Explaining and finding good demonstrations for in-context learning,” arXiv preprint arXiv:2301.11916, vol. 1, p. 15, 2023

    arXiv 2023

  6. [6]

    Long-context llms struggle with long in-context learning,

    T. Li, G. Zhang, Q. D. Do, X. Yue, and W. Chen, “Long-context llms struggle with long in-context learning,” arXiv preprint arXiv:2404.02060, 2024

    arXiv 2024

  7. [7]

    Babilong: Testing the limits of llms with long context reasoning-in-a-haystack,

    Y . Kuratov, A. Bulatov, P. Anokhin, I. Rodkin, D. Sorokin, A. Sorokin, and M. Burtsev, “Babilong: Testing the limits of llms with long context reasoning-in-a-haystack,” Advances in Neural Information Processing Systems, vol. 37, pp. 106 519–106 554, 2024

    work page 2024

  8. [8]

    In-context learning creates task vectors,

    R. Hendel, M. Geva, and A. Globerson, “In-context learning creates task vectors,” in Findings of the Association for Computational Linguistics: EMNLP 2023, H. Bouamor, J. Pino, and K. Bali, Eds., Dec. 2023, pp. 9318–9333

    work page 2023

Show all 52 references
  1. [9]

    Editing models with task arithmetic,

    G. Ilharco, M. T. Ribeiro, M. Wortsman, L. Schmidt, H. Hajishirzi, and A. Farhadi, “Editing models with task arithmetic,” in The Eleventh International Conference on Learning Representations, 2023

    2023

  2. [10]

    In-context vectors: making in context learning more effective and controllable through latent space steering,

    S. Liu, H. Ye, L. Xing, and J. Zou, “In-context vectors: making in context learning more effective and controllable through latent space steering,” inProceedings of the 41st International Conference on Machine Learning, 2024, pp. 32 287–32 307

    2024

  3. [11]

    Implicit in-context learning,

    Z. Li, Z. Xu, L. Han, Y . Gao, S. Wen, D. Liu, H. Wang, and D. N. Metaxas, “Implicit in-context learning,” in The Thirteenth International Conference on Learning Representations, 2025

    2025

  4. [12]

    ELICIT: LLM augmentation via external in-context capability,

    F. Wang, J. Yan, Y . Zhang, and T. Lin, “ELICIT: LLM augmentation via external in-context capability,” in The Thirteenth International Conference on Learning Representations, 2025. 10

    2025

  5. [13]

    Task vectors in in-context learning: Emergence, formation, and benefit,

    L. Yang, Z. Lin, K. Lee, D. Papailiopoulos, and R. Nowak, “Task vectors in in-context learning: Emergence, formation, and benefit,” arXiv preprint arXiv:2501.09240, 2025

    2025 arXiv

  6. [14]

    Multimodal task vectors enable many-shot multimodal in-context learning,

    B. Huang, C. Mitra, L. Karlinsky, A. Arbelle, T. Darrell, and R. Herzig, “Multimodal task vectors enable many-shot multimodal in-context learning,” Advances in Neural Information Processing Systems, vol. 37, pp. 22 124–22 153, 2024

    2024

  7. [15]

    Calibrate before use: Improving few-shot per- formance of language models,

    Z. Zhao, E. Wallace, S. Feng, D. Klein, and S. Singh, “Calibrate before use: Improving few-shot per- formance of language models,” in International conference on machine learning . PMLR, 2021, pp. 12 697–12 706

    2021

  8. [16]

    Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity,

    Y . Lu, M. Bartolo, A. Moore, S. Riedel, and P. Stenetorp, “Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity,”arXiv preprint arXiv:2104.08786, 2021

    2021 arXiv

  9. [17]

    Batch-icl: Effective, efficient, and order-agnostic in-context learning,

    K. Zhang, A. Lv, Y . Chen, H. Ha, T. Xu, and R. Yan, “Batch-icl: Effective, efficient, and order-agnostic in-context learning,” arXiv preprint arXiv:2401.06469, 2024

    2024 arXiv

  10. [18]

    When is task vector provably effective for model editing? a generalization analysis of nonlinear transformers,

    H. Li, Y . Zhang, S. Zhang, M. Wang, S. Liu, and P.-Y . Chen, “When is task vector provably effective for model editing? a generalization analysis of nonlinear transformers,” arXiv preprint arXiv:2504.10957, 2025

    2025 arXiv

  11. [19]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V . Le, D. Zhouet al., “Chain-of-thought prompting elicits reasoning in large language models,” Advances in neural information processing systems, vol. 35, pp. 24 824–24 837, 2022

    2022

  12. [20]

    Large language models are zero-shot reasoners,

    T. Kojima, S. S. Gu, M. Reid, Y . Matsuo, and Y . Iwasawa, “Large language models are zero-shot reasoners,” Advances in neural information processing systems, vol. 35, pp. 22 199–22 213, 2022

    2022

  13. [21]

    Least-to-most prompting enables complex reasoning in large language models,

    D. Zhou, N. Schärli, L. Hou, J. Wei, N. Scales, X. Wang, D. Schuurmans, C. Cui, O. Bousquet, Q. Le et al., “Least-to-most prompting enables complex reasoning in large language models,” arXiv preprint arXiv:2205.10625, 2022

    2022 arXiv

  14. [22]

    Rethinking the role of demonstrations: What makes in-context learning work?

    S. Min, X. Lyu, A. Holtzman, M. Artetxe, M. Lewis, H. Hajishirzi, and L. Zettlemoyer, “Rethinking the role of demonstrations: What makes in-context learning work?” in Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 2022, pp. 11 048–11 064

    2022

  15. [23]

    Learning to retrieve prompts for in-context learning,

    O. Rubin, J. Herzig, and J. Berant, “Learning to retrieve prompts for in-context learning,” inProceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2022, pp. 2655–2671

    2022

  16. [24]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,”Advances in neural information processing systems, vol. 30, 2017

    2017

  17. [25]

    Language models are unsupervised multitask learners,

    A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever et al., “Language models are unsupervised multitask learners,” OpenAI blog, vol. 1, no. 8, p. 9, 2019

    2019

  18. [26]

    Lora: Low-rank adaptation of large language models

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, W. Chenet al., “Lora: Low-rank adaptation of large language models.” ICLR, vol. 1, no. 2, p. 3, 2022

    2022

  19. [27]

    Prefix-tuning: Optimizing continuous prompts for generation,

    X. L. Li and P. Liang, “Prefix-tuning: Optimizing continuous prompts for generation,” arXiv preprint arXiv:2101.00190, 2021

    2021 arXiv

  20. [28]

    Why can gpt learn in-context? language models implicitly perform gradient descent as meta-optimizers,

    D. Dai, Y . Sun, L. Dong, Y . Hao, S. Ma, Z. Sui, and F. Wei, “Why can gpt learn in-context? language models implicitly perform gradient descent as meta-optimizers,” arXiv preprint arXiv:2212.10559, 2022

    2022 arXiv

  21. [29]

    Towards a unified view of parameter-efficient transfer learning,

    J. He, C. Zhou, X. Ma, T. Berg-Kirkpatrick, and G. Neubig, “Towards a unified view of parameter-efficient transfer learning,” in International Conference on Learning Representations, 2022

    2022

  22. [30]

    The llama 3 herd of models,

    A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan et al., “The llama 3 herd of models,” arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  23. [31]

    Mistral 7b,

    A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de las Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M.-A. Lachaux, P. Stock, T. L. Scao, T. Lavril, T. Wang, T. Lacroix, and W. E. Sayed, “Mistral 7b,” 2023. [Online]. Available: ht...

    2023 arXiv

  24. [32]

    The probabilistic relevance framework: Bm25 and beyond,

    S. Robertson, H. Zaragoza et al., “The probabilistic relevance framework: Bm25 and beyond,”Foundations and Trends® in Information Retrieval, vol. 3, no. 4, pp. 333–389, 2009

    2009

  25. [33]

    A primer in bertology: What we know about how bert works,

    A. Rogers, O. Kovaleva, and A. Rumshisky, “A primer in bertology: What we know about how bert works,” Transactions of the association for computational linguistics, vol. 8, pp. 842–866, 2021. 11

    2021

  26. [34]

    Bert rediscovers the classical nlp pipeline,

    I. Tenney, D. Das, and E. Pavlick, “Bert rediscovers the classical nlp pipeline,” arXiv preprint arXiv:1905.05950, 2019

    1905 arXiv

  27. [35]

    A mathematical framework for transformer circuits,

    N. Elhage, N. Nanda, C. Olsson, T. Henighan, N. Joseph, B. Mann, A. Askell, Y . Bai, A. Chen, T. Conerly et al., “A mathematical framework for transformer circuits,”Transformer Circuits Thread, vol. 1, no. 1, p. 12, 2021

    2021

  28. [36]

    Commonsenseqa: A question answering challenge targeting commonsense knowledge,

    A. Talmor, J. Herzig, N. Lourie, and J. Berant, “Commonsenseqa: A question answering challenge targeting commonsense knowledge,” arXiv preprint arXiv:1811.00937, 2018

    2018 arXiv

  29. [37]

    Can a suit of armor conduct electricity? a new dataset for open book question answering,

    T. Mihaylov, P. Clark, T. Khot, and A. Sabharwal, “Can a suit of armor conduct electricity? a new dataset for open book question answering,” arXiv preprint arXiv:1809.02789, 2018

    2018 arXiv

  30. [38]

    Hellaswag: Can a machine really finish your sentence?

    R. Zellers, A. Holtzman, Y . Bisk, A. Farhadi, and Y . Choi, “Hellaswag: Can a machine really finish your sentence?” arXiv preprint arXiv:1905.07830, 2019

    1905 arXiv

  31. [39]

    Boolq: Exploring the surprising difficulty of natural yes/no questions,

    C. Clark, K. Lee, M.-W. Chang, T. Kwiatkowski, M. Collins, and K. Toutanova, “Boolq: Exploring the surprising difficulty of natural yes/no questions,” arXiv preprint arXiv:1905.10044, 2019

    1905 arXiv

  32. [40]

    Challenging big-bench tasks and whether chain-of-thought can solve them,

    M. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y . Tay, H. W. Chung, A. Chowdhery, Q. V . Le, E. H. Chi, D. Zhou et al., “Challenging big-bench tasks and whether chain-of-thought can solve them,”arXiv preprint arXiv:2210.09261, 2022

    arXiv

  33. [41]

    Think you have solved question answering? try arc, the ai2 reasoning challenge,

    P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord, “Think you have solved question answering? try arc, the ai2 reasoning challenge,” arXiv preprint arXiv:1803.05457, 2018

    2018 arXiv

  34. [42]

    Mathqa: Towards inter- pretable math word problem solving with operation-based formalisms,

    A. Amini, S. Gabriel, P. Lin, R. Koncel-Kedziorski, Y . Choi, and H. Hajishirzi, “Mathqa: Towards inter- pretable math word problem solving with operation-based formalisms,” arXiv preprint arXiv:1905.13319, 2019

    1905 arXiv

  35. [43]

    Mmlu- pro: A more robust and challenging multi-task language understanding benchmark,

    Y . Wang, X. Ma, G. Zhang, Y . Ni, A. Chandra, S. Guo, W. Ren, A. Arulraj, X. He, Z. Jianget al., “Mmlu- pro: A more robust and challenging multi-task language understanding benchmark,” in The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchm...

    2024

  36. [44]

    Crows-pairs: A challenge dataset for measuring social biases in masked language models,

    N. Nangia, C. Vania, R. Bhalerao, and S. R. Bowman, “Crows-pairs: A challenge dataset for measuring social biases in masked language models,” arXiv preprint arXiv:2010.00133, 2020

    2010 arXiv

  37. [45]

    Bbq: A hand-built bias benchmark for question answering,

    A. Parrish, A. Chen, N. Nangia, V . Padmakumar, J. Phang, J. Thompson, P. M. Htut, and S. R. Bowman, “Bbq: A hand-built bias benchmark for question answering,” arXiv preprint arXiv:2110.08193, 2021

    2021 arXiv

  38. [46]

    Pointer sentinel mixture models,

    S. Merity, C. Xiong, J. Bradbury, and R. Socher, “Pointer sentinel mixture models,” arXiv preprint arXiv:1609.07843, 2016

    2016 arXiv

  39. [47]

    Glue: A multi-task benchmark and analysis platform for natural language understanding,

    A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman, “Glue: A multi-task benchmark and analysis platform for natural language understanding,” arXiv preprint arXiv:1804.07461, 2018

    2018 arXiv

  40. [48]

    Super- glue: A stickier benchmark for general-purpose language understanding systems,

    A. Wang, Y . Pruksachatkun, N. Nangia, A. Singh, J. Michael, F. Hill, O. Levy, and S. Bowman, “Super- glue: A stickier benchmark for general-purpose language understanding systems,” Advances in neural information processing systems, vol. 32, 2019

    2019

  41. [49]

    Analysing mathematical reasoning abilities of neural models,

    D. Saxton, E. Grefenstette, F. Hill, and P. Kohli, “Analysing mathematical reasoning abilities of neural models,” arXiv preprint arXiv:1904.01557, 2019

    1904 arXiv

  42. [50]

    Eleutherai/lm-evaluation-harness: Major refactor,

    L. Sutawika, L. Gao, H. Schoelkopf, S. Biderman, J. Tow, B. Abbasi, ben fattori, C. Lovering, farzanehnakhaee70, J. Phang, A. Thite, Fazz, Aflah, N. Muennighoff, T. Wang, sdtblck, nopperl, gakada, tttyuntian, researcher2, Chris, J. Etxaniz, Z. Kasner, Khalid, J. Hsu, AndyZwei,...

    2023 doi

  43. [51]

    For every pair (hℓ, vsmall) there exist static LoRA factors (Wdown, Wup) and a scale s, all independent of the runtime query, such that ˜hℓ = ˆhℓ for all inputs

    ATV ⇒ LoRA (simulation). For every pair (hℓ, vsmall) there exist static LoRA factors (Wdown, Wup) and a scale s, all independent of the runtime query, such that ˜hℓ = ˆhℓ for all inputs

  44. [52]

    ATV implies LoRA

    LoRA ⇒ ATV (simulation). Conversely, any LoRA update with rank r = ds can be expressed in ATV form by an appropriate choice of (λ, vsmall, Aℓ). Hence, when the rank budgets are matched, ATV and LoRA realize the same class of low-rank additive perturbations to the frozen model;...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.