One Task Vector is not Enough: A Large-Scale Study for In-Context Learning

Elena Tutubalina; Ivan Oseledets; Pavel Tikhonov

REVIEW 3 major objections 5 minor 11 references

One Task Vector is not Enough: A Large-Scale Study for In-Context Learning

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A single task vector does not capture a complex few-shot task; many subtask-specific vectors are needed.

desk verdict A genuinely larger dataset and a confirmed layer-15 peak, but the many-vectors claim is undercut by a control failure in the key experiment. read the letter →

arxiv 2505.23911 v1 pith:7GSOPSVQ submitted 2025-05-29 cs.CL

Pavel Tikhonov , Ivan Oseledets , Elena Tutubalina This is my paper · ORCID

classification cs.CL

keywords in-contextlearningtaskvectorsfunctionhiddenstateinterventionLLMinterpretabilityfew-shotLlama-3-8BQUITEAFEW

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the idea of one task vector—a single hidden-state activation encoding an entire few-shot task—is insufficient for complex tasks. Working with a new dataset of 3,096 tasks, it shows that task-vector performance peaks at an intermediate layer and varies sharply by task type. For compound tasks, the model relies on several specialized vectors at different output-token positions rather than one global vector. If correct, this shifts how we interpret in-context learning: task knowledge is distributed across the model's activations, not compressed into a single point.

What carries the argument

The central object is the task vector, defined as the hidden state at a chosen layer after the model processes the final '->' token of a few-shot prompt. It is applied through causal intervention: replace the hidden state at that position in a zero-shot input and let the model continue generating. The paper introduces a token-wise variant: for each token being generated, substitute the hidden state from the corresponding token position in the last few-shot example's output. This 'sub-task vector' method makes visible which few-shot positions carry which sub-components of a complex answer, and it is the mechanism that demonstrates the insufficiency of the single classic vector.

What would settle it

Re-run the 100-description JSON experiment with human-verified input-output pairs and compare the classic task vector against sub-task vectors. If a single classic task vector then matches the sub-task-vector accuracy, the paper's central insufficiency claim would fail; if the layer-15 peak and the sub-task advantage do not reproduce on a different model family, the generality claim would fail.

Watch

Extended reading notes

Core claim

On Llama-3-8B, injecting a task vector extracted at layer 15 after the final '->' token improves zero-shot outputs over the no-vector baseline, but the gain is uneven across task categories. The paper's central discovery is that for compositional tasks, a single classic task vector fails because the information needed is spread across multiple token positions; replacing the hidden state at the current generation step with the hidden state from the corresponding token of a few-shot output—a 'sub-task vector'—substantially raises the probability of the correct next token. The authors conclude that 'we need to talk not about one task vector for the entire task, but about many task vectors for the task,' meaning task execution for complex outputs relies on a sequence of specialized, context-dependent activations rather than a single overarching representation.

Load-bearing premise

The load-bearing premise is that the Qwen-generated input-output pairs are valid, correctly labeled examples of each task: the dataset was filtered and generated by large language models without human verification, so if those pairs contain systematic errors or biases, every reported task-vector performance number would stop meaning what it claims.

Editorial extensions

If this is right

If the claim is right, evaluations of task vectors should stop reporting one vector's quality in isolation and instead report per-position or per-subtask vectors for compound tasks.
The consistent layer-15 peak across task categories gives a concrete target for future causal studies: intermediate layers of Llama-3-8B are where task information becomes actionable.
Complex tasks can be executed more accurately by sequentially selecting the most relevant few-shot hidden state at each generation step, turning the paper's diagnostic method into a practical intervention.
Single-vector results from simple toy benchmarks such as antonyms or translations do not automatically transfer to realistic instructions that combine multiple operations, so benchmark design matters for interpretability conclusions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Testable extension: the paper's token-matching procedure could be turned into a training-free decoding algorithm that picks the best few-shot hidden state per generated token; the authors do not propose such an algorithm.
Implicit consequence: if task knowledge is distributed across output-token positions, then interpretability probes that read only one position may under-count the task representation, and position-aware probes would be a natural next test.
Implicit consequence: prompt design for in-context learning should attend to the structure and ordering of the example output, because the model appears to key on token positions inside that output.
Testable extension: since the dataset was generated by another LLM without human verification, a small human-validated subset of QUITEAFEW would let a reader separate genuine task-vector effects from biases introduced by the generator.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Two things to know about this paper. First, the QUITEAFEW dataset is a real resource: 3,096 tasks derived from Alpaca, each with 30 example pairs, generated by Qwen-2.5-72B and Qwen-3-235B-A22B. That is a genuine scaling jump over the hand-crafted 6-40-task sets used before, and it lets the authors confirm at scale that task-vector performance peaks at an intermediate layer (layer 15 for Llama-3-8B) and that the peak varies by task category. Second, the headline claim—that complex tasks require many subtask-specific vectors rather than one—is not yet supported by the evidence as presented. The key experiment in Section 4.2 confounds the number of injected vectors with the injection position. The classic task vector is extracted at the final '->' token and injected only at that token in the zero-shot input; the sub-task vectors are extracted from matching output positions and injected at those exact positions during generation. So the comparison differs in both count and location. A single vector injected at every output position might do just as well, which would make the takeaway 'position matters' rather than 'many vectors are needed.' The authors don't include that control. The stress-test note also raises a second issue: the sub-task vectors come from positions that immediately precede the target token in the demonstration, so without a transfer test using different values, one could worry about template-copying. Actually, the text does show a transfer: the few-shot colors green, Berlin, F8 and the query colors red, Vancouver, AMG GT, so the sub-task vector is not copying the token itself. But the position confound remains. This is fixable with a clean experiment: inject the same single vector at all output positions, and compare per-position extraction.

The dataset itself also rests on LLM-generated pairs with no human verification, and the evaluation uses LLM judges with format and correctness scores. The authors acknowledge this in the Limitations section. No variance estimates are reported. Given the size of the dataset, a sample of human-checked pairs would go a long way.

None of this sinks the paper. The layer-15 result is consistent with prior work, and the dataset is a contribution that people in interpretability will use. The sub-task vector observation is interesting and worth chasing, but it needs a properly controlled replication on more than one synthetic domain before claiming distributed representations.

This paper deserves a serious referee, but the current version needs major revision, mostly in Section 4.2 and the dataset validation. I'd send it to review.

Referee Report

3 major / 5 minor

Summary. The paper introduces QUITEAFEW, a dataset of 3,096 few-shot tasks with 30-50 input/output pairs each, automatically derived from Alpaca using Qwen models. Using Llama-3-8B, the authors report three findings: task-vector injection works best at an intermediate layer (around layer 15), the benefit varies strongly across task types, and complex compositional tasks are better served by multiple 'sub-task vectors' extracted from different output tokens than by a single global task vector. The paper interprets the last result as evidence that task knowledge is distributed rather than compressed into one vector.

Significance. If the central claims hold, the paper would make a useful empirical contribution by scaling task-vector analysis from dozens of toy tasks to over three thousand diverse tasks, and by providing a concrete mechanistic hypothesis about compositional task execution. The sub-task-vector experiment is a clean token-level intervention that does not rely on fitted parameters, and the paper is honest about several limitations (single model, fixed temperature, LLM-based evaluation). However, the main interpretive claim -- that many task vectors are needed rather than one -- rests on a comparison that confounds the number of vectors with the injection position, and the generated dataset and LLM-based scoring are not validated against any external or human ground truth. These issues are load-bearing for the paper's headline conclusions.

major comments (3)

[Section 4.2, Fig. 6] The comparison labeled 'Classic Task Vector' extracts a single hidden state at the final '->' token of the few-shot prompt and injects it only at the '->' token of the zero-shot input, whereas 'Sub-Task Vectors' extracts states from several output tokens and injects each at the corresponding output position. These conditions differ simultaneously in the number of vectors and in the injection site, so the observed improvement cannot be attributed to having multiple vectors; it could be entirely due to position-matched injection. Please add control conditions that vary one factor at a time, for example injecting the single '->' vector at every output position, and injecting a single vector extracted from the ':' position at matching positions, and report whether the sub-task advantage survives.
[Section 4.2, Fig. 6] Even with position controlled, the sub-task vectors are taken from the hidden states immediately preceding each token to be produced in the demonstration (e.g., the ':' before the color value). These states are by construction the ones that generated the demonstration's next token, so the experiment does not distinguish a reusable subtask representation from template copying or next-token statistics. The 'dynamic value' claim needs a transfer test in which the demonstration's attribute values differ from the query's values; the paper should report whether the sub-task vector from ':' increases probability of the query's correct value or instead of the demonstration's value. If the latter, the result is about local token prediction, not task vectors.
[Section 2.2, Appendix C, Figs. 1-3 and 6] QUITEAFEW is generated by Qwen-2.5-72B and Qwen-3-235B-A22B without any human verification, and the evaluation uses an LLM judge on a 0-10 scale. The paper reports averages without error bars, confidence intervals, or inter-annotator agreement. Since the layer-15 peak and task-type variability are the paper's first two claims, the absence of any validation of the synthetic data or the judge's reliability is load-bearing. Please provide at least a human or rule-based audit of a random sample of generated pairs and judge scores, and report variance across the 10 repeats and across tasks.

minor comments (5)

[Section 4.1] The sentence 'It is interesting that the first "real" task vector was not in the _G->, but in the {' appears twice, once ending with '{' and once continuing with 'at the beginning of the output sequence'; the duplicate should be removed.
[Fig. 5 and surrounding text] There are typos in the example text: 'Culinan' should be 'Cullinan' and 'Culinan' appears inconsistently.
[Section 3] The opening line '3,096t learning' appears to be a truncated or corrupted phrase; it should read something like 'For ICL, we conduct a series of experiments'.
[General] The paper does not state whether the QUITEAFEW dataset and the experimental code will be released; for reproducibility, please include an availability statement.
[Figs. 1-6] Some figure captions and axis labels are rendered in very small fonts and are hard to read, especially in Fig. 5; please ensure all text is legible in the final version.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's central claims rest on new experiments and external prior work, not on self-citation or definitional reduction.

full rationale

I walked the paper's derivation chain and found no load-bearing circular step. The task-vector methodology follows prior external work (Hendel et al., Todd et al.), but the experiments are run on a newly constructed dataset and are not fitted to the target claim. The layer-15 finding is a descriptive result of a layer-wise sweep, not a parameter fitted to data and then renamed as a prediction; the subsequent use of layer 15 for all other experiments is a design choice, not a claim that layer 15 was predicted. The sub-task vector experiment is a causal intervention that extracts hidden states from specific output positions and measures next-token probabilities; the conclusion that many sub-task vectors are needed is an interpretation of measured probabilities, not a consequence of the definition of a task vector. The paper does not invoke any uniqueness theorem, and its citations to prior work are external to the author set, so there is no self-citation chain forcing the result. The skeptic's concern about confounding vector count with injection position is a legitimate experimental-control issue, but it is not circularity: the observed gains are not guaranteed by construction and could be disambiguated by further controlled experiments. Similarly, the LLM-generated dataset is a data-quality concern, not a circularity concern. Accordingly, the appropriate score is 0.

Assumptions & free parameters 1 free parameters · 5 assumptions · 1 invented entities

The central claims rest on a set of domain assumptions about the validity of LLM-generated data, the reliability of LLM-based scoring, and the causal meaning of hidden-state interventions. No free parameters are fitted to make the results work, but the fixed decoding temperature is a hyperparameter choice. The only new conceptual object is the 'sub-task vector', supported only within the paper.

free parameters (1)

decoding temperature = fixed, value not reported
All generation uses a fixed temperature; varying it might change task vector efficacy, as noted in Limitations.

assumptions (5)

domain assumption QUITEAFEW's LLM-generated input-output pairs are correct and sufficiently diverse to represent each task.
Section 2.2 uses Qwen models to generate pairs without human verification; errors would bias all subsequent evaluation.
domain assumption LLM-based judge scores (format and correctness) are reliable proxies for task quality.
Section 2.3 / Appendix C define the judge; evaluator bias is acknowledged in Limitations.
domain assumption The hidden state of the final separator token in the few-shot prompt encodes task information.
Section 2.1 builds on Hendel et al. (2023) and Todd et al. (2024); the paper does not re-derive this.
domain assumption Intervening on layer 15 hidden states causally reveals which representations are used for generation.
Section 4 uses substitution experiments; assumes linear or direct causality of hidden states.
domain assumption Results on Llama-3-8B generalize to other LLMs.
Limitations explicitly note single-model study; generalization is unverified.

invented entities (1)

sub-task vectors
purpose: Explains why a single task vector underperforms on complex outputs: each JSON attribute or structural token has its own associated hidden-state vector.
Introduced in Section 4 based on one synthetic car-description domain; no external benchmark confirms they exist generally.

how reviews work

0 comments

Cite this review

Pith. "Pith review of One Task Vector is not Enough: A Large-Scale Study for In-Context Learning." pith.science (2026). https://pith.science/paper/7GSOPSVQ

@misc{pith2026250523911,
  author       = {Pith},
  title        = {Pith review of: One Task Vector is not Enough: A Large-Scale Study for In-Context Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7GSOPSVQ}},
  note         = {Machine review of arXiv:2505.23911}
}

read the original abstract

In-context learning (ICL) enables Large Language Models (LLMs) to adapt to new tasks using few examples, with task vectors - specific hidden state activations - hypothesized to encode task information. Existing studies are limited by small-scale benchmarks, restricting comprehensive analysis. We introduce QuiteAFew, a novel dataset of 3,096 diverse few-shot tasks, each with 30 input-output pairs derived from the Alpaca dataset. Experiments with Llama-3-8B on QuiteAFew reveal: (1) task vector performance peaks at an intermediate layer (e.g., 15th), (2) effectiveness varies significantly by task type, and (3) complex tasks rely on multiple, subtask-specific vectors rather than a single vector, suggesting distributed task knowledge representation.

Figures

Figures reproduced from arXiv: 2505.23911 by the authors.

**Figure 1.** Average task vectors performance on QUITEAFEW dataset. hidden state as part of its standard computation. 2.2 Dataset Collection We built QUITEAFEW by expanding the Alpaca dataset (Taori et al., 2023), which contains instruction-following entries from OpenAI’s textdavinci 003. Many Alpaca entries include an instruction paired with an example input and output, providing a structure ideal for generating diverse few-s… view at source ↗

**Figure 2.** Marks distribution on QUITEAFEW dataset. appropriate for creating diverse, high-quality fewshot examples. We used Qwen-2.5-72B with a tailored classification prompt (see Appendix A) to evaluate each Alpaca entry’s suitability. The prompt assessed whether an instruction and its example input could support generating at least 30 distinct inputoutput pairs. The evaluation criteria were: • The instruction must allow … view at source ↗

**Figure 3.** Layer-wise Task Vector Performance across different task categories. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: Dual-axis analysis of task vector effectiveness [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]

**Figure 5.** Figure 5: Token-level influence of few-shot hidden states on zero-shot JSON generation. Rows represent tokens [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

**Figure 6.** Figure 6: Comparison of token prediction probabilities [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

11 extracted references · 11 canonical work pages

[1]

Analyze the instruction and understand its scope

work page
[2]

Generate { num_examples } different , realistic , and diverse inputs that could be used with this instruction

work page
[3]

For each input , provide an appropriate output following the pattern shown in the example

work page
[4]

Ensure each input - output pair is unique and demonstrates different aspects of the instruction

work page
[5]

Format your response exactly as a CSV table with three columns with a header : counter | input | output Requirements : - Generate exactly { num_examples } examples - Ensure all examples are distinct and non - repetitive - Maintain consistent quality across all examples - Follow the same style and format as the provided example - Ensure inputs are realisti...

work page
[6]

Format score (0 -10) : How well the response matches the semantic type required by the task - Score 10: Response provides exactly the type of answer requested ( e . g . , classification label for classification tasks , Yes / No for yes / no questions ) - Score 7 -9: Provides the right type of answer with minor formatting issues - Score 4 -6: Partially att...

work page
[7]

Opinion

Correctness score (0 -10) : How accurate / correct the actual answer is within its task domain - Score 10: Completely correct task completion - Score 7 -9: Mostly correct task completion with minor issues - Score 4 -6: Partial task completion with significant issues - Score 1 -3: Minimal correct task completion - Score 0: No actual task completion ( e . g...

work page
[8]

""{ instruction }

Instruction : """{ instruction }"""

work page

Show all 11 references

[9]

""{ example_input }

Sample Input : """{ example_input }"""
[10]

""{ example_output }

Expected Output : """{ example_output }"""
[11]

""{ o u tp u t _t o _ ev a l ua t e }

Submission to evaluate : """{ o u tp u t _t o _ ev a l ua t e }""" Listing 4: User prompt part of evaluating answer quality prompt

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Analyze the instruction and understand its scope

work page

[2] [2]

Generate { num_examples } different , realistic , and diverse inputs that could be used with this instruction

work page

[3] [3]

For each input , provide an appropriate output following the pattern shown in the example

work page

[4] [4]

Ensure each input - output pair is unique and demonstrates different aspects of the instruction

work page

[5] [5]

Format your response exactly as a CSV table with three columns with a header : counter | input | output Requirements : - Generate exactly { num_examples } examples - Ensure all examples are distinct and non - repetitive - Maintain consistent quality across all examples - Follow the same style and format as the provided example - Ensure inputs are realisti...

work page

[6] [6]

Format score (0 -10) : How well the response matches the semantic type required by the task - Score 10: Response provides exactly the type of answer requested ( e . g . , classification label for classification tasks , Yes / No for yes / no questions ) - Score 7 -9: Provides the right type of answer with minor formatting issues - Score 4 -6: Partially att...

work page

[7] [7]

Opinion

Correctness score (0 -10) : How accurate / correct the actual answer is within its task domain - Score 10: Completely correct task completion - Score 7 -9: Mostly correct task completion with minor issues - Score 4 -6: Partial task completion with significant issues - Score 1 -3: Minimal correct task completion - Score 0: No actual task completion ( e . g...

work page

[8] [8]

""{ instruction }

Instruction : """{ instruction }"""

work page