REVIEW 10 cited by
In-Context Learning Creates Task Vectors
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
In-context learning (ICL) in Large Language Models (LLMs) has emerged as a powerful new learning paradigm. However, its underlying mechanism is still not well understood. In particular, it is challenging to map it to the "standard" machine learning framework, where one uses a training set $S$ to find a best-fitting function $f(x)$ in some hypothesis class. Here we make progress on this problem by showing that the functions learned by ICL often have a very simple structure: they correspond to the transformer LLM whose only inputs are the query $x$ and a single "task vector" calculated from the training set. Thus, ICL can be seen as compressing $S$ into a single task vector $\boldsymbol{\theta}(S)$ and then using this task vector to modulate the transformer to produce the output. We support the above claim via comprehensive experiments across a range of models and tasks.
Forward citations
Cited by 10 Pith papers
-
Multimodal Function Vectors for Visual Relations
Multimodal function vectors extracted from a handful of attention heads in OpenFlamingo-4B encode spatial relations and can be steered, fine-tuned, and composed to improve zero-shot relational reasoning.
-
Bridging the Knowledge-Prediction Gap in LLMs on Multiple-Choice Questions
KAPPA reduces the knowledge-prediction gap in LLMs by aligning a prediction-direction coordinate to a knowledge-direction coordinate in the residual stream, yielding accuracy gains on binary-choice MCQs and modest gai...
-
Train Once, Reuse Everywhere: Generalizable Implicit In-Context Learning by Routing Attention
ICR extracts shared attention directions from in-context learning and routes them at inference time, enabling zero-shot reuse across tasks.
-
Enhancing Project-Specific Code Completion by Inferring Internal API Information
A retrieval-augmented code completion method that infers project-internal APIs from a rough draft and a static knowledge base, beating existing repo-level baselines.
-
Towards Compute-Optimal Many-Shot In-Context Learning
Hybrid demonstration selection that adds 20 similar examples to a large cached random or k-means set matches or beats similarity-only selection at up to 10x lower estimated inference cost in many-shot ICL.
-
Learning without training: The implicit dynamics of in-context learning
A forward pass with in-context examples is exactly equivalent to a context-free forward pass with a minimal rank-1 update to the MLP weights, with an explicit formula for that update.
-
Distinct Computations Emerge From Compositional Curricula in In-Context Learning
When transformer models see easy component examples before a harder combined math problem in one prompt, they solve unseen versions of the combined problem and store intermediate steps internally, unlike models traine...
-
ICM-Fusion: In-Context Meta-Optimized LoRA Fusion for Multi-Task Adaptation
ICM-Fusion uses a conditional VAE plus task-vector guidance to fuse multiple LoRA adapters into one model, reporting marginal average gains on vision and language benchmarks and larger gains in a few-shot long-tail setup.
-
Activation Reward Models for Few-Shot Model Alignment
Mean attention-head activations from a few labeled examples, injected into selected heads, turn a frozen vision-language model into a few-shot reward model that beats prompting and scoring baselines and a new reward-h...
-
InverseScope: Scalable Activation Inversion for Interpreting Large Language Models
A new conditional-generation architecture plus a feature-consistency metric make activation inversion practical for LLMs up to 7B parameters, with experiments on IOI, RAVEL, and in-context learning.
Discussion (0). Sign in to comment.