Pith. sign in

REVIEW 34 cited by

Larger language models do in-context learning differently

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2303.03846 v2 pith:WBLOU7HT submitted 2023-03-07 cs.CL

Larger language models do in-context learning differently

Jerry Wei , Jason Wei , Yi Tay , Dustin Tran , Albert Webson , Yifeng Lu , Xinyun Chen , Hanxiao Liu
show 3 more authors
Da Huang Denny Zhou Tengyu Ma
This is my paper
classification cs.CL
keywords modelspriorssemanticin-contextlanguagelabelsflippedinput-label
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved
0 comments
read the original abstract

We study how in-context learning (ICL) in language models is affected by semantic priors versus input-label mappings. We investigate two setups-ICL with flipped labels and ICL with semantically-unrelated labels-across various model families (GPT-3, InstructGPT, Codex, PaLM, and Flan-PaLM). First, experiments on ICL with flipped labels show that overriding semantic priors is an emergent ability of model scale. While small language models ignore flipped labels presented in-context and thus rely primarily on semantic priors from pretraining, large models can override semantic priors when presented with in-context exemplars that contradict priors, despite the stronger semantic priors that larger models may hold. We next study semantically-unrelated label ICL (SUL-ICL), in which labels are semantically unrelated to their inputs (e.g., foo/bar instead of negative/positive), thereby forcing language models to learn the input-label mappings shown in in-context exemplars in order to perform the task. The ability to do SUL-ICL also emerges primarily with scale, and large-enough language models can even perform linear classification in a SUL-ICL setting. Finally, we evaluate instruction-tuned models and find that instruction tuning strengthens both the use of semantic priors and the capacity to learn input-label mappings, but more of the former.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 34 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. DICE: Entropy-Regularized Equilibrium Selection for Stable Multi-Agent LLM Coordination

    cs.LG 2026-06 unverdicted novelty 7.0

    DICE formalizes multi-agent LLM coordination as discounted incomplete-information Markov games and introduces Heterogeneous Quantal Response Equilibrium (HQRE) to achieve unique stable equilibria with bounded regret, ...

  2. Priors Persist Through Suppression: A Stroop Paradigm for Lexical Override

    cs.CL 2026-05 conditional novelty 7.0

    Lexical override in language models is partial competition: the pretrained meaning persists, predicts interference, and repair works by protecting the contextual target rather than silencing the distractor.

  3. In-Context Fixation: When Demonstrated Labels Override Semantics in Few-Shot Classification

    cs.LG 2026-05 conditional novelty 7.0

    In-context learning binds model outputs to the demonstrated label tokens as an exhaustive vocabulary, overriding semantic plausibility and causing fixation even with homogeneous or nonsense labels.

  4. Task Vector Geometry Underlies Dual Modes of Task Inference in Transformers

    cs.LG 2026-05 unverdicted novelty 7.0

    In a controlled synthetic setting, transformers implement in-distribution task inference via convex combinations of task vectors and out-of-distribution inference via nearly orthogonal extrapolative representations.

  5. Fine-tuning vs. In-context Learning in Large Language Models: A Formal Language Learning Perspective

    cs.CL 2026-04 conditional novelty 7.0

    A controlled formal language task reveals fine-tuning outperforms in-context learning on in-distribution generalization but equals it on out-of-distribution, with ICL showing greater sensitivity to model size and toke...

  6. Fine-tuning vs. In-context Learning in Large Language Models: A Formal Language Learning Perspective

    cs.CL 2026-04 unverdicted novelty 7.0

    Fine-tuning shows higher proficiency than in-context learning on in-distribution generalization in formal languages, with equal out-of-distribution performance and diverging inductive biases at high proficiency.

  7. Better and Worse with Scale: How Contextual Entrainment Diverges with Model Size

    cs.CL 2026-04 unverdicted novelty 7.0

    Contextual entrainment decreases for semantic contexts but increases for non-semantic ones as LLMs scale, following power-law trends with 4x better resistance to misinformation but 2x more copying of arbitrary tokens.

  8. Epistemic Familiarity is Associated With Belief Stability in Large Language Models

    cs.CL 2025-11 conditional novelty 7.0

    Language models retract previously true answers far more often after seeing unfamiliar synthetic statements than after seeing familiar fictional statements, in both internal probes and prompted behavior.

  9. Pre-trained Large Language Models Learn Hidden Markov Models In-context

    cs.LG 2025-06 unverdicted novelty 7.0

    Pre-trained LLMs learn to predict HMM-generated sequences via in-context learning, approaching theoretical optimum on synthetic HMMs and matching expert models on real animal decision data.

  10. HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models

    cs.CV 2023-10 unverdicted novelty 7.0

    HallusionBench shows GPT-4V reaches only 31.42% accuracy on paired questions testing language hallucination and visual illusion in LVLMs, with other models below 16%.

  11. Large Language Models as Optimizers

    cs.LG 2023-09 unverdicted novelty 7.0

    Large language models can optimize by being prompted with histories of past solutions and scores to propose better ones, producing prompts that raise accuracy up to 8% on GSM8K and 50% on Big-Bench Hard over human-des...

  12. DICE: Entropy-Regularized Equilibrium Selection for Stable Multi-Agent LLM Coordination

    cs.LG 2026-06 conditional novelty 6.5

    HQRE entropy regularization makes multi-agent LLM coordination well-posed, yielding unique equilibria, linear mirror convergence, bounded Bayesian regret, and DICE gains of 4.3–8.5 pp on reasoning/planning tasks.

  13. Test-Time Scaling via Error Localization

    cs.LG 2026-07 conditional novelty 6.0

    TTEL uses feedback-induced token probability drops to localize the first error in a failed reasoning trace and branch a new generation from that prefix, improving pass@k per token on coding and math benchmarks.

  14. Sequential Correlations Change In-Context Learning: Effective Context Length and Architectural Mismatch

    stat.ML 2026-07 conditional novelty 6.0

    Within-context token correlations reduce ICL to an effective shorter i.i.d. context length, while query–context correlations lower error and favor softmax over linear attention.

  15. OpenRFM: Dissecting Relational In-Context Learning

    cs.LG 2026-06 unverdicted novelty 6.0

    OpenRFM combines a relational transformer backbone with a batch-level ICL layer and homophily-aware synthetic-plus-real pre-training to improve relational in-context learning by ~30% over prior open models and surpass...

  16. Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retention

    cs.LG 2026-05 unverdicted novelty 6.0

    Larger models succeed on rare and complex tasks by reducing gradient interference from common tasks, allowing rare-task features to accumulate, as shown via synthetic task mixtures and OLMo pretraining from 4M to 4B p...

  17. When Correct Demonstrations Hurt: Rethinking the Role of Exemplars in In-Context Learning

    cs.LG 2026-05 unverdicted novelty 6.0

    Task-preserving perturbations of correct exemplars can degrade ICL performance by changing the effective evidence mixture used for inference.

  18. Format-Constraint Coupling in Knowledge Graph Construction from Statistical Tables

    cs.AI 2026-05 unverdicted novelty 6.0

    Empirical 2x2 factorial study on 6 statistical datasets shows format and schema constraints in LLM-based KG construction from CSV tables produce super-additive fidelity loss up to +1.180, with mismatched pairs falling...

  19. In-Context Learning Operates as Concept Subspace Learning

    cs.LG 2026-05 unverdicted novelty 6.0

    In-context learning decomposes into concept-coordinate regression plus off-subspace leakage, with recoverable task information concentrating in a 68-73 dimensional task-aligned subspace of the residual stream that res...

  20. OPT-BENCH: Evaluating the Iterative Self-Optimization of LLM Agents in Large-Scale Search Spaces

    cs.AI 2026-05 unverdicted novelty 6.0

    OPT-BENCH and OPT-Agent evaluate LLM self-optimization in large search spaces, showing stronger models improve via feedback but stay constrained by base capacity and below human performance.

  21. Dissecting Multimodal In-Context Learning: Modality Asymmetries and Circuit Dynamics in modern Transformers

    cs.CL 2026-01 conditional novelty 6.0

    In a two-modality transformer, a primary-modality pretraining stage installs an induction circuit, so the secondary modality needs only low class diversity to learn in-context from examples.

  22. HalluJudge: A Reference-Free Hallucination Detection for Context Misalignment in Code Review Automation

    cs.SE 2026-01 conditional novelty 6.0

    HalluJudge detects hallucinated LLM code-review comments by scoring context alignment between comment and diff, reaching F1 0.85 on 143 annotated enterprise comments.

  23. Localizing Task Recognition and Task Learning in In-Context Learning via Attention Head Analysis

    cs.CL 2025-09 unverdicted novelty 6.0

    A new framework using Task Subspace Logit Attribution localizes attention heads specialized for task recognition and task learning in in-context learning, showing they align and rotate hidden states within a task subspace.

  24. LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code

    cs.SE 2024-03 unverdicted novelty 6.0

    LiveCodeBench collects 400 recent contest problems to create a contamination-free benchmark evaluating LLMs on code generation and related capabilities like self-repair and execution.

  25. TACTIC-KG: Toward Small Agent Teams for Cyber Threat Intelligence Knowledge Graph Construction

    cs.CR 2026-07 conditional novelty 5.5

    Specialized 3B–8B LLM agents for extraction, typing, verification, and curation outperform much larger monolithic ICL models on CTI knowledge-graph construction.

  26. Data-Efficient Adaptation of LLMs via Attention Head Reweighting

    cs.LG 2026-07 conditional novelty 5.0

    Learning a single scalar per attention head lets LLMs adapt to few-shot text classification better than LoRA, with 200–1000x fewer trainable parameters.

  27. CFALR: Collaborative Filtering-Augmented Large Language Model for Personalized Fashion Outfit Recommendation

    cs.IR 2026-06 unverdicted novelty 5.0

    CFALR augments LLMs with collaborative filtering embeddings via trainable projection layers to outperform prior CF and LLM methods on Polyvore and IQON for personalized outfit tasks.

  28. Constitutional On-Policy Safe Distillation

    cs.LG 2026-06 unverdicted novelty 5.0

    COPSD uses a Cross-SFT cold-start followed by constitution-conditioned distillation to achieve stronger safety-helpfulness balance and lower safety tax on reasoning than prior on-policy self-distillation methods.

  29. Linguistic Productivity in Large Language Models: Models Coerce, but do not Preempt

    cs.CL 2026-06 unverdicted novelty 5.0

    Larger LLMs reproduce constructional productivity via entrenchment in coercion cases with nonce words but fail to use statistical preemption to avoid overgeneralizing semantically plausible but unobserved patterns.

  30. Priors Persist Through Suppression: A Stroop Paradigm for Lexical Override

    cs.CL 2026-05 unverdicted novelty 5.0

    Pretrained lexical priors in language models persist despite explicit remapping rules, as shown by a Stroop paradigm where prior strength predicts interference and activation patching localizes the repair mechanism.

  31. A Systematic Study of Training-Free Methods for Trustworthy Large Language Models

    cs.CL 2026-04 unverdicted novelty 5.0

    Training-free methods for LLM trustworthiness show inconsistent results across dimensions, with clear trade-offs in utility, robustness, and overhead depending on where they intervene during inference.

  32. TRUE: A Trustworthy Unified Explanation Framework for Large Language Model Reasoning

    cs.LG 2026-02 reject novelty 5.0

    TRUE checks whether LLM reasoning traces are self-sufficient by executing them blind, maps neighboring reasoning paths into a DAG, and ranks recurring failure modes by Shapley values.

  33. PaLI-X: On Scaling up a Multilingual Vision and Language Model

    cs.CV 2023-05 unverdicted novelty 4.0

    Scaling a multilingual vision-language model in size and training breadth yields new state-of-the-art results on over 25 benchmarks plus emerging abilities in counting and multilingual detection.

  34. The Prompt Engineering Report Distilled: Quick Start Guide for Life Sciences

    cs.CL 2025-09 unverdicted novelty 3.0

    The paper reduces a broad set of prompt engineering techniques to six core approaches and applies them to life sciences use cases while addressing common LLM pitfalls.