Pith. sign in

REVIEW 18 cited by

In-Context Language Learning: Architectures and Algorithms

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2401.12973 v2 pith:VWGEONGC submitted 2024-01-23 cs.CL cs.LG

Ekin Akyürek , Bailin Wang , Yoon Kim , Jacob Andreas This is my paper · ORCID
classification cs.CLcs.LG
keywords iclllanguagemodelslearningmodelneuralheadsin-context
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Large-scale neural language models exhibit a remarkable capacity for in-context learning (ICL): they can infer novel functions from datasets provided as input. Most of our current understanding of when and how ICL arises comes from LMs trained on extremely simple learning problems like linear regression and associative recall. There remains a significant gap between these model problems and the "real" ICL exhibited by LMs trained on large text corpora, which involves not just retrieval and function approximation but free-form generation of language and other structured outputs. In this paper, we study ICL through the lens of a new family of model problems we term in context language learning (ICLL). In ICLL, LMs are presented with a set of strings from a formal language, and must generate additional strings from the same language. We focus on in-context learning of regular languages generated by random finite automata. We evaluate a diverse set of neural sequence models (including several RNNs, Transformers, and state-space model variants) on regular ICLL tasks, aiming to answer three questions: (1) Which model classes are empirically capable of ICLL? (2) What algorithmic solutions do successful models implement to perform ICLL? (3) What architectural changes can improve ICLL in less performant models? We first show that Transformers significantly outperform neural sequence models with recurrent or convolutional representations on ICLL tasks. Next, we provide evidence that their ability to do so relies on specialized "n-gram heads" (higher-order variants of induction heads) that compute input-conditional next-token distributions. Finally, we show that hard-wiring these heads into neural models improves performance not just on ICLL, but natural language modeling -- improving the perplexity of 340M-parameter models by up to 1.14 points (6.7%) on the SlimPajama dataset.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 18 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Learning In-context n-grams with Transformers: Sub-n-grams Are Near-stationary Points

    cs.LG 2025-08 reject novelty 7.0 of 10

    Sub-n-gram estimators are near-stationary points of the population cross-entropy loss for in-context n-gram learning, offering a theoretical explanation for stage-wise training plateaus.

  2. Decomposing Prediction Mechanisms for In-Context Recall

    cs.LG 2025-07 conditional novelty 7.0 of 10

    In a toy in-context recall task, label-based task initiation and observation-based continuation are distinct mechanisms with separate emergence times, and the same first-token versus second-token gap appears in an OLM...

  3. MesaNet: Sequence Modeling by Locally Optimal Test-Time Training

    cs.LG 2025-06 conditional novelty 7.0 of 10

    MesaNet uses conjugate-gradient-optimal test-time regression in a chunkwise-parallelizable recurrent layer, achieving strong language modeling and benchmark performance at up to 1B scale.

  4. Versatile On-device Adaptation at the Edge by Unifying Few-shot, Zero-shot, Continual, and In-context Learning

    cs.LG 2026-07 conditional novelty 6.0 of 10

    ECL unifies few-shot, continual, zero-shot, and in-context learning on a single TCN-based edge chip, with first hardware baselines on several tasks.

  5. Sequential Correlations Change In-Context Learning: Effective Context Length and Architectural Mismatch

    stat.ML 2026-07 conditional novelty 6.0 of 10

    Within-context token correlations reduce ICL to an effective shorter i.i.d. context length, while query–context correlations lower error and favor softmax over linear attention.

  6. Language Models Need Sleep: Learning to Self-Modify and Consolidate Memories

    cs.LG 2026-06 unverdicted novelty 6.0 of 10

    Sleep-time Knowledge Seeding plus Dreaming lets LLMs expand capacity, distill fragile in-context memories into stable parameters, and self-improve without human labels.

  7. Rethinking Memorization Measures and their Implications in Large Language Models

    cs.LG 2025-07 conditional novelty 6.0 of 10

    Contextual memorization, defined by comparing a string's training loss against the best loss without training on that string, is stricter than counterfactual memorization and suggests that zero-memorization optimal le...

  8. In-Context Learning Strategies Emerge Rationally

    cs.LG 2025-06 conditional novelty 6.0 of 10

    Transformer in-context learning is modeled as a posterior-weighted mixture of memorizing and generalizing Bayesian predictors, with a loss-complexity tradeoff governed by three fitted parameters.

  9. Next-Token Prediction Should be Ambiguity-Sensitive: A Meta-Learning Perspective

    cs.LG 2025-06 conditional novelty 6.0 of 10

    Transformers systematically deviate from the Bayes-optimal predictor under high-ambiguity contexts on a new HMM benchmark, and a Monte Carlo predictor that decouples task inference from token prediction partly closes ...

  10. HAD: Hybrid Architecture Distillation Outperforms Teacher in Genomic Sequence Modeling

    cs.LG 2025-05 conditional novelty 6.0 of 10

    A compact hybrid GDN+attention model distilled from Nucleotide Transformer v2 outperforms similarly sized models and, on several tasks, its 500x larger teacher.

  11. Quantifying Memory Utilization with Effective State-Size

    cs.LG 2025-04 conditional novelty 6.0 of 10

    Effective state-size, the rank of a causal operator submatrix, is proposed and empirically validated as a memory-utilization proxy that beats cache size at predicting recall-task accuracy and reveals how LLMs reset co...

  12. AI for the Open-World: the Learning Principles

    cs.LG 2025-04 conditional novelty 6.0 of 10

    Open-world AI requires rich features, disentangled representations, and inference-time learning; the thesis presents techniques and large-scale experiments supporting these principles.

  13. Task Vectors in In-Context Learning: Emergence, Formation, and Benefit

    cs.LG 2025-01 conditional novelty 6.0 of 10

    Small transformers naturally encode task information in specific layers under limited conditions; a new auxiliary loss places a strong task vector at a chosen layer and improves out-of-distribution robustness.

  14. Randomly Sampled Language Reasoning Problems Elucidate Limitations of In-Context Learning

    cs.LG 2025-01 conditional novelty 6.0 of 10

    On randomly sampled 3-state DFA language tasks, foundation LLMs underperform n-gram baselines under pure in-context-learning prompts.

  15. ReBRAC-v2: The Return of the King

    cs.LG 2026-08 conditional novelty 5.0 of 10

    A fixed-recipe offline RL method combining normalizing-flow actors, categorical critics, staged training, and test-time refinement beats recent flow-based baselines by 22.5 points averaged over ten OGBench categories.

  16. Data-Efficient Adaptation of LLMs via Attention Head Reweighting

    cs.LG 2026-07 conditional novelty 5.0 of 10

    Learning a single scalar per attention head lets LLMs adapt to few-shot text classification better than LoRA, with 200–1000x fewer trainable parameters.

  17. Emergence and Effectiveness of Task Vectors in In-Context Learning: An Encoder Decoder Perspective

    cs.CL 2024-12 conditional novelty 5.0 of 10

    Task Decodability, a k-NN measure of how separable a task is in a model's middle-layer representations, tracks and predicts in-context learning accuracy, and early-layer finetuning improves it more than late-layer finetuning.

  18. A Survey of RWKV

    cs.CL 2024-12 conditional novelty 3.0 of 10

    A review of the RWKV architecture, its versions, applications, benchmarks, and open-source ecosystem; it presents no new experimental results.

Pith tools