Pith. sign in

REVIEW 25 cited by

MetaICL: Learning to Learn In Context

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2110.15943 v2 pith:2WQDSFQN submitted 2021-10-29 cs.CL cs.AI

Sewon Min , Mike Lewis , Luke Zettlemoyer , Hannaneh Hajishirzi This is my paper · ORCID
classification cs.CLcs.AI
keywords meta-traininglearningmetaicltasksin-contexttargetapproachescontext
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

We introduce MetaICL (Meta-training for In-Context Learning), a new meta-training framework for few-shot learning where a pretrained language model is tuned to do in-context learning on a large set of training tasks. This meta-training enables the model to more effectively learn a new task in context at test time, by simply conditioning on a few training examples with no parameter updates or task-specific templates. We experiment on a large, diverse collection of tasks consisting of 142 NLP datasets including classification, question answering, natural language inference, paraphrase detection and more, across seven different meta-training/target splits. MetaICL outperforms a range of baselines including in-context learning without meta-training and multi-task learning followed by zero-shot transfer. We find that the gains are particularly significant for target tasks that have domain shifts from the meta-training tasks, and that using a diverse set of the meta-training tasks is key to improvements. We also show that MetaICL approaches (and sometimes beats) the performance of models fully finetuned on the target task, and outperforms much bigger models with nearly 8x parameters. Finally, we show that MetaICL is complementary to human-written instructions, and the best performance can be achieved by combining both approaches.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 25 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MimicDroid: In-Context Learning for Humanoid Robot Manipulation from Human Play Videos

    cs.RO 2025-09 conditional novelty 7.0 of 10

    Trained only on unlabeled human play videos, MimicDroid lets a GR1 humanoid perform new manipulation tasks from one to three demonstration videos, with roughly twice the real-world success of prior video-conditioned methods.

  2. Can Gradient Descent Simulate Prompting?

    cs.CL 2025-06 conditional novelty 7.0 of 10

    A MAML-style meta-training objective makes a single gradient step on new text recover part of the performance that prompting achieves, on reversal-curse and passage-QA tasks.

  3. Retrieval-Augmented Generation as Noisy In-Context Learning: A Unified Theory and Risk Bounds

    cs.LG 2025-06 conditional novelty 7.0 of 10

    RAG in in-context linear regression has an exact bias-variance tradeoff and a finite-sample bound revealing a generalization ceiling as retrieved examples grow.

  4. ThinkRetrieve: Retrieval-Augmented Reasoning Traces for Test-Time Scaling

    cs.AI 2026-08 conditional novelty 6.0 of 10

    Per-step retrieval of solved exemplars injected into the reasoning trace improves test-time scaling accuracy, with up to 13.4 absolute points gained on AIME 2025.

  5. MicLog: Towards Accurate and Efficient LLM-based Log Parsing via Progressive Meta In-Context Learning

    cs.SE 2026-01 conditional novelty 6.0 of 10

    A small open-source LLM trained with progressive meta in-context learning parses logs more accurately and faster than prior state-of-the-art LLM parsers on Loghub-2.0.

  6. Train Once, Reuse Everywhere: Generalizable Implicit In-Context Learning by Routing Attention

    cs.CL 2025-09 conditional novelty 6.0 of 10

    ICR extracts shared attention directions from in-context learning and routes them at inference time, enabling zero-shot reuse across tasks.

  7. KITE: Kernelized and Information Theoretic Exemplars for In-Context Learning

    cs.LG 2025-09 reject novelty 6.0 of 10

    KITE greedily selects in-context examples by maximizing a kernelized query-specific relevance score plus a D-optimal design diversity term, and reports accuracy gains over KATE, BM25, and DPP baselines on five classif...

  8. Automated Retrosynthesis Planning of Macromolecules Using Large Language Models and Knowledge Graphs

    cs.AI 2025-01 conditional novelty 6.0 of 10

    An LLM-and-knowledge-graph agent automatically constructs retrosynthetic pathway trees for macromolecules and recommends optimal synthesis routes, demonstrated on polyimide.

  9. Geo-LLaVA: A Large Multi-Modal Model for Solving Geometry Math Problems with Meta In-Context Learning

    cs.CV 2024-12 reject novelty 6.0 of 10

    Geo-LLaVA combines retrieval-augmented fine-tuning with in-context learning, reporting 65.25% and 42.36% on selected subsets of GeoQA and the new GeoMath dataset.

  10. PromptRefine: Enhancing Few-Shot Performance on Low-Resource Indic Languages with Example Selection from Related Example Banks

    cs.CL 2024-12 conditional novelty 6.0 of 10

    PromptRefine uses alternating minimization over language-specific retrievers plus diversity-aware DPP fine-tuning to select cross-lingual in-context examples, improving few-shot generation in low-resource Indic languages.

  11. Agentic Services Computing

    cs.SE 2025-09 conditional novelty 5.0 of 10

    A position and survey paper that defines Agentic Services Computing, a lifecycle-based framework for engineering LLM agents as governed, first-class services.

  12. InSQuAD: In-Context Learning for Efficient Retrieval via Submodular Mutual Information to Enforce Quality and Diversity

    cs.LG 2025-08 conditional novelty 5.0 of 10

    A submodular mutual information framework for selecting and training in-context learning exemplars improves average accuracy on nine benchmarks by about five points over the IDEAL baseline.

  13. ICM-Fusion: In-Context Meta-Optimized LoRA Fusion for Multi-Task Adaptation

    cs.CV 2025-08 reject novelty 5.0 of 10

    ICM-Fusion uses a conditional VAE plus task-vector guidance to fuse multiple LoRA adapters into one model, reporting marginal average gains on vision and language benchmarks and larger gains in a few-shot long-tail setup.

  14. DICE: Dynamic In-Context Example Selection in LLM Agents via Efficient Knowledge Transfer

    cs.AI 2025-07 conditional novelty 5.0 of 10

    DICE dynamically retrieves the most relevant in-context demonstrations at each agent step, and in this preprint it raises exact-match and success-rate scores on HotpotQA, ALFWorld, and Webshop across ReAct, Reflexion,...

  15. Divide, Optimize, Merge: Fine-Grained LLM Agent Optimization at Scale

    cs.CL 2025-05 conditional novelty 5.0 of 10

    FGO is a divide-and-conquer framework that runs LLM-based agent optimization on dataset subsets and recursively merges the optimized prompts or tool sets into a final agent system.

  16. E-InMeMo: Enhanced Prompting for Visual In-Context Learning

    cs.CV 2025-04 conditional novelty 5.0 of 10

    Adding a learnable pixel-level perturbation to the in-context pair improves MAE-VQGAN visual in-context learning on segmentation and object detection benchmarks.

  17. InstructRAG: Leveraging Retrieval-Augmented Generation on Instruction Graphs for LLM-Based Task Planning

    cs.AI 2025-04 conditional novelty 5.0 of 10

    InstructRAG combines instruction paths from a graph via reinforcement learning and meta-learned selection, reporting relative gains of 2 to 25 percent over its chosen baselines across four LLM planning benchmarks.

  18. Detecting Zero-Day Attacks in Digital Substations via In-Context Learning

    cs.LG 2025-01 conditional novelty 5.0 of 10

    A GPT-2 transformer trained on synthetic IEC-61850 traffic plus weak pseudo-labels detects four held-out zero-day attack types with 85-95% accuracy without retraining.

  19. TriAdaptLoRA: Brain-Inspired Triangular Adaptive Low-Rank Adaptation for Parameter-Efficient Fine-Tuning

    cs.CL 2025-01 conditional novelty 5.0 of 10

    TriAdaptLoRA, a triangular low-rank adaptation with importance-driven rank growth, achieves small average gains over LoRA, AdaLoRA, and IncreLoRA on GLUE and SQuAD 2.0, with mixed per-task results.

  20. CPRM: A LLM-based Continual Pre-training Framework for Relevance Modeling in Commercial Search

    cs.AI 2024-12 conditional novelty 5.0 of 10

    A continual pre-training framework combining query-item joint training, in-context pre-training on related queries/items, and teacher-generated reading comprehension data improves LLM relevance modeling in commercial search.

  21. Learning Mamba as a Continual Learner: Meta-learning Selective State Space Models for Efficient Continual Learning

    cs.LG 2024-12 conditional novelty 5.0 of 10

    Mamba can be meta-learned as an efficient continual learner, matching or surpassing Transformers on meta-continual learning tasks with a fixed-size state and a selectivity regularizer.

  22. Evaluating LLMs Capabilities Towards Understanding Social Dynamics

    cs.LG 2024-11 conditional novelty 5.0 of 10

    LLMs show improved directionality detection after fine-tuning but perform near chance at cyberbullying and anti-bullying classification, pointing to weak semantic understanding of informal social media language.

  23. Meta-Thinking in LLMs via Multi-Agent Reinforcement Learning: A Survey

    cs.AI 2025-04 conditional novelty 4.0 of 10

    The paper surveys existing work on LLM meta-thinking and argues that multi-agent reinforcement learning is a promising missing ingredient for building self-correcting language models.

  24. Memory-Augmented Agent Training for Business Document Understanding

    cs.CL 2024-12 conditional novelty 4.0 of 10

    Matrix iteratively refines an LLM agent's memory from training trajectories and outperforms prompting and vanilla agent baselines on private UBL invoice transport-reference extraction, though gains on the released ano...

  25. The Rise of Small Language Models in Healthcare: A Comprehensive Survey

    cs.CL 2025-04 conditional novelty 3.0 of 10

    A comprehensive survey of small language models in healthcare, with a taxonomy of building, adapting, and compressing them for clinical NLP tasks.

Pith tools