Pith. sign in

REVIEW 17 cited by

Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluation

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2305.16938 v2 pith:CU5DOTOG submitted 2023-05-26 cs.CL

Marius Mosbach , Tiago Pimentel , Shauli Ravfogel , Dietrich Klakow , Yanai Elazar This is my paper · ORCID
classification cs.CL
keywords modelsfine-tuningin-contextlearningfew-shotfine-tunedgeneralizationnumber
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Few-shot fine-tuning and in-context learning are two alternative strategies for task adaptation of pre-trained language models. Recently, in-context learning has gained popularity over fine-tuning due to its simplicity and improved out-of-domain generalization, and because extensive evidence shows that fine-tuned models pick up on spurious correlations. Unfortunately, previous comparisons of the two approaches were done using models of different sizes. This raises the question of whether the observed weaker out-of-domain generalization of fine-tuned models is an inherent property of fine-tuning or a limitation of the experimental setup. In this paper, we compare the generalization of few-shot fine-tuning and in-context learning to challenge datasets, while controlling for the models used, the number of examples, and the number of parameters, ranging from 125M to 30B. Our results show that fine-tuned language models can in fact generalize well out-of-domain. We find that both approaches generalize similarly; they exhibit large variation and depend on properties such as model size and the number of examples, highlighting that robust task adaptation remains a challenge.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 17 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. POPri: Private Federated Learning using Preference-Optimized Synthetic Data

    cs.LG 2025-04 conditional novelty 7.0 of 10

    POPri uses client similarity scores as RL rewards to DPO-tune an LLM for DP synthetic data generation, outperforming prior private evolution baselines on next-token prediction and classification.

  2. MHA-RAG: Improving Efficiency, Accuracy, and Consistency by Encoding Exemplars as Soft Prompts

    cs.AI 2025-10 conditional novelty 6.0 of 10

    MHA-RAG encodes retrieved exemplars into order-invariant soft prompts via multi-head attention, claiming ~20-point effective-accuracy gains over RAG at ~10x lower inference FLOPs.

  3. Data Whisperer: Efficient Data Selection for Task-Specific LLM Fine-Tuning via Few-Shot In-Context Learning

    cs.CL 2025-05 conditional novelty 6.0 of 10

    Data Whisperer selects fine-tuning data by few-shot in-context learning with attention weighting, achieving better GSM8K accuracy with 10% of the data than with the full training set.

  4. Memorization and Knowledge Injection in Gated LLMs

    cs.CL 2025-04 conditional novelty 6.0 of 10

    MEGa injects episodic memories into separate gated LoRA adapters selected by embedding similarity, mitigating catastrophic forgetting and enabling recall, QA, and compositional questions on two datasets.

  5. The Impact of Model Scaling on Seen and Unseen Language Performance

    cs.CL 2025-01 conditional novelty 6.0 of 10

    Across 204 languages, model scale has little effect on zero-shot multilingual performance, improves two-shot classification linearly, and helps translation mainly for instruction-tuned models.

  6. Does Few-Shot Learning Help LLM Performance in Code Synthesis?

    cs.SE 2024-12 conditional novelty 6.0 of 10

    Few-shot example choice measurably affects LLM code output, and two proposed selectors (a perplexity ranker and a trained MLP ranker) each improve CodeLlama's Pass@1 on HumanEval+ by about five points.

  7. SymDPO: Boosting In-Context Learning of Large Multimodal Models with Symbol Demonstration Direct Preference Optimization

    cs.CV 2024-11 conditional novelty 6.0 of 10

    SymDPO replaces answer text in multimodal demonstrations with meaningless symbols during preference training, forcing models to use image context and improving in-context learning performance on five benchmarks.

  8. Brain-to-Text Decoding with Context-Aware Neural Representations and Large Language Models

    eess.SP 2024-11 conditional novelty 6.0 of 10

    Diphone-based marginalization plus LLM refinement achieves state-of-the-art word error rate (5.77%) and phoneme error rate (15.34%) on the Brain-to-Text 2024 benchmark.

  9. Mimicking or Reasoning: Rethinking Multi-Modal In-Context Learning in Vision-Language Models

    cs.CV 2025-06 conditional novelty 5.0 of 10

    Vision-language models improve little, often not at all, when given demonstrations, even when demonstrations contain explicit reasoning steps.

  10. Leveraging LLMs for Predictive Insights in Food Policy and Behavioral Interventions

    cs.AI 2024-11 conditional novelty 5.0 of 10

    A fine-tuned GPT-3.5 Turbo model predicts the direction of held-out food-policy experiments with 79% accuracy, but only 55% on preregistered unpublished studies.

  11. Leveraging Large Language Models for enzymatic reaction prediction and characterization

    cs.AI 2025-05 conditional novelty 4.0 of 10

    Fine-tuned Llama-3.1 models can perform enzymatic reaction prediction tasks, and multitask learning improves forward and retrosynthesis over single-task training.

  12. Retrieval Augmented Generation Evaluation for Health Documents

    cs.IR 2025-05 conditional novelty 4.0 of 10

    A European Commission team built and evaluated RAGEv, a RAG pipeline for health documents, reporting high yes/no and long-answer scores but recommending human verification of outputs.

  13. In a Few Words: Comparing Weak Supervision and LLMs for Short Query Intent Classification

    cs.IR 2025-04 conditional novelty 4.0 of 10

    LLMs outperform weak supervision on recall but not precision for short query intent classification, and fine-tuning beats in-context learning.

  14. ChronoLLM: A Framework for Customizing Large Language Model for Digital Twins generalization based on PyChrono

    cs.SE 2025-01 conditional novelty 4.0 of 10

    Fine-tuning LLMs on PyChrono-specific data improves their success rate at generating runnable simulation code from about 40% to about 85%, compared to prompting general models.

  15. DRUM: Learning Demonstration Retriever for Large MUlti-modal Models

    cs.CL 2024-12 reject novelty 4.0 of 10

    Fine-tuning a CLIP retriever with a list-wise loss derived from LVLM feedback improves few-shot performance, although the loss formula as written would reward worse demonstrations, creating an internal contradiction.

  16. Few-Shot Learning in Video and 3D Object Detection: A Survey

    cs.CV 2025-07 conditional novelty 3.0 of 10

    A survey of few-shot learning for video and 3D object detection that reviews architectures, losses, and training strategies, but contains numerous citation errors and unsupported performance claims.

  17. In-Context Learning Distillation for Efficient Few-Shot Fine-Tuning

    cs.CL 2024-12 reject novelty 3.0 of 10

    Applying an existing context distillation method to OPT models on MNLI, the paper claims large gains, but the headline comparison is confounded by teacher capacity and hyperparameter tuning on the test metric.

Pith tools