Pith. sign in

REVIEW 5 cited by

Improving In-Context Few-Shot Learning via Self-Supervised Training

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2205.01703 v2 pith:G5JJVNB3 submitted 2022-05-03 cs.CL

Mingda Chen , Jingfei Du , Ramakanth Pasunuru , Todor Mihaylov , Srini Iyer , Veselin Stoyanov , Zornitsa Kozareva This is my paper · ORCID
classification cs.CL
keywords few-shotlearningself-supervisedin-contextobjectivespretrainingself-supervisiontraining
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Self-supervised pretraining has made few-shot learning possible for many NLP tasks. But the pretraining objectives are not typically adapted specifically for in-context few-shot learning. In this paper, we propose to use self-supervision in an intermediate training stage between pretraining and downstream few-shot usage with the goal to teach the model to perform in-context few shot learning. We propose and evaluate four self-supervised objectives on two benchmarks. We find that the intermediate self-supervision stage produces models that outperform strong baselines. Ablation study shows that several factors affect the downstream performance, such as the amount of training data and the diversity of the self-supervised objectives. Human-annotated cross-task supervision and self-supervision are complementary. Qualitative analysis suggests that the self-supervised-trained models are better at following task requirements.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. ThinkRetrieve: Retrieval-Augmented Reasoning Traces for Test-Time Scaling

    cs.AI 2026-08 conditional novelty 6.0 of 10

    Per-step retrieval of solved exemplars injected into the reasoning trace improves test-time scaling accuracy, with up to 13.4 absolute points gained on AIME 2025.

  2. KITE: Kernelized and Information Theoretic Exemplars for In-Context Learning

    cs.LG 2025-09 reject novelty 6.0 of 10

    KITE greedily selects in-context examples by maximizing a kernelized query-specific relevance score plus a D-optimal design diversity term, and reports accuracy gains over KATE, BM25, and DPP baselines on five classif...

  3. PromptRefine: Enhancing Few-Shot Performance on Low-Resource Indic Languages with Example Selection from Related Example Banks

    cs.CL 2024-12 conditional novelty 6.0 of 10

    PromptRefine uses alternating minimization over language-specific retrievers plus diversity-aware DPP fine-tuning to select cross-lingual in-context examples, improving few-shot generation in low-resource Indic languages.

  4. In-Context Deep Learning via Transformer Models

    cs.LG 2024-11 conditional novelty 5.0 of 10

    An explicit construction shows a transformer-like network with an element-wise multiplication layer can simulate L gradient descent steps of an N-layer ReLU network via in-context learning.

  5. ANROT-HELANet: Adverserially and Naturally Robust Attention-Based Aggregation Network via The Hellinger Distance for Few-Shot Classification

    cs.CV 2025-09 reject novelty 3.0 of 10

    ANROT-HELANet combines Hellinger aggregation, attention, and FGSM/Gaussian robust training for few-shot classification, but its ELBO derivation is invalid and its performance claims are overstated.

Pith tools