Pith. sign in

REVIEW 12 cited by

Supervised Contrastive Learning for Pre-trained Language Model Fine-tuning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2011.01403 v3 pith:LV2UE2G4 submitted 2020-11-03 cs.CL cs.LG

Beliz Gunel , Jingfei Du , Alexis Conneau , Ves Stoyanov This is my paper
classification cs.CLcs.LG
keywords fine-tuningdatacross-entropylanguagelearninglossmodelcontrastive
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

State-of-the-art natural language understanding classification models follow two-stages: pre-training a large language model on an auxiliary task, and then fine-tuning the model on a task-specific labeled dataset using cross-entropy loss. However, the cross-entropy loss has several shortcomings that can lead to sub-optimal generalization and instability. Driven by the intuition that good generalization requires capturing the similarity between examples in one class and contrasting them with examples in other classes, we propose a supervised contrastive learning (SCL) objective for the fine-tuning stage. Combined with cross-entropy, our proposed SCL loss obtains significant improvements over a strong RoBERTa-Large baseline on multiple datasets of the GLUE benchmark in few-shot learning settings, without requiring specialized architecture, data augmentations, memory banks, or additional unsupervised data. Our proposed fine-tuning objective leads to models that are more robust to different levels of noise in the fine-tuning training data, and can generalize better to related tasks with limited labeled data.

Discussion (0). Sign in to comment.

Forward citations

Cited by 12 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. SynthFix: Adaptive Neuro-Symbolic Code Vulnerability Repair

    cs.SE 2026-04 unverdicted novelty 7.0 of 10

    SynthFix adaptively routes LLM code repairs to supervised fine-tuning or symbolic-reward fine-tuning, yielding up to 32% higher exact match on JavaScript and C vulnerability benchmarks.

  2. Climber-Pilot: A Non-Myopic Generative Recommendation Model Towards Better Instruction-Following

    cs.IR 2026-02 conditional novelty 6.0 of 10

    A generative recommender combining time-masked multi-item training with condition-guided sparse attention improves long-horizon retrieval and instruction following, with a 4.24% like-rate lift online.

  3. Will Annotators Disagree? Identifying Subjectivity in Value-Laden Arguments

    cs.CL 2025-09 conditional novelty 6.0 of 10

    Directly predicting whether annotators will disagree on a value label outperforms inferring disagreement from per-annotator value predictions on the Touché23-ValueEval dataset.

  4. Cross Paraphrastic Invariance Learning for Hallucination Detection

    cs.CL 2026-06 unverdicted novelty 5.0 of 10

    CPIL is a contrastive two-stage method that enforces paraphrase invariance on limited labeled data to outperform baselines in hallucination detection across 11 tasks.

  5. G-Loss: Graph-Guided Fine-Tuning of Language Models

    cs.CL 2026-04 unverdicted novelty 5.0 of 10

    G-Loss builds a document-similarity graph and uses semi-supervised label propagation to guide fine-tuning of language models, yielding higher accuracy than standard losses on five classification benchmarks.

  6. SynthFix: Adaptive Neuro-Symbolic Code Vulnerability Repair

    cs.SE 2026-04 unverdicted novelty 5.0 of 10

    A router mixes supervised and reward fine-tuning with compiler/security feedback so small code LLMs produce more functionally correct and security-cleared vulnerability patches on three repair benchmarks.

  7. SEFRQO: A Self-Evolving Fine-Tuned RAG-Based Query Optimizer

    cs.DB 2025-08 unverdicted novelty 5.0 of 10

    SEFRQO claims a self-evolving fine-tuned LLM with retrieval and execution feedback reduces query latency versus PostgreSQL, but the provided body is a different paper, blocking verification.

  8. MetaExplainer: A Framework to Generate Multi-Type User-Centered Explanations for AI Systems

    cs.HC 2025-08 conditional novelty 5.0 of 10

    A neuro-symbolic pipeline decomposes user questions, delegates to model explainers, and synthesizes natural-language explanations, achieving moderate stage-wise scores on a diabetes dataset.

  9. Towards Privacy-Preserving and Personalized Smart Homes via Tailored Small Language Models

    cs.CR 2025-07 conditional novelty 5.0 of 10

    HomeLLaMA fine-tunes a local 8B language model on cloud-generated smart home data to provide personalized, privacy-preserving device control, with optional obfuscated cloud queries when the local model falls short.

  10. Contrastive Regularization for Accent-Robust ASR

    cs.SD 2026-05 unverdicted novelty 4.0 of 10

    Supervised contrastive learning as an auxiliary loss during CTC fine-tuning improves accent robustness in ASR, yielding up to 29% relative WER reduction on unseen accents.

  11. LLMSniffer: Detecting LLM-Generated Code via GraphCodeBERT and Supervised Contrastive Learning

    cs.SE 2026-04 unverdicted novelty 4.0 of 10

    LLMSniffer improves detection of LLM-generated code on GPTSniffer and Whodunit benchmarks by fine-tuning GraphCodeBERT via two-stage supervised contrastive learning plus preprocessing and MLP classification.

  12. Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges

    cs.AI 2025-07 reject novelty 1.0 of 10

    A broad survey of LLM alignment that catalogs objectives, benchmarks, SFT/RLHF/DPO methods, and safety challenges, without contributing new experimental or theoretical results.

Pith tools