Pith. sign in

REVIEW 13 cited by

Contrastive Decoding: Open-ended Text Generation as Optimization

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2210.15097 v2 pith:23S2GPBX submitted 2022-10-27 cs.CL cs.AIcs.LG

Xiang Lisa Li , Ari Holtzman , Daniel Fried , Percy Liang , Jason Eisner , Tatsunori Hashimoto , Luke Zettlemoyer , Mike Lewis This is my paper
classification cs.CLcs.AIcs.LG
keywords decodingcontrastivetextobjectiveacrosscalledconstraintdifference
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Given a language model (LM), maximum probability is a poor decoding objective for open-ended generation, because it produces short and repetitive text. On the other hand, sampling can often produce incoherent text that drifts from the original topics. We propose contrastive decoding (CD), a reliable decoding approach that optimizes a contrastive objective subject to a plausibility constraint. The contrastive objective returns the difference between the likelihood under a large LM (called the expert, e.g. OPT-13B) and a small LM (called the amateur, e.g. OPT-125M), and the constraint ensures that the outputs are plausible. CD is inspired by the fact that the failures of larger LMs (e.g., repetition, incoherence) are even more prevalent in smaller LMs, and that this difference signals which texts should be preferred. CD requires zero additional training, and produces higher quality text than decoding from the larger LM alone. It also works across model scales (OPT-13B and GPT2-1.5B) and significantly outperforms four strong decoding algorithms (e.g., nucleus, top-k) in automatic and human evaluations across wikipedia, news and story domains.

Discussion (0). Sign in to comment.

Forward citations

Cited by 13 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Adaptive Perturbation Selection for Contrastive Audio Decoding

    cs.SD 2026-06 unverdicted novelty 6.0 of 10

    A learned per-example router over a 105-perturbation audio library improves contrastive decoding for audio-LLM hallucination, with task-dependent best distortions (e.g., reverse audio for temporal order).

  2. T5-CSBoost: Adversarial Perturbation Resistant LLM Fingerprinting

    cs.CL 2026-05 conditional novelty 6.0 of 10

    Adding a margin-based triplet loss to T5-Sentinel's decoder embeddings improves LLM source attribution robustness to word/character edits, paraphrasing, and unseen models/domains.

  3. Distribution Preference Optimization: A Fine-grained Perspective for LLM Unlearning

    cs.LG 2025-10 conditional novelty 6.0 of 10

    DiPO is a distribution-level unlearning method that constructs preference distributions from the model's own high-confidence logits and achieves state-of-the-art forget quality on TOFU while preserving utility.

  4. INTER: Mitigating Hallucination in Large Vision-Language Models by Interaction Guidance Sampling

    cs.CV 2025-07 conditional novelty 6.0 of 10

    INTER is a training-free logit-correction method that adds Harsanyi interaction scores to selected keyword tokens, lowering hallucination on six LVLM benchmarks.

  5. What makes Reasoning Models Different? Follow the Reasoning Leader for Efficient Decoding

    cs.CL 2025-06 conditional novelty 6.0 of 10

    FoReaL-Decoding lets a strong reasoning model generate the first few tokens of each sentence and a weaker model complete the sentence, cutting theoretical FLOPs by 30-55% while retaining 86-100% of accuracy on four ma...

  6. Token Signature: Predicting Chain-of-Thought Gains with Token Decoding Feature in Large Language Models

    cs.CL 2025-06 conditional novelty 6.0 of 10

    The monotonicity of token probabilities during initial decoding predicts chain-of-thought gains, enabling dynamic selection between CoT and direct answers.

  7. Dual Debiasing for Noisy In-Context Learning for Text Generation

    cs.CL 2025-05 conditional novelty 6.0 of 10

    A dual-debiasing method normalizes perplexity by the model's prior knowledge and a query-specific baseline, detecting noisy ICL demonstrations even at 80% noise.

  8. Avoidance Decoding for Diverse Multi-Branch Story Generation

    cs.CL 2025-09 conditional novelty 5.0 of 10

    Avoidance Decoding penalizes token choices that resemble previously generated story branches, using a hybrid concept-level and narrative-level similarity penalty, and reports large diversity gains across several LLMs.

  9. Enhancing Chain-of-Thought Reasoning with Critical Representation Fine-tuning

    cs.CL 2025-07 conditional novelty 5.0 of 10

    CRFT selects critical internal representations via attention and saliency scores and fine-tunes only them, improving GSM8K accuracy over ReFT from 29.0% to 32.8% on LLaMA-2-7B.

  10. DecoRTL: A Run-time Decoding Framework for RTL Code Generation with LLMs

    cs.PL 2025-07 conditional novelty 5.0 of 10

    DecoRTL combines token-class-aware temperature adjustment with contrastive top-K reranking to improve synthesizability and functional correctness of LLM-generated Verilog.

  11. In-situ Autoguidance: Eliciting Self-Correction in Diffusion Models

    cs.LG 2025-10 reject novelty 4.0 of 10

    On ImageNet 512, dropout-based self-guidance yields FID 2.57 vs 2.56 unguided and FDDINOv2 90.05 vs 68.64, providing no evidence of the claimed guidance benefit.

  12. CARE: Decoding Time Safety Alignment via Rollback and Introspection Intervention

    cs.LG 2025-09 conditional novelty 4.0 of 10

    CARE uses guard-model detection, token-buffer rollback, and self-reflective prompting to reduce harmful responses while preserving response quality.

  13. Reducing Object Hallucination in Large Audio-Language Models via Audio-Aware Decoding

    eess.AS 2025-06 conditional novelty 4.0 of 10

    Audio-Aware Decoding, a contrastive decoding method that uses silent audio as the no-context baseline, reduces object hallucination and improves accuracy across three large audio-language models.

Pith tools