The reviewed record of science sign in
Pith

arxiv: 1904.09751 · v2 · pith:MMMH3ZIP · submitted 2019-04-22 · cs.CL

The Curious Case of Neural Text Degeneration

Ari Holtzman , Jan Buys , Li Du , Maxwell Forbes , Yejin Choi This is my paper

Reviewed by Pith2026-05-12 06:13 UTCgrok-4.3pith:MMMH3ZIPopen to challenge →

classification cs.CL
keywords neural text generationnucleus samplingtext degenerationdecoding strategieslanguage modelssampling methodsdiversitycoherence
0
0 comments X

The pith

Nucleus sampling draws from the dynamic high-probability set to generate more diverse and coherent text than beam search or top-k methods.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Neural language models achieve strong results on understanding tasks when trained to maximize likelihood, yet the same models produce repetitive and uninteresting text when used to generate sequences. The paper demonstrates that this degeneration arises largely from the choice of decoding strategy rather than from flaws in the trained model. It identifies clear distributional mismatches between human-written text and machine-generated text. The authors introduce nucleus sampling, which selects the smallest set of tokens whose cumulative probability reaches a threshold and then samples from that set. This dynamic truncation removes the unreliable tail while preserving variety, producing output that humans rate as more fluent and human-like.

Core claim

The paper shows that the quality of text generated by a fixed neural language model depends heavily on the decoding procedure. Human text and machine text exhibit different probability distributions, with machine outputs often assigning overly high probability to repetitive tokens. The central contribution is nucleus sampling: at each step the model forms the smallest nucleus of tokens whose probabilities sum to at least p (commonly 0.9), then samples the next token from within that nucleus. This procedure yields text with greater lexical diversity and coherence than greedy decoding, beam search, or fixed top-k sampling, while avoiding the blandness that results from always choosing the most

What carries the argument

Nucleus sampling: the procedure that, at each generation step, identifies the smallest set of tokens whose cumulative probability meets or exceeds a threshold p and draws the next token uniformly from within that set, thereby truncating the low-probability tail.

Load-bearing premise

The learned probability distribution places lower-quality tokens in the tail, so removing that tail improves rather than harms the generated text.

What would settle it

Human raters scoring nucleus-sampled continuations as less diverse or less coherent than continuations produced by ancestral sampling or carefully tuned top-k sampling on the same model and prompts.

read the original abstract

Despite considerable advancements with deep neural language models, the enigma of neural text degeneration persists when these models are tested as text generators. The counter-intuitive empirical observation is that even though the use of likelihood as training objective leads to high quality models for a broad range of language understanding tasks, using likelihood as a decoding objective leads to text that is bland and strangely repetitive. In this paper, we reveal surprising distributional differences between human text and machine text. In addition, we find that decoding strategies alone can dramatically effect the quality of machine text, even when generated from exactly the same neural language model. Our findings motivate Nucleus Sampling, a simple but effective method to draw the best out of neural generation. By sampling text from the dynamic nucleus of the probability distribution, which allows for diversity while effectively truncating the less reliable tail of the distribution, the resulting text better demonstrates the quality of human text, yielding enhanced diversity without sacrificing fluency and coherence.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

0 major / 3 minor

Summary. The paper observes that neural language models produce degenerate text (repetitive and bland) under standard decoding methods such as greedy search and beam search, despite strong performance on likelihood-based training objectives. It documents distributional differences between human-written text and model-generated text, then introduces Nucleus Sampling: at each step, tokens are sampled from the smallest set whose cumulative probability mass exceeds a threshold p, thereby truncating the unreliable tail while preserving diversity. Controlled experiments on the same models across datasets compare this method against greedy, beam, top-k, and other baselines using both automatic diversity metrics and human judgments of fluency, coherence, and quality.

Significance. If the empirical results hold, the work is significant for open-ended neural text generation. It supplies a simple, parameter-light decoding rule that demonstrably improves human-judged output quality and diversity over widely used baselines, without requiring changes to model training. The controlled experimental design (identical models, multiple datasets, both automatic and human evaluation) provides reproducible evidence that decoding strategy alone can substantially affect generation quality.

minor comments (3)
  1. Abstract and §3: the phrase 'dynamic nucleus of the probability distribution' is introduced without an immediate formal definition or reference to the precise cumulative-probability rule; a one-sentence definition at first use would improve readability.
  2. Evaluation sections: human judgments are reported on a moderate scale and some automatic metrics are heuristic; adding error bars or statistical significance tests for the human ratings would strengthen the presentation without altering the central claim.
  3. Figure captions and tables: several plots compare multiple decoding strategies but lack explicit indication of which model size or dataset each panel corresponds to; consistent labeling would aid quick comprehension.

Simulated Author's Rebuttal

0 responses · 0 unresolved

We thank the referee for the positive and accurate summary of our work, as well as the recommendation for minor revision. The report correctly identifies the core issues with standard decoding methods and the benefits of nucleus sampling for improving diversity and quality in neural text generation.

Circularity Check

0 steps flagged

No significant circularity detected

full rationale

The paper's central contribution is an empirical analysis of distributional differences between human and machine text, followed by the definition of nucleus sampling as a direct function of the model's softmax probabilities (the smallest set of tokens whose cumulative probability mass exceeds threshold p). This definition contains no fitted parameters derived from the target evaluation metrics, no self-referential equations, and no load-bearing self-citations. Quality and diversity improvements are measured with independent human judgments and automatic metrics (e.g., distinct-n, self-BLEU) that are not algebraically entailed by the sampling rule itself. The derivation chain is therefore self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

1 free parameters · 1 axioms · 0 invented entities

The contribution rests on one tunable hyperparameter and a standard domain assumption about model probabilities; no new entities or fitted constants are introduced.

free parameters (1)
  • p (nucleus probability threshold)
    User-chosen hyperparameter (commonly 0.9) that controls the size of the sampling set; not learned from data in the paper.
axioms (1)
  • domain assumption The neural language model's softmax probabilities meaningfully rank token quality for generation.
    Invoked when justifying truncation of the probability tail as removing unreliable tokens.

pith-pipeline@v0.9.0 · 5462 in / 1211 out tokens · 66969 ms · 2026-05-12T06:13:35.190353+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 60 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Molmo2: Open Weights and Data for Vision-Language Models with Video Understanding and Grounding

    cs.CV 2026-01 unverdicted novelty 8.0

    Molmo2 delivers state-of-the-art open-weight video VLMs with new grounding datasets and training methods that outperform prior open models and match or exceed some proprietary ones on pointing and tracking tasks.

  2. Large Language Diffusion Models

    cs.CL 2025-02 unverdicted novelty 8.0

    LLaDA is a scalable diffusion-based language model that matches autoregressive LLMs like LLaMA3 8B on tasks and surpasses GPT-4o on reversal poem completion.

  3. TinyStories: How Small Can Language Models Be and Still Speak Coherent English?

    cs.CL 2023-05 conditional novelty 8.0

    Tiny language models under 10M parameters trained on a synthetic children's story dataset generate fluent, consistent, multi-paragraph English text with near-perfect grammar and reasoning.

  4. PAL: Program-aided Language Models

    cs.CL 2022-11 conditional novelty 8.0

    PAL improves few-shot reasoning accuracy by having LLMs generate executable programs rather than text-based chains of thought, outperforming much larger models on math and logic benchmarks.

  5. Language Models are Few-Shot Learners

    cs.CL 2020-05 accept novelty 8.0

    GPT-3 shows that scaling an autoregressive language model to 175 billion parameters enables strong few-shot performance across diverse NLP tasks via in-context prompting without fine-tuning.

  6. HellaSwag: Can a Machine Really Finish Your Sentence?

    cs.CL 2019-05 unverdicted novelty 8.0

    HellaSwag dataset shows state-of-the-art models fail commonsense inference tasks that humans solve easily, built via adversarial filtering of distractors.

  7. MeshFlow: Mesh Generation with Equivariant Flow Matching

    cs.GR 2026-06 unverdicted novelty 7.0

    MeshFlow applies equivariant optimal-transport flow matching to generate triangle meshes as soups, matching autoregressive quality with an 18x inference speedup.

  8. Can AI Agents Synthesize Scientific Conclusions?

    cs.AI 2026-06 unverdicted novelty 7.0

    A new benchmark and clean-room harness show frontier AI agents reach only 0.337 factual F1 when synthesizing conclusions from scientific evidence.

  9. PrivCode++: Latent-Conditioned Differentially Private Code Generation for Comprehensive Guarantees

    cs.CR 2026-06 unverdicted novelty 7.0

    PrivCode++ introduces the first DP code generation method protecting both prompts and code via latent-conditioned two-stage training, claiming higher utility and stronger privacy than prior baselines.

  10. Before and After Temperature: A Distributional View of Creative LLM Generation

    cs.CL 2026-05 unverdicted novelty 7.0

    A per-token feature from temperature-induced changes in LLM token distributions predicts within-prompt creativity rank at Spearman rho 0.918 vs LLM judges and 0.870 vs humans, outperforming perplexity, entropy, top-1 ...

  11. Probabilistic Attribution For Large Language Models

    cs.CL 2026-05 unverdicted novelty 7.0

    Develops a model-agnostic attribution score as the log-ratio of conditional response probabilities with and without a marginalized prompt token, derived via Bayes inversion of next-token distributions, and relates it ...

  12. Structured Layout Priors for Robust Out-of-Distribution Visual Document Understanding

    cs.CV 2026-05 conditional novelty 7.0

    Injecting pre-computed layout priors from RT-DETR into VLM prompts raises markdown F1 from 0.37 to 0.92 on a 10k-page OOD benchmark and cuts infinite-loop failures across domains.

  13. BOOKMARKS: Efficient Active Storyline Memory for Role-playing

    cs.CL 2026-05 unverdicted novelty 7.0

    BOOKMARKS introduces searchable bookmarks as reusable answers to storyline questions, enabling active initialization and passive synchronization for more consistent role-playing agent memory than recurrent summarization.

  14. StepCodeReasoner: Aligning Code Reasoning with Stepwise Execution Traces via Reinforcement Learning

    cs.SE 2026-05 unverdicted novelty 7.0

    StepCodeReasoner aligns code reasoning with verifiable stepwise execution traces via print anchors and bi-level GRPO reinforcement learning, reaching SOTA results on CRUXEval (91.1%) and LiveCodeBench (86.5%) for a 7B model.

  15. Self-Attention as a Covariance Readout: A Unified View of In-Context Learning and Repetition

    cs.LG 2026-05 unverdicted novelty 7.0

    Self-attention acts as a covariance readout that unifies in-context learning via population gradient descent and repetitive generation via asymptotic Markov behavior.

  16. CUDABeaver: Benchmarking LLM-Based Automated CUDA Debugging

    cs.LG 2026-05 unverdicted novelty 7.0

    CUDABeaver shows LLM CUDA debuggers often degenerate code for test-passing at the cost of speed, with protocol-aware metrics shifting success rates by up to 40 percentage points.

  17. PairAlign: A Framework for Sequence Tokenization via Self-Alignment with Applications to Audio Tokenization

    cs.LG 2026-05 unverdicted novelty 7.0

    PairAlign learns compact variable-length token sequences for audio via self-alignment on paired content-preserving views, achieving 55% fewer archive tokens than VQ while preserving edit-distance retrieval at 12.71 tokens/s.

  18. PairAlign: A Framework for Sequence Tokenization via Self-Alignment with Applications to Audio Tokenization

    cs.LG 2026-05 unverdicted novelty 7.0

    PairAlign learns compact audio token sequences via self-alignment of paired content views using an autoregressive decoder, achieving strong cross-view consistency and edit-distance preservation while reducing token co...

  19. Ex Ante Evaluation of AI-Induced Idea Diversity Collapse

    cs.AI 2026-05 unverdicted novelty 7.0

    Frontier LLMs generate creative ideas with excess population-level crowding below human-relative parity across tasks, but targeted generation protocols can reduce it.

  20. CP-SynC: Multi-Agent Zero-Shot Constraint Modeling in MiniZinc with Synthesized Checkers

    cs.AI 2026-05 unverdicted novelty 7.0

    CP-SynC uses coordinated LLM agents to generate, validate via synthesized checkers, and select MiniZinc models from natural language, substantially outperforming baselines on a 100-problem benchmark.

  21. Toward a Principled Framework for Agent Safety Measurement

    cs.CR 2026-05 unverdicted novelty 7.0

    BOA uses budgeted search over agent trajectories to report the probability an LLM agent stays safe, finding unsafe paths that sampling misses.

  22. Efficient Test-Time Inference via Deterministic Exploration of Truncated Decoding Trees

    cs.LG 2026-04 unverdicted novelty 7.0

    Distinct Leaf Enumeration (DLE) replaces stochastic self-consistency sampling with deterministic traversal of a truncated decoding tree to enumerate distinct leaves, increasing coverage and reducing redundant computat...

  23. Post-Selection Distributional Model Evaluation

    stat.ML 2026-03 unverdicted novelty 7.0

    PS-DME is a new framework that controls post-selection false coverage rate for distributional KPI estimates via e-values and is provably more sample-efficient than data splitting under explicit conditions.

  24. Improving Sampling for Masked Diffusion Models via Information Gain

    cs.CL 2026-02 unverdicted novelty 7.0

    Info-Gain Sampler improves MDM decoding by using bidirectional information gain to reduce cumulative uncertainty, outperforming greedy samplers on reasoning accuracy and creative writing tasks.

  25. Geometry-Aware Decoding with Wasserstein-Regularized Truncation and Mass Penalties for Large Language Models

    cs.CL 2026-02 unverdicted novelty 7.0

    Top-W applies Wasserstein-regularized truncation on token-embedding geometry to create a closed-form optimal crop for LLM sampling that outperforms prior methods by up to 33.7% on GSM8K, GPQA, AlpacaEval, and MT-Bench.

  26. On the Overscaling Curse of Parallel Thinking: System Efficacy Contradicts Sample Efficiency

    cs.LG 2026-01 unverdicted novelty 7.0

    Parallel thinking in LLMs suffers from overscaling where fixed global budgets waste samples; LanBo predicts per-sample budgets from latent states to raise utilization without hurting accuracy.

  27. OPT-Engine: Benchmarking the Limits of LLMs in Optimization Modeling via Complexity Scaling

    cs.CL 2026-01 accept novelty 7.0

    OPT-Engine shows pure-text chain-of-thought reasoning in LLMs loses robustness as optimization complexity grows, external tools fix only local arithmetic, and solver-integrated methods are bottlenecked by automated co...

  28. Digitizing Nepal's Written Heritage: A Comprehensive HTR Pipeline for Old Nepali Manuscripts

    cs.LG 2025-12 unverdicted novelty 7.0

    The first HTR pipeline for Old Nepali manuscripts achieves 4.9% character error rate with released training code and scripts for low-resource historical scripts.

  29. BEAVER: An Efficient Deterministic LLM Verifier

    cs.AI 2025-12 unverdicted novelty 7.0

    BEAVER is the first practical deterministic verifier that maintains sound probability bounds on LLM safety properties using token tries and frontier data structures, finding 2-3x more violations than sampling at 1/10 ...

  30. TSVer: A Benchmark for Fact Verification Against Time-Series Evidence

    cs.CL 2025-11 unverdicted novelty 7.0

    TSVer is a new benchmark dataset for fact verification against time-series evidence, with 304 annotated real-world claims, 400 time series, verdicts, and justifications, plus baseline results showing current models struggle.

  31. Top-H Decoding: Adapting the Creativity and Coherence with Bounded Entropy in Text Generation

    cs.CL 2025-09 unverdicted novelty 7.0

    Top-H decoding is a computationally efficient greedy algorithm for an entropy-constrained mass maximization problem that improves the creativity-coherence trade-off over min-p sampling in LLM text generation.

  32. Constrained Decoding for Safe Robot Navigation Foundation Models

    cs.RO 2025-09 unverdicted novelty 7.0

    SafeDec uses constrained decoding to ensure autoregressive robot navigation foundation models generate actions that provably satisfy STL safety specifications under assumed dynamics.

  33. Moshi: a speech-text foundation model for real-time dialogue

    eess.AS 2024-09 accept novelty 7.0

    Moshi is the first real-time full-duplex spoken large language model that casts dialogue as speech-to-speech generation using parallel audio streams and an inner monologue of time-aligned text tokens.

  34. Chronos: Learning the Language of Time Series

    cs.LG 2024-03 conditional novelty 7.0

    Chronos pretrains transformer models on tokenized time series to deliver strong zero-shot forecasting across diverse domains.

  35. Self-Rewarding Language Models

    cs.CL 2024-01 conditional novelty 7.0

    Iterative self-rewarding via LLM-as-Judge in DPO training on Llama 2 70B improves instruction following and self-evaluation, outperforming GPT-4 on AlpacaEval 2.0.

  36. Learning to summarize from human feedback

    cs.CL 2020-09 conditional novelty 7.0

    Reinforcement learning on a reward model trained from human summary comparisons produces summaries humans prefer over supervised fine-tuning or human references on TL;DR and transfers to CNN/DM.

  37. When More Sampling Hurts: The Modal Ceiling and Correlation Ceiling of Test-Time Scaling

    cs.LG 2026-06 unverdicted novelty 6.0

    Test-time sampling improves coverage but stalls at modal and correlation ceilings for answer selection, with the effective number of samples as the practical limit.

  38. Brevity is the Soul of Inference Efficiency: Inducing Concision in VLMs via Data Curation

    cs.LG 2026-06 unverdicted novelty 6.0

    Curating pretraining data for concision in VLMs produces models with up to 35x lower cost-of-pass at matched accuracy by reducing output token count.

  39. Brevity is the Soul of Inference Efficiency: Inducing Concision in VLMs via Data Curation

    cs.LG 2026-06 unverdicted novelty 6.0

    Curating concise data for VLMs induces brevity, delivering 35x lower Cost-of-Pass at near-identical accuracy and higher matched-length accuracy than uncurated baselines.

  40. TextHOI-3D: Text-to-3D Hand-Object Interaction via Discrete Multi-View Generation and Joint Mesh Optimization

    cs.CV 2026-06 unverdicted novelty 6.0

    TextHOI-3D generates text-conditioned 3D hand-object meshes using a VQ token space and CLIP-conditioned autoregressive multi-view prediction followed by joint mesh optimization, reporting large reductions in object CD...

  41. Synthetic Data from Cross-Domain Events for Large-Scale Recommendation Systems

    cs.IR 2026-05 unverdicted novelty 6.0

    SCALR generates synthetic cross-domain events to augment recommendation training data and reports statistically significant improvements in industrial A/B tests.

  42. Optimising Factual Consistency in Summarisation via Preference Learning from Multiple Imperfect Metrics

    cs.CL 2026-05 unverdicted novelty 6.0

    Aggregating imperfect factuality metrics into preference data from lexically similar summaries yields consistent factuality gains across model sizes, allowing smaller models to approach larger ones.

  43. Convex Optimization for Alignment and Preference Learning on a Single GPU

    cs.LG 2026-05 unverdicted novelty 6.0

    COALA applies convex optimization reformulations of neural networks to direct preference optimization, claiming single-GPU training with ~18% of DPO's TFLOPs and competitive performance on multiple datasets and models...

  44. STiTch: Semantic Transition and Transportation in Collaboration for Training-Free Zero-Shot Composed Image Retrieval

    cs.CV 2026-05 unverdicted novelty 6.0

    STiTch refines LLM captions via embedding transition and uses set-to-set bidirectional transportation alignment to improve training-free zero-shot composed image retrieval.

  45. Staging by the Book: Automatic Sleep Stage Classification Using Scoring Rules

    eess.SP 2026-05 conditional novelty 6.0

    A rule-based sleep staging method operationalizing AASM scoring rules achieves 60.5% agreement with human majority-vote consensus on 50 PSG recordings while providing epoch-level explanations.

  46. Formal Methods Meet LLMs: Auditing, Monitoring, and Intervention for Compliance of Advanced AI Systems

    cs.AI 2026-05 unverdicted novelty 6.0

    Combines LTL formal methods with LLMs for auditing, predictive monitoring, and runtime intervention on temporally extended behavioral constraints, outperforming LLM baselines and reducing violations.

  47. Asking Back: Interaction-Layer Antidistillation Watermarks

    cs.CR 2026-05 unverdicted novelty 6.0

    Interaction-layer antidistillation watermarks use system-prompt-induced behavioral markers like explicit follow-up questions that transfer to distilled student models at 45-89% relative fidelity and can be audited via...

  48. Why are language models less surprised than humans? Testing the Parse Multiplicity Mismatch Hypothesis

    cs.CL 2026-05 conditional novelty 6.0

    Varying the number of simultaneous parses in RNNGs increases predicted garden-path effects but does not fully reconcile LM surprisal with human reading times.

  49. TextSeal: A Localized LLM Watermark for Provenance & Distillation Protection

    cs.CR 2026-05 unverdicted novelty 6.0

    TextSeal provides a localized, distortion-free LLM watermark that outperforms baselines in detection strength, remains effective in mixed human-AI text, preserves model performance, and transfers through distillation ...

  50. TextSeal: A Localized LLM Watermark for Provenance & Distillation Protection

    cs.CR 2026-05 unverdicted novelty 6.0

    TextSeal provides a localized, distortion-free LLM watermark that enables provenance tracking and distillation detection while preserving performance and text quality.

  51. SOMA: Efficient Multi-turn LLM Serving via Small Language Model

    cs.CL 2026-05 unverdicted novelty 6.0

    SOMA estimates a local response manifold from early turns and adapts a small surrogate model via divergence-maximizing prompts and localized LoRA fine-tuning for efficient multi-turn serving.

  52. Adversarial SQL Injection Generation with LLM-Based Architectures

    cs.CR 2026-05 unverdicted novelty 6.0

    RADAGAS-GPT4o achieves a 22.73% bypass rate against 10 WAFs, succeeding more against AI/ML-based firewalls than rule-based ones.

  53. Annotations Mitigate Post-Training Mode Collapse

    cs.CL 2026-05 unverdicted novelty 6.0

    Annotation-anchored training reduces semantic diversity collapse in post-trained language models by a factor of six compared to standard supervised fine-tuning while preserving instruction-following and improving with scale.

  54. APCD: Adaptive Path-Contrastive Decoding for Reliable Large Language Model Generation

    cs.CL 2026-05 unverdicted novelty 6.0

    APCD reduces LLM hallucinations by expanding decoding paths adaptively when entropy signals uncertainty and by contrasting divergent paths to control their interaction.

  55. Structured Recurrent Mixers for Massively Parallelized Sequence Generation

    cs.CL 2026-05 unverdicted novelty 6.0

    Structured Recurrent Mixers enable algebraic switching between parallel training and recurrent inference representations, delivering higher efficiency, information capacity, and throughput than other linear-complexity models.

  56. Structured Recurrent Mixers for Massively Parallelized Sequence Generation

    cs.CL 2026-05 conditional novelty 6.0

    Structured Recurrent Mixers enable algebraic switching between parallel training and recurrent inference representations, yielding higher throughput, concurrency, and training efficiency than comparable linear-complex...

  57. Structured Recurrent Mixers for Massively Parallelized Sequence Generation

    cs.CL 2026-05 unverdicted novelty 6.0

    Structured Recurrent Mixers provide a dual parallel-recurrent representation for sequence models, claiming superior training efficiency, information capacity, and inference throughput over linear complexity alternatives.

  58. CUDABeaver: Benchmarking LLM-Based Automated CUDA Debugging

    cs.LG 2026-05 unverdicted novelty 6.0

    CUDABEAVER benchmark and pass@k(M,C,A) metric show LLM CUDA debugging success drops by up to 40 percentage points under strict performance requirements.

  59. Continuous Latent Diffusion Language Model

    cs.CL 2026-05 unverdicted novelty 6.0

    Cola DLM proposes a hierarchical latent diffusion model that learns a text-to-latent mapping, fits a global semantic prior in continuous space with a block-causal DiT, and performs conditional decoding, establishing l...

  60. Diversity in Large Language Models under Supervised Fine-Tuning

    cs.LG 2026-04 unverdicted novelty 6.0

    TOFU loss mitigates the narrowing of generative diversity in LLMs after supervised fine-tuning by addressing neglect of low-frequency patterns and forgetting of prior knowledge.

Reference graph

Works this paper leans on

16 extracted references · 16 canonical work pages · cited by 109 Pith papers · 2 internal anchors

  1. [1]

    Neural machine translation by jointly learning to align and translate

    Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. Proceedings of the 2015 International Conference on Learning Representations,

    work page 2015

  2. [2]

    Language gans falling short

    Massimo Caccia, Lucas Caccia, William Fedus, Hugo Larochelle, Joelle Pineau, and Laurent Char- lin. Language gans falling short. In Critiquing and Correcting Trends in Machine Learning: NeurIPS 2018 Workshop,

    work page 2018

  3. [3]

    Language GANs falling short

    URL http://arxiv.org/abs/1811.02549. Yining Chen, Sorcha Gilroy, Andreas Maletti, Jonathan May, and Kevin Knight. Recurrent neu- ral networks as weighted language recognizers. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 2261–2...

    work page arXiv 2018

  4. [4]

    Elizabeth Clark, Yangfeng Ji, and Noah A. Smith. Neural text generation in stories using entity rep- resentations as context. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 2250–2260, New Orleans, Louisiana, June

    work page 2018

  5. [5]

    Hierarchical neural story generation

    10 Published as a conference paper at ICLR 2020 Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pp. 889–898,

    work page 2020

  6. [6]

    Hashimoto, Hugh Zhang, and Percy Liang

    Tatsunori B. Hashimoto, Hugh Zhang, and Percy Liang. Unifying human and statistical evaluation for natural language generation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,

    work page 2019

  7. [7]

    A simple, fast diverse decoding algorithm for neural generation

    Jiwei Li, Will Monroe, and Dan Jurafsky. A simple, fast diverse decoding algorithm for neural generation. arXiv preprint arXiv:1611.08562, 2016a. Jiwei Li, Will Monroe, Alan Ritter, Dan Jurafsky, Michel Galley, and Jianfeng Gao. Deep rein- forcement learning for dialogue generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Lan...

    work page internal anchor Pith review arXiv 2016

  8. [8]

    Sparse forward-backward using minimum diver- gence beams for fast training of conditional random fields

    Chris Pal, Charles Sutton, and Andrew McCallum. Sparse forward-backward using minimum diver- gence beams for fast training of conditional random fields. In 2006 IEEE International Confer- ence on Acoustics Speech and Signal Processing Proceedings, volume 5, May

    work page 2006

  9. [9]

    Steven T Piantadosi

    doi: 10.18653/v1/W18-1505. Steven T Piantadosi. Zipfs word frequency law in natural language: A critical review and future directions. Psychonomic bulletin & review, 21(5):1112–1130,

    work page doi:10.18653/v1/w18-1505

  10. [10]

    Unpublished manuscript

    URL https: //d4mucfpksywv.cloudfront.net/better-language-models/language_ models_are_unsupervised_multitask_learners.pdf. Unpublished manuscript. Stanislau Semeniuta, Aliaksei Severyn, and Sylvain Gelly. On accurate evaluation of gans for lan- guage generation. arXiv preprint arXiv:1806.04936,

    work page internal anchor Pith review arXiv

  11. [11]

    Style transfer from non-parallel text by cross-alignment

    11 Published as a conference paper at ICLR 2020 Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi Jaakkola. Style transfer from non-parallel text by cross-alignment. In Advances in neural information processing systems, pp. 6830–6841,

    work page 2020

  12. [12]

    Felix Stahlberg and Bill Byrne. On nmt search errors and model errors: Cat got your tongue? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pp. 3347–3353,

    work page 2019

  13. [13]

    Evaluating text gans as language models

    Guy Tevet, Gavriel Habib, Vered Shwartz, and Jonathan Berant. Evaluating text gans as language models. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 2241–2247,

    work page 2019

  14. [14]

    Challenges in data-to-document generation

    Sam Wiseman, Stuart Shieber, and Alexander Rush. Challenges in data-to-document generation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 2253–2263, Copenhagen, Denmark, September

    work page 2017

  15. [15]

    Diversity-promoting gan: A cross-entropy based generative adversarial network for diversified text generation

    Jingjing Xu, Xuancheng Ren, Junyang Lin, and Xu Sun. Diversity-promoting gan: A cross-entropy based generative adversarial network for diversified text generation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pp. 3940–3949, Brussels, Belgium, oct

    work page 2018

  16. [16]

    S" series. I have a question about the new \

    12 Published as a conference paper at ICLR 2020 A B EAM WIDTH EFFECT Figure 10: The total number of trigrams produced by Beam Search with varying beam widths, with gold (human) data for comparison. Note how the average length of generations goes down linearly with beam width, while the number of distinct trigrams stays constant and extremely low in compar...

    work page 2020