REVIEW 5 cited by
Why Exposure Bias Matters: An Imitation Learning Perspective of Error Accumulation in Language Generation
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Why Exposure Bias Matters: An Imitation Learning Perspective of Error Accumulation in Language Generation
read the original abstract
Current language generation models suffer from issues such as repetition, incoherence, and hallucinations. An often-repeated hypothesis is that this brittleness of generation models is caused by the training and the generation procedure mismatch, also referred to as exposure bias. In this paper, we verify this hypothesis by analyzing exposure bias from an imitation learning perspective. We show that exposure bias leads to an accumulation of errors, analyze why perplexity fails to capture this accumulation, and empirically show that this accumulation results in poor generation quality. Source code to reproduce these experiments is available at https://github.com/kushalarora/quantifying_exposure_bias
Forward citations
Cited by 5 Pith papers
-
Cross-Tokenizer On-Policy Distillation via Byte-Prefix Marginalization
Byte-Prefix Marginalization maps a teacher's next-token distribution onto the student's vocabulary through shared byte prefixes plus an explicit residual, giving a mass-preserving target for on-policy distillation acr...
-
Neural operator discovery from heterogeneous trajectories
Trajectory grouping plus a low-dimensional latent bottleneck lets a neural operator discover each system's hidden governing factors and extrapolate to unseen systems.
-
Protein Autoregressive Modeling via Multiscale Structure Generation
PAR is a multi-scale autoregressive transformer framework for protein backbone generation that uses coarse-to-fine prediction, noisy context learning, and flow-based decoding to achieve high-quality unconditional and ...
-
Flow marching for a generative PDE foundation model
Flow Marching jointly samples noise and physical time to learn a velocity field for generative PDE modeling, paired with a latent autoencoder and efficient transformer for large-scale pretraining on 2.5M trajectories.
-
When Do Autoregressive Sequence Models Forecast Physical Wavefields? A Controlled Study on Synthetic Seismograms
Multi-token prediction accounts for nearly all rollout stability gains on synthetic three-component seismograms, with sharp dependence on context covering the full P-S interval and magnitude-based losses unable to pre...
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.