REVIEW 7 cited by
What is Wrong with Perplexity for Long-context Language Modeling?
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Handling long-context inputs is crucial for large language models (LLMs) in tasks such as extended conversations, document summarization, and many-shot in-context learning. While recent approaches have extended the context windows of LLMs and employed perplexity (PPL) as a standard evaluation metric, PPL has proven unreliable for assessing long-context capabilities. The underlying cause of this limitation has remained unclear. In this work, we provide a comprehensive explanation for this issue. We find that PPL overlooks key tokens, which are essential for long-context understanding, by averaging across all tokens and thereby obscuring the true performance of models in long-context scenarios. To address this, we propose \textbf{LongPPL}, a novel metric that focuses on key tokens by employing a long-short context contrastive method to identify them. Our experiments demonstrate that LongPPL strongly correlates with performance on various long-context benchmarks (e.g., Pearson correlation of -0.96), significantly outperforming traditional PPL in predictive accuracy. Additionally, we introduce \textbf{LongCE} (Long-context Cross-Entropy) loss, a re-weighting strategy for fine-tuning that prioritizes key tokens, leading to consistent improvements across diverse benchmarks. In summary, these contributions offer deeper insights into the limitations of PPL and present effective solutions for accurately evaluating and enhancing the long-context capabilities of LLMs. Code is available at https://github.com/PKU-ML/LongPPL.
Forward citations
Cited by 7 Pith papers
-
How Data Shapes RoPE Frequency Usage: From Positional Scale Matching to Length Generalization
RoPE frequency usage is determined by a data-induced dependency width W, with the optimal frequency scaling as π/W, explaining both learned spectra and the success of position interpolation.
-
MesaNet: Sequence Modeling by Locally Optimal Test-Time Training
MesaNet uses conjugate-gradient-optimal test-time regression in a chunkwise-parallelizable recurrent layer, achieving strong language modeling and benchmark performance at up to 1B scale.
-
Pretraining Recurrent Networks without Recurrence
SMT trains nonlinear RNNs by imitating one-step memory-transition labels generated by a Transformer, replacing BPTT's unrolled credit assignment with time-parallel supervised learning.
-
BehaviorBox: Automated Discovery of Fine-Grained Performance Differences Between Language Models
A pipeline that combines contextual embeddings with two LMs' per-word probabilities and sparse autoencoders to automatically find interpretable slices where one model outperforms another.
-
A Silver Bullet or a Compromise for Full Attention? A Comprehensive Study of Gist Token-based Context Compression
Gist-token compression is near-lossless for fuzzy long-context tasks but fails at exact recall; fine-grained autoencoding and token importance weighting recover some of the gap.
-
When Precision Meets Position: BFloat16 Breaks Down RoPE in Long-Context Training
BFloat16 rounding breaks RoPE's positional shift invariance, and AnchorAttention, which fixes the first token as a shared anchor, improves long-context training accuracy and speed.
-
Discrete Diffusion Models for Language Generation
This thesis reports an empirical D3PM versus autoregressive comparison on WikiText-103, but the claimed speed advantage is unsupported because the speed numbers duplicate NLL values.
Discussion (0). Continue with ORCID to comment.