REVIEW 4 cited by
Can Perplexity Reflect Large Language Model's Ability in Long Text Understanding?
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Recent studies have shown that Large Language Models (LLMs) have the potential to process extremely long text. Many works only evaluate LLMs' long-text processing ability on the language modeling task, with perplexity (PPL) as the evaluation metric. However, in our study, we find that there is no correlation between PPL and LLMs' long-text understanding ability. Besides, PPL may only reflect the model's ability to model local information instead of catching long-range dependency. Therefore, only using PPL to prove the model could process long text is inappropriate. The local focus feature of PPL could also explain some existing phenomena, such as the great extrapolation ability of the position method ALiBi. When evaluating a model's ability in long text, we might pay more attention to PPL's limitation and avoid overly relying on it.
Forward citations
Cited by 4 Pith papers
-
MesaNet: Sequence Modeling by Locally Optimal Test-Time Training
MesaNet uses conjugate-gradient-optimal test-time regression in a chunkwise-parallelizable recurrent layer, achieving strong language modeling and benchmark performance at up to 1B scale.
-
Estimating the Effects of Sample Training Orders for Large Language Models without Retraining
A framework using Taylor expansions and random projections estimates LLM performance under arbitrary training batch orders from one reference run.
-
Hierarchical Tree Search-based User Lifelong Behavior Modeling on Large Language Model
HiT-LBM chunks lifelong user behavior, uses process-rating-guided tree search to select interest summaries, and fuses them temporally to improve recommendation performance.
-
Theoretical Benefit and Limitation of Diffusion Language Model
Masked diffusion language models have a metric-dependent efficiency tradeoff: near-optimal perplexity in constant steps, but sequence-level correctness needs linearly many steps in the worst case.
Discussion (0). Sign in to comment.