ESLM keeps only high-loss or high-entropy tokens in each batch via a value-at-risk threshold, cutting pretraining FLOPs by about 6% while roughly matching perplexity and downstream accuracy.
Title resolution pending
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
fields
cs.LG 1years
2025 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
ESLM: Risk-Averse Selective Language Modeling for Efficient Pretraining
ESLM keeps only high-loss or high-entropy tokens in each batch via a value-at-risk threshold, cutting pretraining FLOPs by about 6% while roughly matching perplexity and downstream accuracy.