Removing dropout from single-epoch language model pretraining improves downstream task performance and model editability across multiple architectures and benchmarks.
Batching was done sequentially with the Pytorch Data Loader, sequence lengths are capped at 512 tokens
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
fields
cs.CL 1years
2025 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Drop Dropout on Single-Epoch Language Model Pretraining
Removing dropout from single-epoch language model pretraining improves downstream task performance and model editability across multiple architectures and benchmarks.