Pith. sign in

REVIEW 3 major objections 6 minor 17 references

Drop Dropout on Single-Epoch Language Model Pretraining

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read In single-epoch pretraining of language models, removing dropout entirely beats every dropout setting tested.

desk verdict Useful empirical study with a clear signal, but the reported training budgets contradict the single-epoch premise, so the paper's central claim as stated is not actually tested. read the letter →

arxiv 2505.24788 v1 pith:IANF3QSI submitted 2025-05-30 cs.CL cs.AI

Houjun Liu , John Bauer , Christopher D. Manning This is my paper
classification cs.CLcs.AI
keywords dropoutsingle-epochpretraininglanguagemodelingregularizationearlymodeleditingBERTPythia
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that dropout, a regularization technique designed to reduce overfitting, is not only unnecessary but harmful when language models are pretrained for a single epoch. Across masked (BERT) and autoregressive (Pythia 160M and 1.4B) models, the authors find that every dropout setting tested, including the recently proposed early dropout, produces worse models than no dropout at all. No-dropout models score better on language modeling loss, the BLiMP grammaticality benchmark, SQuAD question answering, and MNLI natural language inference, and they are easier to edit with a gradient-based editing method. The practical upshot is that practitioners pretraining in the single-epoch regime can save the cost of tuning a dropout rate and expect better models by setting it to zero.

What carries the argument

The argument rests on controlled pretraining comparisons in which only the dropout configuration varies: static dropout at $p = 0.0$, $p = 0.1$, and $p = 0.3$ on attention and MLP blocks, plus an early-dropout schedule that disables dropout after the first 35% of training. The same models are then evaluated on the pretraining loss, on BLiMP minimal-pair judgments, on fine-tuned SQuAD and MNLI, and on two model-editing interventions: MEND, which learns to project fine-tuning gradients into localized weight updates, and ReFT, which intervenes in an orthogonal representation subspace. A hypothesized mechanism ties the results together: dropout reduces feature co-adaptation by forcing the network to maintain multiple independent feature pathways, which makes stored knowledge less localized and therefore less capable on downstream tasks and harder to edit.

What would settle it

Measure the training-loss and held-out-loss curves during single-epoch pretraining for dropout rates $p = 0.0$ and $p = 0.3$; if the no-dropout model's held-out loss gap widens relative to the dropout model's as training progresses, or if extending training beyond one epoch or repeating data makes dropout models win, the paper's central claim is falsified.

Watch

Extended reading notes

Core claim

The paper's central claim is that in single-epoch pretraining, dropout degrades downstream capability and editability rather than improving them. The authors pretrain BERT-style masked language models and Pythia decoder models at dropout rates of $p = 0.0$, $p = 0.1$, and $p = 0.3$, and also with early dropout that is switched off after 35% of training. They report that the $p = 0.0$ models achieve lower held-out language modeling loss and higher scores on BLiMP, fine-tuned SQuAD, and MNLI, and that early dropout performs worse than no dropout. On model editing, no-dropout models show higher MEND edit success once training is sufficient and equivalent ReFT edit success. The paper concludes that dropout should be dropped from single-epoch language model pretraining.

Load-bearing premise

The load-bearing premise is that a single-epoch pass over the data produces so little overfitting that dropout's regularization is unnecessary; the paper does not directly measure the train-versus-held-out loss gap that would confirm this.

Editorial extensions

If this is right

  • Standard single-epoch pretraining recipes for both masked and autoregressive language models should set dropout to zero in attention and MLP blocks.
  • Early dropout, recently proposed as a stabilization technique, should not be used in single-epoch pretraining because it degrades performance relative to no dropout.
  • Removing dropout during pretraining improves gradient-based model editing with MEND and leaves representation-based editing with ReFT unchanged.
  • The performance penalty grows with dropout rate, so even a small dropout like $p = 0.15$ slows learning and destabilizes convergence compared with $p = 0.0$.
  • The benefit of no-dropout holds consistently across capability measures, including language modeling loss, BLiMP, fine-tuned SQuAD, and MNLI.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the no-dropout recommendation is likely bounded to the single-epoch regime; if data is repeated or training is extended to multiple epochs, overfitting pressure rises and dropout may regain its value.
  • Beyond the paper: the higher MEND edit success for no-dropout models suggests that dropout produces redundant or diffuse knowledge representations; this could be tested directly by probing MLP-layer localizability in no-dropout versus dropout models.
  • Beyond the paper: the observed monotone relationship between dropout rate and performance gap implies the optimal rate is exactly zero in this regime, and a fine-grained sweep at small rates such as $p = 0.05$ could confirm the boundary.
  • Beyond the paper: because the largest model tested is 1.4B parameters, extending the comparison to models above 10B parameters would test whether the conclusion survives as model capacity and overfitting potential grow.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper reports pretraining experiments with BERT-base (110M) and Pythia 160M/1.4B under static dropout rates and an early-dropout schedule, evaluating LM loss, BLiMP, SQuAD, MNLI, and model editability via MEND and ReFT. It concludes that dropout, including early dropout, should be omitted in single-epoch pretraining. The reported evidence consists of learning curves across checkpoints, a three-rate dropout sweep, and edit-success comparisons.

Significance. If the single-epoch result were established, it would be a useful, practical finding: it consolidates the field's informal move away from dropout and extends that move to early dropout, with a testable recommendation. The paper's strengths include releasing code and models, covering both masked and autoregressive architectures, including an editability analysis, and running a rate sweep with three seeds. However, the central claim's scope is not matched by the experimental setup, and the headline comparisons lack seed-level statistics, so the practical recommendation is not yet supported.

major comments (3)
  1. [§3.1, Appendices B and C] The reported hyperparameters imply multiple passes over the training data, directly contradicting the single-epoch scope of the claim. For the BERT model, a 10B-token FineWeb slice with joint batch 384 at sequence length 512 gives one epoch of about 50,900 steps, yet Figure 1 plots BERT checkpoints out to 250k–300k steps, i.e., roughly six epochs. The Pythia descriptions are also internally inconsistent: Appendix C says '1.5 million steps' while Section D says the early-dropout run used roughly 220,000 total steps, and the token totals implied by the given batch sizes do not match 'around 1.1BT.' Because the central recommendation is explicitly scoped to single-epoch pretraining, the paper must either restrict the analysis to the first epoch (the plotted checkpoints appear to include those steps) or revise the claim.
  2. [§4 and Figure 1] The main capability comparisons are shown as single traces without error bars or significance tests; only the rate sweep in Figure 3 has three seeds and only MEND reports a p-value. Given that the LM-loss, BLiMP, and SQuAD gaps are the primary evidence for 'drop dropout,' the absence of seed-level variability makes it impossible to rule out run-to-run noise, and this is especially relevant for MNLI, which the text itself describes as only 'marginally better' while the abstract states that MNLI 'improves.'
  3. [§1 and §6] The premise that single-epoch pretraining yields minimal overfitting (Abstract and Section 1) is load-bearing but is never directly checked: the paper does not report training-versus-validation loss gaps, and the multi-epoch training described above means that the regularization argument for dropping dropout cannot be assumed. Without measuring overfitting or restricting the evaluation to one epoch, the recommendation could fail in settings where data is repeated or training is longer.
minor comments (6)
  1. [Appendix B vs §3.1] Appendix B calls the model BERT-Large, while Section 3.1 and Figure 1 call it BERT-base/110m; this inconsistency should be corrected.
  2. [Appendix C] Appendix C says the Pythia models are trained for '1.5 million steps (around 1.1BT),' but the batch-size and sequence-length values yield a very different token count; please state the dataset size, the number of epochs, and the step counts consistently.
  3. [Tables 1 and 2] Tables 1 and 2 share the caption 'Details of the Small Scale Model'; they should be retitled for the BERT and Pythia settings respectively.
  4. [Section D] Section D's early-dropout schedule of 77,821 out of roughly 220,000 steps is inconsistent with the 1.5-million-step training described in Section C.
  5. [Figure 3] The Figure 3 caption says the loss is averaged across three seeds, but the figure does not show error bars or per-seed curves; adding them would strengthen the rate-sweep conclusion.
  6. [Abstract and §4] The abstract's statement that MNLI 'improves' should be aligned with Section 4's 'marginally better,' ideally with a significance test or confidence interval.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the dropout comparison is an independent empirical test, not a derivation from its own inputs.

full rationale

This paper reports an empirical ablation: it pretrains BERT and Pythia models with dropout rates p = 0.0, 0.1, and 0.3, including an early-dropout schedule taken from Liu et al. (2023), and then compares downstream metrics. The central claim, that removing dropout improves single-epoch pretraining, is established by directly testing the alternatives rather than assuming them. The early-dropout baseline is implemented from the authors' prior work, but that prior result is not treated as evidence for the conclusion; it is itself the intervention being evaluated and is found to underperform no dropout, so the self-citation is not load-bearing. No fitted parameter, normalization choice, or scoring rule is constructed so that the outcome is forced by definition; the losses and benchmark scores are measured from held-out data and standard fine-tuning procedures. The paper's premise that single-epoch pretraining yields minimal overfitting is an assumption, and the appendix numbers may raise a question about whether all runs were truly single-epoch, but that is a correctness or scope concern, not a circularity. Likewise, the absence of train-versus-validation gap reporting weakens the rationale but does not make the empirical comparison circular. There is no equation in the paper that reduces to its own inputs, and no uniqueness theorem or prior result is invoked to forbid alternatives. The finding is an honest empirical comparison with no significant circularity.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper contains no derivations and fits no free parameters; its central claim is an empirical comparison. The listed axioms are domain assumptions inherited from prior literature: that single-epoch pretraining is overfitting-free, that dropout reduces feature co-adaptations, that factual knowledge is stored in MLP layers, and that the chosen benchmarks and editing methods measure the intended capabilities.

assumptions (4)
  • domain assumption Single-epoch pretraining yields minimal overfitting, so regularization is unnecessary.
    Stated in the abstract and Introduction to motivate the study; the paper does not measure train-validation gaps, so this premise is assumed rather than demonstrated.
  • domain assumption Dropout reduces feature co-adaptation (Hinton et al., 2012), which is used to hypothesize that dropout leads to less localized knowledge.
    Invoked in Section A and Section 5 to explain editability results; it is a prior theoretical claim borrowed from the literature.
  • domain assumption Factual knowledge in transformers is stored in the MLP layers after attention (Geva et al., 2021).
    Used in Section 3.3 to justify the editing experiments; without this assumption, the editability analysis loses its interpretation.
  • domain assumption The selected benchmarks (BLiMP, SQuAD, MNLI) and editing methods (MEND, ReFT) are valid measures of downstream capability and editability.
    The paper relies on the standard validity of these benchmarks and methods without additional justification.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Drop Dropout on Single-Epoch Language Model Pretraining." pith.science (2026). https://pith.science/paper/IANF3QSI

@misc{pith2026250524788,
  author       = {Pith},
  title        = {Pith review of: Drop Dropout on Single-Epoch Language Model Pretraining},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IANF3QSI}},
  note         = {Machine review of arXiv:2505.24788}
}
read the original abstract

Originally, dropout was seen as a breakthrough regularization technique that reduced overfitting and improved performance in almost all applications of deep learning by reducing overfitting. Yet, single-epoch pretraining tasks common to modern LLMs yield minimal overfitting, leading to dropout not being used for large LLMs. Nevertheless, no thorough empirical investigation has been done on the role of dropout in LM pretraining. Through experiments in single-epoch pretraining of both masked (BERT) and autoregressive (Pythia 160M and 1.4B) LMs with varying levels of dropout, we find that downstream performance in language modeling, morpho-syntax (BLiMP), question answering (SQuAD), and natural-language inference (MNLI) improves when dropout is not applied during pretraining. We additionally find that the recently-introduced "early dropout" also degrades performance over applying no dropout at all. We further investigate the models' editability, and find that models trained without dropout are more successful in gradient-based model editing (MEND) and equivalent in representation-based model editing (ReFT). Therefore, we advocate to drop dropout during single-epoch pretraining.

Figures

Figures reproduced from arXiv: 2505.24788 by the authors.

Figure 2
Figure 2. top: MEND edit success rate for MLM at varying levels of dropout ((p < 0.0001) across 5 seeds and edited concepts; margin of error are within floating point differences); bottom: ReFT edit success rate for MLM at varying levels of dropout (no significant differ￾ence across 5 seeds and edited concepts). Formalism In this work, we define “knowl￾edge” as tuples (a, r, b) (“subject-relation-object”); then, if a model M … view at source ↗
Figure 3
Figure 3. Sweeping dropout rate on the 1.4B parameter [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

17 extracted references · 13 canonical work pages

  1. [4]

    InProceedings of the 2021 Confer- ence on Empirical Methods in Natural Language Pro- cessing, pages 5484–5495, Online and Punta Cana, Dominican Republic

    Transformer feed-forward layers are key- value memories. InProceedings of the 2021 Confer- ence on Empirical Methods in Natural Language Pro- cessing, pages 5484–5495, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. Adam Goodkind and Klinton Bicknell

    work page 2021

  2. [7]

    Association for Computational Linguistics

    Language models as knowl- edge bases? InProceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2463–2473, Hong Kong, China. Association for Computational Linguistics. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario A...

    work page 2019

  3. [10]

    A broad-coverage challenge corpus for sen- tence understanding through inference. InProceed- ings of the 2018 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, V olume 1 (Long Papers), pages 1112–1122, New Orleans, Louisiana. Association for Computational Linguis- tics. Thomas Wolf, L...

    work page 2018

  4. [11]

    InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online

    Trans- formers: State-of-the-art natural language processing. InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics. Zhengxuan Wu, Aryaman Arora, Zheng Wang, Atti- cus Geiger, Dan Jurafsky, Christopher D Manning, and Christopher Potts

    work page 2020

  5. [12]

    An adapted version of the official evaluation script was used to obtain the dev-slice results reported in this work

    Dropout rate was set to 0.1 dur- ing training and evaluation regardless of whether pretraining used dropout as is reported in (Devlin et al., 2019). An adapted version of the official evaluation script was used to obtain the dev-slice results reported in this work. Questions and an- swers are separated by the [MASK] token, which was previously used for ML...

    work page 2019

  6. [13]

    When MLP dropout is used, p= 0.1

    im- plementation of BERT-Large (Devlin et al., 2019), and only modified it to disable attention and MLP dropout globally. When MLP dropout is used, p= 0.1 . Optimization was donewith regulariza- tionusing AdamW (Loshchilov and Hutter,

    work page 2019

  7. [14]

    Batching was done sequentially with the Pytorch Data Loader, sequence lengths are capped at 512 tokens

    following published parameters: Parameter Value LR linear warmup 10k, linear decay, peak1×10 −4 Adamβ(0.9, 0.999) Adamϵ1×10 −6 Table 1: Details of the Small Scale Model The model was trained on the officially sam- pled 10BT slice of Huggingface FineWeb (Hug- gingFaceFW, 2024), running with fully-sharded data-parallel over 4 GPUs for a joint batch size of ...

    work page 2024

  8. [15]

    Optimization was donewith regulariza- tionusing AdamW (Loshchilov and Hutter,

    implementation of the Pythia suite of models (Biderman et al., 2023); the only modifica- tions we performed involves modulating attention and MLP dropout globally as needed for each ex- periment. Optimization was donewith regulariza- tionusing AdamW (Loshchilov and Hutter,

    work page 2023

Show all 17 references
  1. [17]

    Batch size was set to 128, and dropout rate was set to 0.15 regardless of whether pretrain- ing the BERT model used dropout consistent with previous approaches

    dataset available 2 on Huggingface for 3 epochs using the adam optimizer, at a fixed learning rate of 2×10 −5. Batch size was set to 128, and dropout rate was set to 0.15 regardless of whether pretrain- ing the BERT model used dropout consistent with previous approaches. Premi...

    2023

  2. [2012]

    Geoffrey E Hinton

    Improving neural networks by preventing co-adaptation of feature de- tectors.arXiv preprint arXiv:1207.0580. Geoffrey E Hinton

    arXiv

  3. [2014]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al

    Dropout: a simple way to prevent neural networks from overfitting.The journal of machine learning research, 15(1):1929–1958. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal...

    1929

  4. [2018]

    InProceedings of the 8th Workshop on Cognitive Modeling and Com- putational Linguistics (CMCL 2018), pages 10–18, Salt Lake City, Utah

    Predictive power of word surprisal for reading times is a linear function of language model quality. InProceedings of the 8th Workshop on Cognitive Modeling and Com- putational Linguistics (CMCL 2018), pages 10–18, Salt Lake City, Utah. Association for Computational Linguistic...

    2018

  5. [2019]

    BERT: Pre-training of deep bidirectional transformers for language under- standing. InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, V olume 1 (Long and Short Papers), pages 4171...

    2019

  6. [2020]

    Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy

    The pile: An 800gb dataset of diverse text for lan- guage modeling.arXiv preprint arXiv:2101.00027. Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy

    arXiv

  7. [2021]

    InPro- ceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing, pages 6491– 6506, Online and Punta Cana, Dominican Republic

    Edit- ing factual knowledge in language models. InPro- ceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing, pages 6491– 6506, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. Jacob Devlin, Ming-Wei Chang, K...

    2021

  8. [2023]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin

    Llama: Open and effi- cient foundation language models.arXiv preprint arXiv:2302.13971. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin

    arXiv

  9. [2024]

    A Motivating the Studying of Dropout Through Knowledge Storage The central claim of Hinton et al

    Reft: Representa- tion finetuning for language models.arXiv preprint arXiv:2404.03592. A Motivating the Studying of Dropout Through Knowledge Storage The central claim of Hinton et al. (2012) is that dropout reduces feature co-adaptations. Framing dropout under this lens impli...

    2012 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.