{"id":"ec2cff2e-7d40-46ff-bb02-1922b601c8b2","arxiv_id":"1907.11692","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"With better hyperparameters, more data, and longer training, an unchanged BERT-Large architecture matches or exceeds XLNet and other successors on GLUE, SQuAD, and RACE.","lead":"A careful replication of BERT pretraining shows the original model was significantly undertrained: with longer training, more data, larger batches, dynamic masking, and no next-sentence-prediction loss, the same architecture matches or beats every post-BERT method on GLUE, SQuAD, and RACE. The takeaway is that several reported architectural improvements may have been confounded by training budget.","discovery_kind":"replication","skeptic_critique":{"model":"claude-opus-4-7","headline":"At matched pretraining data (BOOKS+WIKI), Table 4 shows XLNet_LARGE actually beats RoBERTa on SQuAD 2.0 (87.8 vs 87.3) and MNLI-m (88.4 vs 89.0 — within noise), so the \"MLM competitive with permutation LM\" claim is only clearly supported once RoBERTa is given more data and longer training.","rationale":"The reader's verdict (ACCEPT, low correctness risk, moderate novelty) is well-calibrated. The replication is careful, the ablations on masking/NSP/batch size are clean (seed medians, controlled data), code and checkpoints are released, and the practical artifact has held up in the field. The reader also correctly flagged the one place where the rhetoric outruns the evidence: the cross-method \"MLM ≈ permutation LM\" comparison is not compute- or data-matched, and the authors themselves note this only in a footnote.\n\nI'd keep the verdict at ACCEPT because (a) the within-paper ablations are sound and do support the narrower claim that BERT was undertrained and that NSP and static masking were suboptimal; (b) the cross-method claim is hedged honestly in footnote 2; and (c) the contribution is primarily methodological discipline plus a reproducible artifact, not a novel objective. The load-bearing concern is real but does not change the disposition — it changes the strength with which one should read the paper's framing. Specifically, on matched BOOKS+WIKI data (Table 4), XLNet_LARGE outperforms RoBERTa on SQuAD 2.0 (87.8 vs 87.3) and is comparable on MNLI; RoBERTa's clear lead emerges only at 160GB/500K steps. So a more defensible reading is \"MLM scales competitively with permutation LM when given more data,\" not \"MLM is intrinsically competitive.\"\n\nSecondary concern not raised by the reader: the byte-level 50K BPE adds ~20M parameters to RoBERTa_LARGE relative to BERT_LARGE's 30K WordPiece, and Section 4.4 dismisses this without a table. Some portion of the BERT_LARGE → RoBERTa-BOOKS+WIKI gap on SQuAD/MNLI could be embedding-capacity rather than training-discipline. This is minor but unresolved.\n\nNo issues with circularity, derivation gaps, or ethics. Novelty score of 5 is appropriate — the components (dynamic masking, dropping NSP, byte-BPE, large batches) all predate this paper; the contribution is integration, scale, and the controlled study itself.","tokens_in":3401,"tokens_out":3344,"duration_ms":51885,"concrete_test":"Run a single matched-budget comparison: pretrain RoBERTa and XLNet on identical data (BOOKS+WIKI, 16GB), identical tokenizer and vocab size (either both 30K WordPiece or both 50K byte-BPE), identical batch size and step count (e.g., 8K × 100K), identical optimizer settings, and report SQuAD 2.0 / MNLI / SST-2 medians over 5 seeds. If XLNet ≥ RoBERTa on 2 of 3 tasks (consistent with Table 4's BOOKS+WIKI row), the \"MLM is competitive with permutation LM\" claim should be restated as \"MLM scales as well as permutation LM given more data,\" which is a weaker and more accurate conclusion. A cheaper partial test: ablate the 50K byte-BPE in RoBERTa-BOOKS+WIKI back to BERT's 30K WordPiece and check whether the +2.7 SQuAD / +2.4 MNLI gain over BERT_LARGE survives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The reader's weakest_assumption is the right concern; I'd sharpen it with the paper's own numbers. Table 4 contains the cleanest head-to-head: at BOOKS+WIKI (13–16GB), 256 or comparable batch, 1M steps, XLNet_LARGE reports 94.0/87.8 on SQuAD 1.1/2.0 and 88.4 MNLI-m, while RoBERTa-BOOKS+WIKI at 8K×100K reports 93.6/87.3 and 89.0. RoBERTa loses or ties on SQuAD 2.0 and is roughly even on MNLI. RoBERTa only moves ahead after adding 144GB more text and 4–5× more optimizer steps (500K). The headline \"MLM ≈ permutation LM\" therefore rests on a setting where RoBERTa's training budget exceeds XLNet's, not on a compute- and data-matched comparison.\n\nA second related issue: even the \"controlled\" BERT_LARGE vs RoBERTa-BOOKS+WIKI row in Table 4 is not compute-matched in the sense of optimizer trajectory. RoBERTa uses an 8K batch with tuned LR/β2/ε and a byte-level 50K BPE that adds ~20M parameters; BERT_LARGE used 256 batch and 30K char-BPE. Some of the 90.9→93.6 SQuAD gap could reflect the extra parameters in the embedding table and the much larger effective batch's known regularization effects, not purely \"BERT was undertrained.\" The paper does not isolate the contribution of the 50K byte-BPE (Section 4.4 explicitly says \"early experiments revealed only slight differences\" but provides no table), so the parameter-count confound is unaddressed.\n\nNeither point overturns the practical finding that RoBERTa is a strong, reproducible model. They do narrow the scientific claim from \"MLM is competitive with permutation LM\" to \"MLM with more data and longer training matches permutation LM with less.\" Footnote 2 acknowledges this but does not test it.","agreement_with_reader":"agree"},"referee_report":{"model":"claude-opus-4-7","summary":"The paper presents a replication and ablation study of BERT pretraining. The authors reimplement BERT in fairseq, sweep four design axes — dynamic vs. static masking (§4.1), input format and the NSP loss (§4.2), batch size (§4.3), and byte-level BPE (§4.4) — and combine the favorable settings with substantially more data (160GB across BOOKS+WIKI, CC-NEWS, OPENWEBTEXT, STORIES) and more optimizer steps (up to 500K at 8K batch). The resulting model, RoBERTa, is reported to match or exceed all post-BERT published systems on GLUE (Table 5), SQuAD v1.1/v2.0 (Table 6), and RACE (Table 7), without multi-task finetuning on GLUE or external QA data on SQuAD. The central scientific claim is that BERT was significantly undertrained and that, with the right training recipe, the MLM objective is competitive with subsequently proposed alternatives such as permutation LM (XLNet).","tokens_in":3817,"tokens_out":2884,"duration_ms":48979,"significance":"If the result holds, the paper materially reshapes how the community attributes credit for the gains reported in 2018–2019: a sizable fraction of post-BERT improvement is attributable to data scale, batch size, and training length rather than to new objectives or architectures. This is a useful corrective and a high-value contribution to a literature where ablations against private data and undisclosed compute budgets have made comparisons unreliable. Concrete strengths are: (i) the ablations in Tables 1–3 are clean and use medians over five seeds; (ii) the introduction of CC-NEWS partially closes the public-data gap with concurrent work; (iii) models, code, and a documented hyperparameter recipe (Tables 9–10) are released, enabling third-party replication. The released artifact has in fact become a widely used baseline, which is itself evidence of the practical claim. The paper is appropriately modest in footnote 2 about the limits of its comparisons.","major_comments":[{"comment":"The headline claim that MLM is 'competitive with' permutation LM is not cleanly supported by the most controlled row of Table 4. At matched BOOKS+WIKI data, XLNet_LARGE reports 94.0/87.8 on SQuAD 1.1/2.0 and 88.4 on MNLI-m, while RoBERTa-BOOKS+WIKI reports 93.6/87.3 and 89.0 — RoBERTa loses on SQuAD 2.0 and is within noise on MNLI. RoBERTa only clearly surpasses XLNet after adding ~10× more text and 5× more updates (500K), at which point XLNet itself is also no longer at its matched-data setting. Please either (a) restate the conclusion as 'MLM is competitive once given comparable or larger training budget,' or (b) report a compute- and data-matched comparison (same corpus, same token count seen, same batch and step budget). The current phrasing in §1 and §7 overstates what Table 4 shows.","section":"§5, Table 4"},{"comment":"The switch to a 50K byte-level BPE adds approximately 20M parameters to BERT_LARGE (the paper's own estimate in §4.4). This confounds the BERT_LARGE → RoBERTa-BOOKS+WIKI comparison in Table 4 (90.9/81.8 → 93.6/87.3 on SQuAD), since part of the gap may reflect added embedding capacity rather than 'BERT was undertrained.' §4.4 states 'early experiments revealed only slight differences' but provides no table. A small ablation isolating 30K char-BPE vs 50K byte-BPE at otherwise matched settings (one row would suffice) would close this gap and is important because the BPE choice is one of the four pillars of the recipe.","section":"§4.4 / Table 4"},{"comment":"The large-batch comparison varies batch size, step count, and learning rate jointly while reporting only perplexity and two GLUE dev metrics. The 2K-batch/125K-step setting outperforms both 256/1M and 8K/31K on perplexity (3.68 vs 3.99 vs 3.77), yet the paper adopts 8K for downstream experiments citing parallelization. Please clarify why 2K is not the preferred choice on the evidence presented, or report SQuAD/RACE numbers for the three settings so the choice is grounded in end-task performance rather than engineering convenience.","section":"§4.3, Table 3"},{"comment":"The conclusion that removing NSP 'matches or slightly improves' downstream task performance is drawn from differences that are often within plausible seed variance (e.g., FULL-SENTENCES 84.7 vs SEGMENT-PAIR+NSP 84.0 on MNLI-m; 92.5 vs 92.9 on SST-2). Reported numbers are medians over five seeds but no spread is given. Please report standard deviations or min/max across seeds for Table 2 so the reader can judge whether the NSP-removal effect exceeds noise; this matters because removing NSP is one of the four headline modifications.","section":"§4.2, Table 2"}],"minor_comments":[{"comment":"CC-NEWS filtering is described in one sentence ('76GB after filtering'). A short description of the filter (language ID, dedup, boilerplate removal) would help replication, especially since the dataset is presented as a contribution.","section":"§3.2"},{"comment":"The 'data' column lists 13GB for XLNet and 16GB for RoBERTa under 'BOOKS+WIKI'; footnote 3 attributes this to Wikipedia cleaning differences. Worth restating in the Table 4 caption so a casual reader does not mistake this for a data-budget mismatch in RoBERTa's favor at the matched row.","section":"Table 4"},{"comment":"The WNLI procedure (margin ranking with spaCy-extracted candidates, SuperGLUE reformatting) is non-standard and excludes negative training examples. Given the 91.3 dev / 89.0 test number contributes to the average, a sentence explicitly flagging that this score is not directly comparable to other systems' WNLI numbers would be appropriate.","section":"§5.1, WNLI"},{"comment":"The dynamic-vs-static gap is small (e.g., 78.7 vs 78.3 SQuAD 2.0; 84.0 vs 84.3 MNLI). Calling dynamic masking 'comparable or slightly better' is fair, but the abstract and §1 list dynamic masking as one of four key improvements; consider softening the framing to match Table 1.","section":"§4.1, Table 1"},{"comment":"Several places contain OCR-like artifacts in the submitted PDF ('Y ang', 'Y ou', 'V aswani', 'B OOK CORPUS'); please verify font/encoding in the camera-ready.","section":"Typography"},{"comment":"The Appendix hyperparameters (Tables 9–10) are useful; consider also reporting the total wall-clock and GPU-hours per pretraining run so future replications can budget appropriately. The text mentions '1024 V100 GPUs for approximately one day' but only for one configuration.","section":"§5"}],"recommendation":"accept","confidential_remarks":"The manuscript is essentially an engineering/empirical contribution rather than a methodological novelty, and its scientific claim ('MLM is competitive with permutation LM') is somewhat over-stated relative to the controlled evidence in Table 4 — at matched data, XLNet is at least as good. However, the paper is unusually transparent about its own caveats (footnote 2; §5 notes data/diversity are conflated), the ablations are clean, and the released artifact is a genuine community resource. I recommend acceptance, ideally after the authors soften the 'competitive with permutation LM' framing or add a more strictly matched comparison, and add the missing BPE ablation. None of these are blocking for a venue that values reproducible empirical work."},"author_rebuttal":{"model":"claude-opus-4-7","summary":"We thank the referee for the careful and substantive report, and in particular for distinguishing the empirical contribution (the recipe and the released artifact) from the rhetorical framing of the central claim. We accept the four major points essentially as stated. The referee is correct that (i) our 'competitive with permutation LM' claim is properly conditioned on training budget rather than asserted at matched data, (ii) the 50K byte-level BPE adds parameters and partially confounds the BERT_LARGE -> RoBERTa comparison in Table 4, (iii) the choice of 8K batch over 2K in Table 3 is motivated by parallelization rather than by a clean end-task win, and (iv) several of the NSP-removal contrasts in Table 2 are within seed noise and the prose should reflect this. We will revise §1, §4.2, §4.3, §4.4, §5 and §7 accordingly, add per-seed spread to Table 2, add the BASE-scale BPE comparison and SQuAD numbers for the Table 3 settings to the appendix, and explicitly bound the headline claim. Two items — a strictly token-matched re-run against XLNet, and a LARGE-scale char-BPE vs byte-BPE ablation — we cannot produce within the revision window; we list these as standing objections and will disclose them rather than overclaim.","responses":[{"response":"The referee is right that our matched-data row is the appropriate basis for the strongest version of the claim, and that on that row RoBERTa is essentially tied with (and slightly behind on SQuAD 2.0) XLNet_LARGE rather than dominating it. Our intended claim was the weaker one the referee articulates: that MLM remains competitive with permutation LM when given a comparable or larger training budget, and that a substantial portion of the post-BERT gains attributed to new objectives can be recovered by training scale alone. We will revise §1 and §7 to state this more precisely, replacing 'match or exceed every model published after it' in unqualified form with language that explicitly conditions on training budget. We will also add a sentence to §5 noting that at matched BOOKS+WIKI / 1M-equivalent budget, RoBERTa and XLNet_LARGE are within ~0.5 points on SQuAD/MNLI, and that the larger-budget rows of Table 4 are not budget-matched against XLNet's own larger-budget row (94.5/88.8, 89.8 with 126GB / 500K / batch 2K). A strictly token-matched re-run against XLNet is unfortunately outside what we can produce within the revision window — XLNet's permutation training has a different effective tokens-per-step accounting, and we do not have access to their exact data composition — but we will state this limitation explicitly rather than paper over it.","revision_made":"yes","referee_comment":"Major #1 [§5, Table 4]: Headline claim that MLM is 'competitive with' permutation LM is not cleanly supported at matched BOOKS+WIKI. RoBERTa loses on SQuAD 2.0 (87.3 vs 87.8) and is within noise on MNLI; the win only emerges after ~10x data and 5x steps, by which point XLNet is also off its matched-data setting. Restate to 'competitive once given comparable or larger budget,' or run a strictly compute- and data-matched comparison."},{"response":"We agree this is a real confound and that §4.4's qualitative remark is not a substitute for a number. Our internal early experiments compared the two encodings at BERT_BASE scale with otherwise matched settings and did not show systematic gains for byte-level BPE (in fact slightly worse on some tasks, as noted), which is why we framed the choice as motivated by universality rather than accuracy. We will add a row to the appendix giving the head-to-head dev numbers we have at BASE scale, and we will explicitly flag in §4.4 and in the discussion of Table 4 that the ~20M-parameter increase at LARGE is a confound for the BERT_LARGE -> RoBERTa-BOOKS+WIKI delta, so that readers do not attribute the full 90.9 -> 93.6 SQuAD 1.1 gap to 'undertraining.' We do not have a fully matched 30K-char vs 50K-byte run at LARGE scale, and we will say so rather than overclaim.","revision_made":"partial","referee_comment":"Major #2 [§4.4 / Table 4]: The 50K byte-level BPE adds ~20M parameters to BERT_LARGE, confounding the BERT_LARGE -> RoBERTa-BOOKS+WIKI comparison. §4.4 asserts 'only slight differences' but shows no table. Provide a one-row ablation isolating 30K char-BPE vs 50K byte-BPE at matched settings."},{"response":"The referee has correctly identified that Table 3 does not on its face justify 8K over 2K on accuracy grounds. The honest statement of our reasoning is engineering: at the scale of the §5 experiments (1024 V100s, 500K steps, 160GB), 8K batches were materially easier to keep utilization high under distributed data-parallel training, and the dev-set differences we observed between 2K and 8K at this controlled BASE-scale setup were small and did not consistently favor 2K on downstream tasks beyond what Table 3 shows. We will (i) explicitly state in §4.3 that the choice of 8K over 2K is driven by parallelization rather than by an accuracy advantage on Table 3, (ii) add SQuAD numbers for the three Table 3 settings to the appendix where we have them, and (iii) soften the implication that 8K is optimal on the evidence presented. We agree this is a fair correction.","revision_made":"yes","referee_comment":"Major #3 [§4.3, Table 3]: Batch size, steps, and learning rate vary jointly; only ppl + two GLUE metrics are reported. 2K/125K beats 8K/31K on ppl but 8K is adopted citing parallelization. Justify on end-task performance or report SQuAD/RACE for the three settings."},{"response":"This is well taken. Our claim in §4.2 is deliberately phrased as 'matches or slightly improves' rather than 'improves,' precisely because for several of the cells the gap is within what we observe across seeds, and the stronger statement we make is the negative one — that retaining NSP does not help and that SENTENCE-PAIR (which forces short inputs) clearly hurts. We will add per-cell spread (std and min/max over the five seeds) to Table 2 in the revision, for both the NSP and the input-format rows, so the reader can see directly which contrasts are above seed noise (SEGMENT-PAIR vs SENTENCE-PAIR; SEGMENT-PAIR vs DOC-SENTENCES on SQuAD/RACE) and which are not (FULL-SENTENCES vs SEGMENT-PAIR+NSP on MNLI/SST-2). We will also adjust the prose in §4.2 and §7 so that the headline summary about NSP is 'removing NSP does not hurt, and removing it together with the SENTENCE-PAIR format helps,' rather than implying a uniform improvement.","revision_made":"yes","referee_comment":"Major #4 [§4.2, Table 2]: NSP-removal effects are within plausible seed variance (e.g., FULL-SENTENCES 84.7 vs SEGMENT-PAIR+NSP 84.0 on MNLI-m; 92.5 vs 92.9 on SST-2). Report std / min-max across the five seeds so readers can judge whether the effect exceeds noise."}],"tokens_in":3587,"tokens_out":2959,"duration_ms":44588,"standing_objections":["A strictly token-, batch-, and step-matched head-to-head against XLNet (Major #1) is not feasible within the revision window: we lack access to XLNet's exact data composition and the permutation objective's per-step token accounting differs from MLM's. We will instead bound the claim and disclose the limitation, rather than produce a comparison we cannot run cleanly.","We do not have a fully matched 30K char-BPE vs 50K byte-BPE ablation at LARGE scale (Major #2). We can add the BASE-scale comparison we did run, and we will flag the parameter-count confound at LARGE explicitly, but a LARGE-scale matched ablation is beyond the compute we can commit to this revision."]},"desk_editor":{"model":"claude-opus-4-7","letter":"This is the RoBERTa paper. You probably already know the bottom line: take BERT, train it longer on more data with bigger batches, drop NSP, use dynamic masking, switch to byte-level BPE, and you match or beat XLNet, SpanBERT, MT-DNN on GLUE/SQuAD/RACE. No architectural change. Code and checkpoints released.\n\nWhat's good. The ablations are clean and reported as medians over five seeds, which was not standard practice at the time. The NSP-vs-no-NSP comparison (Table 2) is careful — they distinguish SEGMENT-PAIR, SENTENCE-PAIR, FULL-SENTENCES, DOC-SENTENCES, which lets them isolate \"remove NSP loss\" from \"change input format,\" and the conclusion that earlier NSP-removal results were confounded by also shortening inputs is a genuinely useful piece of methodological hygiene. The batch-size and masking studies are small but informative. As an empirical artifact RoBERTa became a standard backbone, and deservedly.\n\nWhere I'd push back. The headline framing — \"MLM is competitive with permutation LM\" — is not supported at matched compute. Look at Table 4: at BOOKS+WIKI with comparable steps, XLNet_LARGE actually edges RoBERTa on SQuAD 2.0 (87.8 vs 87.3) and is ahead on MNLI in their own numbers. RoBERTa only clearly passes XLNet after ingesting an extra ~145GB of text and running 5× more optimizer steps. Footnote 2 concedes other methods would likely also benefit from more tuning, which is honest but unresolved. The stress-test note has this right.\n\nA smaller point the stress test also flags: the 50K byte-level BPE adds ~20M parameters to the embedding table, and Section 4.4 waves at this with \"early experiments revealed only slight differences\" but ships no table. So a sliver of the BERT_LARGE → RoBERTa gap is parameter count, not training discipline. Minor, but the paper should have isolated it.\n\nNeither concern overturns the practical claim that BERT was undertrained, which is well-supported. They do narrow the scientific claim.\n\nRecommendation. Useful paper, well executed within its scope, honestly flagged caveats. The methodological message — control compute before crediting objectives — is worth taking seriously and citing. Worth referees' time; worth a reading group slot for the ablation methodology alone.","headline":"Solid, reproducible replication study showing BERT was undertrained; the cross-method \"MLM ≈ permutation LM\" claim is weaker than the abstract suggests but the practical contribution is real.","tokens_in":4729,"tokens_out":1124,"would_cite":true,"duration_ms":21968,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":{"model":"claude-opus-4-7","evidence":[{"relation":"unclear","rs_module":"N/A — empirical ML methodology, no overlap with RS forcing chain (Foundation/LogicAsFunctionalEquation, Cost/FunctionalEquation, PhiForcing, DimensionForcing, EightTick).","rs_theorem":null,"paper_passage":"We carefully evaluate a number of design decisions when pretraining BERT models. We find that performance can be substantially improved by training the model longer, with bigger batches over more data; removing the next sentence prediction objective; training on longer sequences; and dynamically changing the masking pattern."}],"headline":"Empirical NLP pretraining replication study; no contact with RS forcing structure.","alignment":"orthogonal","rationale":"RoBERTa is an empirical engineering study of BERT-style language model pretraining. Its central claims concern training budget, batch size, masking strategy, and dataset scale on benchmarks (GLUE, SQuAD, RACE). There is no cost-functional analysis, no ratio symmetry, no golden-ratio or φ-ladder structure, no 8-tick periodicity, no parameter-free derivation of constants, and no claim that touches anything RS forces. The paper has many tunable hyperparameters (learning rate, batch size, β2, ε, vocab size, training steps), which is the opposite of RS's zero-parameter posture, but this is a domain mismatch rather than a contradiction — RS makes no prediction about transformer pretraining. The framework is silent on this domain.","tokens_in":22874,"confidence":"high","tokens_out":643,"duration_ms":16676,"cache_read_input_tokens":415308,"cache_creation_input_tokens":0},"lean_confirmation":{"model":"claude-opus-4-7","status":"out_of_scope","citations":[],"rationale":"The load-bearing premise is an empirical machine-learning measurement (downstream task accuracies from a controlled training-budget comparison). Lean cannot machine-check that one neural training recipe outperforms another on GLUE; this is wet-lab-equivalent for ML. The shape-of-logic corpus contains no modules on language modeling, transformers, or empirical NLP benchmarks. This paper is squarely out_of_scope for a Lean confirmation badge — which is the honest status, not a failure.","tokens_in":21965,"confidence":"high","tokens_out":802,"duration_ms":17707,"inferential_bridge":"The paper's central claim is an empirical/experimental finding about neural network training: hyperparameter and data choices, rather than architectural novelty, drive recent NLP benchmark gains. Establishing this requires running pretraining + fine-tuning and measuring scores on GLUE/SQuAD/RACE — it is not a mathematical/structural identity that admits a Lean proof. The shape-of-logic library is a Lean 4 corpus about cost-functional uniqueness, φ-forcing, dimension forcing, gauge structure from cube symmetry, and related Recognition Science structural theorems. Nothing in it speaks to transformer pretraining, masked language modeling losses, byte-pair encoding, or benchmark accuracy. There is no theorem that could even in principle stand in for the empirical comparison.","load_bearing_premise":"Empirical claim: under matched (or larger) compute and data, BERT's masked-language-modeling pretraining (with dynamic masking, no NSP, larger batches, byte-level BPE, and longer training over ~160GB of text) matches or exceeds the downstream GLUE/SQuAD/RACE scores of post-BERT methods such as XLNet, SpanBERT, and MT-DNN. The premise is a statistical comparison of fine-tuned benchmark accuracies obtained from actually training neural networks on text corpora.","cache_read_input_tokens":415178,"cache_creation_input_tokens":0},"pith_extraction":{"msc":[],"pacs":[],"model":"claude-opus-4-7","headline":"A careful retraining of BERT — longer, on more data, with dynamic masking and no next-sentence loss — matches or beats every model published after it on GLUE, SQuAD, and RACE.","keywords":["masked language modeling","BERT","pretraining","GLUE benchmark","SQuAD","RACE","byte-level BPE","dynamic masking"],"falsifier":"Retrain a competing model (e.g. XLNet or a permutation/span-based variant) under matched data (160GB), matched batch size (8K), and matched step count (500K) using the same byte-level BPE and dynamic masking, and compare GLUE/SQuAD/RACE numbers head-to-head. If the competitor still beats RoBERTa by a clear margin under matched compute, the claim that masked language modeling is competitive with the alternatives fails.","tokens_in":4603,"feed_emoji":"📚","tokens_out":2411,"duration_ms":35939,"temperature":0.7,"pith_summary":"The paper is a replication and tuning study of BERT pretraining. Its central move is to hold the architecture and the masked-language-modeling objective fixed and vary only the things that are usually treated as background: how long to train, on how much text, with what batch size, what masking schedule, what tokenizer, and whether to keep the next-sentence-prediction auxiliary loss. Under that controlled sweep, a retrained BERT — called RoBERTa — matches or surpasses every model published after BERT on GLUE, SQuAD, and RACE, including ones that introduced new pretraining objectives. The authors take this as evidence that the original BERT was significantly undertrained, that next-sentence prediction is unnecessary, and that masked language modeling remains competitive with permutation, span, and autoregressive alternatives once the training budget is matched. A sympathetic reader should care because the result reframes a year of reported \"objective\" gains as substantially gains in compute and data.","feed_headline":"BERT was undertrained: same model, more steps, beats its successors","feed_subtitle":"Longer training, more data, dynamic masking, and no next-sentence loss match or pass XLNet on GLUE, SQuAD, RACE.","key_machinery":"A controlled ablation over BERT's training recipe rather than its architecture: (1) dynamic masking instead of a fixed precomputed mask, (2) packing full sentences across document boundaries and dropping the next-sentence-prediction auxiliary loss, (3) batch sizes of 8K sequences with retuned learning rate and Adam β₂=0.98, (4) a 50K byte-level BPE vocabulary with no language-specific preprocessing, and (5) scaling pretraining data to 160GB (BookCorpus+Wikipedia plus CC-News, OpenWebText, and Stories) and pretraining for up to 500K steps. The architecture and the masked-language-modeling objective are held fixed at BERT_LARGE.","core_discovery":"The paper argues that BERT, as originally released, was undertrained, and that a careful replication holding architecture and objective fixed — while training longer, on roughly ten times more text, with larger batches, dynamic masking, no next-sentence-prediction loss, and a byte-level BPE vocabulary — matches or surpasses every post-BERT model published up to that point on GLUE, SQuAD, and RACE. The implication the authors press is that gains attributed to newer pretraining objectives or architectures may instead be explained by training budget and data scale.","pith_inferences":["<parameter name=\"0\">If most apparent progress over BERT is explained by training budget","then benchmark leaderboards in this period are partly tracking compute spend rather than modeling ideas — a methodological caution that extends well beyond NLP."],"forward_implications":["<parameter name=\"0\">Reported gains from newer pretraining objectives over BERT should be re-examined against compute-matched baselines","since training budget alone closes most of the gap."],"fun_headline_variants":[],"cache_read_input_tokens":22941,"weakest_assumption_plain":"That fixing the architecture and objective while changing data, steps, batch size, and tokenizer constitutes a fair attribution of credit — the comparison with competing methods does not retune those methods under matched compute, so the claim that masked language modeling is \"competitive\" with newer objectives rests on the assumption that the competitors would not pull ahead again under the same scaling treatment.","fun_headline_variants_meta":null,"cache_creation_input_tokens":0},"created_at":"2026-05-09T01:10:08.388354+00:00","model_set":{"reader":"claude-opus-4-7"},"falsifier":"Retrain a competing model (e.g. XLNet or a permutation/span-based variant) under matched data (160GB), matched batch size (8K), and matched step count (500K) using the same byte-level BPE and dynamic masking, and compare GLUE/SQuAD/RACE numbers head-to-head. If the competitor still beats RoBERTa by a clear margin under matched compute, the claim that masked language modeling is competitive with the alternatives fails.","supporting_citations":[],"review_version":1}