Pith Number

pith:EMCX3KNY

pith:2019:EMCX3KNYQSYAMONWQMUKITMAQZ

not attested not anchored not stored refs resolved

RoBERTa: A Robustly Optimized BERT Pretraining Approach

Danqi Chen, Jingfei Du, Luke Zettlemoyer, Mandar Joshi, Mike Lewis, Myle Ott, Naman Goyal, Omer Levy, Veselin Stoyanov, Yinhan Liu

A careful retraining of BERT — longer, on more data, with dynamic masking and no next-sentence loss — matches or beats every model published after it on GLUE, SQuAD, and RACE.

arxiv:1907.11692 v1 · 2019-07-26 · cs.CL

Open paper page JSON Open Graph Bundle Merged state Verified badge What is a Pith Number?

Add to your LaTeX paper

\usepackage{pith}
\pithnumber{EMCX3KNYQSYAMONWQMUKITMAQZ}

Prints a linked badge after your title and injects PDF metadata. Compiles on arXiv. Learn more · Embed verified badge

Record completeness

1 Bitcoin timestamp

2 Internet Archive

3 Author claim open · sign in to claim

4 Citations open

5 Replications open

✓ Portable graph bundle live · download bundle · merged state

The bundle contains the canonical record plus signed events. A mirror can host it anywhere and recompute the same current state with the deterministic merge algorithm.

Claims

C1strongest claim

Under controlled comparison, BERT's masked-language-modeling objective with the original architecture, when trained longer on more data with larger batches, dynamic masking, no NSP loss, and byte-level BPE, matches or exceeds the downstream performance of every published post-BERT method (XLNet, SpanBERT, MT-DNN, etc.) on GLUE, SQuAD, and RACE — implying that previously reported gains over BERT are substantially attributable to training budget rather than architectural or objective novelty.

C2weakest assumption

That holding "architecture and objective" fixed while varying data, steps, batch size, and masking constitutes a fair attribution of credit. The XLNet comparison in particular conflates multiple axes (RoBERTa uses 160GB vs. XLNet's 126GB, different step counts, different vocabularies), and the authors acknowledge they did not retune XLNet under matched compute. The claim that MLM is "competitive" with permutation LM rests on this, and the paper itself notes (footnote 2) that other methods could likely also improve with more tuning.

C3one line summary

With better hyperparameters, more data, and longer training, an unchanged BERT-Large architecture matches or exceeds XLNet and other successors on GLUE, SQuAD, and RACE.

References

51 extracted · 51 resolved · 3 Pith anchors

[1] Eneko Agirre, Llu' i s M`arquez, and Richard Wicentowski, editors. 2007. Proceedings of the Fourth International Workshop on Semantic Evaluations (SemEval-2007) 2007

[2] Cloze-driven Pretraining of Self-attention Networks 2019 · arXiv:1903.07785

[3] Roy Bar-Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. 2006. The second PASCAL recognising textual entailment challenge. In Proceedings of the second 2006

[4] Luisa Bentivogli, Ido Dagan, Hoa Trang Dang, Danilo Giampiccolo, and Bernardo Magnini. 2009. The fifth PASCAL recognizing textual entailment challenge 2009

[5] Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large annotated corpus for learning natural language inference. In Empirical Methods in Natural Language Processing 2015

Formal links

1 machine-checked theorem link

Cited by

575 papers in Pith

UCSC NLP at SemEval-2026 Task 10: Boundary-Aware Span Extraction and RoBERTa Classification for Conspiracy Detection

Recovering Latent Structures after Variational Bayesian Variable Selection: Fit Assessment and Factor-Number Selection in Partially Exploratory Factor Analysis

A Unified Detection Framework for AI-Related Content and Artifacts

Multi-Class vs. Multi-Label BERT for CVE-to-CWE Mapping: How Taxonomy Structure Shapes the Errors

Asymmetric Focal Loss Improves Graph Neural Network Prediction of Drug-Drug Interactions

Receipt and verification

First computed	2026-07-04T23:49:50.768566Z
Builder	pith-number-builder-2026-05-17-v1
Signature	Pith Ed25519 (`pith-v1-2026-05`) · public key
Schema	pith-number/v1.0

Canonical hash

23057da9b884b00639b68328a44d80867d59be79695e1c06d66dbe2e3142337e

Aliases

arxiv: 1907.11692 · arxiv_version: 1907.11692v1 · doi: 10.48550/arxiv.1907.11692 · pith_short_12: EMCX3KNYQSYA · pith_short_16: EMCX3KNYQSYAMONW · pith_short_8: EMCX3KNY

Agent API

Resolver JSON Graph JSON Events JSON Schema Signing key

Verify this Pith Number yourself

curl -sH 'Accept: application/ld+json' https://pith.science/pith/EMCX3KNYQSYAMONWQMUKITMAQZ \
  | jq -c '.canonical_record' \
  | python3 -c "import sys,json,hashlib; b=json.dumps(json.loads(sys.stdin.read()), sort_keys=True, separators=(',',':'), ensure_ascii=False).encode(); print(hashlib.sha256(b).hexdigest())"
# expect: 23057da9b884b00639b68328a44d80867d59be79695e1c06d66dbe2e3142337e

Canonical record JSON

{
  "metadata": {
    "abstract_canon_sha256": "28bcebc417de7b07736f5b8236aee8283e3f9f07155471aeb3002e6eb8878753",
    "cross_cats_sorted": [],
    "license": "http://arxiv.org/licenses/nonexclusive-distrib/1.0/",
    "primary_cat": "cs.CL",
    "submitted_at": "2019-07-26T17:48:29Z",
    "title_canon_sha256": "a6658a1fd9390b3fb8d3fcc8e7edeaea97c38c2de7818cf45883a5ff37d20dc4"
  },
  "schema_version": "1.0",
  "source": {
    "id": "1907.11692",
    "kind": "arxiv",
    "version": 1
  }
}