{"paper":{"title":"ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators","license":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/","headline":"ELECTRA pre-trains text encoders as discriminators that detect replaced tokens, producing stronger contextual representations than BERT with the same model size, data, and compute.","cross_cats":[],"primary_cat":"cs.CL","authors_text":"Christopher D. Manning, Kevin Clark, Minh-Thang Luong, Quoc V. Le","submitted_at":"2020-03-23T21:17:42Z","abstract_excerpt":"Masked language modeling (MLM) pre-training methods such as BERT corrupt the input by replacing some tokens with [MASK] and then train a model to reconstruct the original tokens. While they produce good results when transferred to downstream NLP tasks, they generally require large amounts of compute to be effective. As an alternative, we propose a more sample-efficient pre-training task called replaced token detection. Instead of masking the input, our approach corrupts it by replacing some tokens with plausible alternatives sampled from a small generator network. Then, instead of training a m"},"claims":{"count":4,"items":[{"kind":"strongest_claim","text":"the contextual representations learned by our approach substantially outperform the ones learned by BERT given the same model size, data, and compute","source":"verdict.strongest_claim","status":"machine_extracted","claim_id":"C1","attestation":"unclaimed"},{"kind":"weakest_assumption","text":"That the replaced-token detection objective produces transferable contextual representations superior to those from masked language modeling when model size, data, and compute are held fixed.","source":"verdict.weakest_assumption","status":"machine_extracted","claim_id":"C2","attestation":"unclaimed"},{"kind":"one_line_summary","text":"ELECTRA replaces masked language modeling with replaced token detection, yielding contextual representations that outperform BERT at equal compute and match larger models like RoBERTa with far less compute.","source":"verdict.one_line_summary","status":"machine_extracted","claim_id":"C3","attestation":"unclaimed"},{"kind":"headline","text":"ELECTRA pre-trains text encoders as discriminators that detect replaced tokens, producing stronger contextual representations than BERT with the same model size, data, and compute.","source":"verdict.pith_extraction.headline","status":"machine_extracted","claim_id":"C4","attestation":"unclaimed"}],"snapshot_sha256":"3cc8f1b2416205746285a883c24cc451aa32a4c3a5d66dbfcd1cb4a146cc2af7"},"source":{"id":"2003.10555","kind":"arxiv","version":1},"verdict":{"id":"b9d9fd5e-2cb7-49e7-9afc-f2e4d564e628","model_set":{"reader":"grok-4.3"},"created_at":"2026-05-16T10:22:50.482051Z","strongest_claim":"the contextual representations learned by our approach substantially outperform the ones learned by BERT given the same model size, data, and compute","one_line_summary":"ELECTRA replaces masked language modeling with replaced token detection, yielding contextual representations that outperform BERT at equal compute and match larger models like RoBERTa with far less compute.","pipeline_version":"pith-pipeline@v0.9.0","weakest_assumption":"That the replaced-token detection objective produces transferable contextual representations superior to those from masked language modeling when model size, data, and compute are held fixed.","pith_extraction_headline":"ELECTRA pre-trains text encoders as discriminators that detect replaced tokens, producing stronger contextual representations than BERT with the same model size, data, and compute."},"references":{"count":17,"sample":[{"doi":"","year":null,"title":"arXiv preprint arXiv:1811.02549 , year=","work_id":"851f20c8-1e5a-4d79-8bac-46b38d0a46aa","ref_index":1,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2020,"title":"10 Published as a conference paper at ICLR 2020 Daniel M","work_id":"969a10a2-c713-40b3-b33b-8d7788afee97","ref_index":2,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":1909,"title":"TinyBERT: Distilling BERT for natural language understanding.arXiv preprint arXiv:1909.10351","work_id":"40d9fbb8-3c66-44bf-955c-5b5560f1e2f8","ref_index":3,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":1907,"title":"SpanBERT: Improving pre-training by representing and predicting spans","work_id":"1d90c45c-05d0-44dc-909b-2b6e2a406c24","ref_index":4,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":1909,"title":"ALBERT: A Lite BERT for Self-supervised Learning of Language Representations","work_id":"aedf7950-7c35-4e28-a32d-bec290f51669","ref_index":5,"cited_arxiv_id":"1909.11942","is_internal_anchor":true}],"resolved_work":17,"snapshot_sha256":"4461250c60d659f6c3c202767f933f81fca4df6a10ed43f03fd711861748fce2","internal_anchors":4},"formal_canon":{"evidence_count":2,"snapshot_sha256":"2f7f114414d9ddf586384dbb7ad0ee3ec5210b4b4bede11639a161d6d2bcf1de"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"}