Pith Number

pith:SRNS5FXN

pith:2020:SRNS5FXN35VG3OA55F6GX4BVX5

not attested not anchored not stored refs resolved

ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators

Christopher D. Manning, Kevin Clark, Minh-Thang Luong, Quoc V. Le

ELECTRA pre-trains text encoders as discriminators that detect replaced tokens, producing stronger contextual representations than BERT with the same model size, data, and compute.

arxiv:2003.10555 v1 · 2020-03-23 · cs.CL

Open paper page JSON Open Graph Bundle Merged state Verified badge What is a Pith Number?

Add to your LaTeX paper

\usepackage{pith}
\pithnumber{SRNS5FXN35VG3OA55F6GX4BVX5}

Prints a linked badge after your title and injects PDF metadata. Compiles on arXiv. Learn more · Embed verified badge

Record completeness

1 Bitcoin timestamp

2 Internet Archive

3 Author claim open · sign in to claim

4 Citations open

5 Replications open

✓ Portable graph bundle live · download bundle · merged state

The bundle contains the canonical record plus signed events. A mirror can host it anywhere and recompute the same current state with the deterministic merge algorithm.

Claims

C1strongest claim

the contextual representations learned by our approach substantially outperform the ones learned by BERT given the same model size, data, and compute

C2weakest assumption

That the replaced-token detection objective produces transferable contextual representations superior to those from masked language modeling when model size, data, and compute are held fixed.

C3one line summary

ELECTRA replaces masked language modeling with replaced token detection, yielding contextual representations that outperform BERT at equal compute and match larger models like RoBERTa with far less compute.

References

17 extracted · 17 resolved · 4 Pith anchors

[1] arXiv preprint arXiv:1811.02549 , year=

[2] 10 Published as a conference paper at ICLR 2020 Daniel M 2020

[3] TinyBERT: Distilling BERT for natural language understanding.arXiv preprint arXiv:1909.10351 1909

[4] SpanBERT: Improving pre-training by representing and predicting spans 1907

[5] ALBERT: A Lite BERT for Self-supervised Learning of Language Representations 1909 · arXiv:1909.11942

Formal links

2 machine-checked theorem links

Cited by

33 papers in Pith

Explaining the Explainers in Graph Neural Networks: a Comparative Study

On the Power of Foundation Models

Secret Leak Detection in Software Issue Reports using LLMs: A Comprehensive Evaluation

Mitigating Extrinsic Gender Bias for Bangla Classification Tasks

Social media polarization during conflict: Insights from an ideological stance dataset on Israel-Palestine Reddit comments

Receipt and verification

First computed	2026-05-17T23:38:48.196704Z
Builder	pith-number-builder-2026-05-17-v1
Signature	Pith Ed25519 (`pith-v1-2026-05`) · public key
Schema	pith-number/v1.0

Canonical hash

945b2e96eddf6a6db81de97c6bf035bf5017e48b6718f5217e1b05104d4ea43c

Aliases

arxiv: 2003.10555 · arxiv_version: 2003.10555v1 · doi: 10.48550/arxiv.2003.10555 · pith_short_12: SRNS5FXN35VG · pith_short_16: SRNS5FXN35VG3OA5 · pith_short_8: SRNS5FXN

Agent API

Resolver JSON Graph JSON Events JSON Schema Signing key

Verify this Pith Number yourself

curl -sH 'Accept: application/ld+json' https://pith.science/pith/SRNS5FXN35VG3OA55F6GX4BVX5 \
  | jq -c '.canonical_record' \
  | python3 -c "import sys,json,hashlib; b=json.dumps(json.loads(sys.stdin.read()), sort_keys=True, separators=(',',':'), ensure_ascii=False).encode(); print(hashlib.sha256(b).hexdigest())"
# expect: 945b2e96eddf6a6db81de97c6bf035bf5017e48b6718f5217e1b05104d4ea43c

Canonical record JSON

{
  "metadata": {
    "abstract_canon_sha256": "ee9a436437f5b3b17c4c329508a03483153ebbdb3e8d04a29b25d6a7ad806e7c",
    "cross_cats_sorted": [],
    "license": "http://arxiv.org/licenses/nonexclusive-distrib/1.0/",
    "primary_cat": "cs.CL",
    "submitted_at": "2020-03-23T21:17:42Z",
    "title_canon_sha256": "2c4d718974014f01447a4ae698420439e4c1e639b3db756ab33c4e51fee81152"
  },
  "schema_version": "1.0",
  "source": {
    "id": "2003.10555",
    "kind": "arxiv",
    "version": 1
  }
}