Pith Number

pith:P3AFXVTD

pith:2020:P3AFXVTDDTINLPZKZVHH3X2O6L

not attested not anchored not stored refs resolved

DeBERTa: Decoding-enhanced BERT with Disentangled Attention

Jianfeng Gao, Pengcheng He, Weizhu Chen, XiaoDong Liu

DeBERTa uses separate vectors for word content and position to compute attention, plus absolute positions in the mask decoder, yielding better NLP performance than RoBERTa with less data and the first single-model score above human average.

arxiv:2006.03654 v6 · 2020-06-05 · cs.CL · cs.LG

Open paper page JSON Open Graph Bundle Merged state Verified badge What is a Pith Number?

Add to your LaTeX paper

\usepackage{pith}
\pithnumber{P3AFXVTDDTINLPZKZVHH3X2O6L}

Prints a linked badge after your title and injects PDF metadata. Compiles on arXiv. Learn more · Embed verified badge

Record completeness

1 Bitcoin timestamp

2 Internet Archive

3 Author claim open · sign in to claim

4 Citations open

5 Replications open

✓ Portable graph bundle live · download bundle · merged state

The bundle contains the canonical record plus signed events. A mirror can host it anywhere and recompute the same current state with the deterministic merge algorithm.

Claims

C1strongest claim

the single DeBERTa model surpass the human performance on the SuperGLUE benchmark for the first time in terms of macro-average score (89.9 versus 89.8)

C2weakest assumption

That the reported performance gains on MNLI, SQuAD, RACE, and SuperGLUE are caused by the disentangled attention and enhanced mask decoder rather than differences in training data volume, hyperparameters, or other unreported factors.

C3one line summary

DeBERTa improves BERT-style models by separating content and relative position in attention and adding absolute positions to the decoder, yielding consistent gains on NLU and NLG tasks and the first single-model superhuman score on SuperGLUE.

References

44 extracted · 44 resolved · 8 Pith anchors

[1] Layer Normalization · arXiv:1607.06450

[2] Longformer: The Long-Document Transformer 2004 · arXiv:2004.05150

[3] Language Models are Few-Shot Learners 2005 · arXiv:2005.14165

[4] SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation 2017 · arXiv:1708.00055

[5] Natural-to formal-language generation using tensor product representations 1910

Formal links

3 machine-checked theorem links

Cited by

74 papers in Pith

COBART: Controlled, Optimized, Bidirectional and Auto-Regressive Transformer for Ad Headline Generation

Structure Before Collapse: Transient semantic geometry in next-token prediction

CoVStream: Edge-Cloud Collaboration for Understanding of Long Video Streams

ITNet: A Learnable Integral Transform That Subsumes Convolution, Attention, and Recurrence

BamiBERT: A New BERT-based Language Model for Vietnamese

Receipt and verification

First computed	2026-07-05T03:20:32.636279Z
Builder	pith-number-builder-2026-05-17-v1
Signature	Pith Ed25519 (`pith-v1-2026-05`) · public key
Schema	pith-number/v1.0

Canonical hash

7ec05bd6631cd0d5bf2acd4e7ddf4ef2e5fde23008daff11fe2b75c0a846fce8

Aliases

arxiv: 2006.03654 · arxiv_version: 2006.03654v6 · doi: 10.48550/arxiv.2006.03654 · pith_short_12: P3AFXVTDDTIN · pith_short_16: P3AFXVTDDTINLPZK · pith_short_8: P3AFXVTD

Agent API

Resolver JSON Graph JSON Events JSON Schema Signing key

Verify this Pith Number yourself

curl -sH 'Accept: application/ld+json' https://pith.science/pith/P3AFXVTDDTINLPZKZVHH3X2O6L \
  | jq -c '.canonical_record' \
  | python3 -c "import sys,json,hashlib; b=json.dumps(json.loads(sys.stdin.read()), sort_keys=True, separators=(',',':'), ensure_ascii=False).encode(); print(hashlib.sha256(b).hexdigest())"
# expect: 7ec05bd6631cd0d5bf2acd4e7ddf4ef2e5fde23008daff11fe2b75c0a846fce8

Canonical record JSON

{
  "metadata": {
    "abstract_canon_sha256": "8e7df1e877f440f3daef317368609fe04000ad2ee869f6a7d3561ae21169fdbd",
    "cross_cats_sorted": [
      "cs.LG"
    ],
    "license": "http://arxiv.org/licenses/nonexclusive-distrib/1.0/",
    "primary_cat": "cs.CL",
    "submitted_at": "2020-06-05T19:54:34Z",
    "title_canon_sha256": "e220bd04a5976cdab32ba1b0a07868de86e7816a5391460e7d85f99ac0ca3b49"
  },
  "schema_version": "1.0",
  "source": {
    "id": "2006.03654",
    "kind": "arxiv",
    "version": 6
  }
}