Pith Number

pith:5N2N3GXH

pith:2019:5N2N3GXHAVGJ3DIAERBA2ED736

not attested not anchored not stored refs resolved

HellaSwag: Can a Machine Really Finish Your Sentence?

Ali Farhadi, Ari Holtzman, Rowan Zellers, Yejin Choi, Yonatan Bisk

HellaSwag shows state-of-the-art models still fail at commonsense sentence completion that humans solve easily.

arxiv:1905.07830 v1 · 2019-05-19 · cs.CL

Open paper page JSON Open Graph Bundle Merged state Verified badge What is a Pith Number?

Add to your LaTeX paper

\usepackage{pith}
\pithnumber{5N2N3GXHAVGJ3DIAERBA2ED736}

Prints a linked badge after your title and injects PDF metadata. Compiles on arXiv. Learn more · Embed verified badge

Record completeness

1 Bitcoin timestamp

2 Internet Archive

3 Author claim open · sign in to claim

4 Citations open

5 Replications open

✓ Portable graph bundle live · download bundle · merged state

The bundle contains the canonical record plus signed events. A mirror can host it anywhere and recompute the same current state with the deterministic merge algorithm.

Claims

C1strongest claim

Though its questions are trivial for humans (>95% accuracy), state-of-the-art models struggle (<48%).

C2weakest assumption

That adversarial filtering produces examples requiring genuine commonsense reasoning rather than merely exploiting specific weaknesses or distributional artifacts in the models used during filtering.

C3one line summary

HellaSwag dataset shows state-of-the-art models fail commonsense inference tasks that humans solve easily, built via adversarial filtering of distractors.

References

19 extracted · 19 resolved · 2 Pith anchors

[1] Yonatan Belinkov and Yonatan Bisk. 2018. Synthetic and natural noise both break neural machine translation. In ICLR. ICLR 2018

[2] Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2017. Enhanced lstm for natural language inference. In Proceedings of the 55th Annual Meeting of the Association for Computa 2017

[3] BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding 2018 · arXiv:1810.04805

[4] Max Glockner, Vered Shwartz, and Yoav Goldberg. 2018. Breaking nli systems with sentences that require simple lexical inferences. In Proceedings of the 56th Annual Meeting of the Association for Compu 2018

[5] Jonathan Gordon and Benjamin Van Durme. 2013. Reporting bias and knowledge acquisition. In Proceedings of the 2013 workshop on Automated knowledge base construction, pages 25--30. ACM 2013

Cited by

154 papers in Pith

Improved Large Language Diffusion Models

SharQ: Bridging Activation Sparsity and FP4 Quantization for LLM Inference

GRINQH: Graded Input-based Quantization Hierarchy for Efficient LLM Generation

Tapered Language Models

On the Expressive Power of Weight Quantization in Large Language Models

Receipt and verification

First computed	2026-07-04T23:38:28.263807Z
Builder	pith-number-builder-2026-05-17-v1
Signature	Pith Ed25519 (`pith-v1-2026-05`) · public key
Schema	pith-number/v1.0

Canonical hash

eb74dd9ae7054c9d8d0024420d107fdf956cf16e993c09c33a7d78713303065d

Aliases

arxiv: 1905.07830 · arxiv_version: 1905.07830v1 · doi: 10.48550/arxiv.1905.07830 · pith_short_12: 5N2N3GXHAVGJ · pith_short_16: 5N2N3GXHAVGJ3DIA · pith_short_8: 5N2N3GXH

Agent API

Resolver JSON Graph JSON Events JSON Schema Signing key

Verify this Pith Number yourself

curl -sH 'Accept: application/ld+json' https://pith.science/pith/5N2N3GXHAVGJ3DIAERBA2ED736 \
  | jq -c '.canonical_record' \
  | python3 -c "import sys,json,hashlib; b=json.dumps(json.loads(sys.stdin.read()), sort_keys=True, separators=(',',':'), ensure_ascii=False).encode(); print(hashlib.sha256(b).hexdigest())"
# expect: eb74dd9ae7054c9d8d0024420d107fdf956cf16e993c09c33a7d78713303065d

Canonical record JSON

{
  "metadata": {
    "abstract_canon_sha256": "927c4889782be6b6bd5c205399e62f8e4869cac9638c87bfc36eb6830589af43",
    "cross_cats_sorted": [],
    "license": "http://arxiv.org/licenses/nonexclusive-distrib/1.0/",
    "primary_cat": "cs.CL",
    "submitted_at": "2019-05-19T23:57:23Z",
    "title_canon_sha256": "e9e53d4b2c8e24587b3aa33ee5bfedbfafdf21af4617107e422e70fe02959910"
  },
  "schema_version": "1.0",
  "source": {
    "id": "1905.07830",
    "kind": "arxiv",
    "version": 1
  }
}