Pith Number

pith:A2JKDCHH

pith:2026:A2JKDCHHRVJTZNMZSYNZ4DFZPJ

not attested not anchored not stored refs resolved

Unsteady Metrics and Benchmarking Cultures of AI Model Builders

Christo Buschek, Maty Bohacek, Stefan Baack

AI builders select benchmarks to fit marketing narratives rather than enable consistent scientific comparison.

arxiv:2605.14164 v1 · 2026-05-13 · cs.AI

Open paper page JSON Open Graph Bundle Merged state Verified badge What is a Pith Number?

Add to your LaTeX paper

\usepackage{pith}
\pithnumber{A2JKDCHHRVJTZNMZSYNZ4DFZPJ}

Prints a linked badge after your title and injects PDF metadata. Compiles on arXiv. Learn more · Embed verified badge

Record completeness

1 Bitcoin timestamp

2 Internet Archive

3 Author claim open · sign in to claim

4 Citations open

5 Replications open

✓ Portable graph bundle live · download bundle · merged state

The bundle contains the canonical record plus signed events. A mirror can host it anywhere and recompute the same current state with the deterministic merge algorithm.

Claims

C1strongest claim

We argue that highlighted benchmarks function less as standardized measurement tools and more as flexible narrative devices prioritizing market positioning over scientific evaluation.

C2weakest assumption

That the 139 model releases from 11 major builders and the 231 highlighted benchmarks they chose accurately capture the dominant evaluation practices and narrative strategies across the industry in 2025.

C3one line summary

AI model builders mostly highlight unique benchmarks that act as flexible narrative tools for market positioning rather than standardized scientific measurements.

References

69 extracted · 69 resolved · 10 Pith anchors

[1] Mohamed Abdalla and Moustafa Abdalla. 2021. The Grey Hoodie Project: Big Tobacco, Big Tech, and the Threat on Academic Integrity. InProceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Socie 2021 · doi:10.1145/3461702.3462563

[2] Norah Alzahrani, Hisham Alyahya, Yazeed Alnumay, Sultan Alrashed, Shaykhah Alsubaie, Yousef Almushayqih, Faisal Mirza, Nouf Alotaibi, Nora Al-Twairesh, Areeb Alowisheq, et al. 2024. When benchmarks ar 2024

[3] Concrete Problems in AI Safety 2016 · doi:10.48550/arxiv.1606.06565

[4] Anthropic. 2025. Claude 3.7 Sonnet System Card 2025

[5] InCOLING 2004: Pro- ceedings of the 20th International Conference on Computational Linguistics, pages 106–112, Geneva, Switzerland 2025

Receipt and verification

First computed	2026-05-17T23:39:11.436409Z
Builder	pith-number-builder-2026-05-17-v1
Signature	Pith Ed25519 (`pith-v1-2026-05`) · public key
Schema	pith-number/v1.0

Canonical hash

0692a188e78d533cb599961b9e0cb97a6079a414ecfdd85b52ac905d354f5bc7

Aliases

arxiv: 2605.14164 · arxiv_version: 2605.14164v1 · doi: 10.48550/arxiv.2605.14164 · pith_short_12: A2JKDCHHRVJT · pith_short_16: A2JKDCHHRVJTZNMZ · pith_short_8: A2JKDCHH

Agent API

Resolver JSON Graph JSON Events JSON Schema Signing key

Verify this Pith Number yourself

curl -sH 'Accept: application/ld+json' https://pith.science/pith/A2JKDCHHRVJTZNMZSYNZ4DFZPJ \
  | jq -c '.canonical_record' \
  | python3 -c "import sys,json,hashlib; b=json.dumps(json.loads(sys.stdin.read()), sort_keys=True, separators=(',',':'), ensure_ascii=False).encode(); print(hashlib.sha256(b).hexdigest())"
# expect: 0692a188e78d533cb599961b9e0cb97a6079a414ecfdd85b52ac905d354f5bc7

Canonical record JSON

{
  "metadata": {
    "abstract_canon_sha256": "11b0518dd17c08084c46cafee5b859f185a629992a6d0fd01e68ea2bfd3d041f",
    "cross_cats_sorted": [],
    "license": "http://creativecommons.org/licenses/by-nc-sa/4.0/",
    "primary_cat": "cs.AI",
    "submitted_at": "2026-05-13T22:39:10Z",
    "title_canon_sha256": "9e49ad973a3a1b8668c06496813326b3b941c1390bdc917806d74f9199b4820e"
  },
  "schema_version": "1.0",
  "source": {
    "id": "2605.14164",
    "kind": "arxiv",
    "version": 1
  }
}