Pith Number

pith:4PQYXXNT

pith:2022:4PQYXXNT3XJLFDYBVQC2WNCDH6

not attested not anchored not stored refs resolved

Holistic Evaluation of Language Models

Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher R\'e, Deepak Narayanan, Diana Acosta-Navas, Dilara Soylu, Dimitris Tsipras, Drew A. Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue Wang, Keshav Santhanam, Laurel Orr, Lucia Zheng, Mert Yuksekgonul, Michihiro Yasunaga, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri Chatterji, Omar Khattab, Percy Liang, Peter Henderson, Qian Huang, Rishi Bommasani, Ryan Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Tony Lee, Vishrav Chaudhary, William Wang, Xuechen Li, Yian Zhang, Yifan Mai, Yuhuai Wu, Yuhui Zhang, Yuta Koreeda

Language models are now densely benchmarked on the same 42 scenarios and 7 metrics under standardized conditions for all 30 models evaluated.

arxiv:2211.09110 v2 · 2022-11-16 · cs.CL · cs.AI · cs.LG

Open paper page JSON Open Graph Bundle Merged state Verified badge What is a Pith Number?

Add to your LaTeX paper

\usepackage{pith}
\pithnumber{4PQYXXNT3XJLFDYBVQC2WNCDH6}

Prints a linked badge after your title and injects PDF metadata. Compiles on arXiv. Learn more · Embed verified badge

Record completeness

1 Bitcoin timestamp

2 Internet Archive

3 Author claim open · sign in to claim

4 Citations open

5 Replications open

✓ Portable graph bundle live · download bundle · merged state

The bundle contains the canonical record plus signed events. A mirror can host it anywhere and recompute the same current state with the deterministic merge algorithm.

Claims

C1strongest claim

We improve this to 96.0%: now all 30 models have been densely benchmarked on the same core scenarios and metrics under standardized conditions. Our evaluation surfaces 25 top-level findings.

C2weakest assumption

The selection of a broad but feasible subset of scenarios and metrics from the full taxonomy is sufficient to deliver a holistic view, even while the paper explicitly notes missing or underrepresented areas such as question answering for neglected English dialects and metrics for trustworthiness.

C3one line summary

HELM establishes a multi-metric evaluation covering 30 language models on 42 scenarios (16 core) to raise average scenario coverage from 17.9% to 96% under uniform conditions while releasing all prompts, completions, and a toolkit.

References

21 extracted · 21 resolved · 7 Pith anchors

[1] Language Models are Few-Shot Learners 2021 · doi:10.18653/v1/2021.naacl-main.385

[2] doi: 10.18653/v1/2021.acl-long.150 2021 · doi:10.18653/v1/2021.acl-long.150

[3] URLhttps://glottolog.org/accessed2021-08-08 2018 · doi:10.5281/zenodo.4761960

[4] Measuring Coding Challenge Competence With APPS 2021 · doi:10.18653/v1/2021.eacl-main.225

[5] In Christopher Hitchcock & Alan Hajek, edi- tors: Oxford Handbook of Probability and Philosophy , Oxford University Press, pp 2021 · doi:10.1093/oxfordhb/9780199286546.001.0001/

Cited by

185 papers in Pith

From Prompts to Contracts: Harness Engineering for Auditable Enterprise LLM Agents

A Red Teaming Framework for Large Language Models: A Case Study on Faithfulness Evaluation

Age of LLM: A Strategic 1v1 Benchmark for Reasoning, Diplomacy and Reliability of Large Language Models under Fog of War

DiffusionBench: On Holistic Evaluation of Diffusion Transformers

DualEval: Joint Model-Item Calibration for Unified LLM Evaluation

Receipt and verification

First computed	2026-07-05T06:55:50.810187Z
Builder	pith-number-builder-2026-05-17-v1
Signature	Pith Ed25519 (`pith-v1-2026-05`) · public key
Schema	pith-number/v1.0

Canonical hash

e3e18bddb3ddd2b28f01ac05ab34433faca427f4e0532cbe6708657207dca654

Aliases

arxiv: 2211.09110 · arxiv_version: 2211.09110v2 · doi: 10.48550/arxiv.2211.09110 · pith_short_12: 4PQYXXNT3XJL · pith_short_16: 4PQYXXNT3XJLFDYB · pith_short_8: 4PQYXXNT

Agent API

Resolver JSON Graph JSON Events JSON Schema Signing key

Verify this Pith Number yourself

curl -sH 'Accept: application/ld+json' https://pith.science/pith/4PQYXXNT3XJLFDYBVQC2WNCDH6 \
  | jq -c '.canonical_record' \
  | python3 -c "import sys,json,hashlib; b=json.dumps(json.loads(sys.stdin.read()), sort_keys=True, separators=(',',':'), ensure_ascii=False).encode(); print(hashlib.sha256(b).hexdigest())"
# expect: e3e18bddb3ddd2b28f01ac05ab34433faca427f4e0532cbe6708657207dca654

Canonical record JSON

{
  "metadata": {
    "abstract_canon_sha256": "5e8efe20a353e260589415162a56e5c0872f9e62d4f58f9e1aa97f12c642bd32",
    "cross_cats_sorted": [
      "cs.AI",
      "cs.LG"
    ],
    "license": "http://creativecommons.org/licenses/by/4.0/",
    "primary_cat": "cs.CL",
    "submitted_at": "2022-11-16T18:51:34Z",
    "title_canon_sha256": "8b61ecb21f40f5050d219aa6879d32a1f08e0e2a55ff5e7ad0fa2e141ccbc56c"
  },
  "schema_version": "1.0",
  "source": {
    "id": "2211.09110",
    "kind": "arxiv",
    "version": 2
  }
}