Pith Number

pith:65R7KA66

pith:2026:65R7KA66TKKJELWLEL6GCNEDY5

not attested not anchored not stored refs pending

STABLEVAL: Disagreement-Aware and Stable Evaluation of AI Systems

Akash Bonagiri, Angelina Lai, Devang Borkar, Gerard Janno Anderias, Gezheng Kang, Houman Homayoun, Ishant Gandhi, Saee Patil, Setareh Rafatirad

STABLEVAL models latent item correctness and annotator confusion to produce stable AI system rankings where majority vote fails.

arxiv:2605.02122 v2 · 2026-05-04 · cs.LG · cs.AI

Open paper page JSON Open Graph Bundle Merged state Verified badge What is a Pith Number?

Add to your LaTeX paper

\usepackage{pith}
\pithnumber{65R7KA66TKKJELWLEL6GCNEDY5}

Prints a linked badge after your title and injects PDF metadata. Compiles on arXiv. Learn more · Embed verified badge

Record completeness

1 Bitcoin timestamp

2 Internet Archive

3 Author claim open · sign in to claim

4 Citations open

5 Replications open

✓ Portable graph bundle live · download bundle · merged state

The bundle contains the canonical record plus signed events. A mirror can host it anywhere and recompute the same current state with the deterministic merge algorithm.

Claims

C1strongest claim

Across controlled synthetic experiments and multiple real-world human-annotated benchmarks, majority vote exhibits increasing score error and ranking instability under annotator heterogeneity and adversarial noise, while STABLEVAL yields more stable and statistically grounded system rankings.

C2weakest assumption

That the chosen probabilistic model of latent item correctness and annotator-specific confusion patterns will produce posteriors that genuinely reflect real-world stability rather than artifacts of the modeling assumptions or fitting procedure.

C3one line summary

STABLEVAL models latent correctness and annotator confusion to deliver more stable and uncertainty-aware AI system rankings than majority-vote aggregation.

Receipt and verification

First computed	2026-06-02T03:05:05.527338Z
Builder	pith-number-builder-2026-05-17-v1
Signature	Pith Ed25519 (`pith-v1-2026-05`) · public key
Schema	pith-number/v1.0

Canonical hash

f763f503de9a94922ecb22fc613483c77e919378d6ab724a677a01b80d15fa60

Aliases

arxiv: 2605.02122 · arxiv_version: 2605.02122v2 · doi: 10.48550/arxiv.2605.02122 · pith_short_12: 65R7KA66TKKJ · pith_short_16: 65R7KA66TKKJELWL · pith_short_8: 65R7KA66

Agent API

Resolver JSON Graph JSON Events JSON Schema Signing key

Verify this Pith Number yourself

curl -sH 'Accept: application/ld+json' https://pith.science/pith/65R7KA66TKKJELWLEL6GCNEDY5 \
  | jq -c '.canonical_record' \
  | python3 -c "import sys,json,hashlib; b=json.dumps(json.loads(sys.stdin.read()), sort_keys=True, separators=(',',':'), ensure_ascii=False).encode(); print(hashlib.sha256(b).hexdigest())"
# expect: f763f503de9a94922ecb22fc613483c77e919378d6ab724a677a01b80d15fa60

Canonical record JSON

{
  "metadata": {
    "abstract_canon_sha256": "5d3e78a0e5f93f1b7a658e9ec2bb4ec9d46f3592a37701079bba0462aebb27e9",
    "cross_cats_sorted": [
      "cs.AI"
    ],
    "license": "http://creativecommons.org/licenses/by/4.0/",
    "primary_cat": "cs.LG",
    "submitted_at": "2026-05-04T01:03:48Z",
    "title_canon_sha256": "207f94c72b77219b78e201130c30d6d1464da1323029d97f049bada399110d29"
  },
  "schema_version": "1.0",
  "source": {
    "id": "2605.02122",
    "kind": "arxiv",
    "version": 2
  }
}