Pith Number

pith:GWX5LQSW

pith:2025:GWX5LQSWIQNRJJU6TGMAMCE2DK

not attested not anchored not stored refs pending

Let's Measure Information Step-by-Step: AI-Based Evaluation Beyond Vibes

Sanmi Koyejo, Zachary Robertson

Treating AI evaluation as mutual information estimation via prompting lets total variation distance resist adversarial attacks without ground truth.

arxiv:2508.05469 v4 · 2025-08-07 · cs.LG · cs.IT · math.IT

Open paper page JSON Open Graph Bundle Merged state Verified badge What is a Pith Number?

Add to your LaTeX paper

\usepackage{pith}
\pithnumber{GWX5LQSWIQNRJJU6TGMAMCE2DK}

Prints a linked badge after your title and injects PDF metadata. Compiles on arXiv. Learn more · Embed verified badge

Record completeness

1 Bitcoin timestamp

2 Internet Archive

3 Author claim open · sign in to claim

4 Citations open

5 Replications open

✓ Portable graph bundle live · download bundle · merged state

The bundle contains the canonical record plus signed events. A mirror can host it anywhere and recompute the same current state with the deterministic merge algorithm.

Claims

C1strongest claim

Certain f-divergences such as total variation distance maintain polynomial guarantees under attack for mutual information estimation in AI evaluation, with TVD-MI retaining AUC 0.70-0.77 under adversarial attacks while other approaches decay toward chance.

C2weakest assumption

The assumption that treating the overseer as a strategic player estimating mutual information by prompting makes truthful agent reporting an optimal strategy, which underpins the claim that the method resists adversarial manipulation without ground truth.

C3one line summary

The paper introduces mutual evaluation with f-divergences like total variation distance to robustly estimate information relationships in AI systems without ground truth, showing better resistance to adversarial attacks than other methods.

Formal links

2 machine-checked theorem links

Cited by

1 paper in Pith

Scoring Rules! Statistical and Strategic Alignment for Text Evaluation Metrics

Receipt and verification

First computed	2026-06-23T03:13:46.603272Z
Builder	pith-number-builder-2026-05-17-v1
Signature	Pith Ed25519 (`pith-v1-2026-05`) · public key
Schema	pith-number/v1.0

Canonical hash

35afd5c256441b14a69e999806089a1aa987c1e6078a1c93d530b9b956376e7e

Aliases

arxiv: 2508.05469 · arxiv_version: 2508.05469v4 · doi: 10.48550/arxiv.2508.05469 · pith_short_12: GWX5LQSWIQNR · pith_short_16: GWX5LQSWIQNRJJU6 · pith_short_8: GWX5LQSW

Agent API

Resolver JSON Graph JSON Events JSON Schema Signing key

Verify this Pith Number yourself

curl -sH 'Accept: application/ld+json' https://pith.science/pith/GWX5LQSWIQNRJJU6TGMAMCE2DK \
  | jq -c '.canonical_record' \
  | python3 -c "import sys,json,hashlib; b=json.dumps(json.loads(sys.stdin.read()), sort_keys=True, separators=(',',':'), ensure_ascii=False).encode(); print(hashlib.sha256(b).hexdigest())"
# expect: 35afd5c256441b14a69e999806089a1aa987c1e6078a1c93d530b9b956376e7e

Canonical record JSON

{
  "metadata": {
    "abstract_canon_sha256": "e70c7b7aeceee79bb18742f113955ba86826c00c591beba8aae5dbd05dc83b69",
    "cross_cats_sorted": [
      "cs.IT",
      "math.IT"
    ],
    "license": "http://creativecommons.org/licenses/by/4.0/",
    "primary_cat": "cs.LG",
    "submitted_at": "2025-08-07T15:11:43Z",
    "title_canon_sha256": "f63bbda0467da21521db117d5cf6bd28ba7585450ee8b5740d2afb5112ae82fc"
  },
  "schema_version": "1.0",
  "source": {
    "id": "2508.05469",
    "kind": "arxiv",
    "version": 4
  }
}