{"paper":{"title":"Let's Measure Information Step-by-Step: AI-Based Evaluation Beyond Vibes","license":"http://creativecommons.org/licenses/by/4.0/","headline":"Treating AI evaluation as mutual information estimation via prompting lets total variation distance resist adversarial attacks without ground truth.","cross_cats":["cs.IT","math.IT"],"primary_cat":"cs.LG","authors_text":"Sanmi Koyejo, Zachary Robertson","submitted_at":"2025-08-07T15:11:43Z","abstract_excerpt":"We evaluate artificial intelligence (AI) systems without ground truth by exploiting a link between strategic gaming and information loss. Building on established information theory, we analyze which mechanisms resist adversarial manipulation. This motivates mutual evaluation, where the overseer is treated as a strategic player estimating mutual information by prompting, making truthful agent reporting an optimal strategy. We show that certain f-divergences, such as total variation distance (TVD), maintain polynomial guarantees under attack, building on an established exponential barrier for es"},"claims":{"count":4,"items":[{"kind":"strongest_claim","text":"Certain f-divergences such as total variation distance maintain polynomial guarantees under attack for mutual information estimation in AI evaluation, with TVD-MI retaining AUC 0.70-0.77 under adversarial attacks while other approaches decay toward chance.","source":"verdict.strongest_claim","status":"machine_extracted","claim_id":"C1","attestation":"unclaimed"},{"kind":"weakest_assumption","text":"The assumption that treating the overseer as a strategic player estimating mutual information by prompting makes truthful agent reporting an optimal strategy, which underpins the claim that the method resists adversarial manipulation without ground truth.","source":"verdict.weakest_assumption","status":"machine_extracted","claim_id":"C2","attestation":"unclaimed"},{"kind":"one_line_summary","text":"The paper introduces mutual evaluation with f-divergences like total variation distance to robustly estimate information relationships in AI systems without ground truth, showing better resistance to adversarial attacks than other methods.","source":"verdict.one_line_summary","status":"machine_extracted","claim_id":"C3","attestation":"unclaimed"},{"kind":"headline","text":"Treating AI evaluation as mutual information estimation via prompting lets total variation distance resist adversarial attacks without ground truth.","source":"verdict.pith_extraction.headline","status":"machine_extracted","claim_id":"C4","attestation":"unclaimed"}],"snapshot_sha256":"3414d5498b7a321e96edf0fb29c1ca4cf0aa453f744a83c80d3ac4383065e287"},"source":{"id":"2508.05469","kind":"arxiv","version":4},"verdict":{"id":"760b1f5f-804d-4630-85e0-eab79172a198","model_set":{"reader":"grok-4.3"},"created_at":"2026-05-18T23:41:16.558764Z","strongest_claim":"Certain f-divergences such as total variation distance maintain polynomial guarantees under attack for mutual information estimation in AI evaluation, with TVD-MI retaining AUC 0.70-0.77 under adversarial attacks while other approaches decay toward chance.","one_line_summary":"The paper introduces mutual evaluation with f-divergences like total variation distance to robustly estimate information relationships in AI systems without ground truth, showing better resistance to adversarial attacks than other methods.","pipeline_version":"pith-pipeline@v0.9.0","weakest_assumption":"The assumption that treating the overseer as a strategic player estimating mutual information by prompting makes truthful agent reporting an optimal strategy, which underpins the claim that the method resists adversarial manipulation without ground truth.","pith_extraction_headline":"Treating AI evaluation as mutual information estimation via prompting lets total variation distance resist adversarial attacks without ground truth."},"integrity":{"clean":true,"summary":{"advisory":0,"critical":0,"by_detector":{},"informational":0},"endpoint":"/pith/2508.05469/integrity.json","findings":[],"available":true,"detectors_run":[],"snapshot_sha256":"c28c3603d3b5d939e8dc4c7e95fa8dfce3d595e45f758748cecf8e644a296938"},"references":{"count":0,"sample":[],"resolved_work":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57","internal_anchors":0},"formal_canon":{"evidence_count":2,"snapshot_sha256":"b6d8228ae0fdd687baafdead3d1d16faf3261e616b7d1c1e1d9114fda09251b3"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"}