{"paper":{"title":"AI Evaluation Should Require Standardized Item-Level Data Releases","license":"http://creativecommons.org/licenses/by/4.0/","headline":"Item-level AI benchmark data is essential for a rigorous science of AI evaluation.","cross_cats":["cs.CY","cs.DB"],"primary_cat":"cs.AI","authors_text":"Dongyao Zhu, Han Jiang, Sang T. Truong, Sanmi Koyejo, Susu Zhang, Xiaoyuan Yi, Xing Xie, Yuzhuo Bai, Ziang Xiao","submitted_at":"2026-02-27T04:31:30Z","abstract_excerpt":"This position paper argues that standardized item-level benchmark data should become the default infrastructure for AI evaluation. Current evaluations suffer from underspecified item selection, construct misalignment, and poor generalization. The root cause of these failures is a misplaced focus on aggregate model scores. Without item-level evidence, validity claims cannot be assessed, resulting in inflated capability claims, misdirected research, and unwarranted trust in deployed systems. Our position is that designing valid evaluations requires empirical evidence from item-level model respon"},"claims":{"count":4,"items":[{"kind":"strongest_claim","text":"item-level AI benchmark data is essential for establishing a rigorous science of AI evaluation. Item-level analysis enables fine-grained diagnostics and principled validation of benchmarks.","source":"verdict.strongest_claim","status":"machine_extracted","claim_id":"C1","attestation":"unclaimed"},{"kind":"weakest_assumption","text":"That the systemic validity failures described are primarily caused by the absence of item-level data and that supplying such data will by itself enable principled validation without additional changes to benchmark design or metric selection.","source":"verdict.weakest_assumption","status":"machine_extracted","claim_id":"C2","attestation":"unclaimed"},{"kind":"one_line_summary","text":"Item-level benchmark data is essential for rigorous AI evaluation because it enables fine-grained diagnostics and principled validation of benchmarks that aggregate scores cannot provide.","source":"verdict.one_line_summary","status":"machine_extracted","claim_id":"C3","attestation":"unclaimed"},{"kind":"headline","text":"Item-level AI benchmark data is essential for a rigorous science of AI evaluation.","source":"verdict.pith_extraction.headline","status":"machine_extracted","claim_id":"C4","attestation":"unclaimed"}],"snapshot_sha256":"907bcc88b8abe52928adeb7b8b7a1d8bb0f1cf80236a38e1ea8d7325ce606c66"},"source":{"id":"2604.03244","kind":"arxiv","version":2},"verdict":{"id":"6fc69267-01d2-4646-ae5d-773dd2ce9884","model_set":{"reader":"grok-4.3"},"created_at":"2026-05-15T19:13:38.408450Z","strongest_claim":"item-level AI benchmark data is essential for establishing a rigorous science of AI evaluation. Item-level analysis enables fine-grained diagnostics and principled validation of benchmarks.","one_line_summary":"Item-level benchmark data is essential for rigorous AI evaluation because it enables fine-grained diagnostics and principled validation of benchmarks that aggregate scores cannot provide.","pipeline_version":"pith-pipeline@v0.9.0","weakest_assumption":"That the systemic validity failures described are primarily caused by the absence of item-level data and that supplying such data will by itself enable principled validation without additional changes to benchmark design or metric selection.","pith_extraction_headline":"Item-level AI benchmark data is essential for a rigorous science of AI evaluation."},"integrity":{"clean":true,"summary":{"advisory":0,"critical":0,"by_detector":{},"informational":0},"endpoint":"/pith/2604.03244/integrity.json","findings":[],"available":true,"detectors_run":[],"snapshot_sha256":"c28c3603d3b5d939e8dc4c7e95fa8dfce3d595e45f758748cecf8e644a296938"},"references":{"count":0,"sample":[],"resolved_work":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57","internal_anchors":0},"formal_canon":{"evidence_count":2,"snapshot_sha256":"fb29018738a2e6f6e1c31e566aa5cacd611df5094d66cbae08fa0ac0e209813f"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"}