{"paper":{"title":"ContractBench: Can LLM Agents Preserve Observation Contracts?","license":"http://creativecommons.org/licenses/by/4.0/","headline":"LLM agents must preserve observation contracts like tokens and presigned URLs, yet current models routinely fail at this separate capability.","cross_cats":["cs.AI"],"primary_cat":"cs.SE","authors_text":"Arkaprava De, Hanwen Xing, Hao Chen, Jicheng Wang, Yifeng He, Zili Wang","submitted_at":"2026-05-17T06:37:04Z","abstract_excerpt":"Tool-augmented LLM agents call APIs whose intermediate outputs, such as presigned URLs, session tokens, and OAuth state parameters, are observation contracts: artifacts whose later use is constrained by the external system that produced them. We show that observation contract compliance (preserving the temporal validity and byte-level integrity) is an emergent, regression-prone capability: it is neither guaranteed by general tool-use ability nor consistently improved by larger or newer models. To measure this, we introduce ContractBench, a benchmark of 33 dual-axis tasks that probe two orthogo"},"claims":{"count":4,"items":[{"kind":"strongest_claim","text":"no evaluated model clears 80%, with Claude-Opus-4.6 leading at 77.8%, revealing that current frontier models still fail to comply with observation contracts","source":"verdict.strongest_claim","status":"machine_extracted","claim_id":"C1","attestation":"unclaimed"},{"kind":"weakest_assumption","text":"The 33 dual-axis tasks and their failure labels drawn from real-world API specifications sufficiently capture the observation-contract compliance problem that arises in deployed tool-augmented agents.","source":"verdict.weakest_assumption","status":"machine_extracted","claim_id":"C2","attestation":"unclaimed"},{"kind":"one_line_summary","text":"ContractBench shows that LLM agents frequently violate observation contracts by using expired artifacts or corrupting their byte integrity, with no model exceeding 80% success and notable scaling irregularities across families.","source":"verdict.one_line_summary","status":"machine_extracted","claim_id":"C3","attestation":"unclaimed"},{"kind":"headline","text":"LLM agents must preserve observation contracts like tokens and presigned URLs, yet current models routinely fail at this separate capability.","source":"verdict.pith_extraction.headline","status":"machine_extracted","claim_id":"C4","attestation":"unclaimed"}],"snapshot_sha256":"8a260291f250f1411f401f72766fe4b8cd014846e2636cda4027f988ebe50706"},"source":{"id":"2605.17281","kind":"arxiv","version":1},"verdict":{"id":"5bc5e912-8b8f-4b3c-8c96-72ebaa0fab0b","model_set":{"reader":"grok-4.3"},"created_at":"2026-05-19T22:56:39.462455Z","strongest_claim":"no evaluated model clears 80%, with Claude-Opus-4.6 leading at 77.8%, revealing that current frontier models still fail to comply with observation contracts","one_line_summary":"ContractBench shows that LLM agents frequently violate observation contracts by using expired artifacts or corrupting their byte integrity, with no model exceeding 80% success and notable scaling irregularities across families.","pipeline_version":"pith-pipeline@v0.9.0","weakest_assumption":"The 33 dual-axis tasks and their failure labels drawn from real-world API specifications sufficiently capture the observation-contract compliance problem that arises in deployed tool-augmented agents.","pith_extraction_headline":"LLM agents must preserve observation contracts like tokens and presigned URLs, yet current models routinely fail at this separate capability."},"integrity":{"clean":true,"summary":{"advisory":0,"critical":0,"by_detector":{},"informational":0},"endpoint":"/pith/2605.17281/integrity.json","findings":[],"available":true,"detectors_run":[{"name":"doi_compliance","ran_at":"2026-05-19T23:01:55.266047Z","status":"completed","version":"1.0.0","findings_count":0},{"name":"doi_title_agreement","ran_at":"2026-05-19T23:01:19.691587Z","status":"completed","version":"1.0.0","findings_count":0},{"name":"claim_evidence","ran_at":"2026-05-19T22:01:57.826415Z","status":"completed","version":"1.0.0","findings_count":0},{"name":"ai_meta_artifact","ran_at":"2026-05-19T21:33:23.771987Z","status":"skipped","version":"1.0.0","findings_count":0}],"snapshot_sha256":"7aa72b3cde359d45cbd2ac55640b44b9d8777d9c8f58c8e1c83134c0c50ccae9"},"references":{"count":44,"sample":[{"doi":"","year":2026,"title":"2026 , eprint =","work_id":"f42c6a1e-2720-44a2-9b07-d0d85070e8f4","ref_index":1,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2026,"title":"2026 , eprint =","work_id":"4de62ded-8c5f-4635-892f-e124acbaf237","ref_index":2,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"10.1145/365230.365252","year":1966,"title":"Programming semantics for multiprogrammed computations.Commun","work_id":"853d34ab-9346-40c5-aa7c-a65cb42958e5","ref_index":3,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2024,"title":"2024 , url =","work_id":"e979d9bc-7642-42e6-9755-75dd0e116f94","ref_index":4,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2024,"title":"AgentBench: Evaluating","work_id":"98c5d123-b91d-45c7-ae4f-14f91745e06c","ref_index":5,"cited_arxiv_id":"","is_internal_anchor":false}],"resolved_work":44,"snapshot_sha256":"3e322bba5d1c93429439a374bdd7e24c3c321ac4bb6abcca52e419396e03bdf0","internal_anchors":1},"formal_canon":{"evidence_count":2,"snapshot_sha256":"4484454e7eed5f4c186d02123f11525561d4f7836ef5bff1e3f2dabc305f3017"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"}