{"paper":{"title":"CTFusion: A CTF-based Benchmark for LLM Agent Evaluation","license":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/","headline":"Reused CTF challenges allow data contamination that inflates LLM agent scores, which CTFusion fixes by streaming evaluations from live events.","cross_cats":["cs.CR"],"primary_cat":"cs.LG","authors_text":"Dongjun Lee, Ga-eun Bae, Insu Yun","submitted_at":"2026-05-12T04:23:42Z","abstract_excerpt":"Recent advances in Large Language Models (LLMs) have enabled agentic systems for complex, multi-step tasks; cybersecurity is emerging as a prominent application. To evaluate such agents, researchers widely adopt Capture The Flag (CTF) benchmarks. However, current CTF benchmarks reuse existing challenges, which exposes them to data contamination and potential cheating. Notably, we confirmed these issues in practice by integrating web search tools into an existing agent. To address these limitations, we present CTFusion, a streaming evaluation framework built on Live CTFs. To achieve this, CTFus"},"claims":{"count":4,"items":[{"kind":"strongest_claim","text":"existing CTF benchmarks can be unreliable in assessing LLM-based agents, while CTFusion can serve as a robust solution for evaluating cybersecurity agents.","source":"verdict.strongest_claim","status":"machine_extracted","claim_id":"C1","attestation":"unclaimed"},{"kind":"weakest_assumption","text":"That live CTF events remain uncontaminated and that the per-agent independence plus first-flag forwarding fully eliminates the contamination and competition-impact problems demonstrated with web-search tools.","source":"verdict.weakest_assumption","status":"machine_extracted","claim_id":"C2","attestation":"unclaimed"},{"kind":"one_line_summary","text":"CTFusion is a live-CTF streaming benchmark that prevents data contamination by forwarding only the first correct flag per challenge under a shared team account.","source":"verdict.one_line_summary","status":"machine_extracted","claim_id":"C3","attestation":"unclaimed"},{"kind":"headline","text":"Reused CTF challenges allow data contamination that inflates LLM agent scores, which CTFusion fixes by streaming evaluations from live events.","source":"verdict.pith_extraction.headline","status":"machine_extracted","claim_id":"C4","attestation":"unclaimed"}],"snapshot_sha256":"1fe892b9d84b8bc28fe98a66b577c919ddd98e59ac1c6e813a5424e945aac632"},"source":{"id":"2605.11504","kind":"arxiv","version":2},"verdict":{"id":"3dc7618e-7e16-4d92-9d47-1a96c996e63c","model_set":{"reader":"grok-4.3"},"created_at":"2026-05-13T01:33:21.599669Z","strongest_claim":"existing CTF benchmarks can be unreliable in assessing LLM-based agents, while CTFusion can serve as a robust solution for evaluating cybersecurity agents.","one_line_summary":"CTFusion is a live-CTF streaming benchmark that prevents data contamination by forwarding only the first correct flag per challenge under a shared team account.","pipeline_version":"pith-pipeline@v0.9.0","weakest_assumption":"That live CTF events remain uncontaminated and that the per-agent independence plus first-flag forwarding fully eliminates the contamination and competition-impact problems demonstrated with web-search tools.","pith_extraction_headline":"Reused CTF challenges allow data contamination that inflates LLM agent scores, which CTFusion fixes by streaming evaluations from live events."},"integrity":{"clean":true,"summary":{"advisory":0,"critical":0,"by_detector":{},"informational":0},"endpoint":"/pith/2605.11504/integrity.json","findings":[],"available":true,"detectors_run":[{"name":"claim_evidence","ran_at":"2026-05-20T04:02:00.704837Z","status":"completed","version":"1.0.0","findings_count":0},{"name":"ai_meta_artifact","ran_at":"2026-05-19T12:34:54.349487Z","status":"completed","version":"1.0.0","findings_count":0},{"name":"doi_title_agreement","ran_at":"2026-05-19T09:31:19.106952Z","status":"completed","version":"1.0.0","findings_count":0},{"name":"doi_compliance","ran_at":"2026-05-19T08:20:57.289989Z","status":"completed","version":"1.0.0","findings_count":0}],"snapshot_sha256":"c6e3a7bf7fbf0d9f0f065f377e15c141ab1d90951248af6412c92059876cb89b"},"references":{"count":0,"sample":[],"resolved_work":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57","internal_anchors":0},"formal_canon":{"evidence_count":1,"snapshot_sha256":"3a24f1dee57e1f71f53a9097ee69bb3595ab7487d4fa48626f552c2966fd40fb"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"}