REVIEW 4 major objections 7 minor 20 references
The Next Challenge for Agentic Cybersecurity: A Realistic, Contamination-Free Reverse Engineering Benchmark
T0 review · 4 major / 7 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read SRE-Bench, a reverse-engineering benchmark built entirely from scratch, shows today's best AI agents can fully reverse engineer only about a third of real-world-scale binaries.
desk verdict A serious benchmark with a real contamination blind spot: the paper never discloses whether the LLM API providers could train on the private binaries, which would break the durable property that is its main selling point. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is a clean-room evaluation pipeline: 19 target programs written from scratch against private specifications, a 27,000-line in-house protection suite with 44 primitives (per-page authenticated encryption, lazy decryption, measurement-keyed anti-debugging, loader-logic virtualization, and anti-dump deception among them), and deterministic graders that are themselves iteratively hardened by adversarial reward-hacking audits. The pipeline turns each program into multiple binary instances and gives every instance exactly six verifiable tasks, so the benchmark can be scored without human judgment and without giving the agent any feedback during a run. Its governing design rule, "fold, don't branch," feeds tampering, debugging, and observation measurements into page-key derivation rather than into ordinary condition checks, so an agent that attempts to observe or modify the binary gets silent key corruption instead of a check it can patch away.
What would settle it
Ask a frontier model for the withheld instance-specific secrets (archive passwords, trigger constants, firmware fuse words) without giving it the corresponding binary; if any model can name them, or if a training-data membership probe recognizes any SRE-Bench program, the contamination-free premise fails.
Extended reading notes
Core claim
SRE-Bench is presented as the first reverse-engineering benchmark that is both realistic and contamination-free: 19 programs written from scratch by reverse-engineering experts, averaging 16,915.8 lines of code, compiled into 262 binary instances and hardened with an in-house suite of 44 anti-analysis primitives, with every instance scored by a deterministic grader across six tasks, for 1,572 tasks total. The paper reports that on this benchmark agentic reverse engineering is largely unsolved: the strongest evaluated model averages 61.4% of the available score and fully recovers 31.5% of instances, while the weakest model never fully recovers one. The in-house protections are the sharpest obstacle, halving the strongest model's score and driving all others to near zero, while compiler optimization and static linking — traditionally hard for humans — cost agents almost nothing and stripped symbols cost the most. Controlled ablations show the two design requirements are load-bearing: a publicly derived target and a small clean-room target are each solved in minutes for a few dollars, while a target that is both private and at real-world scale cleanly separates the models.
Load-bearing premise
The benchmark's validity rests on every artifact — all 19 programs, the 27,000-line protection suite, constants, and reference answers — being genuinely new and private, never present in any model's training data, and this privacy cannot be independently checked because the artifacts are not released.
Editorial extensions
If this is right
- Source-code security scores should not be read as binary reverse-engineering capability; the gap measured here is direct evidence that the two need separate evaluation.
- Future RE benchmarks should require both clean-room provenance and real-world scale, because the ablation shows relaxing either one collapses difficulty.
- Protection and obfuscation, not compiler optimization, are the dominant obstacle for current agents, so progress in defeating layered protections should move overall scores more than better handling of optimization or linking.
- The benchmark's deterministic six-task grading makes agent progress comparable across models and over time, including by domain and by build factor.
Reading between the lines
- A direct test of the paper's interpretation would be to strip or rename all remaining lexical anchors in a hardened binary and measure score loss; if the model's advantage depends on names and strings rather than instruction-level reasoning, removing them should reproduce the protection-induced collapse.
- Because a large share of exploited vulnerabilities come from vendors that do not release source code, a real RE gap would directly cap autonomous defense and offense on much real-world security work; this makes RE capability a bottleneck worth measuring even if the benchmark's exact numbers change.
- The secret-withholding design could double as a contamination audit: future releases could plant canary secrets and ask whether any model can name them without seeing the binary, turning the unpublishable privacy assumption into a testable one.
- The 19-program pool is small, especially for firmware; a natural extension is to run the same clean-room pipeline over more domains and languages to check whether the reported difficulty ordering (malware hardest, network protocol easiest) is stable or an artifact of program selection.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper presents SRE-Bench, an agentic binary reverse-engineering (RE) benchmark that combines two design properties the authors argue are jointly missing from prior work: contamination-freedom (target programs not present in LLM pretraining corpora) and real-world scale. The benchmark consists of 19 from-scratch programs averaging 16,915.8 LoC across the domains of network protocols, games, file formats, malware, and firmware; an in-house 27K-LoC anti-analysis suite with 44 protection primitives; and 262 binary instances (16 non-firmware programs x 16 builds plus 6 firmware variants), each with six deterministically graded tasks, for 1,572 tasks total. The evaluation runs five frontier LLM agents (GPT-5.6-sol, Claude-Opus-5, GPT-5.5, Grok-4.5, GLM-5.2) in a uniform bash-only harness at a reported cost of $31.4K; the strongest model scores 3.69/6 (61.4%) and fully solves 80 of 254 gradeable instances (31.5%). A controlled three-program ablation (Table 5) shows that a public gzip-based target and a 1.1K-LoC clean-room target are both solved cheaply and quickly, whereas the full private target separates models, supporting the claim that both contamination control and scale are load-bearing. The paper concludes that strong source-code security capability does not transfer to binary analysis and positions SRE-Bench as a durable testbed.
Significance. If the benchmark is as described, it addresses a genuine gap: existing agentic RE benchmarks are either CTF-derived with contamination risk and entry-level difficulty, or contamination-controlled but toy-scale (36-64 LoC on average). SRE-Bench's scale (16.9K LoC on average, 30-470x prior benchmarks), its deterministic six-task grading per instance, its adversarial reward-hacking audits during construction, and its controlled ablation in Table 5 are real strengths; the internal arithmetic (instance counts, LoC totals, cost, and percentage breakdowns in Table 4) checks out, and the paper honestly reports a negative result on anti-agent deterrents in Section B.8. The headline measurement, with the strongest frontier agent at 61.4% and 31.5% full solves, is a useful falsifiable data point. The significance, however, hinges on three unverified or under-documented pillars: the clean-room provenance, which is not independently auditable without artifacts; the data-use terms under which the private instances were exposed to provider APIs during evaluation; and the paper's self-audit claim that its own disclosures do not contaminate future runs.
major comments (4)
- [Section 3 (footnote 2), Section 4 (Models and Harness)] The paper never discloses the data-use terms under which the 262 private instances were sent to the five LLM providers' API endpoints, nor the terms for the Opus 4.8 adversarial audits and the Codex GPT-4 mini verification described in Section 3, footnote 1. If any provider's default policy permits training on API inputs, the act of evaluation itself releases the 'private' instances into training pipelines, which would destroy the benchmark's advertised contamination-freedom for every future evaluation. The current headline numbers are not invalidated, because the evaluated models' training predates the calls, but the paper's durable claims ('SRE-Bench therefore remains contamination-free in practice' in footnote 2; 'the first realistic, contamination-free RE benchmark' in the Abstract) are not established without documentation of explicit zero-retention or no-training agreements or an equivalent safeguard. This is a load-bearing gap for the central claim, and the authors should either disclose the provider agreements or revise the contamination-freedom claim to be scoped to the present evaluation with a stated mechanism for preserving future freshness.
- [Section 5 (Limitation: Benchmark Breadth), Abstract] The paper gives no access or release mechanism for SRE-Bench. The instances are described as private, instance-specific secrets are withheld in Section 3.1, and Section 5 discusses only construction cost as a constraint; yet the Abstract and Conclusion call SRE-Bench 'a rigorous testbed to measure progress,' which requires that third parties be able to run it. Without a stated access route (for example, controlled API access, staged release, or NDA-based distribution), the benchmark's utility and the community's ability to verify the reported measurements cannot be assessed. The paper should specify how and under what terms the benchmark will be made available, and how contamination-freedom is preserved under that access model.
- [Abstract, Section 4, Section 5 (Conclusion)] The abstract's claim that 'strong source-code security capabilities do not yet transfer to binary analysis' is not supported by measurements in this paper. The paper does not evaluate the five models (or any model) on source-code security tasks; Section 2 only cites other benchmarks showing strong source-code performance. The present data show that frontier models score modestly on SRE-Bench, but a model could be weak at both source-code and binary tasks, so the gap measured here does not by itself establish non-transfer. Either add a same-model source-code control or reframe the conclusion as 'RE remains largely unsolved for frontier models' without asserting the transfer claim.
- [Section 3, footnote 1] Footnote 1 asserts that all information disclosed in Section 3.1 and Appendix A is recoverable from the binaries by Codex GPT-4 mini within 200 LLM requests, with outcomes 'manually checked.' Because the paper itself will enter future training corpora, this self-audit is the only support for the claim that the paper's disclosures add no contamination; however, the protocol is too vague to audit, as it omits the prompts, the checking rubric, the failure cases, and any repeated-run details. Please report the actual audit protocol and its failures, or strengthen the redaction policy instead of relying on this claim; the current presentation does not let a reader verify the central contamination-freedom argument.
minor comments (7)
- [Section 3.3 (Instance Generation)] The instance-generation description does not state how many instances the firmware programs yield; the total of 262 can only be reconciled with the stated 16 non-firmware programs x (8 unprotected + 8 protected) = 256 by using Section 5's 'three programs yield only six instances.' Please state the firmware instance count explicitly in Section 3.3.
- [Table 4] The 'Graded' column excludes runs that produced no gradeable result, with refusals and context-window failures merged in the text but not separated. Because refusals may be systematic (for example, the cyber-security framing of the malware and firmware tasks), please report the per-cause counts so readers can assess bias in domain-level averages.
- [Section 3.1 (Malware, Firmware); Abstract; Section 4] There are several typos and sentence fragments: 'prevernt' in the Malware paragraph, 'A fresh device secret generated for each session' in the Firmware paragraph (missing verb), 'fully recovers only 31.5% the instances' in Section 1 (missing 'of'), and missing spaces in '1.9x the next model' and 'lost44of262runs' in Section 4.
- [Figure 3 caption] The caption states that the first three panels use 'the 128 unhardened instances' without noting that this is the subset of 16 non-firmware programs x 8 builds, excluding firmware; please make the subset explicit so the reader can reconcile it with Figure 2, which includes firmware.
- [Section 3.2, Appendix B] Several protection primitives rely on ptrace, /proc, watchdog processes, launch metadata, and page-fault handling, but the paper does not state that these primitives were validated inside the exact container image used for evaluation. If the evaluation containers impose seccomp or ptrace restrictions, some primitives may not have exercised the intended mechanisms; please confirm that the protected instances behave identically in the eval containers.
- [Section 3.2 (Novelty)] The claim that more than half of the 44 primitives have no public implementations is unverifiable as stated; listing which primitives are claimed novel would make the comparison checkable and strengthen the 'in-house' claim.
- [Figure 4, Section B.9] GLM-5.2 scores exactly 0.00 on all eight presets and Grok-4.5 is effectively at the floor; a sentence noting that the protection axis currently has little resolution for the weaker models would help readers interpret the per-preset ordering as a property of the strongest model only.
Circularity Check
No circular derivation: the paper's results are direct measurements on a deterministic grader, and the only self-reference is a minor, non-load-bearing citation plus an unverifiable but non-circular contamination-freedom premise.
full rationale
SRE-Bench's headline results (scores, solve rates, zero rates, cost, time) are measurements produced by a deterministic grader over 262 instances; there is no fitted parameter later renamed as a prediction, and no equation whose output is defined by its input. The ablations in Table 5 are empirical experiments: Minimal, Gzip-variant, and RevCompress are distinct programs with measured costs and scores, so the conclusion that scale and contamination-freedom are load-bearing is supported by observation rather than by construction. The contamination-freedom claim in §3 (footnote 2: 'SRE-Bench therefore remains contamination-free in practice') rests on the authors' own withheld secrets and unreleased artifacts, which is an auditability/external-validity limitation, not a circular reduction: the paper does not use that assertion as the evidence for its capability results. One minor self-citation appears when the paper cites Ding et al. (2024) for a general point about contamination risk in cybersecurity tasks; that citation is incidental and not load-bearing for the benchmark's design or conclusions. No uniqueness theorem is imported from the authors' prior work, and no ansatz is smuggled in via citation. A genuine caveat is that the evaluation protocol sends the private binaries to five LLM API providers, and the paper does not disclose whether no-training/zero-retention agreements were in place, which could affect the benchmark's future contamination-freedom; however, that is a data-governance risk rather than a circularity in the paper's derivation chain. Overall, the derivation chain is self-contained as a measurement study, so circularity is minimal.
Assumptions & free parameters
assumptions (4)
- ad hoc to paper The private, clean-room provenance of all 19 programs and the protection suite guarantees that instances are contamination-free.
- domain assumption Deterministic graders with no in-run feedback measure genuine reverse engineering capability rather than reward hacking.
- domain assumption Performance on toy or publicly derived programs does not predict performance on real-world-scale private binaries.
- domain assumption Optimization and static linking are the classical obstacles for human reverse engineers, making agent insensitivity to them a meaningful divergence.
Cite this review
Pith. "Pith review of The Next Challenge for Agentic Cybersecurity: A Realistic, Contamination-Free Reverse Engineering Benchmark." pith.science (2026). https://pith.science/paper/2JZWUMUO
@misc{pith2026260811469,
author = {Pith},
title = {Pith review of: The Next Challenge for Agentic Cybersecurity: A Realistic, Contamination-Free Reverse Engineering Benchmark},
year = {2026},
howpublished = {\url{https://pith.science/paper/2JZWUMUO}},
note = {Machine review of arXiv:2608.11469}
}
read the original abstract
AI agents are rapidly improving in cybersecurity capabilities when the source code is available for analysis, yet much of the software most consequential to cybersecurity, including malware, firmware, and proprietary applications, is available only as binaries. Analyzing such software requires reverse engineering(RE): recovering program semantics before the analysis can be meaningfully performed. However, evaluating agentic RE poses a fundamental challenge: benchmark instances must be unseen as source code in the LLMs' training data to prevent models from taking shortcuts by recognizing them rather than really analyzing them, while also matching the scale and anti-analysis protections of real software. Unfortunately, however, existing benchmarks do not jointly satisfy these requirements. To this end, we introduce SRE-Bench, the first realistic, contamination-free RE benchmark. Built entirely from scratch by RE experts with over 5,000 hours, SRE-Bench comprises 19 private, real-world-scale programs averaging 16.9K lines of code. We further developed 44 in-house anti-analysis primitives, yielding 262 binary instances and 1572 deterministically graded tasks. Our evaluation across five frontier LLMs (GPT-5.6-sol,Claude-Opus-5,GPT-5.5,Grok-4.5, and GLM-5.2) shows that RE remains largely unsolved: the strongest model, GPT-5.6-sol, scores 61.4% per instance, and fully solves only 31.5% of the instances. Our analysis further reveals that agents behave differently from human engineers, where agents are relatively insensitive to compiler optimization and static linking. Controlled ablations also confirm that both contamination control and realistic scale are essential. These results indicate that strong source-code security capabilities do not yet transfer to binary analysis, highlighting RE as an important frontier for agentic cybersecurity and SRE-Bench as a rigorous testbed to measure progress.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[4]
Vulnerability detection with code language models: How far are we?arXiv preprint arXiv:2403.18624,
Yangruibo Ding, Yanjun Fu, Omniyyah Ibrahim, Chawin Sitawarin, Xinyun Chen, Basel Alomair, David Wagner, Baishakhi Ray, and Yizheng Chen. Vulnerability detection with code language models: How far are we?arXiv preprint arXiv:2403.18624,
-
[7]
Livecodebench: Holistic and contamination free evalua- tion of large language models for code
Naman Jain, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar- Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evalua- tion of large language models for code. InInternational Conference on Learning Representations, volume 2025, pp. 58791–58831,
work page 2025
-
[8]
REStack: A Large-Scale Dataset of Reverse Engineering Discussions from Stack Exchange
Md Humaun Kabir, Md Rakibul Islam, and Farha Kamal. Restack: A large-scale dataset of reverse engineering discussions from stack exchange.arXiv preprint arXiv:2606.05493,
-
[9]
Nicolas Koller et al. Reforge: A method for benchmarking llms’ reverse engineering capabilities in decompiled binary function naming.arXiv preprint arXiv:2607.07738,
-
[10]
SEC-bench Pro: Can Language Models Solve Long-Horizon Software Security Tasks?
Hwiwon Lee, Jiawei Liu, Dongjun Kim, Ziqi Zhang, Chunqiu Steven Xia, and Lingming Zhang. Sec-bench pro: Can language models solve long-horizon software security tasks?arXiv preprint arXiv:2605.26548, 2026a. Hwiwon Lee, Ziqi Zhang, Hanxiao Lu, and Lingming Zhang. Sec-bench: Automated benchmarking of llm agents on real-world software security tasks.Advances...
-
[11]
Seunghyun Lee and David Brumley. Exploitbench: A capability ladder benchmark for llm cyberse- curity agents.arXiv preprint arXiv:2605.14153,
-
[12]
Yu Liu, Lang Gao, Mingxin Yang, Yu Xie, Ping Chen, Xiaojin Zhang, and Wei Chen. Vulde- tectbench: Evaluating the deep capability of vulnerability detection with large language models. arXiv preprint arXiv:2406.07595,
-
[13]
Juefei Pu, Xingyu Li, Zhengchuan Liang, Jonathan Cox, Yifan Wu, Kareem Shehada, Arrdya Sri- vastav, and Zhiyun Qian. Patch-to-poc: A systematic study of agentic llm systems for linux kernel n-day reproduction.arXiv preprint arXiv:2602.07287,
Show all 20 references
-
[16]
Exploitgym: Can ai agents turn security vulnerabilities into real attacks?arXiv preprint arXiv:2605.11086,
Zhun Wang, Nico Schiller, Hongwei Li, Srijiith Sesha Narayana, Milad Nasr, Nicholas Carlini, Xiangyu Qi, Eric Wallace, Elie Bursztein, Luca Invernizzi, et al. Exploitgym: Can ai agents turn security vulnerabilities into real attacks?arXiv preprint arXiv:2605.11086,
-
[17]
Rebench: A procedural, fair-by-construction benchmark for llms on stripped-binary types and names (extended version).arXiv preprint arXiv:2604.27319,
Jun Yeon Won, Xin Jin, Shiqing Ma, and Zhiqiang Lin. Rebench: A procedural, fair-by-construction benchmark for llms on stripped-binary types and names (extended version).arXiv preprint arXiv:2604.27319,
-
[18]
Cybench: A framework for evaluating cyber- security capabilities and risks of language models
Andy K Zhang, Neil Perry, Riya Dulepet, Joey Ji, Celeste Menders, Justin Lin, Eliot Jones, Gashon Hussein, Samantha Liu, Donovan Jasper, et al. Cybench: A framework for evaluating cyber- security capabilities and risks of language models. InInternational Conference on Learning...
2025
-
[19]
Cve-bench: a benchmark for ai agents’ ability to exploit real-world web application vulnerabilities.arXiv preprint arXiv:2503.17332,
Yuxuan Zhu, Antony Kellermann, Dylan Bowman, Philip Li, Akul Gupta, Adarsh Danda, Richard Fang, Conner Jensen, Eric Ihli, Jason Benn, et al. Cve-bench: a benchmark for ai agents’ ability to exploit real-world web application vulnerabilities.arXiv preprint arXiv:2503.17332,
-
[20]
Training language model agents to find vulnerabilities with ctf-dojo.arXiv preprint arXiv:2508.18370,
12 Terry Yue Zhuo, Dingmin Wang, Hantian Ding, Varun Kumar, and Zijian Wang. Training language model agents to find vulnerabilities with ctf-dojo.arXiv preprint arXiv:2508.18370,
-
[1994]
Crebench: Evaluating large language models in cryptographic binary reverse engineering.arXiv preprint arXiv:2604.03750,
Baicheng Chen, Yu Wang, Ziheng Zhou, Xiangru Liu, Juanru Li, Yilei Chen, and Tianxing He. Crebench: Evaluating large language models in cryptographic binary reverse engineering.arXiv preprint arXiv:2604.03750,
-
[2005]
The concept assignment problem in program understanding
Ted J Biggerstaff, Bharat G Mitbander, and Dallas Webster. The concept assignment problem in program understanding. In[1993] Proceedings Working Conference on Reverse Engineering, pp. 27–43. IEEE,
1993
-
[2008]
Benchmarking binary type inference techniques in decompilers
Vedant Soni, Audrey Dutcher, Tiffany Bao, and Ruoyu Wang. Benchmarking binary type inference techniques in decompilers. InProceedings of the 2025 Workshop on Software Understanding and Reverse Engineering, pp. 48–60,
2025
-
[2011]
De- compilebench: A comprehensive benchmark for evaluating decompilers in real-world scenarios
Zeyu Gao, Yuxin Cui, Hao Wang, Siliang Qin, Yuanda Wang, Zhang Bolun, and Chao Zhang. De- compilebench: A comprehensive benchmark for evaluating decompilers in real-world scenarios. InFindings of the Association for Computational Linguistics: ACL 2025, pp. 23250–23267,
2025
-
[2020]
Cy- bergym: Evaluating ai agents’ real-world cybersecurity capabilities at scale.arXiv preprint arXiv:2506.02548,
Zhun Wang, Tianneng Shi, Jingxuan He, Matthew Cai, Jialin Zhang, and Dawn Song. Cy- bergym: Evaluating ai agents’ real-world cybersecurity capabilities at scale.arXiv preprint arXiv:2506.02548,
-
[2025]
Look what you made us patch: 2025 zero-days in review.https://cloud.google
10 Google. Look what you made us patch: 2025 zero-days in review.https://cloud.google. com/blog/topics/threat-intelligence/2025-zero-day-review,
2025
-
[2026]
Crackmebench: Binary reverse engineering for agents.arXiv preprint arXiv:2605.10597,
Isaac David and Arthur Gervais. Crackmebench: Binary reverse engineering for agents.arXiv preprint arXiv:2605.10597,
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.