{"id":"95f3d811-9054-4983-9251-9efd666c3324","arxiv_id":"2608.11469","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"SRE-Bench, a contamination-controlled reverse-engineering benchmark with 262 realistic binary instances, shows the strongest tested AI agent fully solves only 31.5% of instances.","lead":"The paper introduces SRE-Bench, a private reverse-engineering benchmark built from scratch with 19 real-world-scale programs and 262 protected binary instances, so AI models cannot solve tasks by recognizing public code. It reports that the best tested model fully solves only 31.5% of instances, suggesting current AI security agents cannot yet analyze binaries as well as source code.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Sending the private binaries to LLM APIs may have contaminated SRE-Bench: the paper never discloses whether the five providers were barred from training on the instances.","rationale":"The paper is internally consistent and the arithmetic checks out. The central claim is that SRE-Bench is the first benchmark that is simultaneously realistic and contamination-free, and that frontier agents fail it. I agree with the reader that the contamination-free premise is the load-bearing assumption. But I think the more specific and actionable vulnerability is not merely the absence of an external audit, but the evaluation pipeline itself: the 262 private binaries were sent, in full, to the API endpoints of five commercial model providers. Unless the authors used zero-retention/no-training agreements, those providers now possess the binaries and may include them in future training sets. This would make the benchmark contaminated for exactly the later evaluations it is meant to support. The paper is silent on this point; it only says artifacts are 'never released' (publicly) and 'remain contamination-free in practice.' Sending data to an API is a release to the provider, and the paper gives no evidence that the providers committed to not training on it. The reported results for the five specific model versions are not affected, because those models' training predates the API calls, so the headline gap stands for those versions. However, the benchmark's advertised durable property—contamination-freedom—is compromised by the act of using it, unless explicit safeguards were in place. The fix is simple and verifiable: disclose the data-processing agreements. I therefore keep the reader's CONDITIONAL verdict, adding the API-exposure condition to the existing reproducibility condition. If the authors can document no-training agreements, the concern is resolved; if not, the benchmark's central claim should be narrowed to 'contamination-free for the five evaluated model versions at evaluation time' and not as a durable benchmark property.","tokens_in":19280,"tokens_out":9991,"duration_ms":91375,"concrete_test":"Request from the authors the data-processing agreements governing each of the five model evaluations (GPT-5.6-sol, Claude-Opus-5, GPT-5.5, Grok-4.5, GLM-5.2), specifically whether zero-retention / no-training options were enabled when the binary instances were submitted to the providers' APIs. If any provider's terms allow training on API inputs, the contamination-free property fails for future evaluations. If all five are documented as no-training, the concern is fully resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that SRE-Bench is 'contamination-free' is undercut by the evaluation protocol itself. All 262 private binaries were shipped, inside isolated containers, to the API endpoints of five LLM providers (GPT-5.6-sol, Claude-Opus-5, GPT-5.5, Grok-4.5, GLM-5.2) for scoring. The paper nowhere discloses whether those API calls were made under zero-retention / no-training agreements, or whether the providers' default data-use policies apply. If any provider stores or trains on API inputs, then the 'private' instances have been released to third-party training pipelines, and the benchmark is no longer contamination-free for any future evaluation after the first run. The paper's assertion in §3 that 'SRE-Bench therefore remains contamination-free in practice' is thus not established: the instances may have ceased to be private at the moment they were sent to the model providers. The current results for the five specific model versions are unaffected (their training predates the API calls), but the benchmark's advertised central property—contamination-freedom as a durable design attribute—is compromised exactly when the benchmark is used, i.e., by the act of evaluation. This is not merely an auditing problem but a positive mechanism by which the claimed property is violated unless explicit safeguards are in place and documented.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents SRE-Bench, an agentic binary reverse-engineering (RE) benchmark that combines two design properties the authors argue are jointly missing from prior work: contamination-freedom (target programs not present in LLM pretraining corpora) and real-world scale. The benchmark consists of 19 from-scratch programs averaging 16,915.8 LoC across the domains of network protocols, games, file formats, malware, and firmware; an in-house 27K-LoC anti-analysis suite with 44 protection primitives; and 262 binary instances (16 non-firmware programs x 16 builds plus 6 firmware variants), each with six deterministically graded tasks, for 1,572 tasks total. The evaluation runs five frontier LLM agents (GPT-5.6-sol, Claude-Opus-5, GPT-5.5, Grok-4.5, GLM-5.2) in a uniform bash-only harness at a reported cost of $31.4K; the strongest model scores 3.69/6 (61.4%) and fully solves 80 of 254 gradeable instances (31.5%). A controlled three-program ablation (Table 5) shows that a public gzip-based target and a 1.1K-LoC clean-room target are both solved cheaply and quickly, whereas the full private target separates models, supporting the claim that both contamination control and scale are load-bearing. The paper concludes that strong source-code security capability does not transfer to binary analysis and positions SRE-Bench as a durable testbed.","tokens_in":19514,"tokens_out":21036,"duration_ms":158271,"significance":"If the benchmark is as described, it addresses a genuine gap: existing agentic RE benchmarks are either CTF-derived with contamination risk and entry-level difficulty, or contamination-controlled but toy-scale (36-64 LoC on average). SRE-Bench's scale (16.9K LoC on average, 30-470x prior benchmarks), its deterministic six-task grading per instance, its adversarial reward-hacking audits during construction, and its controlled ablation in Table 5 are real strengths; the internal arithmetic (instance counts, LoC totals, cost, and percentage breakdowns in Table 4) checks out, and the paper honestly reports a negative result on anti-agent deterrents in Section B.8. The headline measurement, with the strongest frontier agent at 61.4% and 31.5% full solves, is a useful falsifiable data point. The significance, however, hinges on three unverified or under-documented pillars: the clean-room provenance, which is not independently auditable without artifacts; the data-use terms under which the private instances were exposed to provider APIs during evaluation; and the paper's self-audit claim that its own disclosures do not contaminate future runs.","major_comments":[{"comment":"The paper never discloses the data-use terms under which the 262 private instances were sent to the five LLM providers' API endpoints, nor the terms for the Opus 4.8 adversarial audits and the Codex GPT-4 mini verification described in Section 3, footnote 1. If any provider's default policy permits training on API inputs, the act of evaluation itself releases the 'private' instances into training pipelines, which would destroy the benchmark's advertised contamination-freedom for every future evaluation. The current headline numbers are not invalidated, because the evaluated models' training predates the calls, but the paper's durable claims ('SRE-Bench therefore remains contamination-free in practice' in footnote 2; 'the first realistic, contamination-free RE benchmark' in the Abstract) are not established without documentation of explicit zero-retention or no-training agreements or an equivalent safeguard. This is a load-bearing gap for the central claim, and the authors should either disclose the provider agreements or revise the contamination-freedom claim to be scoped to the present evaluation with a stated mechanism for preserving future freshness.","section":"Section 3 (footnote 2), Section 4 (Models and Harness)"},{"comment":"The paper gives no access or release mechanism for SRE-Bench. The instances are described as private, instance-specific secrets are withheld in Section 3.1, and Section 5 discusses only construction cost as a constraint; yet the Abstract and Conclusion call SRE-Bench 'a rigorous testbed to measure progress,' which requires that third parties be able to run it. Without a stated access route (for example, controlled API access, staged release, or NDA-based distribution), the benchmark's utility and the community's ability to verify the reported measurements cannot be assessed. The paper should specify how and under what terms the benchmark will be made available, and how contamination-freedom is preserved under that access model.","section":"Section 5 (Limitation: Benchmark Breadth), Abstract"},{"comment":"The abstract's claim that 'strong source-code security capabilities do not yet transfer to binary analysis' is not supported by measurements in this paper. The paper does not evaluate the five models (or any model) on source-code security tasks; Section 2 only cites other benchmarks showing strong source-code performance. The present data show that frontier models score modestly on SRE-Bench, but a model could be weak at both source-code and binary tasks, so the gap measured here does not by itself establish non-transfer. Either add a same-model source-code control or reframe the conclusion as 'RE remains largely unsolved for frontier models' without asserting the transfer claim.","section":"Abstract, Section 4, Section 5 (Conclusion)"},{"comment":"Footnote 1 asserts that all information disclosed in Section 3.1 and Appendix A is recoverable from the binaries by Codex GPT-4 mini within 200 LLM requests, with outcomes 'manually checked.' Because the paper itself will enter future training corpora, this self-audit is the only support for the claim that the paper's disclosures add no contamination; however, the protocol is too vague to audit, as it omits the prompts, the checking rubric, the failure cases, and any repeated-run details. Please report the actual audit protocol and its failures, or strengthen the redaction policy instead of relying on this claim; the current presentation does not let a reader verify the central contamination-freedom argument.","section":"Section 3, footnote 1"}],"minor_comments":[{"comment":"The instance-generation description does not state how many instances the firmware programs yield; the total of 262 can only be reconciled with the stated 16 non-firmware programs x (8 unprotected + 8 protected) = 256 by using Section 5's 'three programs yield only six instances.' Please state the firmware instance count explicitly in Section 3.3.","section":"Section 3.3 (Instance Generation)"},{"comment":"The 'Graded' column excludes runs that produced no gradeable result, with refusals and context-window failures merged in the text but not separated. Because refusals may be systematic (for example, the cyber-security framing of the malware and firmware tasks), please report the per-cause counts so readers can assess bias in domain-level averages.","section":"Table 4"},{"comment":"There are several typos and sentence fragments: 'prevernt' in the Malware paragraph, 'A fresh device secret generated for each session' in the Firmware paragraph (missing verb), 'fully recovers only 31.5% the instances' in Section 1 (missing 'of'), and missing spaces in '1.9x the next model' and 'lost44of262runs' in Section 4.","section":"Section 3.1 (Malware, Firmware); Abstract; Section 4"},{"comment":"The caption states that the first three panels use 'the 128 unhardened instances' without noting that this is the subset of 16 non-firmware programs x 8 builds, excluding firmware; please make the subset explicit so the reader can reconcile it with Figure 2, which includes firmware.","section":"Figure 3 caption"},{"comment":"Several protection primitives rely on ptrace, /proc, watchdog processes, launch metadata, and page-fault handling, but the paper does not state that these primitives were validated inside the exact container image used for evaluation. If the evaluation containers impose seccomp or ptrace restrictions, some primitives may not have exercised the intended mechanisms; please confirm that the protected instances behave identically in the eval containers.","section":"Section 3.2, Appendix B"},{"comment":"The claim that more than half of the 44 primitives have no public implementations is unverifiable as stated; listing which primitives are claimed novel would make the comparison checkable and strengthen the 'in-house' claim.","section":"Section 3.2 (Novelty)"},{"comment":"GLM-5.2 scores exactly 0.00 on all eight presets and Grok-4.5 is effectively at the floor; a sentence noting that the protection axis currently has little resolution for the weaker models would help readers interpret the per-preset ordering as a property of the strongest model only.","section":"Figure 4, Section B.9"}],"recommendation":"major_revision","confidential_remarks":"This is a serious benchmark contribution and the internal numerics are consistent, but the paper's central selling point, contamination-freedom, rests on two things the manuscript does not document: the provider data-use terms for the API evaluation and a community access path for the benchmark. I would ask the authors to provide both, and to make the footnote-1 self-audit reproducible. Given that the benchmark artifacts are withheld by design, you may also want to ask for a confidential artifact appendix for reviewers so that the instance count, grading determinism, and clean-room claims can be at least spot-checked. The 'transfer' claim in the abstract and conclusion should be softened regardless of the other fixes."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the useful news: this is a substantial construction effort. Nineteen clean-room programs at 16.9K LoC each, a 27K-LoC protection suite with 44 primitives, deterministic grading, and adversarial reward-hacking audits — that is real work, and it shows in the evaluation. The main empirical finding, that frontier LLMs solve only 31.5% of instances at best and that protection cuts GPT-5.6-sol in half, is plausible and important. The ablations in Table 5 are a good idea: they demonstrate that scale and provenance each matter, and the cost/time numbers are striking. The paper is also honest about its limits, including the alpha-stage protector and the small program pool.\n\nThe soft spots are in proportion to the claims. The most serious is the contamination-freedom guarantee. Section 3.1 and footnote 2 state that SRE-Bench \"remains contamination-free in practice,\" but the paper never discloses whether the five LLM providers were used under zero-retention or no-training agreements. If any provider trains on API inputs, the private binaries ship into training corpora the moment they are evaluated. That does not spoil the reported results for those model versions, which were trained before the API calls, but it breaks the benchmark's durable, load-bearing property. This is not just an audit problem; it is a mechanism by which the claimed property is violated unless safeguards are documented. The fix is simple: state the data policies, or run evaluations on self-hosted models.\n\nThe second weakness is the human-divergence claim. The paper says agents are \"relatively insensitive to compiler optimization and static linking\" compared to human reverse engineers, but I see no human baseline anywhere. Without a controlled study or at least a citation to one, this is an interpretation, not a measured result. It should be framed as a hypothesis, or backed by human data.\n\nThe third is minor: the ablation proving both requirements are essential uses one file-format domain and two models. That is enough to motivate the design, but \"essential\" is stronger than the evidence warrants.\n\nOverall, this deserves serious peer review. The benchmark is a contribution that others will build on, but the authors must disclose API data-use policies, soften or support the human-divergence claim, and ideally release artifacts to a trusted third party for contamination audit. I'd accept with those revisions.","headline":"A serious benchmark with a real contamination blind spot: the paper never discloses whether the LLM API providers could train on the private binaries, which would break the durable property that is its main selling point.","tokens_in":20035,"tokens_out":3847,"would_cite":true,"duration_ms":34308,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SRE-Bench, a reverse-engineering benchmark built entirely from scratch, shows today's best AI agents can fully reverse engineer only about a third of real-world-scale binaries.","keywords":["reverse engineering benchmark","agentic cybersecurity","LLM agents","contamination control","binary analysis","anti-analysis obfuscation","clean-room benchmark","program comprehension"],"falsifier":"Ask a frontier model for the withheld instance-specific secrets (archive passwords, trigger constants, firmware fuse words) without giving it the corresponding binary; if any model can name them, or if a training-data membership probe recognizes any SRE-Bench program, the contamination-free premise fails.","tokens_in":19082,"feed_emoji":"🕵️","tokens_out":11600,"duration_ms":97087,"temperature":0.7,"pith_summary":"SRE-Bench is a reverse-engineering benchmark built so its 262 binary instances cannot be solved by recognizing programs the models already know: all target programs, the protection suite, and the grading infrastructure were written from scratch and kept private, and every instance is scored by a deterministic grader. The paper's central claim is that these scores measure genuine RE rather than recognition, and that on this realistic testbed agentic reverse engineering is largely unsolved. The strongest evaluated model averages 61.4% of the available score and fully recovers only 31.5% of instances; protection layers halve its score, and weaker models collapse. That result is the evidence for the paper's conclusion that strong source-code security capabilities do not yet transfer to binary analysis, making RE a separate frontier worth measuring on its own terms.","feed_headline":"AI agents solve only a third of real reverse-engineering tasks","feed_subtitle":"A from-scratch, sealed benchmark shows binary analysis lags far behind source-code security skills.","key_machinery":"The load-bearing mechanism is a clean-room evaluation pipeline: 19 target programs written from scratch against private specifications, a 27,000-line in-house protection suite with 44 primitives (per-page authenticated encryption, lazy decryption, measurement-keyed anti-debugging, loader-logic virtualization, and anti-dump deception among them), and deterministic graders that are themselves iteratively hardened by adversarial reward-hacking audits. The pipeline turns each program into multiple binary instances and gives every instance exactly six verifiable tasks, so the benchmark can be scored without human judgment and without giving the agent any feedback during a run. Its governing design rule, \"fold, don't branch,\" feeds tampering, debugging, and observation measurements into page-key derivation rather than into ordinary condition checks, so an agent that attempts to observe or modify the binary gets silent key corruption instead of a check it can patch away.","core_discovery":"SRE-Bench is presented as the first reverse-engineering benchmark that is both realistic and contamination-free: 19 programs written from scratch by reverse-engineering experts, averaging 16,915.8 lines of code, compiled into 262 binary instances and hardened with an in-house suite of 44 anti-analysis primitives, with every instance scored by a deterministic grader across six tasks, for 1,572 tasks total. The paper reports that on this benchmark agentic reverse engineering is largely unsolved: the strongest evaluated model averages 61.4% of the available score and fully recovers 31.5% of instances, while the weakest model never fully recovers one. The in-house protections are the sharpest obstacle, halving the strongest model's score and driving all others to near zero, while compiler optimization and static linking — traditionally hard for humans — cost agents almost nothing and stripped symbols cost the most. Controlled ablations show the two design requirements are load-bearing: a publicly derived target and a small clean-room target are each solved in minutes for a few dollars, while a target that is both private and at real-world scale cleanly separates the models.","pith_inferences":["A direct test of the paper's interpretation would be to strip or rename all remaining lexical anchors in a hardened binary and measure score loss; if the model's advantage depends on names and strings rather than instruction-level reasoning, removing them should reproduce the protection-induced collapse.","Because a large share of exploited vulnerabilities come from vendors that do not release source code, a real RE gap would directly cap autonomous defense and offense on much real-world security work; this makes RE capability a bottleneck worth measuring even if the benchmark's exact numbers change.","The secret-withholding design could double as a contamination audit: future releases could plant canary secrets and ask whether any model can name them without seeing the binary, turning the unpublishable privacy assumption into a testable one.","The 19-program pool is small, especially for firmware; a natural extension is to run the same clean-room pipeline over more domains and languages to check whether the reported difficulty ordering (malware hardest, network protocol easiest) is stable or an artifact of program selection."],"forward_implications":["Source-code security scores should not be read as binary reverse-engineering capability; the gap measured here is direct evidence that the two need separate evaluation.","Future RE benchmarks should require both clean-room provenance and real-world scale, because the ablation shows relaxing either one collapses difficulty.","Protection and obfuscation, not compiler optimization, are the dominant obstacle for current agents, so progress in defeating layered protections should move overall scores more than better handling of optimization or linking.","The benchmark's deterministic six-task grading makes agent progress comparable across models and over time, including by domain and by build factor."],"supporting_citations":[{"why":"supplies the prior CTF-derived benchmark whose contamination risk motivates the design.","marker":"Shao et al., 2024"},{"why":"supplies the CTF-derived baseline whose toy difficulty motivates real-world scale.","marker":"Zhang et al., 2025"},{"why":"supplies the contamination-controlled but toy-scale baseline SRE-Bench must beat.","marker":"AgentRE-Bench, 2026"},{"why":"supplies the tiny-program baseline showing textbook-level protections are insufficient.","marker":"David & Gervais, 2026"},{"why":"supplies the crypto-oriented RE baseline with moderate contamination risk and small programs.","marker":"Chen et al., 2026"},{"why":"documents memorisation in code models, grounding the need for contamination control.","marker":"Al-Kaswan et al., 2024"},{"why":"grounds contamination-free evaluation design for code benchmarks.","marker":"Jain et al., 2025"},{"why":"supports the claim that RE is a distinct capability from downstream security tasks.","marker":"Mantovani et al., 2022"},{"why":"is the survey whose domain categories structure the five benchmark domains.","marker":"Kabir et al., 2026"}],"fun_headline_variants":["AI agents solve only 31% of real reverse-engineering instances","New sealed benchmark: top AI model fully solves 31% of RE tasks","Reverse engineering remains unsolved for frontier AI agents","From-scratch binary benchmark: AI scores 61%, solves 31%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's validity rests on every artifact — all 19 programs, the 27,000-line protection suite, constants, and reference answers — being genuinely new and private, never present in any model's training data, and this privacy cannot be independently checked because the artifacts are not released.","fun_headline_variants_meta":{"raw":{"variants":["AI agents solve only 31% of real reverse-engineering instances","New sealed benchmark: top AI model fully solves 31% of RE tasks","Reverse engineering remains unsolved for frontier AI agents","From-scratch binary benchmark: AI scores 61%, solves 31%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000634,"raw_usage":{"total_tokens":3016,"prompt_tokens":1128,"completion_tokens":1888,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":744,"completion_tokens_details":{"reasoning_tokens":1812}},"tokens_in":744,"tokens_out":1888,"duration_ms":13261,"temperature":1.0,"reasoning_tokens":1812,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:12:08.016878+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Ask a frontier model for the withheld instance-specific secrets (archive passwords, trigger constants, firmware fuse words) without giving it the corresponding binary; if any model can name them, or if a training-data membership probe recognizes any SRE-Bench program, the contamination-free premise fails.","supporting_citations":[{"cited_title":"Livecodebench: Holistic and contamination free evalua- tion of large language models for code","cited_arxiv_id":null,"evidence_quote":"grounds contamination-free evaluation design for code benchmarks."}],"review_version":1}