{"id":"c5756cb6-dd10-4691-88a0-8f6f5662a705","arxiv_id":"2506.14682","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A new 70-challenge benchmark measures whether language models can autonomously discover and exploit AI/ML security vulnerabilities, finding Claude-3.7-Sonnet solves 61% of challenges while the best open-source model solves 10%.","lead":"AIRTBench tests language models on 70 capture-the-flag challenges that require hacking AI systems, from prompt injection to model inversion. Frontier models solved up to 61% of challenges, while open-source models solved at most 10%, and agents finished in minutes tasks that take human researchers hours or days.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central leaderboard is at risk from training-data contamination: almost all 70 challenges are public Crucible challenges with no cutoff or contamination check, so the headline capability conclusions may be inflated.","rationale":"The reader's weakest_assumption correctly identifies contamination as the load-bearing risk, and I agree. I considered two other candidates: the opaque human-time baseline behind the 'over 5,000x' claim, and the strong correlation between XML tool-calling syntax errors and performance (Table 9), which confounds cross-model comparisons. Both are real but secondary: the human-efficiency claim is not needed for the benchmark's core capability rankings, and the harness confound affects relative rankings less than a complete memorization failure would. The paper has genuine positive evidence—open-sourced code and dataset, human-verified challenges, per-run metrics, and detailed attack traces for turtle—so the manuscript should not be rejected outright. However, because Section 3.1 concedes that almost all challenges are public and no contamination check is reported, the central leaderboard is not yet fully supported. The right disposition is the reader's CONDITIONAL: accept only after the contamination audit described above, plus the smaller fixes the reader lists. Therefore verdict_should_be is UNCHANGED relative to the reader.","tokens_in":39336,"tokens_out":6462,"duration_ms":72035,"concrete_test":"Run a contamination audit before treating any leaderboard number as a capability measurement: for each of the 70 challenges, give each evaluated model only the challenge prompt (as in Section 3.2) with no Crucible API access and no code-execution environment, and ask it to output the flag or first exploitation step. Then split the observed solve rates by each challenge's public-release date relative to each model's reported training cutoff. If any model emits a correct gAAAAA... flag from the prompt alone, or if solve rates are significantly higher on pre-cutoff public challenges than on matched post-cutoff/private challenges, contamination is confirmed and the headline numbers must be recomputed on a fresh private holdout set.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing assumption is that the 70 publicly hosted Crucible challenges have not leaked into the training data of the evaluated models. Section 3.1 states that 'almost all' of the challenges are 'publicly available', and each challenge is a black-box service with a fixed flag; the paper provides no contamination check, no release-date cutoff relative to model training windows, and no probe for whether models can reproduce flags or solution walkthroughs from memory. The central claim—that the leaderboard measures autonomous discovery and exploitation—requires this assumption. If even a subset of the high-value solutions (e.g., the easy prompt-injection family or the unique Claude solves like cubic, miner, and cluster3) were memorized from public writeups, the headline rates, category comparisons, and the 'clear leader' conclusion would be inflated independently of the benchmark's design quality. Because Section 3.1 itself flags the public availability and the paper ships the full dataset in Appendix J, this is a concrete, checkable vulnerability rather than a speculative one.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"AIRTBench is a benchmark for evaluating language models' ability to autonomously discover and exploit AI/ML security vulnerabilities. It consists of 70 black-box capture-the-flag challenges drawn from the Crucible environment on the Dreadnode platform, each with a verifiable flag. The authors build an agent harness that gives a model a Jupyter kernel and Crucible API tools, evaluate 12 models over 10 passes per challenge (with an acknowledged partial dataset for GPT-4.5), and report suite success rates, overall success rates, category and difficulty breakdowns, token/cost metrics, rate-limit statistics, and a comparison with human solve times. The headline claim is that Claude-3.7-Sonnet is the clear leader with 43 solved challenges (61.4% suite success, 46.9% overall), followed by Gemini-2.5-Pro (39 challenges) and GPT-4.5 (34 challenges), while open-source models lag far behind. The paper also reports that AI agents solve challenges orders of magnitude faster than human operators, with efficiency advantages over 5,000x on hard challenges.","tokens_in":39471,"tokens_out":5886,"duration_ms":62204,"significance":"If the leaderboard is uncontaminated and the human baseline is measured fairly, AIRTBench fills a real gap: it is the first dedicated benchmark for autonomous AI red teaming that uses realistic black-box challenges, verifies flags mechanically, ships open code and data, and provides a difficulty and category taxonomy. The 10-pass protocol, the per-challenge results in the appendices, and the detailed attack traces are genuine strengths, and the paper is careful to report both suite-level and run-level success rates. However, the central capability claims currently rest on unvalidated assumptions about training-data contamination and on an undocumented human-data methodology, and no statistical uncertainty is attached to the leaderboard. These issues are load-bearing for the paper's main conclusions, so the contribution is currently conditional rather than established.","major_comments":[{"comment":"The central claim that AIRTBench measures autonomous discovery and exploitation requires that the 70 challenges, which §3.1 states are 'almost all' publicly available, have not leaked into model training data. The paper provides no contamination check, no release-date cutoff relative to training cutoffs, and no probe for whether models can reproduce flags or solution walkthroughs from memory. Appendix J ships the full dataset, which makes this vulnerability concrete and checkable. If even a subset of the easy prompt-injection family or the unique Claude solves (cubic, miner, cluster3) were memorized from public writeups, the headline rates, the category comparisons, and the 'clear leader' conclusion would all be inflated. Please add a contamination analysis: per-challenge release dates versus training windows, memory-retrieval probes, and ideally a private held-out subset of challenges; until then, the capability conclusions should be explicitly caveated.","section":"§3.1, §5.1, Appendix J"},{"comment":"The 'over 5,000x' human-efficiency claim is not supported by a documented methodology. The human data appear to be aggregate platform statistics, but the paper does not state inclusion criteria, how 'solve time' is defined (active time versus wall-clock time, censoring, hints, multiple users), or how the means are computed. The small samples are especially fragile: turtle has 4 human solves, brig1 has 5, and brig2 has 2, so the reported means of 1,296.5, 2,439.9, and 2,099.3 hours are extremes of highly skewed distributions. A dedicated human-data methodology subsection and a sensitivity analysis (e.g., medians, trimmed means, per-solve distributions) are needed before the 5,000x efficiency claim can be evaluated.","section":"§5.2, Table 3"},{"comment":"None of the headline success rates is accompanied by a confidence interval or a significance test. The overall success rates are proportions over roughly 700 runs per model, but the suite success rates are binary outcomes over only 70 challenges; for example, Claude's 61.4% (43/70) versus Gemini-2.5-Pro's 55.7% (39/70) is a four-challenge difference that falls within ordinary binomial sampling noise. Reporting Wilson intervals and, where meaningful, paired per-challenge comparisons would let readers judge whether the 'clear leader' ranking and the category-level differences are real or within noise.","section":"§4.2, Table 2"},{"comment":"Rate limiting is presented as an intentional environmental constraint, but the between-model asymmetry is large: Gemini-2.5 Pro and Gemini-2.5 Flash saw rate-limit events in 38.1% and 36.7% of attempts, respectively, while Llama models saw under 5%. Since rate limits change the number and timing of actions a model can take, the reported success rates compare models under unequal effective conditions. The paper should either control for rate-limit exposure, model it as a covariate, or report a no-rate-limit subset for a sensitivity check before interpreting the model rankings as pure capability differences.","section":"§4.2, §6.5, Tables 2 and 7"},{"comment":"GPT-4.5-Preview has only a mean of 5.2 runs per challenge (minimum 3, maximum 7), yet it appears in every leaderboard and category table alongside models with 10 runs. The incomplete data are acknowledged in a footnote, but the headline tables do not propagate this uncertainty. Because the suite success rate counts a challenge as solved if any run succeeds, a model with fewer attempts is at a disadvantage; this is a particular concern for a model that is ranked third overall. Please add a sensitivity analysis restricted to the first five runs of all models, or clearly mark GPT-4.5 rows in every table with the reduced denominator.","section":"§4.2 footnote 1, Tables 2-5"}],"minor_comments":[{"comment":"The text says Llama-4-17B solved '11.4% of challenges overall', but Table 2 reports 7 out of 70, which is 10.0%; please correct this inconsistency.","section":"§6.6"},{"comment":"The command in Listing 1 is rendered with spaces inside tokens (for example, 'python -m ai rt ben ch' and '--model'), making it unusable as printed; please fix the formatting.","section":"Listing 1"},{"comment":"The heading 'Code Generation BenchmarksAre as attractive' is missing spacing and capitalization; it should read 'Code Generation Benchmarks Are as Attractive'.","section":"§2"},{"comment":"The multi-row header of Table 3 is difficult to map to the data columns; splitting the human metrics and agent solve-time metrics into separate tables, or using clearer column headers, would improve readability.","section":"Table 3"},{"comment":"Section 5.2 describes cubic and miner as Claude-only solves, while Section 6.1 also lists cluster3 as a Claude-only solve; the lists should be made consistent or explicitly non-exhaustive.","section":"§5.2, §6.1"},{"comment":"Section 3.1 says 'almost all' challenges are publicly available, while Section 3.3 says the benchmark avoids 'scraped challenges'; please clarify which challenges, if any, are private and how that affected the reported results.","section":"§3.1, §3.3"}],"recommendation":"major_revision","confidential_remarks":"The benchmark and the human baseline data both come from the authors' own commercial platform (Dreadnode/Crucible). This is not a logical circularity, but it does create a selection-incentive concern that would be mitigated by third-party replication or by a clearly independent evaluation protocol. Given that the manuscript ships code and data, the requested contamination analysis and statistical uncertainty quantification are feasible in revision and would materially raise confidence in the leaderboard."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: this is the first benchmark I've seen that tries to measure autonomous AI red-teaming ability across a broad challenge taxonomy, and it's open-sourced, so it's worth engaging with. The stress-test note hits the real weak spot: almost all 70 challenges are public Crucible challenges, and the paper provides no contamination check, no release-date cutoff, and no probe for memorized flags. That matters for every headline number in the abstract, because the models have almost certainly crawled the internet, and the challenge flags are fixed.\n\nWhat the paper does well: the benchmark design is more credible than most. Black-box challenges with mechanistic flag verification, 10 passes per model (modest, but better than nothing), a taxonomy mapped to MITRE ATLAS and OWASP, and they openly disclose the GPT-4.5 deprecation and the two excluded challenges. The per-challenge results table in Appendix C is genuinely useful, and the turtle attack traces in Appendix H give real insight into how different models approach the same hard problem. They also ship the harness and dataset, which is exactly what makes the contamination concern checkable.\n\nThe soft spots are real but fixable. The contamination issue is the one I'd want resolved before citing the leaderboard. Public challenges, some around since October 2024, with no temporal cutoff relative to training windows: the absolute solve rates and the 'clear leader' conclusion could be inflated. This doesn't sink the benchmark, but it does make the numbers upper bounds. The 5,000x human-efficiency claim is shakier: Table 3's human times are unexplained averages from platform statistics, with no stated methodology—median vs mean, which users, how many observations, what counts as 'solve time.' For hard challenges where an agent solved in 0.3 minutes, the comparison is a fun headline but not a rigorous result. Also, the XML tool format is a confound: Gemini and Qwen had syntax-error rates above 30% and 99% respectively, so part of the ranking reflects structured-output reliability rather than red-teaming capability. The paper acknowledges the harness is a baseline, but it still shapes the leaderboard.\n\nThe reader's CONDITIONAL verdict is about right. The core artifact is likely sound, and the defects are the kind that peer review can fix. I'd send this to a serious referee, not desk reject it. If the authors add a contamination analysis and document the human baseline properly, this becomes a solid contribution worth citing. As it stands, use it for the dataset and the qualitative insights, but treat the efficiency claims as illustrative.","headline":"A genuinely useful first-of-its-kind benchmark for autonomous AI red teaming, but the contamination risk and the under-documented human baseline mean the headline numbers should be treated as provisional, not facts.","tokens_in":772,"tokens_out":815,"would_cite":true,"duration_ms":34936,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A new 70-challenge benchmark measures when language models can autonomously hack AI systems.","keywords":["AIRTBench","AI red teaming","language model agents","prompt injection","capture-the-flag benchmark","AI/ML security","model inversion","autonomous exploitation"],"falsifier":"Run a contamination probe: give each model the challenge name and ask it to reproduce the flag or solution code without any API access; if a meaningful share of flags can be recalled, the reported solve rates are partly memorization. A cleaner test is to measure models on a fresh set of challenges released after their training cutoffs and compare success rates.","tokens_in":39104,"feed_emoji":"🛡️","tokens_out":7081,"duration_ms":67787,"temperature":0.7,"pith_summary":"The paper introduces AIRTBench, a benchmark built from 70 realistic black-box capture-the-flag challenges hosted on the Crucible platform, and uses it to measure how well twelve large language models can autonomously discover and exploit AI/ML security vulnerabilities. Its central finding is that frontier models are already effective at prompt injection, averaging 49 percent success, while model inversion and system exploitation remain largely unsolved, with most models near zero. The reported leader is Claude-3.7-Sonnet, which solves 43 of 70 challenges (61.4 percent suite success, 46.9 percent per-run success), followed by Gemini-2.5-Pro with 39 challenges and GPT-4.5 with 34. The authors also report that agents solve hard challenges in minutes where human operators take hours or days, an efficiency advantage they estimate at more than 5,000 times on the hardest tasks. If the benchmark measures what it claims, it provides a needed baseline for tracking a capability that will matter as models are deployed against other models.","feed_headline":"70 challenges reveal how well LLMs can hack AI systems","feed_subtitle":"Claude 3.7 Sonnet tops a new 70-challenge AI red-teaming suite at 61 percent; open models trail far behind.","key_machinery":"The load-bearing object is the agent harness: an internet-connected Jupyter kernel in a container, a Crucible API tool-call interface for querying challenges and submitting flags, and the 70 black-box challenges themselves, each an isolated FastAPI service whose flag is verified cryptographically. The harness matters because success is the joint product of model reasoning and tool use; the paper's metrics, including suite success, per-run success, conversation length, token use, and rate-limit adaptation, all depend on this environment. The benchmark's discriminating power comes from challenges designed for human CTF players, with deliberate ambiguity and multi-step attack chains, so a flag cannot be obtained by pattern matching alone.","core_discovery":"The paper's claim is that autonomous AI red teaming can be measured as a concrete, verifiable capability: given a natural-language challenge description, starter code, a Jupyter kernel, and a tool for submitting flags, a language model can independently find and verify exploitable AI/ML weaknesses. In the authors' evaluation, the best model solved 43 of 70 challenges, and performance fell sharply along two axes: challenge difficulty (31.7 percent on easy, 10.7 percent on medium, 1.8 percent on hard) and attack category (prompt injection near 49 percent average success, while model inversion and system exploitation stayed at or below roughly 26 to 32 percent even for the leader). The paper also contends that models reach these results in minutes rather than the hours or days humans need, and that open-source models lag far behind, with the best open model solving 7 challenges at a 1 percent overall run success rate. A handful of hard challenges were solved by only one or two models, and one hard challenge was solved by an open model in under a minute using a clever reframing, leading the authors to conclude that specialized security skills can exist even in models with weak aggregate scores.","pith_inferences":["If challenge content leaks into future training data, these numbers will rise without any real capability gain; a rotating held-out set would keep the benchmark honest.","The per-model success variance on individual hard challenges, where very different models each found a distinct exploit, suggests that an ensemble agent mixing models could outperform any single model.","The observed spurious-elaboration failures, models submitting long narrative flags when stuck, suggest that adding a small verification or stopping module could reap outsized gains in real deployments, independent of the core model.","The heavy rate-limit load that some high-scoring models absorbed indicates that today's leaderboards may be partly measuring API-harness robustness rather than raw offensive reasoning."],"forward_implications":["If the benchmark tracks real capability, security teams can expect autonomous agents to automate the easiest and most common attacks, prompt injection and RAG-based injection, at near-human success rates.","The roughly 5,000x speed advantage on hard challenges means red-teaming workloads that took days can be run in minutes, changing how many attacks can be attempted per engagement.","The large frontier-versus-open gap in suite success rates implies that the most capable automated AI offense will be concentrated among the few providers of frontier models.","Because only the flagship models can solve hard multi-step exploit chains, progress on autonomous red teaming is currently gated by general reasoning ability, not by security-specific training.","The benchmark's difficulty tiers separate models cleanly, so it can serve as a progress metric: the next generation should move the hard-challenge solve rate above 1.8 percent."],"supporting_citations":[{"why":"Supplies the attack-taxonomy categories used to label every challenge.","marker":"[7]"},{"why":"Supplies the LLM-application vulnerability categories used for labelling.","marker":"[8]"},{"why":"Grounds the argument that realistic, verifiable benchmarks track capability better than static tests.","marker":"[15]"},{"why":"Provides an earlier interactive-coding CTF environment that AIRTBench extends to AI-specific challenges.","marker":"[23]"},{"why":"Provides the earlier CTF benchmark for offensive security that motivates this agentic evaluation.","marker":"[24]"},{"why":"Establishes a prior benchmark targeting autonomous adversarial-example defense bypass that AIRTBench complements.","marker":"[26]"},{"why":"Supplies the prior finding on automation advantage in red teaming that the human-versus-agent timing comparison extends.","marker":"[31]"}],"fun_headline_variants":["LLMs hack AI systems in minutes, but open models lag far behind","Autonomous AI red teaming benchmark: Claude tops 70 CTF challenges","Benchmark measures how well LLMs exploit AI security flaws","Frontier models excel at prompt injection, fail hard on model inversion","Open-source AI models solve just 10% of red teaming challenges"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The scores assume that the publicly available Crucible challenges have not appeared in the models' training data; the paper reports no contamination check, so memorized flags or solutions would inflate every leaderboard number.","fun_headline_variants_meta":{"raw":{"variants":["LLMs hack AI systems in minutes, but open models lag far behind","Autonomous AI red teaming benchmark: Claude tops 70 CTF challenges","Benchmark measures how well LLMs exploit AI security flaws","Frontier models excel at prompt injection, fail hard on model inversion","Open-source AI models solve just 10% of red teaming challenges"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000229,"raw_usage":{"total_tokens":1561,"prompt_tokens":1110,"completion_tokens":451,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":726,"completion_tokens_details":{"reasoning_tokens":358}},"tokens_in":726,"tokens_out":451,"duration_ms":4514,"temperature":1.0,"reasoning_tokens":358,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:48:41.193631+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a contamination probe: give each model the challenge name and ask it to reproduce the flag or solution code without any API access; if a meaningful share of flags can be recalled, the reported solve rates are partly memorization. A cleaner test is to measure models on a fresh set of challenges released after their training cutoffs and compare success rates.","supporting_citations":[{"cited_title":"ATLAS - Adversarial Threat Landscape for Artificial-Intelligence Systems","cited_arxiv_id":null,"evidence_quote":"Supplies the attack-taxonomy categories used to label every challenge."},{"cited_title":"OWASP Top Ten for Large Language Model Applications","cited_arxiv_id":null,"evidence_quote":"Supplies the LLM-application vulnerability categories used for labelling."}],"review_version":2}