{"id":"f9659f63-ab58-44a1-82ab-1f1ea0a55af3","arxiv_id":"2602.24009","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Jailbreak Foundry translates 30 published jailbreak attacks into runnable modules with a mean reported-vs-reproduced ASR deviation of +0.26 percentage points, while standardizing evaluation across 10 victim models.","lead":"A research team built an automated pipeline that reads jailbreak papers and turns them into runnable attack modules, then runs all of them against ten language models using one judge. It reports matching original papers' attack success rates closely on average, but the number hides large individual gaps and some tuning toward reported values.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline +0.26 pp fidelity is an aggregate of repo-assisted and paper-only reproductions; the 8 paper-only attacks average about -4.5 pp, so the paper-to-runnable fidelity claim is not established.","rationale":"The reader's weakest assumption points to judge/protocol substitution, which is a real threat to the fidelity comparison. I identify a different, more easily falsifiable confound: the split between repo-assisted and paper-only rows in Table 1. Appendix B.2 directs the auditor to treat the cloned repository as the gold standard and to override the paper when they conflict, so 22 of the 30 rows test 'paper+repo to module', not 'paper to module'. The 8 truly paper-only rows already average about -4.5 pp, and the paper's own no-repo ablation shows drops of up to roughly 35 pp for scaffold-heavy attacks. This is disclosed and ablated, which is credit to the authors, but the title, abstract, and headline mean present a broader claim than the data support. The ASR-targeted refinement pass in Algorithm 1 further means Δ is partly a fitted quantity, and the absence of confidence intervals makes a mean of +0.26 pp weak evidence of high fidelity. I do not think this warrants rejection: the engineering system is plausible, the agent prompts are unusually transparent, and the unified harness is a genuine contribution. A CONDITIONAL verdict is appropriate, with the paper-only reproduction check as the key condition. Hence UNCHANGED relative to the reader's verdict.","tokens_in":31506,"tokens_out":12040,"duration_ms":105988,"concrete_test":"Re-run JBF-FORGE with R=∅ (paper text only) on the 22 attacks that currently have official repositories, using the same matched settings and judges as Table 1, and compare the paper-only Δ distribution with the repo-assisted Δ distribution. If paper-only mean Δ is below -3 pp, or if the fraction of attacks with Δ < -10 exceeds about 25%, then the +0.26 pp headline must be restated as conditional on reference-repository availability; if the paper-only distribution matches the repo-assisted one, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central fidelity claim depends on a confound visible inside Table 1: 22/30 attacks are marked Repo=✓, and Appendix B.2 makes the cloned repository a gold reference that overrides the paper when they conflict. Splitting Table 1 by the Repo column, the 22 repo-assisted rows average about +2.1 pp, while the 8 paper-only rows average about -4.5 pp (SCP -11.8, MAJIC -8.5, PUZZLED -7.5, AIM -6.0, HILL -4.2). Thus the advertised +0.26 pp is mostly a 'paper+repo to module' result, not a 'paper to module' result. The paper's own no-repo ablation in §4.2/Figure 2 confirms the effect is large and method-dependent: e.g., SATA-MLM drops from 88.0 to 51.4 and GTA from 100.0 to 51.4 when the repository is withheld. Additionally, Algorithm 1 (lines 15-22) triggers an enhanced refinement pass whenever Δ < -10 and accepts the patch only if it moves ASR toward ASR_paper, so the reported deviations are partly tuned toward the target number rather than an unbiased measure of implementation fidelity. With no error bars or confidence intervals, the mean is not a robust certificate of 'high fidelity'.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents JAILBREAK FOUNDRY (JBF), a system with three components: JBF-LIB, a shared library of contracts and utilities; JBF-FORGE, a planner–coder–auditor agent pipeline that converts jailbreak papers into executable attack modules; and JBF-EVAL, a standardized evaluation harness. The authors report reproducing 30 attacks with a mean (reproduced−reported) ASR deviation of +0.26 percentage points, a 42% LOC reduction versus original repositories, an 82.5% mean reused-code ratio, and a standardized 30-attack × 10-model AdvBench evaluation under a common GPT-4o judge. The appendices include detailed agent prompts, auditor acceptance criteria, a no-repo ablation, and an enhanced refinement pass.","tokens_in":31869,"tokens_out":7539,"duration_ms":71189,"significance":"If the fidelity claim were fully supported, the system would be a valuable contribution to LLM security evaluation: it would reduce the manual cost of integrating new jailbreak attacks and enable longitudinally comparable benchmarks. Strengths of the paper include a public code release, a broad corpus of 30 attacks, transparency about the agent prompts and refinement protocol, and an explicit ablation comparing with- and without-repository reproduction. The paper is also candid about the role of official repositories for scaffold-heavy attacks. However, the headline fidelity number is not a clean measurement of paper-to-code fidelity: 22 of 30 attacks have an official repository that the pipeline is instructed to treat as a gold reference, and the enhanced refinement pass explicitly optimizes toward the reported ASR. The significance is therefore conditional on reanalysis; the system and released artifacts are nonetheless useful.","major_comments":[{"comment":"The headline mean deviation Δ=+0.26 pp aggregates two very different conditions. In Table 1, 22/30 rows have Repo=✓; per Appendix B.2 and the auditor instructions (Fig. 8), the cloned repository is the gold reference and takes priority over the paper when they conflict. The 8 paper-only rows are predominantly negative (SCP −11.8, MAJIC −8.5, PUZZLED −7.5, AIM −6.0, HILL −4.2; simple mean ≈ −4.5 pp). Thus the headline is a “paper+repo to module” fidelity result, not a “paper to module” result, and the central claim of the title is not established for the paper-only subset. Please report the two subsets separately and either reframe the headline or make paper-only fidelity the primary metric.","section":"§4.2, Table 1, Appendix B.2"},{"comment":"The enhanced refinement pass is invoked exactly when Δ < −10 pp and accepts the patched module only if Δ moves toward ASR_paper (line 19). Consequently, the final Δ is an optimized quantity — the system is tuned toward the target number — rather than an unbiased estimate of reproduction fidelity. No confidence intervals, run counts, or variance estimates are reported for ASR_gen, despite a range of −16.0 to +20.0 pp. A mean of +0.26 pp over a wide range without uncertainty cannot support “high fidelity.” Please report pre- and post-refinement Δ for every attack, and give evaluation variance (e.g., across seeds or bootstrap over samples).","section":"Algorithm 1 (lines 15–22), §3.2, Appendix H"},{"comment":"The matched-setting protocol substitutes a GPT-4o judge whenever the original judge is unavailable or impractical. This affects at least AIR (LLaMA-3-70B judge), MAJIC (LLaMA-2-13B), and AIM/TRIAL/HILL (human judgment). Because ASR is defined by the paper’s success criterion, changing the judge changes the success criterion; any Δ then conflates implementation error with protocol mismatch. The paper calls the comparison “apple-to-apple” (p. 5), which is not justified for these rows. At minimum, flag judge-substituted rows and provide a judge-sensitivity analysis on a subset (e.g., re-judge with the original judge where API-accessible).","section":"§4.1, Table 1 (Judge column)"},{"comment":"The 42% LOC reduction and 82.5% reuse ratio are computed against the generated module only, while JBF-LIB’s 2,014 LOC overhead is treated as fixed and excluded from the per-attack compression denominator. Since every module depends on that library, the end-to-end code size per attack is 2,014 + module LOC, and the reported ratio overstates the reduction. The reuse ratio is in large part a definitional consequence of counting the fixed library as “reused.” Please report the end-to-end footprint (library + modules) and account for functionality that may have been dropped or added during reimplementation (e.g., TrojFill’s increase).","section":"§4.3"},{"comment":"The no-repo ablation is the right experiment, and its results are stark: SATA-MLM drops from 88.0 to 51.4, SATA-ELP from 64.4 to 54.3, and GTA from 100.0 to 51.4 without the repository. This shows that for scaffold-heavy attacks the official repo, not the paper, supplies load-bearing implementation details. The text acknowledges this, but the abstract and conclusion nevertheless present a general “paper-to-module” capability. The no-repo results should be a headline scope condition, and the paper-only subset of Table 1 should be presented as the evidence for paper-only fidelity.","section":"§4.2, Figure 2"}],"minor_comments":[{"comment":"Abstract says “reduces attack-specific implementation code by nearly half,” while §4.3 reports 42%. Please align the wording.","section":"Abstract vs. §4.3"},{"comment":"“Defult Pipeline” should be “Default Pipeline.” Also, Figures 2 and 3 lack error bars and sample sizes; state the number of runs per bar.","section":"Figure 3"},{"comment":"For paper-only rows (Repo=✗), the Gen. LOC column lists generated module size, but ρ is “–” because there is no original repo. Please clarify in the caption that Gen. LOC is the JBF-LIB module size, not necessarily comparable to original repository size.","section":"Table 1"},{"comment":"The text says “26 implementations with variant de-duplication” while Table 1 lists 30 attacks (22 with repos). Please reconcile these counts and define how variants are merged in the LOC/reuse analysis.","section":"§4.3"},{"comment":"The auditor performs static analysis only, and the “100% fidelity” verdict is a self-reported LLM judgment. The main text should acknowledge this as a limitation of the acceptance criterion rather than implying line-level formal verification.","section":"Appendix B.6"},{"comment":"Claims such as “victim-in-the-loop is the strongest search family (mean ASR 60.3% vs. 49.4%)” are presented without confidence intervals or significance tests, and the per-family sample sizes are small (2–8 attacks). These should be framed as exploratory observations.","section":"§5.2"}],"recommendation":"major_revision","confidential_remarks":"The artifact and the authors’ transparency are strengths; my concern is that the paper’s title and abstract overstate paper-only fidelity when 22/30 attacks use a gold-reference repository and the refinement pass tunes toward the reported ASR. If the authors reanalyze the data with the paper-only subset separated and report uncertainty, I would support acceptance. No concerns about novelty or attribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick read of the JBF paper. Bottom line: the engineering is real, but the headline fidelity number is not as advertised. The +0.26pp mean deviation is an aggregate over 22 attacks where the authors cloned the official repository and let the auditor treat it as the gold standard, plus 8 paper-only attacks. Split Table 1: the repo-assisted rows hover around +2pp, while the paper-only rows average about -4.5pp, with SCP, MAJIC, PUZZLED, AIM, and HILL all negative. So the \"paper-to-module\" result is really a \"paper+repo-to-module\" result. The paper's own ablation (Figure 2) shows the same: SATA-MLM drops from 88.0 to 51.4 when the repo is withheld. That is a big, method-dependent confound, and the abstract's \"high fidelity\" language overstates what is demonstrated.\n\nWhat is actually good: the planner-coder-auditor loop, the JBF-LIB contract, and the standardized 10-model harness are a plausible and well-timed answer to the manual-integration bottleneck. The agent prompts in the appendix are unusually transparent, and the line-referenced audit protocol with a bounded loop is a sensible design. The authors also deserve credit for reporting the repo/no-repo difference at all.\n\nSoft spots, in order. First, the enhanced refinement pass in Algorithm 1: when reproduced ASR is more than 10 points below the paper, the system patches the module and keeps the patch only if it moves ASR toward the reported number. That means the average deviation is partly an optimized quantity, not an independent measurement of implementation fidelity. Second, judge substitution: for several papers the evaluation used GPT-4o in place of the original judge or human annotation, which changes the success criterion and makes any deviation ambiguous. Third, no confidence intervals, no run counts, no pinned code or raw results for the headline number. The appendix also notes the auditor is static-only with a smoke-test gate, so a \"100% fidelity\" verdict is a claim about code inspection, not behavior.\n\nWho is this for? People building benchmark infrastructure for LLM safety. It deserves a serious referee, but the revision should report paper-only fidelity separately from repo-assisted fidelity, provide per-run variance, and ideally run the no-repo analysis on all 30 attacks. As-is, I would not rely on the +0.26pp claim.","headline":"Real engineering contribution, but the +0.26pp fidelity headline is a repo-assisted average; the 8 paper-only attacks run about -4.5pp and the refinement pass tunes toward the target ASR.","tokens_in":32328,"tokens_out":2569,"would_cite":true,"duration_ms":26501,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A system that reads jailbreak papers and turns them into runnable benchmark attacks, reproducing reported success rates within 0.26 percentage points on average.","keywords":["jailbreak attacks","LLM safety evaluation","paper-to-code","multi-agent synthesis","reproducible benchmarking","attack success rate","living benchmarks","adversarial robustness"],"falsifier":"Take a reproduced attack whose original paper used human judging or a different automated judge, score the same generated responses with both the original judge/rubric and GPT-4o, and compare the deltas; if the two deltas differ substantially, or if re-running SCP and ISA with their exact original protocols still leaves them 11.8 and 16.0 points below report, then part of the claimed fidelity is an artifact of judge substitution rather than faithful implementation.","tokens_in":31409,"feed_emoji":"🔓","tokens_out":4486,"duration_ms":45157,"temperature":0.7,"pith_summary":"This paper tries to establish that jailbreak research has reached a point where the paper itself is the deliverable: a multi-agent system can read a newly published jailbreak paper and automatically produce a runnable attack module that reproduces the paper's reported attack success rate. Across thirty attacks, the reproduced rates land within +0.26 percentage points of the published numbers on average, and the full translation takes about 28 minutes per attack. If this holds, the bottleneck that keeps benchmarks stale — manual, expert, paper-specific reimplementation — dissolves, and robustness evaluations can be continuously refreshed as new attacks appear. The payoff is a standardized evaluation: the same thirty attacks, run across ten victim models under one dataset and one judge, yielding comparable robustness numbers and exposing which attack mechanisms actually transfer. The paper also argues this is sustainable engineering, since most per-attack code is shared scaffolding rather than novel logic.","feed_headline":"Jailbreak papers become runnable attacks in 28 minutes","feed_subtitle":"Automated pipeline reproduces 30 attacks' success rates within 0.26 points and runs them on 10 models.","key_machinery":"The load-bearing mechanism is the bounded planner–coder–auditor loop in JBF-FORGE. A planner turns the paper (and, when available, the authors' reference repository) into a structured implementation spec; a coder writes a module against the shared JBF-LIB contract; an auditor statically checks the module line-by-line against the spec and contract, accepting only at 100% fidelity, and iterates until acceptance or a cap. A matched-setting attack-success-rate evaluation then measures the reproduced-versus-reported gap, with an enhanced refinement pass triggered on large undershoots. The shared contract — typed parameters, registry, provider-agnostic LLM adapters — is what lets one harness run a","core_discovery":"The paper's central claim is that a jailbreak paper — including its prompts, parameters, and control flow — can be translated into an executable, harness-compatible attack module by an automated planner–coder–auditor workflow, without manual engineering. Across 30 reproduced attacks, run under each paper's representative setting, the mean difference between reproduced and reported attack success rate is +0.26 percentage points, with a range of −16.0 to +20.0. The same modules, placed in a unified evaluation layer with a fixed GPT-4o judge on AdvBench, produce a 30-attack × 10-model success-rate matrix that exposes sharp attack–model interactions. The paper also claims the shared library cuts","pith_inferences":["The judge-substitution design makes the headline fidelity number conditional: where the original paper used human or a different automated judge, part of the +0.26-point agreement could be protocol alignment rather than faithful reimplementation.","Because the paper labels attacks on orthogonal search and carrier axes and finds carrier format drives victim-specific success, the same automated pipeline could test predictions such as 'formal wrappers will beat obfuscation on assistant-style models' against newly published methods.","The dual-use concern named in the paper has a concrete corollary: the same 28-minute automation that helps defenders maintain living benchmarks also lowers the engineering cost for an attacker to stand up many known attacks, so release and rate-limit choices for the harness are part of the safety posture.","A direct extension would be to run the 30-module suite on each new victim model release and flag any attack whose ASR rises past a threshold — a cheap automated red-team tripwire for model updates."],"forward_implications":["Benchmarks can become living: newly published attacks can be integrated in tens of minutes rather than weeks of manual engineering.","Robustness comparisons become apples-to-apples: all attacks share one dataset, judge, and execution protocol, so ASR differences reflect attack-model interaction rather than harness drift.","The same 30-module suite can be re-run as victim models update, serving as a regression test for whether safety updates close specific attack surfaces.","Reproducing a paper's headline numbers no longer requires reimplementing scaffolding: most code is shared infrastructure, and per-attack modules are compact.","Cross-model results can expose narrow blind spots, such as a model with low mean ASR that still falls to a single attack at 82%, which single-model reporting would hide."],"fun_headline_variants":["Automated pipeline turns jailbreak papers into runnable attacks","Jailbreak papers to executable attacks automatically","From paper to attack: automated jailbreak reproduction","Reproduce jailbreak papers as runnable attacks automatically","Automated workflow converts jailbreak papers to executable attacks"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The fidelity claim assumes that substituting a GPT-4o judge for the original judge or human judgment and choosing one representative victim setting per paper preserves the original success criterion; if judge or protocol mismatch changes success rates, the gap between reproduced and reported numbers reflects the mismatch rather than implementation fidelity.","fun_headline_variants_meta":{"raw":{"variants":["Automated pipeline turns jailbreak papers into runnable attacks","Jailbreak papers to executable attacks automatically","From paper to attack: automated jailbreak reproduction","Reproduce jailbreak papers as runnable attacks automatically","Automated workflow converts jailbreak papers to executable attacks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000357,"raw_usage":{"total_tokens":1785,"prompt_tokens":770,"completion_tokens":1015,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":514,"completion_tokens_details":{"reasoning_tokens":939}},"tokens_in":514,"tokens_out":1015,"duration_ms":8621,"temperature":1.0,"reasoning_tokens":939,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T20:04:15.508738+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a reproduced attack whose original paper used human judging or a different automated judge, score the same generated responses with both the original judge/rubric and GPT-4o, and compare the deltas; if the two deltas differ substantially, or if re-running SCP and ISA with their exact original protocols still leaves them 11.8 and 16.0 points below report, then part of the claimed fidelity is an artifact of judge substitution rather than faithful implementation.","supporting_citations":[],"review_version":1}