Pith. sign in

REVIEW 5 major objections 6 minor 1 cited by

Jailbreak Foundry: From Papers to Runnable Attacks for Reproducible Benchmarking

T0 review · 5 major / 6 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A system that reads jailbreak papers and turns them into runnable benchmark attacks, reproducing reported success rates within 0.26 percentage points on average.

desk verdict Real engineering contribution, but the +0.26pp fidelity headline is a repo-assisted average; the 8 paper-only attacks run about -4.5pp and the refinement pass tunes toward the target ASR. read the letter →

arxiv 2602.24009 v4 pith:STH5XJ7X submitted 2026-02-27 cs.CR cs.AIcs.CLcs.LG

Zhicheng Fang , Jingjie Zheng , Chenxu Fu , Wei Xu This is my paper
classification cs.CRcs.AIcs.CLcs.LG
keywords jailbreakattacksLLMsafetyevaluationpaper-to-codemulti-agentsynthesisreproduciblebenchmarkingattacksuccessratelivingbenchmarksadversarialrobustness
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that jailbreak research has reached a point where the paper itself is the deliverable: a multi-agent system can read a newly published jailbreak paper and automatically produce a runnable attack module that reproduces the paper's reported attack success rate. Across thirty attacks, the reproduced rates land within +0.26 percentage points of the published numbers on average, and the full translation takes about 28 minutes per attack. If this holds, the bottleneck that keeps benchmarks stale — manual, expert, paper-specific reimplementation — dissolves, and robustness evaluations can be continuously refreshed as new attacks appear. The payoff is a standardized evaluation: the same thirty attacks, run across ten victim models under one dataset and one judge, yielding comparable robustness numbers and exposing which attack mechanisms actually transfer. The paper also argues this is sustainable engineering, since most per-attack code is shared scaffolding rather than novel logic.

What carries the argument

The load-bearing mechanism is the bounded planner–coder–auditor loop in JBF-FORGE. A planner turns the paper (and, when available, the authors' reference repository) into a structured implementation spec; a coder writes a module against the shared JBF-LIB contract; an auditor statically checks the module line-by-line against the spec and contract, accepting only at 100% fidelity, and iterates until acceptance or a cap. A matched-setting attack-success-rate evaluation then measures the reproduced-versus-reported gap, with an enhanced refinement pass triggered on large undershoots. The shared contract — typed parameters, registry, provider-agnostic LLM adapters — is what lets one harness run a

What would settle it

Take a reproduced attack whose original paper used human judging or a different automated judge, score the same generated responses with both the original judge/rubric and GPT-4o, and compare the deltas; if the two deltas differ substantially, or if re-running SCP and ISA with their exact original protocols still leaves them 11.8 and 16.0 points below report, then part of the claimed fidelity is an artifact of judge substitution rather than faithful implementation.

Watch

Extended reading notes

Core claim

The paper's central claim is that a jailbreak paper — including its prompts, parameters, and control flow — can be translated into an executable, harness-compatible attack module by an automated planner–coder–auditor workflow, without manual engineering. Across 30 reproduced attacks, run under each paper's representative setting, the mean difference between reproduced and reported attack success rate is +0.26 percentage points, with a range of −16.0 to +20.0. The same modules, placed in a unified evaluation layer with a fixed GPT-4o judge on AdvBench, produce a 30-attack × 10-model success-rate matrix that exposes sharp attack–model interactions. The paper also claims the shared library cuts

Load-bearing premise

The fidelity claim assumes that substituting a GPT-4o judge for the original judge or human judgment and choosing one representative victim setting per paper preserves the original success criterion; if judge or protocol mismatch changes success rates, the gap between reproduced and reported numbers reflects the mismatch rather than implementation fidelity.

Editorial extensions

If this is right

  • Benchmarks can become living: newly published attacks can be integrated in tens of minutes rather than weeks of manual engineering.
  • Robustness comparisons become apples-to-apples: all attacks share one dataset, judge, and execution protocol, so ASR differences reflect attack-model interaction rather than harness drift.
  • The same 30-module suite can be re-run as victim models update, serving as a regression test for whether safety updates close specific attack surfaces.
  • Reproducing a paper's headline numbers no longer requires reimplementing scaffolding: most code is shared infrastructure, and per-attack modules are compact.
  • Cross-model results can expose narrow blind spots, such as a model with low mean ASR that still falls to a single attack at 82%, which single-model reporting would hide.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The judge-substitution design makes the headline fidelity number conditional: where the original paper used human or a different automated judge, part of the +0.26-point agreement could be protocol alignment rather than faithful reimplementation.
  • Because the paper labels attacks on orthogonal search and carrier axes and finds carrier format drives victim-specific success, the same automated pipeline could test predictions such as 'formal wrappers will beat obfuscation on assistant-style models' against newly published methods.
  • The dual-use concern named in the paper has a concrete corollary: the same 28-minute automation that helps defenders maintain living benchmarks also lowers the engineering cost for an attacker to stand up many known attacks, so release and rate-limit choices for the harness are part of the safety posture.
  • A direct extension would be to run the 30-module suite on each new victim model release and flag any attack whose ASR rises past a threshold — a cheap automated red-team tripwire for model updates.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. This paper presents JAILBREAK FOUNDRY (JBF), a system with three components: JBF-LIB, a shared library of contracts and utilities; JBF-FORGE, a planner–coder–auditor agent pipeline that converts jailbreak papers into executable attack modules; and JBF-EVAL, a standardized evaluation harness. The authors report reproducing 30 attacks with a mean (reproduced−reported) ASR deviation of +0.26 percentage points, a 42% LOC reduction versus original repositories, an 82.5% mean reused-code ratio, and a standardized 30-attack × 10-model AdvBench evaluation under a common GPT-4o judge. The appendices include detailed agent prompts, auditor acceptance criteria, a no-repo ablation, and an enhanced refinement pass.

Significance. If the fidelity claim were fully supported, the system would be a valuable contribution to LLM security evaluation: it would reduce the manual cost of integrating new jailbreak attacks and enable longitudinally comparable benchmarks. Strengths of the paper include a public code release, a broad corpus of 30 attacks, transparency about the agent prompts and refinement protocol, and an explicit ablation comparing with- and without-repository reproduction. The paper is also candid about the role of official repositories for scaffold-heavy attacks. However, the headline fidelity number is not a clean measurement of paper-to-code fidelity: 22 of 30 attacks have an official repository that the pipeline is instructed to treat as a gold reference, and the enhanced refinement pass explicitly optimizes toward the reported ASR. The significance is therefore conditional on reanalysis; the system and released artifacts are nonetheless useful.

major comments (5)
  1. [§4.2, Table 1, Appendix B.2] The headline mean deviation Δ=+0.26 pp aggregates two very different conditions. In Table 1, 22/30 rows have Repo=✓; per Appendix B.2 and the auditor instructions (Fig. 8), the cloned repository is the gold reference and takes priority over the paper when they conflict. The 8 paper-only rows are predominantly negative (SCP −11.8, MAJIC −8.5, PUZZLED −7.5, AIM −6.0, HILL −4.2; simple mean ≈ −4.5 pp). Thus the headline is a “paper+repo to module” fidelity result, not a “paper to module” result, and the central claim of the title is not established for the paper-only subset. Please report the two subsets separately and either reframe the headline or make paper-only fidelity the primary metric.
  2. [Algorithm 1 (lines 15–22), §3.2, Appendix H] The enhanced refinement pass is invoked exactly when Δ < −10 pp and accepts the patched module only if Δ moves toward ASR_paper (line 19). Consequently, the final Δ is an optimized quantity — the system is tuned toward the target number — rather than an unbiased estimate of reproduction fidelity. No confidence intervals, run counts, or variance estimates are reported for ASR_gen, despite a range of −16.0 to +20.0 pp. A mean of +0.26 pp over a wide range without uncertainty cannot support “high fidelity.” Please report pre- and post-refinement Δ for every attack, and give evaluation variance (e.g., across seeds or bootstrap over samples).
  3. [§4.1, Table 1 (Judge column)] The matched-setting protocol substitutes a GPT-4o judge whenever the original judge is unavailable or impractical. This affects at least AIR (LLaMA-3-70B judge), MAJIC (LLaMA-2-13B), and AIM/TRIAL/HILL (human judgment). Because ASR is defined by the paper’s success criterion, changing the judge changes the success criterion; any Δ then conflates implementation error with protocol mismatch. The paper calls the comparison “apple-to-apple” (p. 5), which is not justified for these rows. At minimum, flag judge-substituted rows and provide a judge-sensitivity analysis on a subset (e.g., re-judge with the original judge where API-accessible).
  4. [§4.3] The 42% LOC reduction and 82.5% reuse ratio are computed against the generated module only, while JBF-LIB’s 2,014 LOC overhead is treated as fixed and excluded from the per-attack compression denominator. Since every module depends on that library, the end-to-end code size per attack is 2,014 + module LOC, and the reported ratio overstates the reduction. The reuse ratio is in large part a definitional consequence of counting the fixed library as “reused.” Please report the end-to-end footprint (library + modules) and account for functionality that may have been dropped or added during reimplementation (e.g., TrojFill’s increase).
  5. [§4.2, Figure 2] The no-repo ablation is the right experiment, and its results are stark: SATA-MLM drops from 88.0 to 51.4, SATA-ELP from 64.4 to 54.3, and GTA from 100.0 to 51.4 without the repository. This shows that for scaffold-heavy attacks the official repo, not the paper, supplies load-bearing implementation details. The text acknowledges this, but the abstract and conclusion nevertheless present a general “paper-to-module” capability. The no-repo results should be a headline scope condition, and the paper-only subset of Table 1 should be presented as the evidence for paper-only fidelity.
minor comments (6)
  1. [Abstract vs. §4.3] Abstract says “reduces attack-specific implementation code by nearly half,” while §4.3 reports 42%. Please align the wording.
  2. [Figure 3] “Defult Pipeline” should be “Default Pipeline.” Also, Figures 2 and 3 lack error bars and sample sizes; state the number of runs per bar.
  3. [Table 1] For paper-only rows (Repo=✗), the Gen. LOC column lists generated module size, but ρ is “–” because there is no original repo. Please clarify in the caption that Gen. LOC is the JBF-LIB module size, not necessarily comparable to original repository size.
  4. [§4.3] The text says “26 implementations with variant de-duplication” while Table 1 lists 30 attacks (22 with repos). Please reconcile these counts and define how variants are merged in the LOC/reuse analysis.
  5. [Appendix B.6] The auditor performs static analysis only, and the “100% fidelity” verdict is a self-reported LLM judgment. The main text should acknowledge this as a limitation of the acceptance criterion rather than implying line-level formal verification.
  6. [§5.2] Claims such as “victim-in-the-loop is the strongest search family (mean ASR 60.3% vs. 49.4%)” are presented without confidence intervals or significance tests, and the per-family sample sizes are small (2–8 attacks). These should be framed as exploratory observations.

Circularity Check

1 steps flagged · score 6.0 of 10

Fidelity metric partly optimized: the enhanced refinement pass patches toward ASR_paper, so the headline +0.26 pp is not an unbiased measurement.

  1. fitted input called prediction [Algorithm 1 (lines 15–22); §3.2 'Enhanced refinement pass'; §4.2 'Enhanced refinement pass']
    "When matched-setting evaluation shows a substantial undershoot ( ∆<−10.0 ), we invoke a single enhanced refinement pass that (i) performs read-only, code-level gap analysis against sp and R, then (ii) applies a tightly scoped patch to mp and, when needed, its paper-specific harness, followed by re-evaluation under the same matched settings e. ... if ASR′gen−ASRpaper ≥∆ then mp←m′p; ∆←ASR′gen−ASRpaper"

    The quantity being validated (∆ = ASR_gen − ASR_paper) is also the objective that triggers and gates refinement: for any attack with ∆<−10, the system patches the module and accepts the patch only when it moves ASR_gen toward ASR_paper. The post-refinement ASR is therefore a fit to the paper-reported number, not an independent measurement of reproduction fidelity. The headline mean deviation +0.26 pp includes these tuned values (Figure 3: mean gap improves from −16.2% to −7.6%), so part of the advertised fidelity is constructed by the evaluation loop itself.

full rationale

The central fidelity claim in the abstract and §4.2 is partially circular. Algorithm 1's enhanced refinement pass explicitly uses ASR_paper as the target of a patch-and-accept loop: it is triggered only when reproduced ASR is at least 10 points below the paper, and a patch is accepted only if the new gap is no worse. For the six attacks shown in Figure 3, the reported 'post-refinement' deviations are therefore optimized quantities rather than unbiased estimates of how well the paper text alone predicts behavior. This is the main circular step. The repo-assisted rows are a separate concern, not a definitional circularity: Appendix B.2 makes the cloned repository a gold reference, and the with-repo/no-repo ablation shows that repo access raises ASR by +19.8 pp on a five-attack subset, so the aggregate +0.26 pp largely reflects paper+repo-to-module translation rather than paper-only translation. That weakens the interpretation of the headline but is not itself circular. The standardized JBF-EVAL cross-model heatmap (Figure 4) is independent of ASR_paper, which prevents a higher score. No load-bearing self-citations or imported uniqueness theorems appear in the derivation chain.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

No new physical or conceptual entities are postulated; JBF-LIB, JBF-FORGE, and JBF-EVAL are software components. The load-bearing choices are protocol decisions: refinement trigger, judge substitution, representative settings, LOC accounting, and reliance on official repositories as gold standard.

free parameters (5)
  • Refinement trigger threshold tau = -10.0 pp = -10.0 percentage points
    Hand-chosen ASR undershoot threshold that decides when the enhanced refinement pass is invoked (Algorithm 1, Section 3.2). Directly controls how much of the reported fidelity is tuned rather than measured.
  • Fixed JBF-LIB overhead = 2,014 LOC = 2014 LOC
    Accounting convention chosen to compute the 82.5% reused-code ratio and 42% LOC compression; the reuse metric depends on this fixed overhead allocation (Section 4.3).
  • Representative victim-model selection rule = "most recent GPT-family model" from the primary results table
    Hand-chosen protocol for picking one configuration per paper; determines which ASR_paper value is used in the fidelity comparison (Section 4.1).
  • Judge substitution policy = GPT-4o when the original judge is unavailable
    ASR comparisons are made with a different judge for several papers (e.g., Human(GPT-4o), LLaMA-2-13B, GPT-4o-mini in Table 1), changing the success criterion the fidelity metric measures (Section 4.1).
  • Cross-model effectiveness thresholds = >=50% ASR on >=6 victims; >=70% ASR on >=6 victims
    Arbitrary thresholds used to claim that automated reproductions are broadly effective (Section 4.4); different thresholds would change the qualitative claim.
assumptions (5)
  • domain assumption Paper-reported ASR values are trustworthy and directly comparable across papers.
    The fidelity metric Delta = ASR_gen - ASR_paper treats the published number as ground truth; if original ASRs are protocol-dependent or unreliable, Delta is not a clean fidelity measure (Section 4.1).
  • domain assumption A single representative victim configuration can stand in for the paper's full evaluation protocol.
    The paper selects one setting from each paper's primary results table and treats the resulting ASR comparison as paper-matched fidelity (Section 4.1).
  • domain assumption A GPT-4o judge with a fixed success rubric reproduces each paper's original success criterion, including human judging.
    Table 1 substitutes GPT-4o for unavailable human or other judges; this assumes judge outputs are interchangeable for ASR (Sections 3.3, 4.1).
  • domain assumption Static source-level auditing plus a one-sample smoke test is sufficient to certify behavioral fidelity.
    The auditor never executes code; acceptance is a line-referenced static check, with only a minimal integration smoke test as a functional gate (Appendix B.6).
  • domain assumption Official reference repositories are faithful to the papers and can serve as a gold standard for implementation details.
    For 22 of 30 attacks, the pipeline uses the official repo to resolve underspecified details, and fidelity is then measured against paper ASR; the no-repo ablation shows this assumption carries substantial weight (Sections 3.2, 4.2, Figure 2).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Jailbreak Foundry: From Papers to Runnable Attacks for Reproducible Benchmarking." pith.science (2026). https://pith.science/paper/STH5XJ7X

@misc{pith2026260224009,
  author       = {Pith},
  title        = {Pith review of: Jailbreak Foundry: From Papers to Runnable Attacks for Reproducible Benchmarking},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/STH5XJ7X}},
  note         = {Machine review of arXiv:2602.24009}
}
read the original abstract

Jailbreak techniques for large language models (LLMs) evolve faster than benchmarks, making robustness estimates stale and difficult to compare across papers due to drift in datasets, harnesses, and judging protocols. We introduce JAILBREAK FOUNDRY (JBF), a system that addresses this gap via a multi-agent workflow to translate jailbreak papers into executable modules for immediate evaluation within a unified harness. JBF features three core components: (i) JBF-LIB for shared contracts and reusable utilities; (ii) JBF-FORGE for the multi-agent paper-to-module translation; and (iii) JBF-EVAL for standardizing evaluations. Across 30 reproduced attacks, JBF achieves high fidelity with a mean (reproduced-reported) attack success rate (ASR) deviation of +0.26 percentage points. By leveraging shared infrastructure, JBF reduces attack-specific implementation code by more than half relative to original repositories and achieves an 82.5% mean reused-code ratio. This system enables a standardized AdvBench evaluation of all 30 attacks across 10 victim models using a consistent GPT-4o judge. By automating both attack integration and standardized evaluation, JBF offers a scalable solution for creating living benchmarks that keep pace with the rapidly shifting security landscape.

Figures

Figures reproduced from arXiv: 2602.24009 by the authors.

Figure 1
Figure 1. JAILBREAK FOUNDRY (JBF) overview. JBF-LIB provides shared contracts and utilities, JBF-FORGE translates papers into runnable modules, and JBF-EVAL evaluates them with fixed datasets, protocols, and judging, enabling comparable cross-attack and cross-model results. enabling directly comparable model robustness results and attack mechanism-level analysis. Our contributions are: (i) Multi-agent paper-to-module translat… view at source ↗
Figure 2
Figure 2. With-repo vs. no-repo reproduction on five selected at￾tacks (recent methods, largest gains in [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. ASR(%) for six attacks with the largest baseline repro￾duction gaps. −16.2% to −7.6%. These gains suggest the dominant fail￾ure mode is implementation-driven underperformance rather than an inherent limit of paper-only descriptions. Improve￾ments concentrate in scaffold-heavy methods with many implicit defaults, where small mismatches in prompt struc￾ture, control flow, or retry logic can materially depress ASR. The… view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: JBF-Eval ASR heatmap on AdvBench: standardized attack success rates (%) for 30 attacks (x-axis) across 10 victim models (y-axis) under a unified harness and judge; warmer colors indicate higher ASR. ter). Savings mainly come from removing paper-specific har￾ness glue, …
Figure 5
Figure 5. Figure 5: Implementation iterations vs. attack performance. Each point denotes an attack (or variant) with its mean ASR averaged over 10 victim models; colors indicate the number of implementation–audit iterations required to reach the auditor’s acceptance. Diamonds and vertical…
Figure 6
Figure 6. Figure 6: Full instruction prompt used for the planning agent. Coding Agent Instructions # Attack Implementation Agent You are an elite AI research engineer specializing in paper-to-code implementation for adversarial ML and jailbreak attacks. Implement precisely and follow the …
Figure 7
Figure 7. Figure 7: Full instruction prompt used for the coding agent. 27 [PITH_FULL_IMAGE:figures/full_fig_p027_7.png]
Figure 8
Figure 8. Figure 8: Full instruction prompt used for the audit agent. 34 [PITH_FULL_IMAGE:figures/full_fig_p034_8.png]
Figure 9
Figure 9. Figure 9: Phase 1 prompt used in the enhanced refinement pass to generate implementation gap notes.md via a read-only gap analysis of the plan, reference artifacts, and the current generated module. Enhanced Refinement Pass (Phase 2) Targeted Refinement Instructions You are Clau…
Figure 10
Figure 10. Figure 10: Phase 2 prompt used in the enhanced refinement pass to apply a tightly scoped patch to the generated attack and, when necessary, its paper-specific test script, following the implementation plan and the gap notes. 36 [PITH_FULL_IMAGE:figures/full_fig_p036_10.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. PixJail: Self-Evolving Paper-to-Pipeline Reproduction for Text-to-Image Jailbreak Evaluation

    cs.CR 2026-06 unverdicted novelty 7.0 of 10

    PixJail automates construction of paper-specific attack modules and unified evaluation pipelines for text-to-image jailbreaks, reproducing eleven methods with 2.1% average and 0% median error.

Reference graph

Works this paper leans on

44 extracted references · 5 linked inside Pith · cited by 1 Pith paper

  1. [1]

    2.Secondary:the JBF-LIBcontractC(interfaces, I/O schema, parameterization hooks, and required metadata)

    Primary:the structured implementation plan/specification sp (the canonical description of what must be implemented). 2.Secondary:the JBF-LIBcontractC(interfaces, I/O schema, parameterization hooks, and required metadata)

  2. [2]

    Do NOT read the paper markdown

    Read the implementation plan and follow it exactly. Do NOT read the paper markdown

  3. [3]

    Partial/Equivalent

    Gold reference (when available):an official runnable reference repository R to resolve underspecified defaults, prompts, and control flow details. Non-invention rule.The auditor must not introduce requirements that are absent from sp and C; in ambiguous cases, it records the ambiguity and evaluates whether the implementation is consistent with the plan an...

  4. [4]

    Li, X., Zhou, Z., Zhu, J., Yao, J., Liu, T., and Han, B

    URL https://openreview.net/forum? id=GLhLU6y1uK. Li, X., Zhou, Z., Zhu, J., Yao, J., Liu, T., and Han, B. Deepinception: Hypnotize large language model to be jailbreaker, 2024. URL https://arxiv.org/abs/ 2311.03191. Liang, Z., Huang, H., and Chen, Z. Equacode: A multi- strategy jailbreak approach for large language models via equation solving and code com...

    arXiv 2024

  5. [5]

    method_name_gen

    Implement the attack per plan, using the framework patterns below. If cloned code is referenced, also inspect any`.ipynb`notebooks in the repo for implementation details and defaults.,→ **Multi-Attempt Support (Required when applicable)**: - The test harness supports multiple attempts per query via CLI args: `--attempts-per-query`and`--attempts-success-th...

  6. [6]

    Starace, G., Jaffe, O., Sherburn, D., Aung, J., Chan, J

    URL https://openreview.net/forum? id=KZLE5BaaOH. Starace, G., Jaffe, O., Sherburn, D., Aung, J., Chan, J. S., Maksin, L., Dias, R., Mays, E., Kinsella, B., Thomp- son, W., Heidecke, J., Glaese, A., and Patwardhan, T. Paperbench: Evaluating ai’s ability to replicate ai research, 2025. URL https://arxiv.org/abs/ 2504.01848. Sun, Z., Zhang, Z., Liang, D., Su...

    arXiv 2025

  7. [7]

    Verdict matches evidence

  8. [8]

    Changes Since Previous Iteration

    If re-audit, verify you have: - Documented status of ALL prior issues - Performed spot-checks on previously-correct components - Actively searched for NEW issues (not just re-checked old ones) - Included the "Changes Since Previous Iteration" section - Prepended (not overwritten) to the existing file ## Final Output At the end, output: ```json { "status":...

Show all 44 references
  1. [12]

    ISBN 979-8-89176-256-5

    Association for Computational Linguistics. ISBN 979-8-89176-256-5. doi: 10.18653/v1/2025.findings-acl

    2025 doi

  2. [16]

    Environment setup (required): ```bash # Activate the project environment (example) source <conda_root>/etc/profile.d/conda.sh conda activate <env_name> ```

  3. [18]

    No jailbreak

    If the plan says "No jailbreak", stop with message: "No jailbreak."

  4. [19]

    If the plan says a repository was found/cloned, read the cloned code in `attacks_paper_info/{arxiv_id}/`and follow it as the gold standard.,→

  5. [21]

    Read`attacks_paper_info/{arxiv_id}/Implementation_verdict.md`

  6. [22]

    Read`src/autojailbreak/attacks/generated/{attack_name}.py`

  7. [23]

    Apply all fixes required by the verdict

  8. [24]

    status":

    Run the test script and fix any new errors. ## Quality Checklist Before completing, verify: - [ ] Read framework code (base.py, factory.py, litellm.py) - [ ] Did NOT read existing attack implementations - [ ] NAME has "_gen" suffix - [ ] File name matches NAME exactly - [ ] Al...

  9. [25]

    **Read Previous Verdict First** - Read the entire existing verdict file - Extract all issues marked as bad or warning - Note all components marked as good - Identify the previous verdict result (100% Fidelity or Not) - If`coverage_analysis.md`exists, compare it against current...

  10. [26]

    **Verify Fixes for Prior Issues** (MANDATORY) - For EVERY issue marked bad or warning in the previous iteration: - Re-examine the specific code location mentioned - Determine: Fixed good | Partially Fixed warning | Still Broken bad | Regressed - Document the status change for ...

  11. [27]

    **Spot-Check Previously-Correct Components** (MANDATORY) - Randomly select 20-30% of components marked good in prior iteration - Re-verify these are still correct (catch regressions) - If any have regressed, document as Regression

  12. [28]

    **Deep Audit of Previously-Problematic Areas** (MANDATORY) - For components that were bad or warning before: - Perform forensic-level re-analysis - Verify the fix addresses the root cause - Check for new issues introduced by the fix

  13. [29]

    **Hunt for NEW Issues** (MANDATORY - CRITICAL) - **This is your most important responsibility** - Review ALL components for issues NOT identified in prior iterations - Pay special attention to: - Code sections modified since last audit - Edge cases not covered in previous anal...

  14. [30]

    Previous audit said 100% Fidelity, so I'll just quickly confirm

    **Analyze Code Changes** - Compare current implementation line numbers vs. previous verdict - If line numbers shifted significantly, code was modified - Focus extra scrutiny on modified sections ### Anti-Complacency Safeguards 29 Jailbreak Foundry: From Papers to Runnable Atta...

  15. [31]

    **Algorithm Fidelity** - Step-by-step correspondence to paper - Control flow patterns (loops, recursion, branching) - Order of operations preserved

  16. [32]

    **Data Structures** - Structure types match paper descriptions - Update operations align with paper - Edge cases handled as specified 30 Jailbreak Foundry: From Papers to Runnable Attacks for Reproducible Benchmarking

  17. [33]

    **Mathematical Translations** - Formulas implemented exactly - Invariants and constraints preserved - Numerical methods match paper

  18. [34]

    target model

    **Parameters** - Names correspond to paper notation - Default values match paper - Types align with descriptions - Behavioral effects match paper - **Victim-as-target handling**: If the paper's "target model" is actually the same as the victim model under attack, confirm the i...

  19. [35]

    **External Dependencies** - Model selection matches paper (e.g., GPT-4, Claude) - **IMPORTANT**: Only verify MODEL name, NOT provider/API (user may use different API access),→ - API calls align with methodology - Sampling strategies match specs

  20. [36]

    ## Audit Iteration 2 - 2025-11-25

    **Additions and Deviations** - Flag code behavior not in paper - Identify optimizations altering semantics - Note defensive programming changing behavior ### Optimization/Training Cache Requirements - If the attack includes reusable optimization or training rounds, verify that...

    2025

  21. [37]

    Read both required files completely (paper + implementation)

  22. [38]

    If re-audit: Read and analyze previous verdict file

  23. [39]

    All Coverage Table rows have exact line references

  24. [40]

    Parameter Mapping is complete

  25. [41]

    Misalignments have specific citations and locations

  26. [42]

    Required Changes are actionable and minimal

  27. [45]

    Verify fixes were implemented correctly (accountability)

  28. [46]

    ## Differences Likely Causing the ASR Gap

    Discover issues missed in prior iterations (continuous improvement) Be thorough, be skeptical, be precise. The quality of the implementation depends on your diligence.,→ Figure 8.Full instruction prompt used for the audit agent. 34 Jailbreak Foundry: From Papers to Runnable At...

  29. [92]

    findings-emnlp.92/

    URL https://aclanthology.org/2025. findings-emnlp.92/. Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., et al. Qwen3 technical report, 2025. URL https://arxiv.org/abs/2505.09388. Yao, Y ., Tong, X., Wang, R., Wang, Y ., Li, L., Liu, L., Teng, Y ., an...

    2025 arXiv

  30. [100]

    findings-acl.100/

    URL https://aclanthology.org/2025. findings-acl.100/. 9 Jailbreak Foundry: From Papers to Runnable Attacks for Reproducible Benchmarking Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., et al. The llama 3 herd of models, 2024. URLhttps: //arxiv.org/abs/2407.2178...

    2025 arXiv

  31. [118]

    naacl-long.118/

    URL https://aclanthology.org/2024. naacl-long.118/. Ding, P., Kuang, J., Sun, W., Wang, Z., Cao, X., Cai, X., Chen, J., and Huang, S. Friend or foe: How llms’ safety mind gets fooled by intent shift attack, 2025. URL https://arxiv.org/abs/2511.00556. Dong, X., Hu, W., Xu, W., ...

    2024

  32. [190]

    emnlp-main.190/

    URL https://aclanthology.org/2025. emnlp-main.190/. Wang, X., Chen, Y ., Li, J., Wang, Y ., Yao, Y ., Gu, T., Li, J., Teng, Y ., Wang, Y ., and Hu, X. Openrt: An open-source red teaming framework for multimodal llms, 2026. URL https://arxiv.org/abs/2601.01592. Wei, A., Haghtal...

    2025

  33. [298]

    findings-acl.298/

    URL https://aclanthology.org/2025. findings-acl.298/. 12 Jailbreak Foundry: From Papers to Runnable Attacks for Reproducible Benchmarking A. Attack Taxonomy To support cross-paper comparison, we annotate each reproduced attack in Table 1 with two orthogonal taxonomy labels: Se...

    2025

  34. [314]

    emnlp-main.314/

    URL https://aclanthology.org/2025. emnlp-main.314/. Yan, Y ., Sun, S., Wang, Z., Lin, Y ., Duan, Z., Zheng, Z., Liu, M., Yin, Z., and Zhang, J. Confusion is the final bar- rier: Rethinking jailbreak evaluation and investigating the real misuse threat of LLMs. In Christodoulopo...

    2025

  35. [408]

    findings-acl.408/

    URL https://aclanthology.org/2025. findings-acl.408/. Zhang, T., Cao, B., Cao, Y ., Lin, L., Mitra, P., and Chen, J. WordGame: Efficient & effective LLM jailbreak via simultaneous obfuscation in query and response. In Chiruzzo, L., Ritter, A., and Wang, L. (eds.),Find- ings of...

    2025 arXiv

  36. [2024]

    ISBN 9798331314385

    Curran Associates Inc. ISBN 9798331314385. Miao, Z., Li, L., Xiong, Y ., Liu, Z., Zhu, P., and Shao, J. Response attack: Exploiting contextual priming to jailbreak large language models, 2025. URL https: //arxiv.org/abs/2507.05248. OpenAI. Introducing gpt-5. 2025a. URL https:/...

    2025

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.