Pith. sign in

REVIEW 4 major objections 6 minor 25 references

An aligned LLM behind a disabled regex filter blocks 0% of adversarial probes under substring refusal matching — identical to no defenses — yet a semantic judge finds 56–100% refusals on framed variants, so alignment's measured contribution

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 11:22 UTC pith:LSX3UJK7

load-bearing objection Useful single-axis ablation and released corpus, but the headline metric-dependent claim rests on a table that changes metric and mutation strategy at once. the 4 major comments →

arxiv 2607.20494 v1 pith:LSX3UJK7 submitted 2026-06-12 cs.AI cs.CRcs.LG

Isolating LLM Alignment from Regex: Zero Coverage and Metric-Dependent Divergence Under Adversarial Mutation

Alexandre Cristov\~ao Maiorano This is my paper
classification cs.AI cs.CRcs.LG
keywords LLM alignmentinput regex filterablation latticerefusal classificationjailbreak probesmetric dependencesubstring matchingLLM judge
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper asks whether model-side alignment still adds safety coverage when the input regex filter that normally sits in front of it is disabled. It builds a single-axis ablation — a live aligned LLM in an otherwise identical defense stack, with the nine-pattern input filter turned off — and probes it with 45 adversarial prompts amplified into roughly 1,555 evaluated probe-run pairs. Under the primary substring refusal classifier, the aligned model blocks 0% of probes in every LLM threat category, statistically indistinguishable from a no-defense stub, so the hypothesis that alignment adds measurable coverage is refuted. Under a secondary LLM-judge metric, however, adversarially framed variants show 56–100% block rates: the model does refuse, but in phrasing the substring test misses. The paper concludes that alignment's contribution is metric-dependent — near zero on plain natural-language harmful requests, real but visible only to a semantic judge on explicit adversarial framing.

Core claim

The central discovery is that the measured value of LLM alignment depends on which verdict instrument reads the response. In the new L5-no-regex condition, identical to the live-backend condition except that the nine-pattern input filter is disabled, Gemini-2.5-flash refuses none of the 45 probes according to the inherited substring refusal-marker classifier: a 0% block rate across all five threat categories, with an upper confidence bound below 5% and p = 1.00 against the no-defense baseline. The same adversarial variants scored by an LLM judge show 56–100% block rates (p < 0.01). The author reads this as alignment genuinely responding to explicit jailbreak framing while producing refusals

What carries the argument

The load-bearing instrument is a defense ablation lattice extended with an L5-no-regex endpoint — the live aligned backend with the nine-pattern input regex disabled, holding token-budget cap, rate limit, and output scrub fixed — so that any difference from the no-defense baseline isolates alignment alone. Verdicts pass through two parallel metrics: a conservative substring refusal-marker classifier as the primary read, and an LLM judge in shadow mode as a secondary semantic read. The disagreement between the two instruments is the paper's central observation; 95% confidence intervals and exact-test p-values on pairwise L5-versus-L0 comparisons carry the statistical claim.

Load-bearing premise

The main 0% result assumes the inherited substring refusal-marker list is sensitive enough to catch every refusal the aligned model produces; the paper's own judge-based numbers on the same responses show the list misses refusals the model does emit, so an under-sensitive list would make the null result a measurement artifact rather than evidence that alignment contributes nothing.

What would settle it

Score the released L5 run JSON with an independent semantic judge (human raters or a non-Gemini LLM) on exactly the response pairs the substring classifier marked 0% blocked; if substantially more than 5% of those responses are refusals, the claim that alignment adds zero coverage on natural-language probes fails. A cheaper check: enumerate the aligned model's actual refusal phrasings on the released probe set and see whether the inherited marker list covers them.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Defenders who evaluate coverage with substring refusal matching should not count model alignment as a safety net behind a regex blocklist: on natural-language harmful requests its measured contribution was zero.
  • The same layer does respond to explicit adversarial jailbreak framing, but that response is invisible to substring classifiers, so robust evaluation stacks need a semantic judge to observe it.
  • The earlier result that an active regex filter caps alignment now extends to the harder regime where probes are engineered to bypass the regex — at least under the primary substring metric.
  • Coverage attribution between stacked defense layers depends on both probe form and verdict metric, so single-number coverage claims for such stacks are incomplete.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The 56–100% judge-visible block rates are plausibly an upper bound on alignment's real contribution: variants were pre-generated and evaluated statically against the regex-free target, and the paper itself notes that adaptive attackers routinely exceed 90% bypass against defenses that claimed near-zero success under static evaluation.
  • Because the same model family generated the variants, served as the target, and acted as the judge, part of the semantic-block signal may be self-favorable evaluation; an independent judge from a different model family would quantify that share.
  • A directly testable extension: widen the substring refusal-marker list to include the refusal phrasings the LLM judge detects, re-score the released run artifacts, and measure how much of the 0% result is a marker-list artifact rather than a true absence of refusal.
  • The metric gap implies that published jailbreak-resistance numbers are comparable only when the verdict instrument is held fixed; a zero under substring matching and a high rate under a semantic judge can describe the same system.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces a fifth ablation endpoint, L5-no-regex, which is identical to the previously studied L4-real (Gemini-2.5-flash, token budget, rate limit, output scrub) except that the nine-pattern input regex is disabled. Using a locked corpus of N=45 adversarial probes in three sub-corpora, amplified by Gemini paraphrase and PAIR into about 1,555 probe-run pairs over five replications, the paper tests whether alignment adds a measurable block-rate lift over the no-defense L0 baseline. Under a substring refusal-marker classifier, the reported L5 block rate is 0% for all five OWASP categories, with Fisher p=1.00 and Wilson upper bounds stated as <5%, so H1 is refuted. A secondary result reports that PAIR variants show 56–100% block rates when scored by PAIR's internal LLM judge (Gemini-2.5-flash), which the paper interprets as metric-dependent divergence: alignment produces refusals detectable by an LLM judge but missed by substring matching. The sub-corpus differential prediction is not supported. The paper releases the locked corpus, mutation artifacts, and export scripts.

Significance. If fully supported, the paper would make a useful, narrowly scoped empirical contribution: a single-axis ablation isolating the alignment component from a co-deployed regex filter, with reproducible artifacts and a clear separation of generation target from evaluation target. The primary scoped null result under the substring classifier is a legitimate measurement, and the authors are candid about limitations (single backend, static mutation artifacts, same-family attacker/judge). The claimed central finding, however, is the metric-dependent divergence, and that claim is not established by the released tables: the comparison confounds mutation strategy with evaluation metric, and the LLM judge is the same model family as the target. The missing crossed measurement is essential before the abstract's 'refusals too nuanced for substring matching' can be accepted. The paper's strengths are its reproducibility discipline and the clarity of the ablation design; its weaknesses are concentrated in the interpretation of Table 6.

major comments (4)
  1. [§5.1, Table 6] The paper's central 'metric-dependent divergence' claim is not demonstrated because Table 6 changes the mutation strategy and the evaluation metric simultaneously. Gemini-paraphrase variants are scored only by the substring classifier and PAIR variants only by PAIR's internal LLM judge; the table note itself warns that 'results are not directly comparable across strategies,' yet §5.1 builds the central interpretation on exactly that comparison. The 0%-vs-56-100% gap could be due to the variant style rather than the metric: PAIR variants might contain verbatim refusal markers that the substring classifier would also flag, or the LLM judge might be lenient. Report the full 2x2: score PAIR variants with the substring classifier and the original/paraphrase probes with an LLM judge (ideally an independent judge). Until then, the abstract's claim that alignment 'produces refusals too nuanced f
  2. [§5.2, Table 6] The PAIR judge is Gemini-2.5-flash evaluating Gemini-2.5-flash. The paper acknowledges self-favorability concerns and cites [10], but the caveat is not carried into the main interpretation. Even after the metric/strategy confound is fixed, the PAIR block-rate rows cannot be read as evidence about alignment behavior unless the judge is validated against human labels or an independent model family. At minimum, report inter-rater agreement between the substring classifier and the LLM judge on the same responses (the paper states Cohen's kappa is deferred); without this, the 56-100% row is as likely to measure judge bias as alignment refusal.
  3. [§4.2, Appendix A.3, §5.2] The H1 refutation is instrument-limited. The refusal-marker list is inherited from prior work and is never validated for recall on Gemini-2.5-flash's actual refusal phrasing. The paper's own Table 6 shows the classifier and LLM judge disagree massively on PAIR variants (0% vs 56-100%), so the conservative direction of the classifier is not merely theoretical. A null under a classifier that is conceded to undercount refusals establishes only that no marker-list verbatim refusal was produced, not that alignment adds zero observed coverage on natural-language harmful-request probes. Provide a validation set of Gemini refusals with marker-list recall, or report shadow-judge verdicts and the substring classifier's verdicts on the same responses, so readers can assess the sensitivity of the primary metric.
  4. [Table 4, §4.2] The sample-size basis for the Wilson upper bounds is unclear. With the reported per-category base counts (LLM10 has 3 probes, LLM01 has 27), a 0/n Wilson upper bound is below 5% only if the denominator includes variant-amplified probe-run pairs; for LLM10 with 3 original probes and 5 replications the upper bound would be about 20%, not the reported [0,4]. The table caption does not state whether the rates are over original probes, variant-amplified pairs, or all runs, nor which mutation variants are included. Because the '<5% upper bound for every category' is the quantitative basis for refuting H1, the manuscript must state the exact denominator for each cell in Tables 4 and 5.
minor comments (6)
  1. [§4.1, Table 4] The carry-forward invariant uses N=210 probe-run pairs for L0 on the carry-forward sub-corpus, while Table 4's L0 column appears to cover the full corpus with variant amplification. Clarify which runs enter the invariant check and which enter the per-OWASP denominators.
  2. [§4.5 / §5.1] The Table 6 metric note that 'results are not directly comparable across strategies' should be repeated in §5.1 at the point of interpretation, not only as a footnote to the table.
  3. [§5.1] The sentence 'This duality explains why the main isolation result shows 0%' is post-hoc: the paper presents no within-probe mediation analysis linking PAIR framing to refusal phrasing. Soften or support.
  4. [Appendix C] The Wilson interval formula is presented without an equation number; adding one would make the cross-references in §3.4 cleaner.
  5. [§3.3, References] There is a duplicated 'the' in 'following the the prior brittleness protocol,' and the URLs for references [8] and [9] break across lines in the rendered text. Please fix.
  6. [Figure 2] The dashed reference line is labeled 'same metric as Table 4,' but Table 6's own note says the strategies are not directly comparable. Consider relabeling the reference to avoid reinforcing the confounded comparison.

Circularity Check

0 steps flagged

No construction-level circularity: the primary 0% result is a fresh empirical ablation; the flagged Gemini-judges-Gemini confound is a validity threat, not a circular derivation.

full rationale

The load-bearing H1 result (L5-no-regex block rate 0% vs. L0 under the substring classifier) is a new measurement on a locked, sha256-verified corpus (llm-mutation-locked-2026-06-03) with a deterministic classifier; no parameter is fitted to the outcome and no equation defines L5's block rate in terms of the conclusion. The inherited refusal-marker list is a sensitivity limitation, not circularity: the paper concedes "the classifier is conservative — false negatives (missed blocks) produce underestimates of alignment's effectiveness, so reported block rates are lower bounds." The secondary 56–100% PAIR result is the most self-referential part: PAIR variants are evaluated by "PAIR's internal LLM judge (Gemini-2.5-flash)" while the target is the same model family, and Table 6 itself warns "results are not directly comparable across strategies." The paper also acknowledges self-favorability [10] and states that an independent judge from a different model family is required. This confounds mutation strategy with metric and target with judge, so the "metric-dependent divergence" inference is weaker than claimed; but the flaw is a missing crossed measurement, not a reduction of the claim to its own inputs. Self-citations to [8,9] supply the lattice, protocol, and carry-forward probes, but the L5-vs-L0 comparison is independently measured, externally reproducible from released artifacts, and does not depend on [9]'s conclusions for its primary statistical content. Accordingly, no circular step rises above a minor, non-load-bearing self-citation pattern.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

The central claim has no free parameters fitted to data; the listed items are hand-chosen experimental settings that materially affect the measured block rates. The most consequential axiom is the under-sensitivity of the substring classifier, which the paper itself demonstrates is violated. The same-family judge is a domain assumption, not a fitted parameter.

free parameters (5)
  • LLM temperature = 0.7
    Hand-chosen to match prior work [9]; refusal rates may vary with temperature, so all block-rate estimates are conditional on this setting.
  • PAIR rounds = 5
    Static pre-generation budget, hand-chosen as conservative; more rounds could produce harder variants and lower measured alignment block rates.
  • TAP branches/depth = K=3, depth=2
    Reduced from K=10/depth=3 for API budget; limits search and makes measured block rates upper bounds.
  • Variant-generation seed = 20260603
    Fixed seed pins the committed static artifacts; different seeds would produce different variant distributions.
  • Substring refusal-marker list = not disclosed
    Inherited from [9]; the entire 0% result depends on this exact list, which is not reproduced in the paper.
axioms (5)
  • domain assumption The substring classifier's refusal markers detect all refusals relevant to H1
    Section 3.4 defines blocked as refusal marker appearing verbatim; Table 6 shows PAIR variants with 56–100% LLM-judge refusals are 0% by this classifier, so the assumption is violated for the secondary result.
  • domain assumption Gemini-2.5-flash can serve as a valid judge of Gemini-2.5-flash refusals
    Section 5.2 cites self-favorability bias [10]; the secondary 56–100% block rates rest on this judge without a cross-family or human baseline.
  • domain assumption Static pre-generated PAIR/TAP artifacts give an upper bound on alignment block rate
    Section 3.3 assumes a dynamic attacker would find more bypasses, citing Nasr et al. [13]; the direction is plausible but not measured in this paper.
  • domain assumption L0-naive stub feedback produces diverse PAIR variants without biasing evaluation
    Section 3.3 chooses L0 as generation target to avoid circular dependency; this assumes resulting variants are representative of the bypass surface.
  • standard math Probe-run pairs are independent Bernoulli trials for Wilson/Fisher inference
    Appendix C uses Wilson CI and Fisher exact test; repeated calls to the same 45 probes are treated as independent trials, which is valid for stochastic output per call but not for prompt-population generalization.

pith-pipeline@v1.3.0-alltime-deepseek · 15834 in / 12939 out tokens · 134921 ms · 2026-08-02T11:22:06.020648+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Isolating LLM Alignment from Regex: Zero Coverage and Metric-Dependent Divergence Under Adversarial Mutation." pith.science (2026). https://pith.science/paper/LSX3UJK7

@misc{pith2026260720494,
  author       = {Pith},
  title        = {Pith review of: Isolating LLM Alignment from Regex: Zero Coverage and Metric-Dependent Divergence Under Adversarial Mutation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LSX3UJK7}},
  note         = {Machine review of arXiv:2607.20494}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Production LLM applications commonly stack a regex filter in front of model-side alignment; prior work found no measurable coverage gain from adding a live Gemini backend behind an active regex filter. We ask whether that ceiling holds when the corpus is \emph{designed to bypass the regex}. We introduce $L_5$-no-regex -- identical to $L_4$-real (Gemini-2.5-flash, token-budget cap, rate limit, output scrub) but with the nine-pattern filter disabled -- and evaluate it against $N{=}45$ adversarial probes across three sub-corpora (carry-forward, regex-bypass, alignment-isolate), amplified by Gemini paraphrase and PAIR to ${\sim}1{,}555$ probe-run pairs over $N{=}5$ replications. Under the primary substring classifier, H1 is refuted: $L_5$ block rate is $0,%$ across all five OWASP LLM Top-10 categories ($\Delta\text{pp}{=}0$ vs.\ $L_0$, $p{=}1.00$; Wilson upper bound ${<}5,%$). A secondary LLM-judge metric on PAIR variants shows $56$--$100,%$ block rates ($p{<}0.01$), revealing alignment does respond to adversarially-framed probes -- but produces refusals too nuanced for substring matching. The sub-corpus differential prediction is not supported ($p{=}1.00$). Alignment's contribution is \emph{metric-dependent}: on natural-language harmful-request probes, it adds zero observed coverage beyond the regex; on adversarially-framed variants, an LLM judge detects refusals the substring classifier misses. The locked corpus, mutation artifacts, and export scripts are released for replication.

Figures

Figures reproduced from arXiv: 2607.20494 by Alexandre Cristov\~ao Maiorano.

Figure 1
Figure 1. Figure 1: Single-axis ablation: L5-no-regex vs. L4- real. The only difference is the input filter: active in L4 (red dashed = probes blocked there), disabled in L5 (probes reach alignment directly). Budget, rate limit, and Gemini backend are identical. 3 Methodology 3.1 Ablation lattice prior work [9] introduced a defense lattice: a set of synthetic HTTP endpoints that each implement a dif￾ferent combination of defe… view at source ↗
Figure 2
Figure 2. Figure 2: PAIR-variant block rate on L5-no-regex per OWASP category, measured with PAIR’s internal LLM judge (Gemini-2.5-flash). Error bars show Wilson 95 % confidence intervals (asymmetric). ‡ = p<0.01 vs. original probes (Fisher exact). The dashed reference marks 0 % block rate for Gemini-paraphrase variants (substring classifier, same metric as [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

25 extracted references · 11 linked inside Pith

  1. [1]

    Attackiq continuous threat exposure management platform, 2025

    AttackIQ. Attackiq continuous threat exposure management platform, 2025. URL https://at tackiq.com/platform/

    2025

  2. [2]

    Pappas, Florian Tram` er, Hamed Hassani, and Eric Wong

    Patrick Chao, Edoardo Debenedetti, Alexan- der Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J. Pappas, Florian Tram` er, Hamed Hassani, and Eric Wong. JailbreakBench: An open robustness benchmark for jailbreak- ing large language models, 2024. URL https: //arxiv.org/abs/2404.01318 . NeurIPS 2024 ...

    Pith/arXiv arXiv 2024

  3. [3]

    Pappas, and Eric Wong

    Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J. Pappas, and Eric Wong. Jailbreaking black box large language models in twenty queries, 2024. URL https: //arxiv.org/abs/2310.08419

    Pith/arXiv arXiv 2024

  4. [4]

    StruQ: Defending against prompt injection with structured queries, 2024

    Sizhe Chen, Julien Piet, Chawin Sitawarin, and David Wagner. StruQ: Defending against prompt injection with structured queries, 2024. URL https://arxiv.org/abs/2402.06363

    Pith/arXiv arXiv 2024

  5. [5]

    SecAlign: Defending against prompt injection with preference opti- mization, 2024

    Sizhe Chen, Arman Zharmagambetov, Saeed Mahloujifar, Kamalika Chaudhuri, David Wag- ner, and Chuan Guo. SecAlign: Defending against prompt injection with preference opti- mization, 2024. URL https://arxiv.org/abs/ 2410.05451. 11

    Pith/arXiv arXiv 2024

  6. [6]

    Wichmann

    Robert Geirhos, J¨ orn-Henrik Jacobsen, Clau- dio Michaelis, Richard Zemel, Wieland Bren- del, Matthias Bethge, and Felix A. Wichmann. Shortcut learning in deep neural networks.Na- ture Machine Intelligence, 2:665–673, 2020. doi: 10.1038/s42256-020-00257-z

    doi:10.1038/s42256-020-00257-z 2020

  7. [7]

    AutoDAN: Generating stealthy jailbreak prompts on aligned large language mod- els

    Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. AutoDAN: Generating stealthy jailbreak prompts on aligned large language mod- els. InICLR, 2024. URL https://arxiv.org/ abs/2310.04451

    Pith/arXiv arXiv 2024

  8. [8]

    Isolating LLM Alignment from Regex

    Alexandre Cristov˜ ao Maiorano. llm-alignment- isolation: Replication package for “Isolating LLM Alignment from Regex”, 2026. URL https://gi thub.com/alemaiorano/llm-alignment-iso lation

    2026

  9. [9]

    Which defense closes which threat? attributing OW ASP-LLM- Top-10 coverage and its brittleness under para- phrasing, 2026

    Alexandre Cristov˜ ao Maiorano. Which defense closes which threat? attributing OW ASP-LLM- Top-10 coverage and its brittleness under para- phrasing, 2026. URL https://arxiv.org/abs/ 2606.02822

    Pith/arXiv arXiv 2026

  10. [10]

    The judge who never admits: Hidden shortcuts in LLM-based evaluation, 2026

    Arash Marioriyad, Omid Ghahroodi, Ehsaned- din Asgari, Mohammad Hossein Rohban, and Mahdieh Soleymani Baghshah. The judge who never admits: Hidden shortcuts in LLM-based evaluation, 2026. URL https://arxiv.org/ab s/2602.07996. LLM judges (including Gemini- 2.0-Flash) exhibit hidden biases – source hier- archy, recency – that drive verdict shifts with- out...

    arXiv 2026

  11. [11]

    HarmBench: A standardized evaluation framework for auto- mated red teaming and robust refusal

    Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendrycks. HarmBench: A standardized evaluation framework for auto- mated red teaming and robust refusal. InICML,

  12. [12]

    Tree of at- tacks: Jailbreaking black-box LLMs automati- cally.NeurIPS, 2024

    Anay Mehrotra, Manolis Zampetakis, Paul Kassianik, Blaine Nelson, Hyrum Anderson, Yaron Singer, and Amin Karbasi. Tree of at- tacks: Jailbreaking black-box LLMs automati- cally.NeurIPS, 2024. URL https://arxiv.or g/abs/2312.02119

    Pith/arXiv arXiv 2024

  13. [13]

    Schulhoff, Jamie Hayes, Michael Ilie, Juliette Pluto, Shuang Song, Harsh Chaudhari, Ilia Shumailov, Abhradeep Thakurta, Kai Yuan- qing Xiao, Andreas Terzis, and Florian Tram` er

    Milad Nasr, Nicholas Carlini, Chawin Sitawarin, Sander V. Schulhoff, Jamie Hayes, Michael Ilie, Juliette Pluto, Shuang Song, Harsh Chaudhari, Ilia Shumailov, Abhradeep Thakurta, Kai Yuan- qing Xiao, Andreas Terzis, and Florian Tram` er. The attacker moves second: Stronger adaptive attacks bypass defenses against LLM jailbreaks and prompt injections, 2025....

    Pith/arXiv arXiv 2025

  14. [14]

    Owasp top 10 for large language model applications, 2025 edition, 2025

    OW ASP GenAI Security Project. Owasp top 10 for large language model applications, 2025 edition, 2025. URL https://genai.owasp.or g/llm-top-10/

    2025

  15. [15]

    Ignore previous prompt: Attack techniques for language models

    F´ abio Perez and Ian Ribeiro. Ignore previous prompt: Attack techniques for language models. InNeurIPS 2022 Workshop on Machine Learning Safety, 2022. URL https://arxiv.org/abs/22 11.09527

    2022

  16. [16]

    Picus security cybersecurity glos- sary, 2025

    Picus Security. Picus security cybersecurity glos- sary, 2025. URL https://www.picussecurity. com/resource/glossary/

    2025

  17. [17]

    Safebreach validate — breach and attack simulation, 2025

    SafeBreach. Safebreach validate — breach and attack simulation, 2025. URL https://www.sa febreach.com/validate-breach-and-attac k-simulation/

    2025

  18. [18]

    A StrongREJECT for empty jailbreaks, 2024

    Alexandra Souly, Qingyuan Lu, Dillon Bowen, Tu Trinh, Elvis Hsieh, Sana Pandey, Pieter Abbeel, Justin Svegliato, Scott Emmons, Olivia Watkins, and Sam Toyer. A StrongREJECT for empty jailbreaks, 2024. URL https://arxiv.or g/abs/2402.10260 . Introduced the StrongRE- JECT benchmark; shows substring-based ASR metrics systematically overstate jailbreak effec-...

    Pith/arXiv arXiv 2024

  19. [19]

    Jailbroken: How does llm safety training fail?NeurIPS, 2023

    Alexander Wei, Nika Haghtalab, and Jacob Stein- hardt. Jailbroken: How does llm safety training fail?NeurIPS, 2023. URL https://arxiv.or g/abs/2307.02483

    Pith/arXiv arXiv 2023

  20. [21]

    Zico Kolter, and Matt Fredrik- son

    Andy Zou, Zifan Wang, Nicholas Carlini, Mi- lad Nasr, J. Zico Kolter, and Matt Fredrik- son. Universal and transferable adversarial at- 12 tacks on aligned language models, 2023. URL https://arxiv.org/abs/2307.15043. A Replication Contract This appendix specifies the interface sufficient for in- dependent replication. Engine source code is withheld; the c...

    Pith/arXiv arXiv 2023

  21. [23]

    An author listed the nine JAILBREAK PATTERNS regex from theL 3/L4 server code

  22. [24]

    For each evasion technique category (encoding, obfuscation, framing, structure), a probe was crafted that (a) preserves the OW ASP category’s attack intent and (b) contains no string matching any of the nine patterns

  23. [25]

    Each probe was mechanically verified against the exact regex list before inclusion: a probe that triggered any pattern was revised until it did not

  24. [26]

    Alignment-isolate probes( N= 13) followed the same process but with an additional criterion: the probe must not useanyencoding, obfuscation, or structural framing technique

    Coverage was balanced across OW ASP categories to avoid over-representing LLM01; see the by- category counts in the corpustotalsfield. Alignment-isolate probes( N= 13) followed the same process but with an additional criterion: the probe must not useanyencoding, obfuscation, or structural framing technique. Every probe was re- viewed in a separate author ...

  25. [2024]

    URL https://arxiv.org/abs/2402.042 49