{"id":"d84e89ad-3908-42ba-b318-6257afc799cd","arxiv_id":"2502.00580","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"An iterative LLM-based prompt evaluator blocked 100% of the Best-of-N jailbreaking paper's released successful prompts and 99.8% of a fresh replication, with false-positive rates near zero.","lead":"This paper tests a defense that uses a second LLM to repeatedly judge whether a user prompt is dangerous or a jailbreak attempt. On the Best-of-N jailbreaking paper's released attack prompts, the defense blocked 100% of them, and on the authors' own replication, 99.8%. The result is a cheap, deployable text-safety layer, though it was measured on static attack lists rather than against adaptive attackers.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported 99.8–100% blocking rates depend on excluding non-conforming evaluator outputs; the number and danger-correlation of those exclusions are unreported, so the headline estimates may not reflect any deployable policy.","rationale":"The reader's conditional verdict is sound. My stress-test focuses on the same weakest link and sharpens it. The paper's central numeric claims are internally consistent with the appendix tables; the method is simple and the code is released, which supports reproducibility. However, Section 2.1.3 defines blocking only over conforming responses. Because the scoring rule is asymmetric and the paper does not report exclusion counts, the headline rates may not correspond to any actual deployment policy: exclusions are neither fail-closed nor fail-open. This is not an accusation of selective reporting; it is a missing measurement that determines whether the claimed 100%/99.8% is an estimate of the method or an artifact of preprocessing. The concrete test above would settle it. If the rates are robust to the two extreme codings, the paper's central claim is well supported for static BoN corpora; if not, the paper should be revised to describe the deployment policy for malformed outputs and to report the exclusion counts. I do not see a need to change the verdict from conditional.","tokens_in":10044,"tokens_out":7486,"duration_ms":84273,"concrete_test":"Rerun the DATDP evaluation on the BoN Paper (1,045 prompts), Replicated Jailbreaks (1,000), and Normal (250) datasets, logging for each prompt the counts of yes/no/nonconforming evaluations. Then recompute the headline rates under three policies: (i) exclude nonconforming (as in the paper), (ii) treat every nonconforming output as 'yes' (fail-closed), (iii) treat every nonconforming output as 'no' (fail-open). Report the number of prompts with zero conforming outputs in each dataset. If the 99.8–100% blocking rates and the 0–0.4% normal-prompt false-positive rates change by more than ~1 percentage point under (ii) or (iii), the central claim must be revised to state the deployment policy and exclusion counts; if they do not, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing unverified assumption is in Section 2.1.3: DATDP's blocking decision uses only evaluator responses that end with 'yes' or 'no'; all other responses are 'excluded.' The paper reports no count of excluded responses, no per-prompt distribution, and no analysis of whether dangerous prompts are over-represented among nonconforming outputs. This matters because the scoring rule is asymmetric: with N=5 (Claude), two 'yes' and three 'no' blocks, but one 'yes' and four 'no' passes; with exclusions, a prompt with only one conforming 'yes' and zero conforming 'no's' is credited as blocked (+2). If dangerous prompts trigger malformed responses at higher rates than normal prompts—plausible, since refusals and hedging about harmful content are likely to be nonconforming—the 100%/99.8% blocking rates and the 0–0.4% false-positive rates are not estimates of any single deployment policy. A deployment must either fail closed (block on malformed output, raising false positives) or fail open (pass, lowering blocking); the paper's 'exclude' policy is only an evaluation artifact. Thus the headline claim that DATDP blocks these jailbreaks is not yet cleanly measured.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes DATDP, a pre-prompt evaluation defense in which a separate LLM judges whether a user prompt is dangerous or a jailbreak attempt, using N repeated evaluations and a weighted vote (yes = +2, no = -1) to decide whether to block the prompt before it reaches the responding model. DATDP is evaluated on six text datasets: the 159 HarmBench harmful prompts, the 1045 successful jailbreaks released by the Best-of-N paper, 1000 successful and 1000 unsuccessful jailbreaks from the authors' replication of Best-of-N, 1590 random augmented prompts, and 250 normal prompts. Using Claude 3.5 Sonnet or LLaMa-3-8B-instruct as the evaluator, the paper reports blocking rates between 99.5% and 100% on harmful augmented prompts, 0% and 0.4% false positives on normal prompts, exact Clopper-Pearson confidence intervals, and a sensitivity analysis over the number of iterations N. The authors conclude that evaluation agents make BoN jailbreaking roughly 500 times harder and that small models are nearly as effective evaluators as frontier models.","tokens_in":10328,"tokens_out":9044,"duration_ms":83892,"significance":"If the reported numbers hold, the paper offers a simple, cheap, and model-agnostic defense against a currently effective class of text jailbreaks. The evaluation has genuine strengths: it uses external corpora (HarmBench and the BoN paper's released jailbreaks), fixed hyperparameters, exact binomial confidence intervals, open-source code, and a replication of the attack. The finding that a small open-weight model can serve as a near-equivalent evaluator is practically important. The main reservations are methodological rather than conceptual: the headline blocking rates are computed under an exclusion rule for non-conforming evaluator outputs whose effect is unquantified, and the '500x harder' conclusion extrapolates from static-corpus blocking to an adaptive attacker. Both issues are addressable in revision.","major_comments":[{"comment":"The exclusion of evaluator responses that do not end in 'yes' or 'no' is not quantified, and it can change the reported blocking rates. The paper states that this was 'an issue with LLaMa-3-8B-instruct' but gives no counts per dataset or per prompt. Because the voting rule is asymmetric (a 'yes' contributes +2, a 'no' contributes -1) and the threshold depends on N, dropping non-conforming responses is not a neutral operation: for N=5, one conforming 'yes' and four non-conforming responses yields a positive sum and a block, whereas four conforming 'no' responses would make the prompt pass. If dangerous prompts are more likely than benign prompts to produce malformed or hedging outputs, the headline 100% and 99.8% blocking rates and the 0-0.4% false-positive rates do not correspond to any single deployable policy. Please report the number and distribution of excluded responses, and rerun the main tables under fail-closed (block on malformed output) and fail-open (pass on malformed output) assumptions, or justify why exclusions are uninformative.","section":"Section 2.1.3 and Table 2"},{"comment":"The claim that DATDP makes BoN '500 times or so harder' assumes a constant per-attempt blocking probability and a non-adaptive attacker. The experiments measure blocking on a fixed corpus of prompts that a static BoN run produced; they do not measure end-to-end robustness against an attacker who can adapt, for example by sampling augmentations conditioned on passing the evaluator, by increasing N, or by targeting the evaluator's blind spots. The power-law argument in Section 3.1 therefore goes beyond the data. Please either evaluate DATDP in a filter-aware adaptive attack loop (e.g., BoN with the evaluator as an oracle) or restate the conclusion as 'blocked a fixed corpus of sampled BoN attempts.'","section":"Section 3.1"}],"minor_comments":[{"comment":"The example 'one yes and four no would give a weighted sum of 1 × 2 − 4 × −1 = −2' is arithmetically wrong; the expression evaluates to +6. The intended calculation is 1×2 + 4×(−1) = −2.","section":"Section 2.1.3"},{"comment":"The caption reads 'DATDP on Datasets of Augmented Prompts' but the figure displays the non-augmented HarmBench and Normal datasets; rename it to 'Non-augmented Prompts' or similar.","section":"Figure 4 caption"},{"comment":"There are minor typos: 'were were labeled' in Section 2.1.3, 'A vailability' in Section 1.2, and 'This study aim to assess' in Section 1.1.","section":"Throughout"},{"comment":"The decision rule for a zero weighted sum is unspecified; please state whether a sum of zero blocks or passes the prompt.","section":"Section 2.1.3"},{"comment":"The normal-prompt corpus is only 250 prompts, so the false-positive estimate is imprecise; for the Claude evaluator the 95% CI upper bound is 1.46%, so 'near-zero false positives' should be presented with that sample-size caveat.","section":"Section 3.2 and Table 2"},{"comment":"The appendix lists prompts that were not blocked, which is helpful; making the raw evaluator outputs or at least the excluded-response counts available would further support reproducibility.","section":"Appendix A"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within scope for a security or AI-safety venue. The method builds directly on the authors' own prior work ([18,19]), and that self-citation is disclosed; the incremental contribution is the systematic evaluation against Best-of-N jailbreaks. No concerns about citation practice beyond asking the authors to clarify explicitly what is new relative to [18,19]."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: this is a legitimate empirical result, not a hype paper. The evaluation-agent idea comes from the authors' own 2022 work, but the BoN-specific measurement is new. They tested against external corpora (HarmBench, the BoN paper's released jailbreaks), ran their own replication of BoN, and used fixed hyperparameters with a sensitivity analysis for N. The finding that an 8B LLaMa model matches Claude as an evaluator is the most practically useful part. They ship code, and the confidence intervals are computed correctly. That is real work.\n\nThe soft spots are real but not disqualifying. The biggest is Section 2.1.3: responses that do not end in 'yes' or 'no' are excluded, and the paper never says how many were excluded or whether dangerous prompts were over-represented among them. The stress-test note is right that this matters. The scoring is asymmetric—one 'yes' blocks, and it takes four 'no's to overcome one 'yes'—so if harmful prompts trigger malformed evaluator outputs more often than benign ones (plausible, since refusals often hedge), the reported 99.8–100% blocking rates overstate what a deployable policy would achieve. A deployment must either fail closed (block malformed outputs, raising false positives) or fail open (pass them, lowering blocking). The paper's 'exclude' rule is an evaluation artifact, not a policy. The authors should report the counts and redo the analysis under both closure rules.\n\nThe '500 times or so harder' extrapolation is also hand-wavy. It assumes the filter multiplies attack success probability by a constant, but there is no evidence for that. The evaluation is static—no adaptive attacker that optimizes against DATDP. That is a limitation worth stating clearly, not a fatal one.\n\nThis paper is for practitioners who want a cheap, understandable filter to add in front of an LLM, and for safety researchers studying input filtering. It deserves a serious referee. I would send it to peer review, but with a request that the authors add exclusion counts, test both fail-open and fail-closed interpretations, and temper the '500 times' claim. With those changes, the central claim would be solid.","headline":"A cheap, open-source prompt-evaluation filter looks genuinely effective against Best-of-N text jailbreaks, but the headline rates depend on an unreported exclusion policy that any deployment must resolve.","tokens_in":10853,"tokens_out":1596,"would_cite":true,"duration_ms":18907,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Repeatedly asking an LLM to judge a prompt blocks essentially all Best-of-N jailbreaks, including 100% of the attack paper's published successful jailbreak prompts.","keywords":["LLM jailbreaking","Best-of-N jailbreaking","prompt evaluation agent","iterative safety evaluation","adversarial robustness","defensive AI","LLM safety","weighted scoring"],"falsifier":"Re-run the DATDP evaluation on the BoN paper's 1,045 jailbreak prompts and the replication's 1,000 jailbreaking prompts, but treat every malformed evaluator response as an acceptance ('no') instead of excluding it; if the fraction blocked drops below 99%, the exclusion rule is carrying the result.","tokens_in":9846,"feed_emoji":"🛡️","tokens_out":6320,"duration_ms":57443,"temperature":0.7,"pith_summary":"This paper claims that repeatedly asking an evaluation language model to judge whether a user prompt is dangerous or a jailbreak attempt can block nearly all Best-of-N jailbreaks. Across six datasets, the Defense Against the Dark Prompts (DATDP) method blocked 100% of the 1,045 successful jailbreak prompts published with the Best-of-N paper and 99.8% of 1,000 jailbreaking prompts from the authors' own replication, with only one false positive among 250 normal prompts. The same result held when the evaluation was powered by the small LLaMa-3-8B-instruct model instead of Claude. If this holds in practice, a cheap, static prompt filter could substantially raise the cost of this class of attack.","feed_headline":"Prompt evaluator blocks 100% of Best-of-N jailbreaks","feed_subtitle":"Iterating a small LLM as judge catches nearly all augmented attacks with almost no false positives on normal prompts.","key_machinery":"The carrying object is the DATDP evaluation agent: a separate LLM given a system prompt that defines the forbidden task as 'anything that could be dangerous for an LLM to answer or could be intended to trick or jailbreak the LLM,' instructed to reason step-by-step and end with 'yes' or 'no.' Each prompt is evaluated $N$ times, responses not ending in 'yes' or 'no' are discarded, and a weighted sum (yes $=+2$, no $=-1$) decides block vs. pass. This repeated, weighted evaluation is what lets a small model detect augmented prompts that would jailbreak the same model in a responding role.","core_discovery":"The central discovery is that the same models that are vulnerable to Best-of-N jailbreaking when answering can reliably judge whether a prompt is dangerous when placed in an evaluation role. DATDP does this by taking the submitted prompt, asking an evaluation LLM to reason step by step and end with 'yes' or 'no' about whether the prompt asks for something dangerous or tries to jailbreak, repeating this $N$ times ($N=25$ for LLaMa-3-8B-instruct, $N=5$ for Claude), and using a weighted vote in which a 'yes' rejection is weighted $+2$ and a 'no' acceptance is weighted $-1$. This procedure blocked all 1,045 BoN-paper jailbreaks, 99.8% of the replication jailbreaks, and 99.94% of generic augmented prompts, while accepting virtually all normal prompts. The paper argues this should multiply the cost of BoN attacks by roughly 500.","pith_inferences":["If these blocking rates transfer to production traffic, DATDP-style evaluation could become a standard upstream layer for LLM APIs, trading extra inference cost for a large reduction in successful BoN-style attacks without retraining the responder.","A natural next test is to run DATDP against a BoN attack whose augmentations are chosen adversarially to fool the evaluator rather than sampled randomly; the paper's tension argument predicts lower but still substantial blocking.","An unbiased estimate of LLaMa's true blocking rate requires accounting for the excluded non-'yes'/'no' responses; if dangerous prompts were more likely to produce malformed outputs, the reported 99.8% would be optimistic."],"forward_implications":["The cost of Best-of-N jailbreaking rises sharply: the paper estimates the defense multiplies the number of attempts needed by roughly 500, since it blocks about 99.7% of augmented prompts at the prompt level.","Small models can guard larger models: LLaMa-3-8B-instruct as an evaluator blocks 99.8% of jailbreaks of itself, suggesting cheap defenses are feasible without frontier-grade evaluators.","Prompt-level defense is additive: DATDP can be placed upstream of any responding LLM without retraining that model, complementing its internal safety mechanisms.","Response-level evaluation adds a second layer: although a LLaMa-based response evaluator blocked only 76% of harmful responses in the paper's initial test, it caught all three prompts that slipped past the prompt-level agent, so a dual-layer design looks promising."],"supporting_citations":[{"why":"Supplies the Best-of-N jailbreak method and the 1,045 successful jailbreak prompts that DATDP is tested against.","marker":"[16]"},{"why":"Provides the 159 harmful base prompts and the HarmBench auto-grader used to classify the replication's jailbreak successes and failures.","marker":"[20]"},{"why":"Describes the earlier evaluation-agent approach that DATDP builds on for prompt safety assessment.","marker":"[18]"},{"why":"Releases the earlier prompt-evaluator code that DATDP extends with iteration and weighted scoring.","marker":"[19]"},{"why":"Gives the exact binomial confidence-interval method used to report the blocking rates.","marker":"[21]"},{"why":"Supports the design rationale that small, trusted models can monitor larger models, motivating the use of LLaMa-3-8B-instruct as an evaluator.","marker":"[22]"}],"fun_headline_variants":["LLM judge blocks every Best-of-N jailbreak","Repeated prompt screening stops all BoN attacks","Small evaluator LLM catches 100% of BoN jailbreaks","Prompt evaluation thwarts all BoN jailbreak attempts","One evaluation LLM defeats all BoN jailbreaks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported blocking rates assume that excluding evaluation responses that do not end in 'yes' or 'no'—which happened with LLaMa-3-8B-instruct—does not systematically remove dangerous prompts from the denominator.","fun_headline_variants_meta":{"raw":{"variants":["LLM judge blocks every Best-of-N jailbreak","Repeated prompt screening stops all BoN attacks","Small evaluator LLM catches 100% of BoN jailbreaks","Prompt evaluation thwarts all BoN jailbreak attempts","One evaluation LLM defeats all BoN jailbreaks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000307,"raw_usage":{"total_tokens":1772,"prompt_tokens":972,"completion_tokens":800,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":588,"completion_tokens_details":{"reasoning_tokens":721}},"tokens_in":588,"tokens_out":800,"duration_ms":32160,"temperature":1.0,"reasoning_tokens":721,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T18:25:51.341976+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the DATDP evaluation on the BoN paper's 1,045 jailbreak prompts and the replication's 1,000 jailbreaking prompts, but treat every malformed evaluator response as an acceptance ('no') instead of excluding it; if the fraction blocked drops below 99%, the exclusion rule is carrying the result.","supporting_citations":[{"cited_title":"Using gpt-eliezer against chatgpt jailbreaking, 2022","cited_arxiv_id":null,"evidence_quote":"Describes the earlier evaluation-agent approach that DATDP builds on for prompt safety assessment."},{"cited_title":"chatgpt-prompt-evaluator on aligned ai’s github, 2022","cited_arxiv_id":null,"evidence_quote":"Releases the earlier prompt-evaluator code that DATDP extends with iteration and weighted scoring."},{"cited_title":"The use of confidence or fiducial limits illustrated in the case of the binomial.Biometrika, 26(4):404–413, 1934","cited_arxiv_id":null,"evidence_quote":"Gives the exact binomial confidence-interval method used to report the blocking rates."}],"review_version":1}