{"id":"504f001a-fbfe-43ca-88df-1366e7007957","arxiv_id":"2508.20848","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"JADES judges jailbreak success by decomposing harmful prompts into weighted sub-questions and scoring each part, claiming 98.5% human agreement and showing prior attack success rates are inflated.","lead":"Researchers present JADES, an evaluation framework that determines if a jailbreak response truly completes a harmful request by splitting the request into scored, weighted sub-questions. They report 98.5% agreement with human raters on a new 400-item benchmark and argue that prior jailbreak success rates were inflated, with one attack on GPT-3.5-Turbo falling from 93% to 69%.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Human gold standard was annotated with a decompositional rubric; 98.5% agreement may measure self-consistency, not superiority over holistic judgment.","rationale":"The paper's central claim is that JADES is a more accurate jailbreak evaluator than existing holistic methods, and that prior ASRs are inflated. For that claim to hold, the human labels used as ground truth must represent an independent judgment of success. Appendix D.2's annotation guidance, however, invites annotators to decompose questions into weighted sub-questions and aggregate—the exact procedure JADES automates. This does not make the evaluation internally inconsistent, but it makes the headline 98.5% agreement partly a self-consistency measure. The paper does not report the split between decompositional and holistic annotations, so the magnitude of the effect is unknown. The reader's conditional verdict captures this correctly. I considered other potential objections—lack of confidence intervals, GPT-4.1 inconsistency, threshold sensitivity, extrapolation to the re-evaluation set—but they are secondary: even if addressed, the benchmark validity issue would remain. The fact-checking extension study on HarmfulQA is a genuinely more objective validation, and the audit-trail case study is helpful, but neither validates the decompositional scoring core against holistic human perception. The proposed re-annotation test is feasible and would settle the concern; until then, the conditional verdict stands.","tokens_in":23053,"tokens_out":8665,"duration_ms":88935,"concrete_test":"Re-annotate a random sample of 150 JailbreakQR pairs with fresh annotators using a purely holistic protocol (no sub-question decomposition; read question and response, label failed/partial/successful). Compute JADES's binary and ternary agreement against these holistic labels and compare with the same baselines. If JADES's agreement drops materially (e.g., >5 points) or its margin over holistic baselines shrinks below the claimed 9%, the headline validation is rubric-dependent. Additionally, report the fraction of original annotations produced via Case A vs Case B if that was logged.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing concern is that the human gold standard used to validate JADES was itself elicited with a decompositional rubric. Appendix D.2 instructs annotators to 'break the question into sub-questions,' mark critical/supporting, score each sub-question, then weigh and aggregate—essentially the JADES mechanism. Although Case B permits a holistic path, the guidance makes decomposition the default, and the paper does not report how many annotations used Case A vs Case B, nor agreement broken down by path. Thus the reported 98.5% agreement may largely measure JADES's fidelity to the very rubric it automates, not its superiority over independent holistic human judgment. Since the binary mapping and ASR re-evaluation inherit these labels, the 'overestimation' conclusion (e.g., 93% vs 69%) is also contingent on this rubric. The HarmfulQA fact-checking study is more objective, but it validates only the extension, not the decompositional core.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes JADES, a multi-agent LLM-based evaluation framework for jailbreak attempts. JADES decomposes a harmful question into weighted sub-questions, cleans and pairs sentences from the jailbreak response, scores each sub-answer on a five-level Likert scale, and aggregates sub-scores into an overall verdict. An optional fact-checking extension splits responses into unit facts and verifies them against web-sourced Wikipedia content to penalize hallucinations. The authors introduce JailbreakQR, a 400-pair human-annotated benchmark, and report 98.5% binary and 86.3% ternary agreement with human labels, outperforming several published baselines. Re-evaluating five attacks on four LLMs, they find prior attack success rates are substantially overestimated (e.g., LAA on GPT-3.5-Turbo drops from 93% to 69%) and that partial successes dominate binary attacks. They also introduce HarmfulQA and report that the fact-check extension raises accuracy from 0.85 to 0.97.","tokens_in":23267,"tokens_out":6270,"duration_ms":65859,"significance":"If the validation were independent, the contribution would be significant: JADES offers a transparent, interpretable, and decomposable evaluation metric, along with two new datasets and a credible demonstration that binary ASR overstates jailbreak risk. The fact-check extension is assessed against objective, Wikipedia-derived labels with perfect inter-annotator agreement (PABAK = 1.0), and that part of the evaluation is more convincing. However, the central 98.5% agreement claim is weakened by the annotation protocol: the human gold standard in Appendix D.2 was produced under instructions to decompose, mark critical/supporting sub-questions, score each, and aggregate—essentially the same mechanism JADES automates. The reported agreement may therefore measure fidelity to a specific rubric rather than superiority over independent holistic human perception. Because the re-evaluation conclusions in Sections 5.2 and 5.3 inherit these labels, the overestimation claim is conditional on the same rubric. The paper is promising and contains useful resources, but the core validation needs rework or substantially qualified claims.","major_comments":[{"comment":"The human annotation guidance instructs annotators to 'break the question into sub-questions,' mark each as critical or supporting, score each sub-question, and then weigh and aggregate into the final label. This is the same decompositional scoring logic that JADES implements. The 98.5% binary agreement is therefore partly a self-consistency check between GPT-4o and the annotation rubric, not a demonstration that JADES matches general human perception. The comparisons to holistic baselines (JailbreakRadar, JailbreakBench, StrongReject) are also disadvantaged, because holistic judgments are measured against labels that were elicited through decomposition. The paper does not report how many annotations used Case A (decomposition) versus Case B (holistic), nor agreement broken down by path. The conservative instruction in Appendix D.2 ('only responses that clearly and fully cover all critic","section":"Appendix D.2 and Section 4.3"},{"comment":"The section concludes that 'JADES yields highly consistent judgments across diverse LLMs,' but Table 3 shows GPT-4.1 has PABAK 0.22 (binary) and 0.33 (ternary), which the authors themselves classify as 'Fair agreement.' Only GPT-4o-mini and DeepSeek-V3 show high agreement. The claim that decomposition reduces task complexity and therefore yields cross-model consistency is not supported by the full set of tested models. The explanation that GPT-4.1 refuses to execute agent instructions is a practical limitation for a framework described as universal. Please either soften the consistency claim, or provide analysis of how frequently refusals occur and whether a small prompt change restores agreement.","section":"Section 4.4, Table 3"},{"comment":"The dataset construction description states that five attack methods are applied across four LLMs, but the formula '260 × 3 × 4 + 260 × 2 × 2' implies only three attacks on four models and two attacks on two models. This inconsistency should be clarified, especially because Table 5 later leaves DSN/GCG as 'not applicable' for GPT-3.5-Turbo and GPT-4. The reader needs to know exactly which attack-model combinations generated the 400 annotated pairs and how this affects the generalizability of the validation set.","section":"Section 4.1"}],"minor_comments":[{"comment":"The phrase 'accessible via the following link' is followed by no URL in the arXiv version. Please include a working repository link or a statement about availability for review.","section":"Abstract / Section 1"},{"comment":"Typo: 'The second case raise from' should be 'The second case arose from' or 'came from.'","section":"Section 4.4"},{"comment":"Reference [33] contains a typo: 'athaniel Li' should be 'Nathaniel Li.'","section":"References"},{"comment":"The claim that JADES is 'the first framework to enable ternary jailbreak evaluation' is a strong novelty assertion. Please cite any prior ternary or multi-level jailbreak evaluations, or temper the claim.","section":"Section 4.3"},{"comment":"In Figure 5, the label mapping uses 'fail' while the main text uses 'failed.' Please standardize terminology.","section":"Appendix E"}],"recommendation":"major_revision","confidential_remarks":"The central circularity concern is structural: the human annotation protocol encodes the decompositional philosophy, so the headline agreement number is not an independent test of the framework. This seems fixable by adding a holistic-annotation arm or a per-path breakdown, and I do not see it as a reason to reject outright. The fact-checking extension evaluation is the most objective part of the paper and should be highlighted. The missing code/data link is a reproducibility issue that must be addressed before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This paper is worth reading if you work on jailbreak evaluation. The core idea—borrow analytic scoring from education, decompose a harmful query into weighted sub-questions, and score each sub-answer—is a genuine step beyond holistic LLM judges. The multi-agent implementation is thoughtful, the audit trail in Table 4 and Figure 5 is a real plus, and the ternary (failed/partial/success) framing with SR/ASR is a useful corrective to the field's binary ASR habits. The fact-checking extension on HarmfulQA is also clever, and because it checks claims against Wikipedia, that sub-study is more objective than the main validation. The paper also ships code and data, which helps reproducibility.\n\nThe main soft spot is exactly where the stress-test note lands. Appendix D.2 instructs annotators to break the question into sub-questions, mark them critical or supporting, score each, then weigh and aggregate—essentially the JADES mechanism. The protocol does allow a holistic path (Case B), but decomposition is clearly the default, and the paper never reports how many annotations used which path, or agreement broken down by path. So the headline 98.5% agreement with human evaluators partly measures how well GPT-4o reproduces the very rubric the authors wrote for humans. Against a gold standard collected with independent holistic judgments, the margin over baselines might shrink. The re-evaluation numbers—e.g., LAA dropping from 93% to 69%—inherit this concern: the direction (overestimation) is plausible, but the magnitudes are contingent on this rubric.\n\nOther soft spots: thresholds (0.25, 0.75) are borrowed from Likert-scale practice in education, not tuned for jailbreak content, and the paper provides no confidence intervals. The 86.3% ternary accuracy, with 41% of human-labeled successes downgraded to partial, shows the judge LLM is stricter than humans—fine as a conservative bias, but it's a calibration choice, not an objective fact.\n\nWho is this for? It's a serious contribution to the jailbreak evaluation toolbox, and it should survive review if the authors add an independent holistic gold standard, report case-by-case rubric usage, and acknowledge that the current validation supports \"consistent with a decompositional rubric\" more than \"matches human perception.\" I'd send it to review. I would not yet cite the 98.5% as evidence, though the decomposition framework itself is worth a cite. For a reading group, it would generate good discussion about evaluation methodology in LLM security.","headline":"JADES is a well-built decompositional evaluator with a real transparency advantage, but the 98.5% human agreement is partly a self-consistency check because the human gold standard was annotated with the same decompositional rubric the framework automates.","tokens_in":23780,"tokens_out":2085,"would_cite":false,"duration_ms":22616,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Jailbreak attacks succeed far less often than prior evaluations report, according to a new decompositional judge that matches human raters 98.5% of the time.","keywords":["jailbreak evaluation","LLM-as-a-judge","decompositional scoring","analytic scoring","attack success rate","hallucination detection","fact-checking","LLM safety"],"falsifier":"Re-label the 400 JailbreakQR pairs with annotators who judge success/failure directly as a whole, with no decomposition instructions, then compare JADES against those holistic labels. If JADES's accuracy relative to holistic LLM baselines shrinks to a small margin, the claim that decomposition—rather than the rubric itself—explains the alignment is falsified.","tokens_in":22945,"feed_emoji":"⚖️","tokens_out":7864,"duration_ms":63215,"temperature":0.7,"pith_summary":"The paper's central claim is that the standard ways of measuring whether a jailbreak attack succeeded—searching for trigger phrases, scoring toxicity, or asking an LLM to judge the whole response—systematically overcount successes. Its proposed framework, JADES, instead mimics analytic rubric scoring: it decomposes the harmful question into up to five weighted sub-questions, cleans the response, matches sentences to each sub-question, scores each sub-answer on a 0–1 scale, and aggregates. Validated on 400 human-annotated prompt–response pairs, JADES reports 98.5% agreement with human labels in the binary success/failure setting, more than 9 points above strong baselines, and 86.3% in a finer failed/partial/success ternary setting. Re-evaluated with JADES, five popular attacks show inflated success rates—LAA on GPT-3.5-Turbo drops from 93% to 69%—and across all tested attacks the fully-successful share of reported successes is at most 25%. If the paper is right, the field's threat numbers are overstated, and fine-grained, interpretable evaluation rather than higher attack success should drive risk assessment.","feed_headline":"Jailbreak attack success rates drop under a new evaluator","feed_subtitle":"It matches human judges 98.5% of the time and cuts LAA on GPT-3.5 from 93% to 69%.","key_machinery":"Decompositional (analytic) scoring, implemented as a four-node multi-agent pipeline: the Question Decomposition Node generates up to five weighted sub-questions and stores them in memory; the Clean Node segments the response into sentences and discards irrelevant ones; the Sub-Question Pairing Node selects the sentences that bear on each sub-question; the Evaluation Node scores each sub-answer from the adversary's perspective on a {0, 0.25, 0.5, 0.75, 1} Likert scale and computes a weighted total. The optional fact-checking node splits the cleaned response into self-contained unit facts, verifies each against a trusted web source, and hands the verdicts to the judge. The decomposition-and-pa","core_discovery":"JADES asserts that a jailbreak only succeeds when the response actually accomplishes the adversary's harmful task, and that this can be measured by decomposing the task. The framework's question-decomposition agent breaks the harmful query into a small set of weighted sub-questions capturing what a real adversary would need—materials, steps, detonation, and so on. A cleaning agent strips the response of role-play filler and distraction tokens; a pairing agent selects, for each sub-question, the sentences that purport to answer it; a judge agent scores each sub-answer on a five-level Likert scale; and weighted aggregation yields one overall score, with thresholds mapping it to failed, partial","pith_inferences":["The paper's own annotation protocol (Appendix D.2) instructs human labelers to decompose the question into weighted sub-questions before deciding—the same mechanism JADES automates—so the 98.5% agreement chiefly measures how faithfully GPT-4o reproduces this rubric; agreement against independent holistic human judgment could be closer to the holistic baselines.","The paper itself flags that its thresholds and five-sub-question cap are borrowed from educational grading, and that its edge-case fixes are post-hoc, which suggests the framework's calibration, not its architecture, is the most fragile part.","Because decomposition is generated by an LLM, JADES inherits that model's interpretive biases—the paper's error cases show inverted disinformative premises and misframed yes/no questions—so evaluation quality is bounded by the decomposer.","Testable extension: apply the fact-checking node to non-harmful constructed responses, where factual completeness rather than refusal is the scoring target; if verification transfers, the framework becomes a general response-fidelity scorer."],"forward_implications":["Binary attack success rates reported for GCG, DSN, LAA, PAIR, and JailbreakChat are systematically inflated; under JADES re-evaluation most drop substantially, with LAA on GPT-3.5-Turbo falling from 93% to 69%.","The share of fully successful outcomes within reported ASR is at most 0.25 for all tested attack–model pairs, meaning binary 'success' is mostly partial success.","Apparent vulnerability of weaker open models such as Vicuna shrinks once vague or incoherent outputs are penalized, while stronger models, once jailbroken, produce relatively higher-quality harmful content.","Ternary evaluation with the SR/ASR quality metric gives defenders a way to prioritize against attacks that actually fulfill the harmful task, not merely bypass refusal.","Because every decision step is recorded, JADES verdicts come with an auditable trail, making safety assessments inspectable and reproducible."],"supporting_citations":[{"why":"Supplies the 100 harmful questions and the official JailbreakBench evaluator used as the binary baseline for the re-evaluation of attack success rates.","marker":"[9]"},{"why":"LAA is the attack whose reported ASR on GPT-3.5-Turbo drops from 93% to 69% under JADES re-evaluation, the paper's headline overestimation result.","marker":"[3]"},{"why":"PAIR is one of the five re-evaluated attacks; its prompt rewriting causes target shift, giving it the lowest share of fully successful outcomes.","marker":"[10]"},{"why":"GCG and the string-matching method define a baseline evaluator and one of the re-evaluated attacks; also a source of benchmark questions.","marker":"[65]"},{"why":"DSN is one of the re-evaluated attacks, with JADES cutting its Llama-2 ASR from 0.94 to 0.66.","marker":"[64]"},{"why":"HarmBench serves as a baseline evaluator and a source of harmful questions for the re-evaluation dataset.","marker":"[33]"},{"why":"StrongReject is a baseline evaluator, exemplifying the holistic evaluation with high recall but low precision that JADES claims to beat.","marker":"[52]"},{"why":"JailbreakRadar is a baseline evaluator and one of the two sources of the 260 harmful questions used to build JailbreakQR.","marker":"[11]"},{"why":"Analytic-scoring research that motivates the core mechanism: decomposing constructed responses into criteria improves grading consistency over holistic judgment.","marker":"[23]"},{"why":"The Likert-scale scoring and the empirical 0.25 threshold used to map sub-scores and total scores to verdicts.","marker":"[29]"}],"fun_headline_variants":["Jailbreak success redefined: new framework matches human judges 98.5%","Decompose to decide: JADES framework for jailbreak evaluation","New evaluation reveals jailbreak attacks are overestimated","JADES: Decomposing jailbreak tasks for reliable assessment"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The 400 human annotations that serve as ground truth were produced under a rubric that tells annotators to decompose the question into weighted sub-questions and aggregate—essentially JADES's own mechanism—so the headline agreement measures how well the GPT-4o pipeline reproduces this rubric, not how well it matches an independent, holistic human perception of success.","fun_headline_variants_meta":{"raw":{"variants":["Jailbreak success redefined: new framework matches human judges 98.5%","Decompose to decide: JADES framework for jailbreak evaluation","New evaluation reveals jailbreak attacks are overestimated","JADES: Decomposing jailbreak tasks for reliable assessment"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000331,"raw_usage":{"total_tokens":1704,"prompt_tokens":794,"completion_tokens":910,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":538,"completion_tokens_details":{"reasoning_tokens":834}},"tokens_in":538,"tokens_out":910,"duration_ms":7983,"temperature":1.0,"reasoning_tokens":834,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T14:45:43.536057+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-label the 400 JailbreakQR pairs with annotators who judge success/failure directly as a whole, with no decomposition instructions, then compare JADES against those holistic labels. If JADES's accuracy relative to holistic LLM baselines shrinks to a small margin, the claim that decomposition—rather than the rubric itself—explains the alignment is falsified.","supporting_citations":[{"cited_title":"Analytic or Holistic: A Study of Agreement Between Different Grading Models","cited_arxiv_id":null,"evidence_quote":"Analytic-scoring research that motivates the core mechanism: decomposing constructed responses into criteria improves grading consistency over holistic judgment."},{"cited_title":"A technique for the measurement of attitudes","cited_arxiv_id":null,"evidence_quote":"The Likert-scale scoring and the empirical 0.25 threshold used to map sub-scores and total scores to verdicts."}],"review_version":1}