{"id":"8b525851-71ef-490a-85d4-4f124751f0e1","arxiv_id":"2607.19424","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"JailMeter, an evidence-extraction multi-agent evaluator, reports 97.27% accuracy on its own 330-instance non-refusal jailbreak benchmark, with a distilled 7B model reaching 95.15%.","lead":"This paper introduces JailMeter, a multi-agent framework that strips jailbreak noise from LLM responses and judges success by whether the response fully answers the original malicious request. On a new 330-sample human-labeled benchmark, it reports 97.27% accuracy, beating existing evaluators.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"JailMeter-Eva uses only GPT-4o-mini responses (330 non-refusal samples from 60 questions); the 97.27% superiority over baselines may not generalize to other target models or refusal-inclusive evaluations.","rationale":"The reader's verdict is CONDITIONAL, and we agree that the central weakness is lack of external validation. We sharpen this to a concrete distributional concern: all test responses come from one target model (GPT-4o-mini), and the evaluator itself is GPT-4o-mini-based. This creates an in-distribution evaluation that may not hold for other models. The paper provides no human-labeled evaluation on other target models, despite using JailMeterSLM to report ASR trends across 12 models (Sec. 5.4). The proposed concrete test would settle the generalization question. Beta tuning on 200 test samples (App. C.1) is a secondary flaw but unlikely to explain the large margin; no significance tests are reported, but the gap is large. Overall, the manuscript's internal evidence is solid; the claim is plausible but not yet established beyond its self-built benchmark, so CONDITIONAL is appropriate.","tokens_in":35914,"tokens_out":6933,"duration_ms":69777,"concrete_test":"Build an independent human-labeled set of ~400 responses from at least three diverse target models (e.g., Llama-3-70B, Claude-3.5-Sonnet, GPT-4o) using the same 11 jailbreak methods plus direct malicious queries, and include explicit refusals. Have 2-3 annotators label each response with the paper's criterion. Re-run the Table 2 comparison (JailMeter, JailMeterSLM, and all 8 baselines) on this set. If JailMeter's accuracy falls below ~90% or its accuracy advantage over the best baseline narrows to less than 5 points, the claimed superiority is benchmark-specific.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline claim rests entirely on JailMeter-Eva (Sec. 3.3), which contains 330 non-refusal responses generated by a single target model, GPT-4o-mini, from 60 original questions. JailMeter's agents also use GPT-4o-mini as the backbone (Sec. 5.2, App. B.1). Although baselines share the same backbone, the multi-agent evidence extraction could be particularly well-matched to stylistic patterns of GPT-4o-mini outputs, and no human-labeled data from other target models is provided to rule this out. The only external check, BenignInstructions (Table 3), is a trivial 100% case. Consequently, the 'substantially outperforming' claim is established for a narrow, self-constructed distribution. The exclusion of refusals further biases against simple refusal-detection baselines, so the margin may shrink in real-world settings. This is a generalization risk, not an internal inconsistency; the ablations and Fleiss' κ support internal validity.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes JailMeter, a multi-agent framework for evaluating jailbreak attacks on LLMs. JailMeter segments a target model's response into semantic chunks, iteratively extracts and refines evidence via relevance and conciseness agents (dual-feedback optimization), and then uses an evaluation agent to judge whether the original malicious request is fully satisfied. The authors construct a human-labeled benchmark, JailMeter-Eva, containing 330 non-refusal responses from GPT-4o-mini generated from 60 malicious questions and 11 jailbreak methods. They report 97.27% accuracy and 93.88% F1 for JailMeter, outperforming eight existing evaluation methods, and a distilled SLM (JailMeterSLM) reaching 95.15% accuracy at substantially lower cost. They also re-evaluate 11 jailbreak methods across 12 target models and observe that ASR decreases for newer models and that some methods suffer from semantic drift.","tokens_in":36167,"tokens_out":4540,"duration_ms":49184,"significance":"If the reported gains are robust, JailMeter addresses a genuine problem: jailbreak evaluation is fragmented across inconsistent criteria and methods, and multi-agent evidence extraction with iterative feedback is a plausible remedy. The paper's strengths include a human-annotated dataset with reported inter-annotator agreement (Fleiss' κ = 0.86, App. A.4), component ablations, released code/data, and a distilled efficient student model. The evaluation criterion is transparent and grounded in whether the response completes the original request, which is a defensible definition of attack success. However, the central quantitative claim is currently supported only on a self-constructed benchmark with a single target model, a single backbone for the evaluator, and a test-set-selected hyperparameter. These issues do not invalidate the framework, but they limit the strength of the 'substantially outperforming' claim and require additional validation before the results can be fully credited.","major_comments":[{"comment":"The headline comparison is performed entirely on JailMeter-Eva, which contains responses from a single target model, GPT-4o-mini, generated from 60 original questions. JailMeter's agents also use GPT-4o-mini as the backbone. This creates a distributional overlap between the evaluator and the target model; the multi-agent evidence extraction may be particularly effective on the stylistic patterns of GPT-4o-mini outputs, and no human-labeled data from other target models (e.g., Llama-2-7B-Chat, GPT-4o, or other open models) is provided to rule out this concern. The 'substantially outperforming' claim is therefore established only for a narrow, self-constructed distribution. I recommend adding a cross-target evaluation with human labels and, if possible, a refusal-inclusive subset.","section":"Sec. 3.3, Table 2, App. B.1"},{"comment":"The balance parameter β is selected by testing 11 values on a random 200-sample subset of JailMeter-Eva, which is the same benchmark used to report the final 97.27% accuracy. This is test-set hyperparameter tuning and can inflate the reported accuracy. The paper notes that β has only a small effect on performance, but the selection protocol should still use a separate validation split or nested cross-validation, with the final accuracy reported on a held-out test set. Without this, the exact margin over baselines is not trustworthy.","section":"Appendix C.1, Table 2"},{"comment":"The JailMeter-Eva dataset excludes all responses containing refusal phrases such as 'I'm sorry' or 'I apologize.' This design choice makes the benchmark challenging, but it systematically removes the case class where refusal-based evaluation methods (e.g., I-FSJ, DAN) are most likely to succeed. In Table 2, these methods achieve 0.00% recall and F1, which appears to be a direct consequence of evaluating them only on non-refusal responses. The reported performance gap between JailMeter and baselines is thus not representative of standard ASR evaluation, where refusals are common. I ask the authors to report results on a sample that includes refusals, or to explicitly state and justify why the non-refusal-only comparison is the appropriate setting for the paper's claims.","section":"Sec. 3.3, Table 2"},{"comment":"No uncertainty quantification is provided for any of the accuracy/F1 numbers. With 330 binary samples, the differences among some methods and ablations may be within sampling noise; for example, in Table 4 the w/o Acon (91.52%) and w/o Arel (91.82%) variants differ by only 0.3 percentage points, and in Table 2 several baselines lie within a few points of each other. I request bootstrap confidence intervals or significance tests (e.g., McNemar's test for paired binary predictions) for the main comparisons in Tables 2, 4, and 5. This is necessary to support the claim of 'substantially outperforming' and to interpret the component-level ablation results.","section":"Table 2, Table 4, Sec. 5.2"}],"minor_comments":[{"comment":"The termination condition compares Quality(t)_score with Quality(t-1)_score and breaks when the current score is not greater than the previous. If scores are equal at the first and second iterations, the algorithm returns E^0; if the score increases monotonically, it stops only at tmax. This behavior is reasonable, but the pseudocode would benefit from an explicit comment stating that the loop assumes non-decreasing quality and that ties are treated as convergence.","section":"Algorithm 1, Sec. 4.1"},{"comment":"The ASR heatmaps are difficult to read in the text version: the model names on the y-axis and method names on the x-axis are not clearly legible, and the color scale is not defined. Please provide a color bar and consider reporting the exact ASR values in a table, since Section 5.4 makes qualitative claims (e.g., ASR decreases for newer models) that would be easier to verify numerically.","section":"Figure 2"},{"comment":"The sensitivity of β is only tested on overall accuracy. Since the framework has separate relevance and conciseness agents, it would be informative to also report how β affects the precision/recall trade-off or the number of refinement iterations. This would help users choose β in practice.","section":"Eq. (1), App. C.1"},{"comment":"The BenignInstructions dataset is described as containing GPT-4 responses, whereas the main benchmark uses GPT-4o-mini. This is not a problem, but the text should clarify why the same target model is not used, and it would be helpful to report the exact prompts or a sample to facilitate reproduction.","section":"Sec. 5.1, BenignInstructions"},{"comment":"The claim that smaller models can exhibit lower ASR because they fail to follow complex multi-step instructions is interesting, but the connection between model size and generated response quality is not directly measured. A short analysis of failure modes (e.g., off-topic vs. incoherent vs. partial responses) would strengthen the interpretation.","section":"Sec. 5.4"}],"recommendation":"major_revision","confidential_remarks":"The framework is promising and the human-labeled dataset is a useful resource, but the current evidence base is too narrow for the strength of the claims. The most urgent fixes are (1) moving the β selection off the test set, (2) adding cross-target human-labeled evaluation, and (3) addressing the refusal-exclusion bias in the baseline comparison. If the authors can provide these, the paper could become a solid contribution. I also note that the 'substantially outperforming' phrasing depends on the non-refusal-only setting; the revision should make this scope explicit."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: this is a useful, well-specified evaluation framework, and the ablations support that the pieces do something. But don't take the 97.27% number at face value. It comes from a self-built 330-sample benchmark with responses from a single target model, and the main balance parameter β was tuned on a 200-sample slice of that same benchmark (Appendix C.1). That's test-set tuning, and it's a real reason to treat the headline as optimistic until they validate on held-out data.\n\nBy the way, the paper does a lot right. It adopts one explicit success criterion — response captures malicious intent and provides a complete answer — and builds the whole pipeline around it. That criterion is also used to label JailMeter-Eva, so it's not an independent ground truth in the strongest sense, but the labels come from three human annotators with Fleiss' κ=0.86, which is solid for this task. The iterative evidence extraction with relevance and conciseness feedback is genuinely new as a combination, and the ablation shows each component contributes. Distilling to a 7B model for cheap evaluation is a nice practical contribution. They also ship code and data and are honest about limitations, including evaluator-targeted attacks.\n\nSoft spots, in proportion: (1) External validity. JailMeter-Eva is 330 instances from 60 questions, all responses from GPT-4o-mini. The evaluation agent also uses GPT-4o-mini as backbone, so the method might be over-fit to the stylistic patterns of that model's outputs. The stress-test note is right that we have no human-labeled data from other target models to rule this out. The BenignInstructions check is 100% for everyone — it doesn't discriminate. (2) The β tuning on a random subset of the same benchmark is a problem; they even show β doesn't matter much, which is good, but they should say it's selection on the test distribution. (3) No significance tests or error bars. Given the benchmark size, a 97% vs 80% gap could still be real, but we can't tell from the paper. (4) Excluding refusals biases against refusal-detection baselines; that's a defensible design choice for a 'challenging' benchmark, but it means the superiority can't be extrapolated to real-world ASR measurement as-is.\n\nOverall: the central idea is sound and the internal evidence is credible, but the headline superiority claim is not yet established. It deserves serious review — the framework is worth building on — but I'd send it back for a larger validation set, more target models, and proper uncertainty reporting.","headline":"JailMeter is a credible, well-specified evaluation framework with supportive ablations, but the headline accuracy rests on a tiny single-model benchmark and test-set tuning, so the 'substantially outperforms' claim needs external validation.","tokens_in":36650,"tokens_out":2710,"would_cite":true,"duration_ms":26059,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"JailMeter claims jailbreak success is best measured by stripping a response of its attack noise and checking whether the stripped content fully delivers the original malicious request — and reports 97.27% accuracy on a human-labeled benchma","keywords":["jailbreak attacks","LLM safety","evaluation framework","evidence extraction","information bottleneck","multi-agent system","knowledge distillation","attack success rate"],"falsifier":"Take a fresh set of jailbreak responses generated from a different question pool and several target models, label them by the paper's own criterion with multiple human annotators, and run JailMeter plus the same baselines: if JailMeter's accuracy falls to roughly the level of a plain judge (around 80%) or the gap to the second-best method disappears, the central claim of substantially more reliable evaluation is refuted. A second, sharper test: adversarially pad a complete response with a misleading but plausible-sounding passage and check whether JailMeter's evidence filtering drops the passa","tokens_in":35832,"feed_emoji":"🔍","tokens_out":8602,"duration_ms":63269,"temperature":0.7,"pith_summary":"JailMeter argues that the main obstacle to reliable jailbreak evaluation is not the attacks themselves but how we measure them: existing methods use inconsistent criteria and are thrown off by the extraneous jailbreak noise attacks wrap around their requests. It proposes a multi-agent framework that iteratively compresses a model response into concise evidence, preserving only content relevant to the original malicious question, and then judges success as the response's fully satisfying that intent. On a 330-instance benchmark of hard, non-refused jailbreak cases, JailMeter reports 97.27% accuracy and 93.88% F1, markedly higher than the eight existing methods it compared against. It also distills the framework into a small model, JailMeterSLM, that keeps 95.15% accuracy with much lower cost, making large-scale audits practical. A reader should care because current attack-success numbers are noisy and hard to trust; if JailMeter's criterion and evidence extraction generalize, jailbreak evaluations can be compared meaningfully.","feed_headline":"Jailbreak evaluator hits 97% accuracy by stripping attack noise","feed_subtitle":"A multi-agent judge checks if a model truly answered the malicious request; a distilled small model keeps accuracy.","key_machinery":"The load-bearing mechanism is the dual-feedback evidence optimization loop: the Extraction Agent picks semantic chunks from the segmented response; Relevance and Conciseness Agents score the current evidence and send reasoning and scores back; a composite quality score s_rel − β(1 − s_con) governs when to stop refining. This loop is the paper's concrete implementation of the Information Bottleneck principle — keep only the minimal content sufficient to judge the original request. The distilled small model, JailMeterSLM, replaces the whole loop with a single forward pass by learning the multi-agent trajectories, achieving near-equal accuracy with much lower latency and token cost.","core_discovery":"The central discovery is that jailbreak evaluation can be made significantly more accurate and interpretable by treating evidence extraction as a semantic compression task. The framework uses five agents — sentence segmentation, extraction, relevance, conciseness, and evaluation — and iteratively refines an evidence subset via dual feedback that balances relevance to the original question against conciseness, terminating when a composite quality score converges. The final evidence is then judged by a three-step evaluation agent that asks whether the response identifies the malicious intent and provides a complete answer. The paper claims this removes the noise that jailbreak prompts insert a","pith_inferences":["The same evidence-compression design could improve other content-moderation or instruction-following evaluations where noise distracts judges; that extension is not in the paper.","The headline accuracy is measured on a benchmark built from 60 questions and one target model, so the 97% is a demonstration; generality across models and question distributions remains an open test.","Because the distilled student learns from the teacher's labels, any backbone-choice bias in the teacher may be baked into the student; comparing student labels with fresh human labels would test this.","The paper's identified vulnerability to evaluator-targeted attacks suggests a stress test: crafted responses that bury incomplete answers in persuasive noise could reveal the framework's limits."],"forward_implications":["Adoption of a single criterion (complete fulfillment of the original malicious intent) would make attack-success rates comparable across studies.","Because JailMeter exposes that some attacks produce affirmative but off-target responses, previously reported attack-success rates for such methods are likely inflated.","The evidence output enables human audit and transparent safety reporting.","The distilled small model makes large-scale evaluation practical: roughly 12x fewer tokens and one model call per instance instead of about ten.","Re-evaluation across model families shows newer models are more resistant, and smaller models can look safer simply because they fail to follow complex prompts."],"fun_headline_variants":["Jailbreak judge strips noise and hits 97%","Evidence-based jailbreak eval hits 97% by denoising","Multi-agent judge distills jailbreak noise to 97%","Jailbreak detection sharpened to 97% via evidence filtering","Small distilled model keeps jailbreak detection at 97%"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The paper's entire accuracy gap rests on the assumption that JailMeter-Eva — 330 non-refused responses from a single target model over 60 malicious questions — fairly represents the jailbreak cases an evaluator will actually face; if it does not, the 97% figure and the gap over other methods may not generalize.","fun_headline_variants_meta":{"raw":{"variants":["Jailbreak judge strips noise and hits 97%","Evidence-based jailbreak eval hits 97% by denoising","Multi-agent judge distills jailbreak noise to 97%","Jailbreak detection sharpened to 97% via evidence filtering","Small distilled model keeps jailbreak detection at 97%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000753,"raw_usage":{"total_tokens":3167,"prompt_tokens":708,"completion_tokens":2459,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":452,"completion_tokens_details":{"reasoning_tokens":2373}},"tokens_in":452,"tokens_out":2459,"duration_ms":16271,"temperature":1.0,"reasoning_tokens":2373,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T16:41:20.451238+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a fresh set of jailbreak responses generated from a different question pool and several target models, label them by the paper's own criterion with multiple human annotators, and run JailMeter plus the same baselines: if JailMeter's accuracy falls to roughly the level of a plain judge (around 80%) or the gap to the second-best method disappears, the central claim of substantially more reliable evaluation is refuted. A second, sharper test: adversarially pad a complete response with a misleading but plausible-sounding passage and check whether JailMeter's evidence filtering drops the passa","supporting_citations":[],"review_version":1}