{"id":"6e160d1a-92b0-4415-84fa-ae08dc07b4b2","arxiv_id":"2507.22037","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"An iterative attacker-defender reinforcement learning method that makes a multimodal LLM refuse more jailbreak prompts without over-refusing ordinary queries.","lead":"SecTOW trains a vision-language model to refuse harmful image-query pairs by repeatedly pitting it against an attacker model that invents new jailbreak prompts. The method cuts attack success rates on four safety benchmarks while keeping accuracy on general benchmarks near baseline.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SecTOW's attacker reward and data filter both rely on Llama-Guard-3 as the sole harm judge; if that judge is biased or gameable, the reported ASR drops may reflect judge-specific overfitting rather than genuine robustness.","rationale":"The reader's weakest assumption is the most load-bearing because both the attacker's gradient signal and the filter that admits data into the defender's training set are functions of the same external classifier. In contrast, the defender's internal reward is a rule-based refusal check, but the labels it checks are themselves produced by the judge-dependent pipeline. Consequently, every reported safety improvement is conditional on Llama-Guard-3 being a faithful proxy for harm. The paper provides no human-annotation study, no agreement statistics for Llama-Guard-3, and no ablation with a different judge; the only evidence is the final ASR on four benchmarks, whose label sources are not described in sufficient detail. That leaves open the possibility of reward hacking against the judge, a well-documented failure mode in RL from AI feedback. I do not think this warrants rejection: the framework is coherent, the ablations show the components matter, and the concern can be resolved empirically. But until the judge-dependence is tested, the central claim should remain conditional. Hence no change to the reader's verdict.","tokens_in":14586,"tokens_out":5936,"duration_ms":71208,"concrete_test":"Sample 500 defender responses from the third-iteration attacker's filtered set (queries where Llama-Guard-3 judged the response unsafe). Have three independent annotators label each response as harmful or not, using the same taxonomy as the safety benchmarks; compute majority-vote agreement with Llama-Guard-3. Separately, on a random 300-item subset of MM-SafetyBench, replace the benchmark's label source with the human majority and recompute ASR for the SecTOW-3 defender and the base model. If human–LlamaGuard agreement is below ~80% or if human-based ASR differs from the reported 0.0298 by more than a few points, the training/evaluation loop is likely overfit to Llama-Guard-3.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central training loop is anchored by a single harm classifier. In §3.2.2 (Eq. 1), the attacker's reward is Score_attack, set to 1 iff an 'independent safety evaluation model' judges the defender's response unsafe; §4.1 identifies that model as Llama-Guard-3. In §3.3.2, the same attack-success criterion (frequency of unsafe responses over n samples) filters which synthetic jailbreak samples enter the defender's training set. Thus every piece of data that teaches the defender to refuse is labeled harmful by Llama-Guard-3's verdict on the defender's output. If Llama-Guard-3 is biased—over-triggering on refusals, missing certain harm categories, or vulnerable to phrasing artifacts—the attacker learns to exploit those biases, and the defender learns to be 'safe' only with respect to that judge. The reported ASR reductions on JailBreakV-28k, FigStep, SafeBench, and MM-SafetyBench would then reflect correlation between Llama-Guard-3's judgments and the benchmarks' labels rather than intrinsic robustness. The paper provides no agreement analysis between Llama-Guard-3 and human ratings or an independent judge, and no alternative-evaluator training run, so this load-bearing assumption is unverified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"SecTOW is an iterative defense-attack training method for multimodal large language models. A defender and an auxiliary attacker, both initialized from Qwen2-VL-7B, are alternately trained with GRPO. The attacker generates or refines jailbreak image-query pairs and receives reward when the defender's response is judged unsafe by Llama-Guard-3; the resulting successful attacks are filtered and added to the defender's training set. The defender is rewarded by a rule-based refusal check against response labels, and quality monitors are used to limit over-refusal and preserve attacker diversity. Experiments report large ASR reductions on JailBreakV-28k, FigStep, SafeBench, and MM-SafetyBench while roughly preserving MMMU and MMMU-Pro accuracy, with ablations supporting the cold start, monitoring, and iterative components.","tokens_in":14835,"tokens_out":5756,"duration_ms":61871,"significance":"The paper proposes a genuinely iterative attacker-defender formulation, uses reward-based data expansion that avoids detailed generative labels, and explicitly separates the defender's reward from the attacker's harm judgment by using an external safety evaluator. The reported effects are large, and the ablations and case studies give some support for the mechanism. If the results are robust, the method is a useful step toward automated, continuously improving safety tuning of MLLMs. However, confidence is tempered by the absence of multiple seeds, error bars, code release, and same-protocol baseline comparisons, and by the heavy reliance on a single external harm classifier in the training loop.","major_comments":[{"comment":"The attacker's reward and the data filter are both driven solely by Llama-Guard-3's unsafe/safe decision on the defender's response. This creates strong selection pressure to exploit whatever artifacts Llama-Guard-3 is sensitive to, and the defender is then trained on samples selected by that same judge. The paper reports no agreement analysis between Llama-Guard-3 and human ratings or an independent judge, and no training run with an alternative evaluator. The reported ASR drops on the four safety benchmarks could therefore reflect correlation between Llama-Guard-3's judgments and the benchmark labels rather than intrinsic robustness. A concrete test is needed: report human or independent-judge agreement on a sample of defender responses, and rerun at least one training iteration with a different harm judge, then report ASR under both judges.","section":"§3.2.2 (Eq. 1), §3.3.2, §4.1"},{"comment":"The comparison with Adashield, MLLM-Protector, MMS, and MIRage uses 'their highest performance reported' rather than re-running those methods under SecTOW's evaluation protocol. Since ASR can depend on the judge model, sampling temperature, number of samples, and prompt template, literature-reported numbers are not necessarily comparable to the numbers produced here. The state-of-the-art claim in Section 4.2 is load-bearing and needs a same-protocol comparison, or at minimum a table documenting the evaluation protocol used in each cited source and the exact protocol used in this paper.","section":"§4.2, Table 2"},{"comment":"The manuscript never states which judge or protocol is used to compute ASR on JailBreakV-28k, FigStep, SafeBench, and MM-SafetyBench. If the benchmark ASRs are computed with Llama-Guard-3, the evaluation is partly circular with the attacker's training reward; if they are computed by each benchmark's native judge, that needs to be stated and the judge models reported. Without this information, the central numbers in Tables 1 and 2 cannot be independently interpreted.","section":"§4.2, Tables 1 and 2"},{"comment":"The 'w/o Iteration mechanism' row reports exactly the same values as SecTOW after one iteration in Table 1 (JailBreakV-28k ASR 0.0261, MM-SafetyBench ASR 0.0522, MMMU ACC 0.5444, ORR 0.0033). This ablation therefore compares one iteration against three iterations; it does not remove the iteration mechanism. The text interprets this as removing the core iteration mechanism, which is not supported by the experimental design. Please rename the setting or add a true no-iteration baseline, such as the cold-start defender only, to isolate the iterative contribution.","section":"§4.5, Table 3"},{"comment":"All safety and general numbers come from a single run with no standard errors, confidence intervals, or significance tests. Several comparisons involve small absolute differences, such as JailBreakV-28k ASR 0.0061 after three iterations versus 0.0130 after two iterations, and MMMU ACC 0.5422 versus the base model's 0.5411. Without multiple seeds or bootstrap intervals, it is not possible to determine whether the reported improvements and the stability of general performance are meaningful. Please report variance over at least three seeds or bootstrap confidence intervals.","section":"§4.2, Tables 1 and 3"}],"minor_comments":[{"comment":"The text 'Dval-general dateset' should read 'dataset,' and the function Refuse(i, q) should be defined operationally rather than only by the verbal description.","section":"§3.4, Eq. (3)"},{"comment":"The early-stopping thresholds (10% diversity drop and 5% ORR) are presented without sensitivity analysis; please state whether the main results are robust to reasonable changes in these thresholds.","section":"§3.4"},{"comment":"The heading misspells 'JailBreakV-28k' as 'JailBreaV-28k'.","section":"Table 2"},{"comment":"The conclusion spells the method name as 'SceTOW' instead of 'SecTOW'.","section":"§5"},{"comment":"The filtering criterion uses n samples, but n is never specified; please report the value of n, the resulting dataset sizes after filtering, and the effect of varying n.","section":"§3.3.2"},{"comment":"The similarity metric is described only by a URL to 'Fuzzy'; please provide a version and specify how similarity is computed for multimodal query-image pairs.","section":"§3.4"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and important problem, and the proposed iterative framework is interesting. However, the single-judge dependency, the unspecified ASR evaluation protocol, and the literature-number comparisons are correctness risks that need to be addressed before the central claims can be trusted. The paper would be substantially stronger with code release, multiple seeds, and same-protocol baseline reruns. The stress-test concern about Llama-Guard-3 being the sole harm judge lands: it is a load-bearing assumption in both the attacker reward and the data filter, and it is currently unvalidated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"SecTOW is a real contribution. The idea of alternating GRPO training between a defender and an attacker that generates and filters jailbreak data is new as far as I know, and the paper does it carefully: rule-based rewards for the defender, a format plus attack-success reward for the attacker, quality monitoring to stop over-refusal and repetitive attacks, and ablations that show each piece matters. The reported gains are large—MM-SafetyBench ASR from 0.67 to 0.03, MMMU accuracy about flat. That is the kind of result that makes you sit up.\n\nThe main soft spot is the one the stress test flags. Both the attacker's reward and the data filter rely on Llama-Guard-3's judgment of whether the defender's response is harmful. That is a single point of failure. If Llama-Guard-3 has systematic biases—over-triggering on refusals, missing certain harm types, or being gameable by phrasing—then the attacker learns to exploit Llama-Guard-3, and the defender learns to be safe only with respect to that judge. The paper shows no agreement analysis between Llama-Guard-3 and human labels or another judge, and no training run with a different evaluator. That said, it is not a fatal circularity: the benchmarks have independent labels, and the reported ASR drops on those benchmarks are consistent with genuine improvement. The concern is about generality, not validity.\n\nOther soft spots are more standard. No error bars or multiple seeds, no code release, and Table 2 imports the best reported numbers from other papers instead of re-running them. The free parameters—filter threshold, early-stop thresholds, refusal vocabulary, number of iterations—are not sensitivity-analyzed. These are fixable in revision.\n\nThe paper is worth a serious referee. The method is new, the experiments are well designed, and the ablations are informative. I would ask for a reproducibility package, variance reporting, and at least one experiment with a second judge (or a human agreement sample) before the state-of-the-art claim is accepted.\n\nWho is this for: people working on MLLM safety and alignment who care about training-time defenses. It's not a must-read for the broader RL community, but it's a solid within-subfield advance.","headline":"A genuinely new iterative attacker-defender RL loop with strong reported numbers, but the single safety judge makes the robustness claim narrower than stated.","tokens_in":15401,"tokens_out":2296,"would_cite":true,"duration_ms":25412,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SecTOW claims an iterative attacker–defender reinforcement loop cuts jailbreak success from 0.6726 to 0.0298 on MM-SafetyBench while preserving general accuracy.","keywords":["multimodal large language models","jailbreak attacks","defense-attack training","GRPO reinforcement learning","iterative adversarial training","over-refusal mitigation","synthetic jailbreak data","multimodal safety"],"falsifier":"Score the three-iteration SecTOW defender on the same jailbreak inputs using human annotators instead of the automated safety judge it was trained against; if human-scored attack success approaches the base model's 0.6726 on MM-SafetyBench, the reported defence is an artefact of judge-specific optimisation rather than genuine robustness.","tokens_in":14373,"feed_emoji":"🛡️","tokens_out":8976,"duration_ms":84217,"temperature":0.7,"pith_summary":"SecTOW is an iterative training method that hardens a multimodal large language model against image–query jailbreak attacks by repeatedly making an attacker better at breaking the model, then retraining the model on the attacker's successful jailbreaks. The paper's central claim is that this tug-of-war—implemented with group-relative policy optimization and simple rule-based rewards—sharply reduces attack success rates on four safety benchmarks while preserving general accuracy, avoiding the over-refusal problem of supervised fine-tuning. After three iterations, for example, MM-SafetyBench attack success drops from 0.6726 to 0.0298 and FigStep from 0.3320 to 0.0, while MMMU accuracy stays within 0.001 of the base model. This matters because existing defences either rely on external guardrails that do not fix intrinsic model vulnerabilities or use supervised fine-tuning that makes models refuse harmless queries.","feed_headline":"Jailbreak success rate falls from 67% to 3% in three rounds","feed_subtitle":"An attacker–defender reinforcement loop hardens a multimodal model without sacrificing accuracy on general benchmarks.","key_machinery":"The machinery is an alternating GRPO training loop—group-relative policy optimization, a reinforcement-learning rule that scores each output against its group-mates rather than an absolute value model. The defender's reward is a rule-based refusal match: 1 when its behavior agrees with the label of whether rejection is required. The attacker's reward multiplies a binary attack-success score from an independent safety classifier by a format score enforcing the think/answer output structure. Filtering keeps only generated jailbreaks that defeat the defender in at least half of n sampled responses, and monitoring stops training when attacker-query diversity drops 10% or defender over-refusal exceeds 5%. This closed loop converts sparse harmful data into dense, targeted synthetic training data.","core_discovery":"The central discovery is that adversarial pressure can be built into the training loop of the model itself. SecTOW pairs a defender with an attacker, both trained by group-relative policy optimization; the attacker is rewarded when the defender produces a harmful response, and its successful jailbreak queries—after filtering and quality monitoring—become the defender's next training batch. On the paper's measurements this loop drives attack success rate from 0.6726 to 0.0298 on MM-SafetyBench and to 0.0 on FigStep after three iterations, while MMMU accuracy remains 0.5422 versus 0.5411. The claim is that intrinsic, guardrail-free security can be improved iteratively without the over-refusal penalty that supervised fine-tuning incurs.","pith_inferences":["Because the attacker is rewarded only when the external safety judge calls the defender's response harmful, SecTOW's measured gains are at risk whenever that judge can be fooled; an immediate test is to re-score the defender's responses with a different judge or human annotators.","The loop can be viewed as an automated red-teaming generator: its third-iteration attacker produces jailbreak queries with a 0.5509 success rate against the base defender, suggesting the same attacker could audit other multimodal models or seed harder evaluation sets.","A natural ablation not reported in the paper would train the defender by supervised fine-tuning on the same filtered synthetic jailbreak set; if the ASR matched SecTOW's, the contribution of GRPO would reduce to data generation rather than policy optimisation.","If the trend continues beyond three iterations, one might expect further ASR reductions until the judge's own blind spots bound the achievable security, at which point hardening the judge itself would become the next step."],"forward_implications":["Three iterations of SecTOW reduce MM-SafetyBench attack success from 0.6726 to 0.0298 and FigStep from 0.3320 to 0.0, so the method's effect is large enough to be visible on individual widely used benchmarks.","General capability holds: MMMU accuracy is 0.5422 after training versus 0.5411 for the base model, with over-refusal at 0.0078 rather than the 0.2056 seen with plain supervised fine-tuning.","The ablations show the iteration mechanism, both monitoring mechanisms, and the cold start are all load-bearing; removing any one raises ASR by roughly 4x to over 10x on some benchmarks.","The attacker's own success rate rises from 0.3393 after one iteration to 0.5509 after three, meaning the defender is being trained against progressively stronger attacks rather than the same fixed set."],"supporting_citations":[{"why":"Supplies GRPO, the group-relative policy optimization algorithm used to train both the defender and the attacker.","marker":"(Shao et al., 2024)"},{"why":"Provides the VL-Guard jailbreak and general datasets that seed the training loop and the supervised fine-tuning cold start.","marker":"(Zong et al., 2024)"},{"why":"Provides the independent safety evaluation model that computes the attacker's attack-success reward and filters generated jailbreak data.","marker":"(Chi et al., 2024)"},{"why":"Contributes the JailBreakV-28k benchmark used to measure attack success rate for both defender and attacker.","marker":"(Luo et al., 2024)"},{"why":"Contributes the FigStep typographic-jailbreak benchmark on which SecTOW reaches zero attack success.","marker":"(Gong et al., 2023)"},{"why":"Contributes the MM-SafetyBench benchmark that shows the headline drop from 0.6726 to 0.0298.","marker":"(Liu et al., 2024b)"},{"why":"Contributes the SafeBench safety benchmark used in the main evaluation table.","marker":"(Ying et al., 2024)"},{"why":"Contributes the MMMU general benchmark used to show accuracy and over-refusal rate stay near base levels.","marker":"(Yue et al., 2024a)"},{"why":"Contributes the MMMU-Pro general benchmark used as the second check that general performance is preserved.","marker":"(Yue et al., 2024b)"},{"why":"Provides Qwen2-VL-7B, the base model from which both the SecTOW defender and attacker are initialized.","marker":"(Wang et al., 2024a)"}],"fun_headline_variants":["Iterative attacker-defender RL cuts jailbreak rate from 67% to 3%","Three-round RL loop drops jailbreak success to 3% on multimodal model","SecTOW attack-defense RL: jailbreak rate falls from 67% to 3%","RL attacker and defender loop hardens MLLM, jailbreak 67% to 3%","Iterative attacker-defender RL training secures model, jailbreak to 3%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire loop assumes the external safety judge used for attacker rewards and data filtering correctly recognises harmful responses; if that judge is biased, noisy, or gameable, the defender learns to satisfy the judge rather than to be genuinely safe.","fun_headline_variants_meta":{"raw":{"variants":["Iterative attacker-defender RL cuts jailbreak rate from 67% to 3%","Three-round RL loop drops jailbreak success to 3% on multimodal model","SecTOW attack-defense RL: jailbreak rate falls from 67% to 3%","RL attacker and defender loop hardens MLLM, jailbreak 67% to 3%","Iterative attacker-defender RL training secures model, jailbreak to 3%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001446,"raw_usage":{"total_tokens":5858,"prompt_tokens":1008,"completion_tokens":4850,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":624,"completion_tokens_details":{"reasoning_tokens":4735}},"tokens_in":624,"tokens_out":4850,"duration_ms":35560,"temperature":1.0,"reasoning_tokens":4735,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T12:04:41.242371+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Score the three-iteration SecTOW defender on the same jailbreak inputs using human annotators instead of the automated safety judge it was trained against; if human-scored attack success approaches the base model's 0.6726 on MM-SafetyBench, the reported defence is an artefact of judge-specific optimisation rather than genuine robustness.","supporting_citations":[],"review_version":1}