{"id":"d638ab4a-ee2f-4d5f-a044-9ab4f4e21037","arxiv_id":"2509.00373","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A proposed VLM defense, SPO-VLM, combines activation steering with sequence-level preference optimization and claims lower jailbreak ASR and toxicity than ASTRA while retaining visual understanding.","lead":"SPO-VLM is a two-stage defense that learns activation steering vectors for vision language models via contrastive prompts plus PPO-style preference optimization, reporting lower toxicity and jailbreak success than ASTRA. Its safety gains are partly evaluated on the same datasets used to train the steering vectors, and the optimization as derived is nonstandard, so the results as stated need repair before they are relied upon.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (5) defines the PPO ratio as a difference of two likelihood ratios, which is not a valid importance sampling ratio; Stage II's gradient update is therefore undefined and the central safety claim lacks a sound optimization basis.","rationale":"The reader's weakest assumption points to Eq. (5) as an unvalidated policy ratio, and I agree that this is the most load-bearing concern. Stage II is the paper's main novel contribution and the mechanism by which the steering vector is supposedly refined. If Eq. (5) is not a valid importance sampling ratio, the PPO objective in Eq. (7) is not a correct surrogate for maximizing the reward, and the entire optimization step is undefined. This is not merely a matter of 'nonstandard' but of internal consistency: the loss does not correspond to any clear objective, so the reported safety improvements cannot be explained by the proposed method. The paper provides no code, no formal derivation, and no ablation that isolates the effect of the ratio. A simple re-derivation or a small-scale reproduction would settle this. The secondary issue with R_visual(qv) is also real but less central; even if that reward were fixed, the invalid ratio would still undermine the method. Accordingly, the reader's REJECT verdict is appropriate, though a repair path exists: replace Eq. (5) with a standard ratio or derive a correct estimator and provide code.","tokens_in":12398,"tokens_out":3261,"duration_ms":42259,"concrete_test":"Independently derive the gradient of Eq. (7) with respect to v for a single quadruplet and compare it to the gradient of a well-defined objective such as J(v)=E_{r~π_v}[R(r)] (or a standard preference objective). If the two gradients differ, or if the ratio in Eq. (5) is not positive, the update is not a valid policy gradient. As an empirical complement, re-run the Qwen2-VL-7B experiment with Eq. (5) replaced by a standard per-response PPO ratio (ratio_T and ratio_O computed separately, each with its own advantage) and check whether the safety results are statistically unchanged; if they are, the claimed benefit does not depend on the proposed ratio.","verdict_should_be":"REJECT","load_bearing_attack":"The central methodological novelty is Stage II, where a steering vector v is refined via a PPO-style loss (Eq. 7). The policy ratio in Eq. (5) is written as the difference between two likelihood ratios: π_{L+1}(r_T | a+v)/π_{L+1}(r_T | a) − π_{L+1}(r_O | a+v)/π_{L+1}(r_O | a). In standard PPO, the ratio must be π_θ(x)/π_old(x) for a single sampled action x. Here the two terms involve different responses r_T and r_O, and the subtraction can produce negative values, which is not a probability ratio. No derivation is given to show that this expression is a valid importance sampling estimator or that its gradient corresponds to the gradient of any well-defined objective. The advantage A is computed from a single reward R(r) (Eq. 6), but the ratio mixes r_T and r_O, so the expectation over D has no consistent Monte Carlo target. If Eq. (5) is not a valid estimator, the update rule in Algorithm 1 is not optimizing the stated reward, and the safety improvements in Table 1 cannot be attributed to the proposed preference-optimization mechanism. Additionally, R_visual(qv) in Eq. (6) depends only on the image, not on the generated response, so it cannot differentiate r_T from r_O; this further weakens the claim that visual grounding is actively optimized, though the invalid ratio is the more fundamental issue.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SPO-VLM, a two-stage defense for vision-language models against jailbreak attacks. Stage I constructs an initial steering vector as a weighted combination of difference-in-mean activation vectors computed from contrastive datasets (RealToxicityPrompt, AdvBench, Anthropic Harmful). Stage II refines this vector through a sequence-level preference-optimization procedure: a PPO-style clipped objective is applied to a 'policy ratio' defined in Eq. (5), with rewards combining a detoxification score and a visual-consistency term. Experiments on MiniGPT-4-13B, Qwen2-VL-7B, and LLaVA-v1.5-13B report lower toxicity and jailbreak ASR than ASTRA while preserving or improving visual-understanding scores on several benchmarks. The paper also reports transfer results on MM-SafetyBench and an ablation on Qwen2-VL-7B.","tokens_in":12867,"tokens_out":1980,"duration_ms":25755,"significance":"If the proposed optimization were valid and the evaluations were unbiased, the work would be a meaningful contribution: activation-space intervention with preference-based refinement is a plausible direction, and the reported safety improvements over ASTRA are substantial. The paper also promises code, model weights, and an evaluation toolkit, which is commendable for reproducibility. However, the central methodological novelty—the PPO-style update of Eq. (7)—rests on a policy ratio that is not a valid importance-sampling estimator, and the visual reward does not depend on the generated response. These issues are load-bearing rather than cosmetic. The evaluation also shares training and evaluation data/reward metrics, making the main safety claims difficult to interpret. The significance of the paper is therefore contingent on a reworking of the optimization objective and a decontaminated evaluation.","major_comments":[{"comment":"The quantity in Eq. (5) is not a PPO policy ratio. Standard PPO uses π_θ(a|s)/π_old(a|s) for a single sampled action a. Here the expression is a difference of two likelihood ratios involving two different responses r_T and r_O. This difference can be negative or zero and does not correspond to the probability ratio of any single sampling distribution. No derivation is given showing that its expectation relates to the gradient of a well-defined objective. Consequently, the clipped objective in Eq. (7) and the gradient update in Algorithm 1 are not justified. Since Stage II is the claimed novelty, this invalidates the central mechanism by which SPO-VLM is said to refine steering vectors.","section":"Eq. (5) and Algorithm 1, line 9"},{"comment":"R_visual(qv) depends only on the visual input qv, not on the generated response r. As written, it assigns the same reward to the preferred response r_T and the dispreferred response r_O. Thus it cannot provide a visual-consistency preference signal between the two responses, contrary to the text's claim that 'visual-consistency rewards based on caption-image alignment' guide optimization. The reward also appears to compare the image to a caption, but no caption variable is defined in the tuple (qt, qv, r_T, r_O). This needs clarification or correction.","section":"Eq. (6)"},{"comment":"The evaluation is circular in two respects. First, the steering vectors are initialized from RealToxicityPrompt, AdvBench, and Anthropic Harmful, and the main safety evaluation measures exactly those benchmarks. Second, the toxicity reward R_detoxify uses the Detoxify classifier, and the toxicity evaluation metric is also Detoxify. Thus the reported in-distribution safety gains may reflect overfitting to the training/evaluation signal rather than generalized defense. The transfer experiment on MM-SafetyBench mitigates this concern partially, but the primary safety claims still need a decontaminated evaluation or a clear separation of steering-vector construction data from evaluation data.","section":"Section 5.1 and Table 1"}],"minor_comments":[{"comment":"The objective is written as min_v over the expectation, which is inconsistent with the usual PPO maximization and with Algorithm 1's gradient-descent update. Please clarify the sign convention and the optimization direction.","section":"Eq. (7)"},{"comment":"The preference dataset D is not described: how are the target responses r_T and undesired responses r_O constructed? Is r_O generated by the unsteered model, by a harmful model, or by a hand-crafted template? This is important for interpreting Eq. (5).","section":"Section 4.2"},{"comment":"Hyperparameters α_j, β, ε, c1, learning rate η, and batch size m are not reported. The paper lists some inference settings but omits the training hyperparameters needed for reproducibility.","section":"Section 5.1"},{"comment":"No standard deviations or multiple-seed results are reported. Given the small differences (e.g., Qwen2-VL AdvBench 7.69 vs. 6.38), it is unclear whether the improvements are statistically significant.","section":"Table 1"},{"comment":"The text states 'SPO-VLM reduces the average ASR on AdvBench by approximately 1.28%' and 'cuts the average ASR on Anthropic Harmful nearly in half'. These numbers appear inconsistent with Table 1; please recompute and state the calculation basis.","section":"Section 5.2"}],"recommendation":"reject","confidential_remarks":"The invalid policy ratio in Eq. (5) is a fundamental error that cannot be fixed by local edits; it requires re-deriving the optimization objective. The circular evaluation (training on the same benchmarks and using Detoxify both as reward and metric) independently weakens the empirical claims. I would not consider acceptance unless the optimization is reformulated and the evaluation is decontaminated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead SPO-VLM (arXiv:2509.00373). The short version: the empirical results are attractive, but the paper has a load-bearing math problem. Eq. (5) defines the PPO policy ratio as a difference of two likelihood ratios, one for the target response and one for the undesired response. That is not a valid importance-sampling ratio. Standard PPO needs π_θ(x)/π_old(x) for the single action you actually sampled. Here the ratio is a function of two different sequences, can go negative, and no derivation is given to show it is an unbiased gradient estimator. Without that, the update rule in Algorithm 1 is not actually optimizing the reward in Eq. (6). This is not a minor typo; it is the core of Stage II.\n\nThere is also a secondary issue: R_visual(qv) depends only on the image, not on the generated response, so it cannot discriminate between target and undesired outputs. As written, it contributes no gradient signal at all.\n\nTo give credit where it is due: the two-stage design (difference-in-means initialization plus reward-driven refinement) is a reasonable new combination, and the authors correctly identify ASTRA's utility degradation as a real problem. The evaluation covers three models, four visual benchmarks, and includes an out-of-distribution transfer test on MM-SafetyBench. That is more than many safety papers do. The ablation is informative and suggests Stage II does something useful on top of Stage I, even if we don't fully understand what.\n\nThe biggest methodological concern beyond the PPO issue is the circularity: the steering vectors are trained and evaluated on the same three safety datasets, and the toxicity reward uses the same Detoxify classifier that defines the toxicity metric. The MM-SafetyBench transfer result is the only clean out-of-distribution evidence, and it's only a single attack family. The paper's claim of \"generalized suppression\" is therefore not fully supported.\n\nThe authors also compare only against ASTRA and the original model. No ShiftDC, no other defense, no statistical significance. That limits the strength of the empirical claims.\n\nOverall: the idea is worth pursuing, and the results might be real, but the paper as written does not establish that the optimization is sound. I would send it to peer review because the problem is important and the empirical story, if reproducible, is a step forward. But the reviewers need to force a proper derivation of the ratio, or a re-framing of the method as a heuristic that happens to work. Right now the central mechanism is not well-defined.\n\nRecommendation: serious referee, expect major revision.","headline":"The empirical story is promising, but the PPO update in Eq. (5) is not a valid importance-sampling ratio, so the central claim that Stage II optimizes the stated reward is not supported as written.","tokens_in":13278,"tokens_out":3245,"would_cite":false,"duration_ms":35465,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SPO-VLM defends vision language models against jailbreaks by initializing steering vectors from contrastive activations, then refining them with sequence-level preference optimization; the paper reports lower attack success than prior steer","keywords":["vision language models","activation steering","jailbreak defense","preference optimization","PPO","multimodal safety","visual grounding"],"falsifier":"Take a set of preference pairs, compute the paper's training-ratio quantity for each pair, and compare it against the actual likelihood ratio of the steered and unsteered models on the same response. If the two disagree in sign or magnitude, the clipped PPO objective is not being applied to a valid probability ratio. Separately, replace the visual-consistency reward with a constant and rerun the main table; if attack success and visual scores do not move, that reward term contributes nothing.","tokens_in":12329,"feed_emoji":"🛡️","tokens_out":11391,"duration_ms":123145,"temperature":0.7,"pith_summary":"Vision-language models are vulnerable to jailbreaks that hide harmful instructions in images. SPO-VLM defends them without retraining: it first computes layer-specific steering vectors from contrastive safe/unsafe prompt data, then refines a combined vector with sequence-level preference optimization under a PPO-style objective. The paper reports that on Qwen2-VL-7B, MiniGPT-4-13B, and LLaVA-v1.5-13B, this lowers toxicity and jailbreak attack success rates below the ASTRA baseline on RealToxicityPrompt, AdvBench, and Anthropic Harmful, while visual benchmarks like MM-Vet, SQA, CogVLM, and MME stay close to or above the unmodified model. If the results hold, the method offers a way to harden multimodal models without weight updates or utility loss.","feed_headline":"Two-stage steering stops multimodal jailbreaks without vision loss","feed_subtitle":"Qwen2-VL-7B jailbreak success falls from 75% to 6.4% while visual benchmarks hold or improve.","key_machinery":"The load-bearing object is the layer-ℓ steering vector v^ℓ, a single direction in the residual stream. Stage I initializes it as v^ℓ = Σ_j α_j v^ℓ_j, with v^ℓ_j a difference-in-mean activation between positive and negative instruction sets; Stage II updates it through Eq. (7), a clipped PPO objective whose policy ratio (Eq. 5) compares the steered and unsteered probabilities of preferred and dispreferred responses. The base VLM stays frozen; all behavioral change comes from this vector.","core_discovery":"SPO-VLM's central claim is that the steering vector itself can be the only learnable parameter in a defense. Stage I forms it as a weighted sum of difference-in-mean activations from safe and unsafe prompt sets; Stage II applies it to late-layer residual streams during rollouts and optimizes it with sequence-level PPO. The reward combines a toxicity penalty from Detoxify and a visual-consistency term based on cosine similarity between image and caption tokens. Results: on Qwen2-VL-7B, AdvBench jailbreak ASR drops from 75.00 to 6.38 and MM-Vet rises from 49.13 to 49.80; on MiniGPT-4-13B and LLaVA-v1.5-13B, ASR is lower than ASTRA with less visual degradation, and the defense transfers to unse","pith_inferences":["The recipe is not safety-specific: swapping the reward terms could steer other behavior axes—hallucination, sycophancy, or refusal calibration—without retraining the base model. The paper demonstrates only safety.","Making the Stage-I weights α learnable in Stage II, instead of the fixed 0.5/0.4/0.4 combination, would let the data decide how much each toxicity attribute contributes.","The same sequence-level ratio could support online adaptation: a deployed model could update its steering vector on fresh preference pairs without weight updates, if Eq. (5)'s estimator is valid.","Applying the sequence-level PPO objective to text-only LLMs would clarify how much of the gain comes from the Eq. (5) ratio versus the visual rewards, since text-only steering baselines are more developed."],"forward_implications":["SPO-VLM reports the lowest jailbreak ASR among the compared methods on all three VLMs; for Qwen2-VL-7B, AdvBench ASR falls from 75.00 (original) to 7.69 (ASTRA) to 6.38 (SPO-VLM).","The defense generalizes to structure-based attacks from MM-SafetyBench that were not used in training, including combined SD+OCR attacks.","Visual understanding is preserved or improved: Qwen2-VL-7B MM-Vet rises to 49.80 from 49.13, while ASTRA drops to 48.66; MiniGPT-4 and LLaVA also retain much more of their visual scores than ASTRA.","Because only the steering vector is updated, the defense can be applied inference-time without changing model weights, making it modular and cheaper than full RLHF fine-tuning.","Ablation shows Stage I alone reduces Qwen2-VL-7B AdvBench ASR from 75.00 to 10.96; adding Stage II cuts it to 6.38, showing the preference-optimization stage is load-bearing for the final safety gain."],"supporting_citations":[{"why":"Provides the ASTRA baseline that SPO-VLM is compared against and whose adaptive steering formulation Stage I extends.","marker":"(Wang et al., 2024a)"},{"why":"Supplies the clipped policy-gradient objective that Stage II adapts to steering-vector updates.","marker":"(Schulman et al., 2017)"},{"why":"Provides the Detoxify toxicity scores used in the R_detoxify reward and in RealToxicityPrompt evaluation.","marker":"(Hanu and Unitary team, 2020)"},{"why":"RealToxicityPrompts benchmark used for Stage I steering data and toxicity evaluation.","marker":"(Gehman et al., 2020)"},{"why":"AdvBench jailbreak benchmark and attack prompts used to measure attack success rate.","marker":"(Zou et al., 2023)"},{"why":"Anthropic Harmful dataset used for Stage I steering data and ASR evaluation.","marker":"(Ganguli et al., 2022)"},{"why":"HarmBench classifier used to compute the jailbreak attack success rate.","marker":"(Mazeika et al., 2024)"},{"why":"MM-SafetyBench structure-based attack benchmark used to test cross-attack transfer.","marker":"(Liu et al., 2024b)"},{"why":"BiPO preference optimization of steering vectors motivates the use of preference-labeled pairs to refine v.","marker":"(Cao et al., 2024)"},{"why":"InferAligner inference-time activation steering is the antecedent for Stage I's intervention.","marker":"(Wang et al., 2024c)"}],"fun_headline_variants":["Jailbreak rate on VLM drops 75% to 6% with learned steering vector","SPO-VLM: single learnable steering vector blocks VLM jailbreaks","Steering vector plus PPO cuts VLM jailbreaks to 6.4%","Adaptive steering beats contrastive prompts in VLM defense","Preference-optimized steering defends VLMs without vision loss"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The load-bearing premise is that the quantity called the policy ratio in the training update can be used exactly like the probability ratio PPO is designed for, and that the visual-consistency reward, which by its formula depends only on the image, can tell good responses from bad ones; if either premise fails, the second-stage update does not do what the paper claims.","fun_headline_variants_meta":{"raw":{"variants":["Jailbreak rate on VLM drops 75% to 6% with learned steering vector","SPO-VLM: single learnable steering vector blocks VLM jailbreaks","Steering vector plus PPO cuts VLM jailbreaks to 6.4%","Adaptive steering beats contrastive prompts in VLM defense","Preference-optimized steering defends VLMs without vision loss"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000946,"raw_usage":{"total_tokens":3924,"prompt_tokens":838,"completion_tokens":3086,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":582,"completion_tokens_details":{"reasoning_tokens":2996}},"tokens_in":582,"tokens_out":3086,"duration_ms":24665,"temperature":1.0,"reasoning_tokens":2996,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T13:39:19.439432+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of preference pairs, compute the paper's training-ratio quantity for each pair, and compare it against the actual likelihood ratio of the steered and unsteered models on the same response. If the two disagree in sign or magnitude, the clipped PPO objective is not being applied to a valid probability ratio. Separately, replace the visual-consistency reward with a constant and rerun the main table; if attack success and visual scores do not move, that reward term contributes nothing.","supporting_citations":[],"review_version":1}