{"id":"f62418c0-a5ee-453a-b8c6-97de1b4652cf","arxiv_id":"2608.10513","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Training LVLMs to produce safety-relevant image captions before answering, with a frozen-LLM caption reward, raises multimodal safety average by up to 19 points without lowering vision utility.","lead":"SafeCap trains a vision-language model to caption an image before answering, and rewards captions that let a separate text-only model reach the same safe conclusion. It raises average safety scores by up to 19 points under its caption-based protocol while keeping vision-utility scores about flat.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SafeCap's training reward is image-blind: caption, answer, and alignment judges see no image, so visual grounding is never rewarded; the alignment term can be satisfied from question text alone, leaving the claimed 'expose visual cues' mechanism unverified.","rationale":"The paper is a coherent empirical study with a clear protocol, consistent DirectCap gains, ablations, a 3-seed stability check, and a 100-sample human review. These are real supporting evidence. The reader's weakest assumption — that the reward and judge models are trustworthy safety arbiters — is correct, and my reading finds the issue more specific: every reward component is image-blind, and the alignment term g can be satisfied from question text alone. This does not disprove the benchmark results, but it means the central mechanistic claim (captions expose visual cues) is not actually verified by the reward. The safety gains could instead come from an image-blind objective that rewards rubric-compliant text and agreement. The caption-replacement and image-grounded human evaluation described above would settle whether the gains are grounded in visual evidence. Since the paper already acknowledges the caption factuality limitation in Appendix D and the empirical results are otherwise consistent, I keep the reader's CONDITIONAL verdict rather than escalating to rejection; the additional conditions are (1) an independent image-grounded safety/grounding audit and (2) confirmation that the alignment signal depends on caption content, not just question text.","tokens_in":20121,"tokens_out":15300,"duration_ms":141316,"concrete_test":"Concrete test: Using the released Qwen3.5-4B-Base SafeCap checkpoint, sample 300 prompts from MM-SafetyBench, VLSBench, and FigStep. For each SafeCap DirectCap output, record policy answer a, caption c, frozen-LLM answer a_f, and alignment score g. Re-run the frozen LLM plus safety-alignment judge with (i) the question only (no caption) and (ii) a caption with the safety-relevant object/attribute removed, keeping a fixed. If g=1 for a substantial fraction of harmful cases under either replacement, the caption reward does not enforce caption-carried visual evidence. Separately, have annotators who see the original image rate final-answer safety and caption factual accuracy; if automatic safety scores stay high while caption grounding is low (e.g., <90% accurate), the image-blind reward is improving judged safety without reliable visual grounding.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Load-bearing concern: the training reward is computed entirely without access to the image. In Appendix J, the answer-retention and answer-risk judges receive only the question and the policy's text; the caption-retention judge receives only the caption and is explicitly instructed 'do NOT judge factual correctness'; the frozen text-only LLM and the safety-alignment judge likewise see only text. Appendix D concedes: 'Since the frozen LLM does not observe the image, the reward cannot by itself identify factual errors in a caption.' This is not merely a caption-side limitation: the final-answer reward is image-blind too, so a plausible but hallucinated refusal can earn high retention and low risk. The problem is compounded by a confound in the alignment term g: for prompts whose harmful intent is stated in the question (common in MM-SafetyBench/MSSBench), the frozen LLM recognizes the risk from the question alone, so g=1 even when the caption omits the safety-relevant visual content. The 'caption-mediated' reward then degenerates into a descriptive-coverage reward, and the claimed mechanism — exposing visual cues — is not actually enforced. The 100-sample human review (Appendix G) and manual trajectory monitoring are useful but small, informal checks, not systematic grounding verification. SafeCap's benchmark gains are real under the DirectCap protocol, but they do not by themselves establish that the model is safer because it perceives visual risk; they may reflect an image-blind RL objective that teaches rubric-compliant refusal and plausible text.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SafeCap, a reinforcement-learning framework that trains a vision-language model to emit a structured <caption>...answer output. The training objective combines a template gate, a caption-mediated reward (coverage score plus a frozen text-only LLM's safety-alignment consistency answer), and a direct answer reward with an exponential risk discount. The method is evaluated on five multimodal safety benchmarks and six vision-utility benchmarks under three inference protocols (Direct, DirectCap, Prism) across four Qwen3.5 model settings, and is compared against safety SFT, DPO, and SafeGRPO. The main claim is that SafeCap improves aggregate safety under its intended DirectCap protocol by 3.7–19.0 points while maintaining or improving vision utility.","tokens_in":20430,"tokens_out":5711,"duration_ms":58911,"significance":"If the reported gains are robust and not artifacts of the evaluation protocol, SafeCap is a practical and inexpensive alignment method: it uses a public dataset, a frozen text-only LLM and judge, and requires no additional visual supervision. The paper includes useful strengths: consistent DirectCap gains across four model settings, ablations showing both reward components matter, seed-level robustness at an early checkpoint, alternative reward-form comparisons, refusal-rate diagnostics, and a small human review. However, the central mechanistic claim — that the caption reward teaches the model to expose genuine visual cues — is not directly verified, because the reward and most safety evaluations are produced by text-only LLM judges. The paper's significance therefore hinges on additional evidence connecting the reward to factual visual grounding.","major_comments":[{"comment":"The caption reward is computed entirely without access to the image. The caption-retention judge is explicitly instructed \"do NOT judge factual correctness,\" the frozen LLM never sees the image, and the safety-alignment judge compares only the two text answers. Thus a detailed but hallucinated caption — or even a caption that is unrelated to the image — can receive a high coverage score and, whenever the frozen answer agrees with the policy answer on risk status, a high caption reward. For prompts whose harmful intent is stated in the question (common in MM-SafetyBench and MSSBench), the alignment gate g can equal 1 regardless of the caption content. The claimed mechanism that the caption \"exposes visual cues relevant to safe response generation\" is therefore not enforced by Eq. (5); the objective is, at best, a descriptive-coverage reward with a consistency gate. The ablation in Table 4 shows the caption reward contributes to scores, but it does not show the contribution arises from factual visual grounding rather than generic descriptive style. I request evidence of caption-image factual consistency on safety-critical samples (e.g., VQA-style factual checks or human factual ratings) and a diagnostic that varies the caption content while holding the question fixed to demonstrate that the alignment gate is actually sensitive to the caption's visual content.","section":"Caption-Mediated Reward, Eq. (5); Appendix J"},{"comment":"The training reward uses an LLM judge (gpt-oss-20b), and the evaluation pipeline relies on automatic LLM judges for at least FigStep and VLSBench. Because the reward and evaluation pipelines are built from the same LLM-as-judge technology, correlated judge biases (e.g., preference for certain refusal phrasing or verbose descriptive style) can inflate the reported safety gains without reflecting genuine refusal of harmful requests. The 100-sample human review in Appendix G is a welcome check, but it is small relative to the full benchmark suite and reports only an overall consensus, not per-benchmark or per-category agreement. I ask for either a larger human-validated subset reported separately for the safety-critical benchmarks, or for at least one safety benchmark scored by a rule-based or fully human protocol, to rule out judge-family bias as the source of the headline gains.","section":"Benchmarks and metrics; Appendix G"},{"comment":"The main training results in Table 2 are single-run point estimates. The three-seed robustness check is conducted only at a matched 100-step early checkpoint, not at the 200-step final checkpoints that produce the headline numbers (e.g., the +19.0-point DirectCap safety gain for 4B-Base). Because the paper's central claim is built on these aggregate deltas, I request variance information at the final training step, or at least a clear statement that the final checkpoint was selected by a fixed schedule rather than by inspecting benchmark results. Without this, the reader cannot distinguish a stable method from a favorable random seed.","section":"Table 2; Appendix A"}],"minor_comments":[{"comment":"Panel (a) has a typo: \"Visaul Safety Gap\" should be \"Visual Safety Gap.\"","section":"Figure 1"},{"comment":"The sentence \"The SFT and DPO baselines in Appendix F use the same backbone ... and preform same training steps\" contains a typo: \"preform\" should be \"perform.\"","section":"Appendix A"},{"comment":"The tables are dense because deltas are embedded as superscripts next to absolute scores. A separate delta table or a cleaner two-row layout per model/protocol would improve readability and reduce the risk of misreading sign and magnitude.","section":"Tables 1 and 2"},{"comment":"The notation uγh is ambiguous because γ is a real coefficient and h is an integer exponent. Please write u(a)γ^{h(a)} and define the convention explicitly (e.g., integer power) to avoid confusion with a subscript or concatenation.","section":"Eq. (3)"}],"recommendation":"major_revision","confidential_remarks":"The empirical contribution is solid and the paper is generally well written, but the headline claim is stated in terms of a caption-mediated safety mechanism that the reward design does not actually enforce. The requested grounding evidence is feasible: evaluate caption-image factual consistency on safety-critical samples, and show that the alignment gate is sensitive to the caption's visual content. If those experiments support the mechanism, I would be willing to recommend acceptance. The single-run main tables and the correlated-judge issue are also addressable within a revision. No concerns about novelty or citation pattern beyond the usual need to cite the original benchmark protocols when modifying them."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: SafeCap is a real, well-tested training-time method for LVLM safety, and its headline result—consistent safety gains under the DirectCap protocol across four model settings—holds up under scrutiny. Second thing: the stress-test note is basically right. The reward never sees the image. Every judge, the frozen LLM, the alignment check—all text-only. So the paper's claim that caption-mediated RL teaches the model to 'expose visual cues' is not actually enforced by the training objective. A detailed but hallucinated caption can earn full caption reward on any prompt where the question itself is benign. The authors concede this in Appendix D ('the reward cannot by itself identify factual errors in a caption'), and they try to patch it with manual trajectory monitoring and a 100-sample human review, but that review checks judge agreement, not caption-image grounding. This is a load-bearing gap in the mechanism story, not a minor footnote.\n\nWhat's genuinely new: the combination of a caption-mediated reward with a frozen text-only LLM as the safety arbiter, plus decoupled normalization and a direct answer reward. The ablation study is well done—every component matters, and the risk-coefficient ablation is informative. The multi-protocol evaluation (Direct, DirectCap, Prism) is a good idea, and the paper is honest about which protocol is the intended one. The 3-seed early-checkpoint robustness check (t-test p=0.004) is better than nothing, but it's at 100 steps, not the full 200-step run, so the main tables remain single-seed.\n\nThe weakest part beyond grounding is the judge overlap: the same LLM-as-judge family scores both training rewards and several evaluation benchmarks. The 100/100 human agreement on evaluation labels is reassuring but small. And the SafeGRPO comparison mixes protocols (SafeCap under DirectCap vs SafeGRPO under Base), though SafeCap also wins under Base, so it's not fatal.\n\nWho should read this: anyone working on multimodal safety alignment or RL for LVLMs. The method is directly usable and the failure mode analysis is useful. I'd send it to review—it's a solid empirical contribution that needs revision, not a desk reject. The reviewers should demand (a) an image-grounded caption check on a held-out sample, (b) full-seed statistics for the main tables, and (c) refusal-rate comparisons against zero-training.","headline":"SafeCap's DirectCap safety gains are real and the ablations are solid, but the reward is computed entirely in text, so the 'visual grounding' mechanism is asserted rather than demonstrated; worth serious review with mandatory grounding checks.","tokens_in":21001,"tokens_out":5055,"would_cite":true,"duration_ms":47097,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Caption-mediated reinforcement learning improves large vision-language model safety by up to 19 points on aggregate benchmarks while preserving vision utility.","keywords":["LVLM safety","image captioning","reinforcement learning","jailbreak defense","caption-mediated reward","multimodal safety alignment","group-relative policy optimization","safety benchmarks"],"falsifier":"Feed a SafeCap-trained model a set of images where the generated caption contains plausible but hallucinated safety-critical details (for example, a weapon not present in the image) and check whether the refusal rate tracks the actual risk; a high refusal rate on benign images with hallucinated hazards would show the reward learned to fabricate evidence rather than recognize real danger.","tokens_in":19946,"feed_emoji":"🛡️","tokens_out":11273,"duration_ms":83968,"temperature":0.7,"pith_summary":"SafeCap claims that large vision-language models (LVLMs) can be made safer against jailbreak attacks by training them to first generate a safety-relevant caption of the input image, then answer the user's question. The caption is optimized not for factual accuracy alone, but for whether it enables a frozen text-only language model to reach a safety-aligned decision that agrees with the model's own answer. Across five multimodal safety benchmarks and six vision-utility benchmarks, SafeCap improves aggregate safety by 3.7 to 19.0 percentage points across four model settings while maintaining comparable or improved vision utility. If correct, this means captioning can serve as a trainable safety interface that preserves helpfulness, rather than a simple refusal wrapper.","feed_headline":"Captioning-first training boosts LVLM safety by up to 19 points","feed_subtitle":"A caption-before-answer path keeps vision performance intact while blocking jailbreaks across 11 benchmarks.","key_machinery":"The load-bearing mechanism is the caption-mediated reward, a composite of three components: (1) a template gate that enforces exactly one well-formed caption block and a non-empty answer; (2) a caption reward that combines a frozen text-only LLM's safety-alignment agreement with the policy's answer (a binary judge signal) and a rubric-based descriptive-coverage score; and (3) a direct answer reward using the exponential risk discount $S(u,h)=u\\gamma^h$ with $\\gamma=0.35$. These values are group-normalized component-wise before being combined into the policy-gradient advantage. This design steers the policy to expose visual evidence that is both detailed and sufficient for a text-only reasoner to make a safe decision, rather than simply refusing or parroting generic disclaimers.","core_discovery":"SafeCap's central discovery is that a caption-mediated reinforcement learning objective raises LVLM safety more effectively than direct refusal supervision or standard safety fine-tuning. The policy is trained with a variant of group-relative policy optimization to emit a structured response: a caption block followed by the answer. The caption is rewarded when it lets a frozen text-only LLM produce an answer whose safety status agrees with the policy's own answer, and when it covers concrete visual details; the answer is rewarded with an exponential risk discount, $S(u,h)=u\\gamma^h$, that suppresses risky content without zeroing out useful safe answers. The intended operating point, DirectCap, shows the most consistent safety gains, while the diagnostic Direct and Prism paths are more model-dependent. The authors conclude that learned self-captioning, rather than inference-time caption wrapping or direct refusal, is the reliable path to multimodal safety alignment.","pith_inferences":["The paper's claim that judges do not check caption factuality raises a testable risk: if captions hallucinate safety-relevant details, training could inflate judged safety without improving real-world safety; a follow-up could measure refusal rates on benign images with hallucinated hazards.","The exponential risk-discount reward form might generalize beyond LVLMs to any alignment problem where helpfulness and safety must be balanced without rewarding cheap refusals; one could test it on text-only safety alignment.","Because the caption is trained to support a frozen LLM's safety decision, SafeCap captions could be reused as a lightweight inspection interface for human oversight or for training smaller safety classifiers.","The method's reliance on a frozen LLM as judge suggests its safety gains may be bounded by the judge's own risk recognition; replacing the frozen LLM with a stronger one could push gains further, but also risks introducing judge-specific bias."],"forward_implications":["If SafeCap is correct, multimodal safety alignment can be achieved through a trainable captioning interface that preserves vision utility.","The consistent safety gains across four model settings suggest the method transfers across different backbones and initialization states.","Under matched training data and steps, SafeCap outperforms safety SFT, DPO, and a recent rule-governed GRPO baseline, indicating that caption-mediated RL is a stronger safety objective.","The Prism diagnostic, where a frozen text-only LLM answers from the caption alone, shows captions retain enough evidence to transfer safety judgments to a stronger frozen reasoner."],"supporting_citations":[{"why":"Supplies the public safety-preference dataset used to train SafeCap in all main runs.","marker":"Zhang et al. 2024"},{"why":"Defines the group-relative policy optimization algorithm that SafeCap uses to update the policy.","marker":"Shao et al. 2024"},{"why":"Introduces caption-conditioned reinforcement learning that SafeCap adapts from perceptual VQA to safety alignment.","marker":"Xing et al. 2025"},{"why":"ECSO, the prior caption-mediated defense that motivates the caption-channel idea and whose limitations SafeCap addresses.","marker":"Gou et al. 2024"},{"why":"SafeGRPO, a strong RL safety baseline that SafeCap is compared against on matched data and steps.","marker":"Rong et al. 2025"},{"why":"Provides the MM-SafetyBench benchmark used to measure refusal of visually embedded harmful instructions.","marker":"Liu et al. 2024"},{"why":"FigStep, a typographic-image jailbreak benchmark that tests whether captions expose visible text hazards.","marker":"Gong et al. 2023"},{"why":"GDPO, whose decoupled group normalization inspires the component-wise normalization in the reward.","marker":"Liu et al. 2026"}],"fun_headline_variants":["SafeCap: caption-based RL lifts LVLM safety by up to 19 points","Caption-first reinforcement learning beats direct safety tuning for LVLMs","Reinforced self-captioning adds up to 19 safety points to LVLMs","LVLM safety jumps 19 points with caption-mediated RL","SafeCap: learn to caption, then answer safely—up to 19 points"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reward signal assumes the frozen text-only language model and the judge models are trustworthy safety arbiters; the caption-retention judge explicitly does not verify factual accuracy, so if captions are detailed but hallucinated, or if judges reward wordy refusals, training can raise judged safety without raising real safety.","fun_headline_variants_meta":{"raw":{"variants":["SafeCap: caption-based RL lifts LVLM safety by up to 19 points","Caption-first reinforcement learning beats direct safety tuning for LVLMs","Reinforced self-captioning adds up to 19 safety points to LVLMs","LVLM safety jumps 19 points with caption-mediated RL","SafeCap: learn to caption, then answer safely—up to 19 points"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000265,"raw_usage":{"total_tokens":1586,"prompt_tokens":903,"completion_tokens":683,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":519,"completion_tokens_details":{"reasoning_tokens":584}},"tokens_in":519,"tokens_out":683,"duration_ms":5828,"temperature":1.0,"reasoning_tokens":584,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:18:36.595846+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Feed a SafeCap-trained model a set of images where the generated caption contains plausible but hallucinated safety-critical details (for example, a weapon not present in the image) and check whether the refusal rate tracks the actual risk; a high refusal rate on benign images with hallucinated hazards would show the reward learned to fabricate evidence rather than recognize real danger.","supporting_citations":[{"cited_title":"T.; and Zhang, Y","cited_arxiv_id":null,"evidence_quote":"ECSO, the prior caption-mediated defense that motivates the caption-channel idea and whose limitations SafeCap addresses."}],"review_version":1}