{"id":"de84db72-94a4-478c-b17d-f89f1a40bae9","arxiv_id":"2505.24208","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Reducing the modality gap between image and text embeddings during LVLM pretraining reduces unsafe response rates by up to 16.3% across models and benchmarks.","lead":"This paper finds that vision-language models are more unsafe when their image and text representations are far apart, and that this gap appears during pretraining and persists. The authors add a small regularizer during pretraining that pulls image and text embeddings closer, reducing unsafe responses by up to 16.3% without hurting benchmark performance.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The main threat is that REGAP's L2 regularizer (Eq. 5) may improve safety by collapsing the image pathway rather than by closing the modality gap; aggregate utility in Table 3 does not rule this out.","rationale":"The paper's central claim is causal: the magnitude of the pretraining modality gap drives downstream safety degradation, so closing that gap improves safety. REGAP's regularizer, however, is not a precise gap-closing operation; Eq. 5 is an unnormalized pairwise L2 loss that, at its optimum, collapses all image tokens onto the text-token centroid. Since alpha is chosen to make Lsim comparable to Lpre, the projector gets a strong signal to discard image-specific structure. A model whose visual pathway has been partially collapsed would behave more like the text-only LLM when given any image, which would lower unsafe rate on HADES, MM-SafetyBench, and FigStep without addressing the modality-gap mechanism. Utility Table 3 is the only evidence against this, and aggregate benchmark scores are insensitive to exactly the kind of visual blunting that matters here: a model can still caption and answer common questions while losing the ability to process the specific image content that disrupts safety. The paper also lacks error bars, multiple seeds, or exact alpha values, and the authors note reduced data for ShareGPT4V and missing MiniGPT-4 data, so cross-architecture claims are not the strongest support. The proposed control experiment would settle whether the mechanism is real. This does not change the reader's CONDITIONAL verdict: the intervention is simple, cheap, and partially effective regardless of mechanism, but full acceptance should wait for the control.","tokens_in":909,"tokens_out":880,"duration_ms":115021,"concrete_test":"Retrain LLaVA-1.5-7B-LoRA under the same pretraining setup, but replace Eq. 5 with a visual-collapse control Lvar = (1/m) sum_a ||f_v^a - mean_b(f_v^b)||^2, using the same alpha schedule and seeds. If this control achieves unsafe-rate reductions comparable to REGAP on HADES Toxic and HADES Original, the safety gain is attributable to loss of visual information rather than to modality-gap reduction; if REGAP is substantially safer, the mechanism survives. Report per-seed means and standard deviations for unsafe rate and for the Table 3 utility benchmarks.","verdict_should_be":"UNCHANGED","load_bearing_attack":"To make the paper's causal claim that reducing the pretraining modality gap improves LVLM safety, one must show that REGAP's safety gain is not just a byproduct of degrading the model's use of visual input. Eq. 5 minimizes the average squared distance between every image token and every text token. In isolation its optimum is collapse: all image tokens map to the mean text-token vector. The schedule in Sec. 5 scales Lsim to match Lpre (Fig. 10 shows Lsim is significantly larger), so the projector is strongly pulled toward that collapse during pretraining. If image tokens become less discriminative, the LVLM can fall back on the frozen LLM's text-only safety behavior whenever a harmful prompt is paired with an image, exactly the outcome attributed to gap reduction. Table 3 reports aggregate utility over ten benchmarks, but those averages do not measure whether the model still extracts image content in safety-relevant settings; a model with blunted vision can lose fine-grained visual sensitivity and still score within roughly one point on MMBench, TextVQA, and similar benchmarks. The paper provides no analysis of refusal rates, no image-absent or irrelevant-image controls on the REGAP checkpoints, and no token-diversity statistics, so the confound is unresolved.","agreement_with_reader":"agree"},"referee_report":null,"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read arXiv:2505.24208. The real contribution here is empirical: the paper measures the modality gap (MIR) across checkpoints and shows a strong negative correlation with unsafe rate, and it demonstrates that the gap appears during pretraining and persists. That is new, and the proposed REGAP regularizer—an L2 alignment of image and text token embeddings during pretraining—is simple, architecture-agnostic, and delivers consistent safety gains across three models and several benchmarks with modest utility loss. The stacking with existing defenses (SimCLIP, RobustCLIP, CMRM) is also a practical plus.\n\nThe load-bearing causal claim, however, is not yet supported. The L2 loss (Eq. 5) has an optimum that collapses image tokens toward the text mean; with α scaled to match the pretraining loss (Fig. 10 shows L_sim is much larger), the projector is strongly pushed in that direction. Utility drops in Table 3 are small on aggregate, but they don't measure whether the model still extracts image content in safety-relevant settings. No image-absent or irrelevant-image controls, no refusal-rate analysis, no token-diversity statistics—so \"reducing the modality gap improves safety\" remains one plausible story, and \"blunting vision improves safety\" is another. The authors even note that applying the regularizer during fine-tuning collapses the model, which suggests the same mechanism could be partially at play during pretraining.\n\nThe statistical basis is also thin: the correlations are n ≈ 4 to 6 checkpoints without confidence intervals, and the main tables are single runs without error bars or seeds. Appendix 9.8 promises exact α values but the tables there list only standard training hyperparameters. No code is released. These are fixable.\n\nNone of this is fatal. The direction is promising and the consistency across architectures is encouraging. But the paper should only be accepted conditionally: release code and hyperparameters, add multiple seeds and variance, and specifically test whether REGAP preserves visual discriminability in safety-relevant contexts. If the blunted-vision confound is ruled out, the modality-gap mechanism becomes credible.\n\nThis is worth a serious referee—the regularizer is cheap and could become a default pretraining step for open VLMs—but with the expectation of heavy revision. I'd bring it to a reading group to dig into the confound.\n\nNet: engage with it, but hold the causal language until that alternative is addressed.","headline":"Useful correlation analysis and a simple pretraining regularizer, but the causal claim is undercut by an unresolved blunted-vision confound.","tokens_in":16309,"tokens_out":3191,"would_cite":true,"duration_ms":35946,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that the representational distance between image and text embeddings—the modality gap—is inversely correlated with vision-language model safety, and that shrinking it during pretraining with an L2 regularizer…","keywords":["modality gap","vision-language model safety","safety alignment","pretraining regularization","multimodal jailbreak","large vision-language model","representation alignment"],"falsifier":"Compare REGAP-pretrained and baseline models on the same harmful prompts with images replaced by blank or white-noise images and on fine-grained visual-reasoning tasks: if the unsafe-rate drop persists on blank images while visual utility drops disproportionately, the regularizer is creating blunt visual insensitivity rather than fixing the modality gap.","tokens_in":15305,"feed_emoji":"🛡️","tokens_out":9821,"duration_ms":95194,"temperature":0.7,"pith_summary":"Safety-aligned language models become easier to jailbreak once a vision encoder and projector are added, and this paper's thesis is that the root cause is measurable and addressable: the modality gap, or the unusually large distance between image token embeddings and text token embeddings in the model's input space. Drawing on five open models, the authors show a strong inverse correlation between this gap and safety, that pretrained checkpoints already carry the gap, and that the gap after pretraining predicts the gap after fine-tuning with a correlation of 0.93. They introduce REGAP, a pretraining-time regularizer that adds the mean pairwise L2 distance between image and text token embeddings to the captioning loss, scaled so it does not overwhelm the original objective. Across LLaVA-1.5, ShareGPT4V, and MiniGPT-4, REGAP reduces unsafe response rates by up to 16.3%, preserves utility on standard benchmarks, and stacks with existing inference-time defenses to improve them by up to 18.2%.","feed_headline":"Shrinking the image-text gap cuts harmful VLM answers by 16.3%","feed_subtitle":"A distance-based pretraining regularizer keeps aligned LLMs safe after gaining vision, without hurting utility.","key_machinery":"The central objects are the Modality Integration Rate (MIR) and the REGAP regularizer. MIR is a named metric: the logarithm of the summed Fréchet Inception Distance between image and text token embedding distributions across transformer layers, after text-centric $\\ell^2$ normalization and outlier removal, so lower MIR means faster cross-modal integration. MIR is too costly to optimize directly, so REGAP uses its computable proxy, the pairwise $L_2$ distance $L_{\\mathrm{sim}} = \\frac{1}{mn}\\sum_{a=1}^{m} \\sum_{b=1}^{n} \\|f^v_a - f^t_b\\|_2^2$ between all image and text token embeddings from the same image-caption pair, scaled by $\\alpha = L_{\\mathrm{pre}}/L_{\\mathrm{sim}}$ after a warm-up so it does not overwhelm the captioning loss. The leverage comes from the pretraining setup: the vision encoder and LLM are frozen, so the projector alone decides where image embeddings land, and aligning the input layer propagates a smaller gap into deeper layers.","core_discovery":"The central claim is that the modality gap is not a symptom but a controllable cause of safety degradation: models with a larger image–text embedding gap are systematically more likely to answer harmful prompts, and the gap is created during the projector pretraining stage and survives instruction tuning. Using Modality Integration Rate (MIR), a layer-wise Fréchet distance between image and text token distributions, the paper reports strong correlations between MIR and unsafe rate (0.71 after fine-tuning, 0.78 at pretraining) and a 0.93 correlation between the pretraining gap and the fine-tuned gap. REGAP addresses the gap at the input layer, where the projector is the only trainable component, by adding a scaled pairwise L2 distance between image and text tokens to the pretraining loss. The paper reports that this intervention reduces unsafe rate by up to 16.3% on the LLaVA-1.5 family, reduces it by up to 24.3% across harmful prompt categories on other architectures, and that combining REGAP with defense methods such as SimCLIP, RobustCLIP, and CMRM yields up to 18.2% further improvement, approaching text-only LLM safety levels.","pith_inferences":["Editorial inference: if the modality gap is causal rather than merely predictive, then other pretraining-time interventions that move image embeddings into the text-token distribution—contrastive alignment, shared embedding training, or different projector initialization—should reproduce comparable safety gains; REGAP's particular loss is probably not the only way to get them.","Editorial inference: a concrete probe of the mechanism is to measure unsafe rates on blank or irrelevant images, which the paper notes already trigger failures; if REGAP's main effect is to make the model ignore the image, blank-image prompts should show the full safety gain, whereas if the gap itself matters, gains should concentrate on prompts where image content is actually processed.","Editorial inference: the observation that regularizing fine-tuning collapses output while pretraining regularization helps implies that the projector stage is a uniquely permissive intervention window, since later alignment must fight against consolidated cross-modal representations.","Editorial inference: a testable prediction for future work is that REGAP's complementarity with steering methods comes from geometry—REGAP removes the static representation gap, while steering methods correct per-prompt activation shifts—so they address different parts of the same failure."],"forward_implications":["REGAP offers model builders a cheap, data-free knob: adding one regularizer to the existing projector-pretraining objective recovers a large fraction of the safety lost when a text-only LLM gains vision.","Because the method works on both MLP-based projectors like LLaVA and ShareGPT4V and the Q-Former-based MiniGPT-4, modality-gap reduction generalizes as a pretraining design principle rather than a fix tied to one architecture.","REGAP composes with inference-time steering and robust-encoder defenses, so the two families target complementary failure modes and can be combined to approach text-only LLM safety.","The reported safety gains do not come at the expense of general capability: on the utility benchmarks tabulated, the REGAP model stays within about one point of the no-defense base on average while reducing unsafe rate substantially.","REGAP is a first-layer intervention, and the authors find that pushing the same regularization into deeper layers or into fine-tuning adds no safety benefit and can even hurt the model, which narrows where future pretraining-time safety fixes should be applied."],"supporting_citations":[{"why":"Supplies the Modality Integration Rate (MIR) metric used to quantify the modality gap.","marker":"[Huang et al., 2024]"},{"why":"Supplies the HADES jailbreak dataset and toxic image-prompt pairs used in safety evaluation.","marker":"[Li et al., 2024b]"},{"why":"Defines the Unsafe Rate metric, documents the safety-degradation phenomenon, and provides the CMRM baseline.","marker":"[Liu et al., 2024a]"},{"why":"Offers the earlier hypothesis that the modality gap contributes to VLM safety failures.","marker":"[Gao et al., 2024]"},{"why":"Provides the ShareGPT4V architecture and dataset used to test generalization across pretraining data.","marker":"[Chen et al., 2024a]"},{"why":"Provides the MiniGPT-4 Q-Former-based architecture used to test generalization across projector designs.","marker":"[Zhu et al., 2023]"},{"why":"Provides the Beaver-dam judge model used to decide which responses count as unsafe.","marker":"[Ji et al., 2023]"},{"why":"Provides the gradient-based visual perturbation method used to construct adversarial images for the HADES extension.","marker":"[Qi et al., 2023]"}],"fun_headline_variants":["Modality gap drives VLM safety failures—closing it helps","Pretraining gap predicts harmful VLM responses; shrink it","Cut unsafe VLM answers 16.3% by reducing image-text gap","VLM safety restored by shrinking pretraining modality gap","Lower image-text gap, lower unsafe rate: up to 16.3%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the safety gains come from closing the image-text gap itself, not from the regularizer making the model process visual input more poorly.","fun_headline_variants_meta":{"raw":{"variants":["Modality gap drives VLM safety failures—closing it helps","Pretraining gap predicts harmful VLM responses; shrink it","Cut unsafe VLM answers 16.3% by reducing image-text gap","VLM safety restored by shrinking pretraining modality gap","Lower image-text gap, lower unsafe rate: up to 16.3%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000202,"raw_usage":{"total_tokens":1411,"prompt_tokens":1000,"completion_tokens":411,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":616,"completion_tokens_details":{"reasoning_tokens":320}},"tokens_in":616,"tokens_out":411,"duration_ms":4574,"temperature":1.0,"reasoning_tokens":320,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:30:12.694604+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare REGAP-pretrained and baseline models on the same harmful prompts with images replaced by blank or white-noise images and on fine-grained visual-reasoning tasks: if the unsafe-rate drop persists on blank images while visual utility drops disproportionately, the regularizer is creating blunt visual insensitivity rather than fixing the modality gap.","supporting_citations":[],"review_version":1}