Pith. sign in

REVIEW 3 major objections 4 minor 65 references

This paper argues that the dominant jailbreak route against video LLMs is not blindness to harmful content but an understanding-refusal coupling failure: models recognize the harm, yet harmful visual evidence activates weaker refusal signal

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 08:17 UTC pith:NDVLCFVI

load-bearing objection A useful diagnostic framing for a real video-LLM safety anomaly, but the 'understands yet doesn't refuse' premise is not yet proven because the understanding test is safety-cued and the intent-match criterion is undefined. the 3 major comments →

arxiv 2607.21151 v2 pith:NDVLCFVI submitted 2026-07-23 cs.AI

V-DEAL: Diagnosing Video Safety De-Calibration as an Understanding-Refusal Coupling Failure

Zhetong Zhang , Honghao Fu , Miao Xu , Yiwei Wang , Yujun Cai This is my paper
classification cs.AI
keywords video large language modelssafety alignmentjailbreak attacksrefusal mechanismhidden-state analysisrepresentation engineeringprompt-based defensemultimodal safety
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper claims that jailbreak risk in video LLMs is not primarily a perception problem. Across six models and three public benchmarks, models identify harmful video content with over 81% accuracy, yet when a harmful video is paired with a benign query the average attack success rate reaches 48.33% — higher than when the same video is paired with an explicitly harmful query. The authors diagnose this as an understanding-refusal coupling failure: visual evidence is recognized, but it activates weaker refusal-related internal representations than text does. They make the diagnosis actionable with a training-free prompt intervention that forces explicit visual reasoning before a safety decision, reducing average attack success from 48.33% to 0.80%.

Core claim

The central claim is that the VH-BQ (harmful video + benign query) vulnerability does not stem from insufficient video understanding. Using a three-stage diagnostic — behaviour, understanding, representation — the paper rules out perception failure and shows that harmful video inputs produce lower refusal-alignment scores than harmful text inputs in hidden-state space. Even when a model can accurately describe the harmful video and its intent, the internal refusal signal is too weak to trigger refusal reliably. The paper's intervention, which asks the model to explicitly reason about the harmful visual evidence before deciding, cuts attack success to below 1% on average across six models, wh

What carries the argument

The central instrument is a layer-wise 'refusal direction': the normalized difference between the mean hidden representations of refusal and non-refusal samples, estimated from external text and image-text safety data. Each hidden state is projected onto this direction to yield a scalar refusal score, averaged over the top-m most separable layers. This converts an unobservable internal refusal tendency into a measurable axis, allowing the paper to compare refusal alignment across text-only, harmful-text, harmful-video, and benign-query conditions and to locate where the understanding-to-refusal coupling fails.

Load-bearing premise

The load-bearing premise is that the refusal direction computed from external text and image-text safety data transfers to video-conditioned hidden states and genuinely reflects each model's internal refusal mechanism for video inputs; if that direction mainly captures text-elicited refusal, the lower refusal scores observed under harmful-video-with-benign-query would be an artifact of the probe.

What would settle it

Estimate the refusal direction from video-conditioned refusal responses alone, rather than from text and image-text data. If harmful video inputs then refuse-score as high as harmful text inputs, the paper's core evidence of weaker visual refusal activation would be an artifact of a text-biased probe. A second falsifier: find a model that can describe a harmful video in detail and still produce step-by-step harmful instructions when the prompt explicitly forces visual reasoning before answering.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Safety evaluations of video LLMs should stop treating understanding accuracy as a proxy for safety; a model can fully grasp harmful content yet still comply.
  • The harmful-video-with-benign-query condition is a systematically stronger attack than explicit harmful queries and should be a standard evaluation setting.
  • Hidden-state refusal scores can diagnose safety de-calibration: a low refusal score alongside high understanding accuracy identifies the coupling failure rather than a perception gap.
  • A training-free prompt intervention that makes visual evidence explicit can close most of the safety gap without weight updates, offering an immediate stopgap for deployed models.
  • Alignment training should aim to make video-conditioned hidden states activate refusal signals as strongly as text-conditioned states do.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Our inference: the same coupling failure may generalize to other modalities — an image or audio clip whose harmful content is recognized but under-activates refusal could show the same benign-query vulnerability; the V-DEAL diagnostic transfers by re-estimating the refusal direction on that modality's data.
  • Our inference: refusal scores could be used as a runtime guardrail — if a video-conditioned hidden state lands in a low-refusal region, an external check can force a refusal before generation proceeds.
  • We read the paper's closing limitation — fixed manual prompt and demonstrations — as pointing to portability across model families as the open question, with automated prompt search as the natural next test.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces V-DEAL, a three-stage diagnostic framework for a video-specific safety vulnerability: harmful videos paired with benign queries (VH-BQ) yield higher attack success rates than the same videos paired with explicitly harmful queries (VH-HQ). Across six open-source Video LLMs and three video safety benchmarks, the authors report an average VH-BQ ASR of 48.33%, compared with 35.84% under VH-HQ. They then evaluate video understanding through structured visual-attribute labeling and summary generation, reporting average structured-label accuracy above 81% and high summary similarity. A representation-level analysis computes refusal-alignment scores by projecting hidden states onto a refusal direction estimated from external text and image-text safety data, finding that harmful video evidence activates weaker refusal-related signals than harmful text. Finally, a prompt-based intervention that instructs the model to reason about visual evidence before deciding safety reduces average ASR to 0.80% under zero-shot prompting, with performance comparable to prior fine-tuning baselines on a matched SEA-style setting. The paper concludes that the VH-BQ vulnerability is not fully explained by insufficient video understanding, but reflects a failure to translate recognized visual harm into refusal behavior.

Significance. If the central claim holds, the paper makes a useful contribution: it documents a counterintuitive and consistent behavioral anomaly across six model families, proposes a three-level diagnostic structure that goes beyond attack success rates, and introduces a training-free intervention with large measured reductions in ASR. The representation analysis is strengthened by a transfer audit with random-direction controls (Appendix D.2), and the behavioral results are reported transparently with per-condition and per-model tables. However, the paper's headline conclusion—'even when Video LLMs can understand harmful video content, they may still fail to reliably trigger refusal'—depends critically on the validity of the understanding evaluation. That evaluation currently rests on perceptual attribute accuracy and an under-specified partial-match intent validation, so the central premise is not yet established with the rigor the paper claims. The work is significant as a diagnostic proposal and an empirical phenomenon report, but it needs stronger construct validation before the mechanistic conclusion can be accepted.

major comments (3)
  1. [§4.2, Appendix C.9, Table 8] The central premise—that models recognize harmful video content—is not yet supported. Table 2 reports accuracy on four structured attributes (main_environment, visible_weapon, physical_contact, num_visible_people); these are perceptual attributes, not judgments of harmful intent. The only intent-level evidence (Table 8, Appendix C.9) uses an undefined 'Partial-or-Exact Intent Match' and does not specify the evaluation prompt, the matching rule, or how reference rationales were generated. With an unspecified partial-match criterion, rates of 82–96% could reflect loose topical overlap. Moreover, the understanding prompts in Appendix C.4 instruct models to act as 'safety-oriented video understanding assistant[s]' and to look for 'weapons, violence, or threatening context,' whereas VH-BQ uses a benign query without such cues. The understanding evaluation may therefore overestimate real-world
  2. [§4.3, Eq. (1), Appendix D.2] The refusal direction w_l is estimated from external text and image-text safety data (XSTest, MM-SafetyBench) and then applied to video-conditioned hidden states. The claim that harmful video activates weaker refusal-related signals than harmful text assumes that w_l transfers across modalities and that the projection is calibrated across the four conditions. The transfer audit in Appendix D.2 shows AUC above a random-direction control, which is a good check, but it only tests binary separation on labelled video-conditioned responses; it does not establish that the lower VH-BQ refusal scores are not driven by the benign query text dominating the last-token representation. A control experiment with fixed text and varying video content, or an analysis that ablates the contribution of the text token to the projected score, is needed. Without it, the representation-level conclusion may be pa
  3. [§5, Appendix E.3.1] Section 5 frames the prompt intervention as 'Realignment as Diagnostic Validation,' implying that the ASR reduction confirms the understanding-refusal coupling. However, the safety system prompt explicitly instructs the model to treat any request with 'ANY realistic chance' of harm as harmful, to 'MUST err on the side of refusing,' and to ground the decision in the video. A large ASR drop is expected from these instructions alone, independent of the hypothesized mechanism. The intervention also bundles a system prompt, eight demonstrations, and an explicit video-grounding instruction, with no ablation isolating the video-grounding component. The defense result is useful in its own right, but it should be presented as a defense outcome rather than as direct evidence for the diagnostic mechanism, unless ablations are added.
minor comments (4)
  1. [Table 8, Appendix C.9] The column headed 'Matched Samples' is opaque; it appears to count partial-or-exact matches, but the caption should state this explicitly. The matching criterion itself should be defined even in the caption, since the main text currently refers only to an 'adopted partial-or-exact matching criterion.'
  2. [Appendix D.4] Figure 6 and the accompanying text refer to 'Exp1 judge labels,' but 'Exp1' is not defined anywhere in the paper. Please clarify which experimental run or labelling protocol this refers to.
  3. [Table 6, §4.2] The 'over 81% accuracy' claim is an average over four attributes, which can hide substantial attribute-level failures. For MiniCPM-V-4.5, num_visible_people accuracy is only 0.389 while visible_weapon accuracy is 0.972. Reporting the aggregate alone is misleading; per-attribute results should be emphasized in the main text.
  4. [Table 1, §4.3] The caption for Table 1 says 'R-Score is not averaged across models because score scales are model-dependent.' This is sensible, but the caption should also state that refusal scores are computed on a balanced 1,250-sample subset per condition, as described later in Appendix D.1, so the reader can interpret the summary table correctly.

Circularity Check

0 steps flagged

No significant circularity: the refusal-direction probe is fit to external data and audited against random controls; the diagnostic conclusion is not forced by construction. Remaining concerns are construct validity and reporting, not circularity.

full rationale

The central diagnostic chain is not circular by construction. The refusal direction w_l (Eq. 1) is estimated from external safety corpora (XSTest, MM-SafetyBench) that are disjoint from the VH-BQ video conditions under study, and refusal scores are then computed by projection (Eq. 2) and compared across conditions. Nothing in Eq. 1 forces VH-BQ scores to be lower than T-HQ or VH-HQ; the observed ordering is an empirical result. The external-to-video transfer audit (Appendix D.2) checks the direction against a random-direction control with AUCs above the 95th percentile, so the probe is at least partially validated on video-conditioned states. The understanding analysis is a separate proxy task (structured labels/summaries) and does not by definition imply any particular refusal behaviour, so the claim that strong understanding co-occurs with weak refusal is not tautological. The prompt-based realignment is a direct intervention, not a fitted prediction; it is a weak validation of the mechanism because the system prompt explicitly instructs refusal, but it is not a reduction to the inputs. There are minor self-citations in the reference list (e.g., [17,49,54,56,62]) but none is load-bearing; the stability of the diagnosis does not rest on these. Two genuine limitations are non-circularity concerns: (i) Appendix C.9's 'Partial-or-Exact Intent Match' is never defined, so the harmful-intent premise is under-supported; and (ii) layers are selected by ROC-AUC on the same external corpus used to fit the direction, which risks overfitting, though the transfer audit and random control mitigate this. These are validity/reporting issues, not evidence that any 'prediction' equals its input by construction.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

The paper introduces no new physical or conceptual entities; the 'refusal direction' is a known construct from prior representation engineering. The main load-bearing choices are the fitted refusal direction and the hand-selected analysis attributes.

free parameters (5)
  • m (top-layers averaging count) = 4
    The number of layers averaged in RefusalScore (Eq. 2) is set a priori to 4 based on pilot runs; affects the aggregated refusal scores and thus the representation-level conclusion.
  • Per-model top-m layer indices = e.g., Qwen2-VL-7B: [24,25,26,27]
    Layers are selected by ROC-AUC on the external safety corpus (Appendix D.1); this is a fit to data that determines which hidden states are used in the refusal score.
  • Refusal direction vector w_l (per layer) = Normalized class-mean difference (Eq. 1)
    The direction is estimated from XSTest and MM-SafetyBench class means; it is the central probe used to compare refusal tendency across conditions.
  • Core attribute selection for understanding accuracy = four attributes (env, weapon, contact, people)
    The aggregate understanding metric excludes primary_focus and camera_view (Appendix C.2); this hand-chosen selection affects the headline 81%+ accuracy claim.
  • 8-shot demonstrations = 8 GPT-4.1 generated examples
    The few-shot intervention uses a fixed set of demonstrations generated by a stronger model; the choice of examples could affect the reported ASR improvement.
axioms (5)
  • domain assumption GPT-4o-mini judge labels (HARMFUL/REFUSAL/SAFE) are correct
    All behavioural metrics (ASR, refusal rate, safe rate) rely on the judge's classification; no validation of the judge against human labels is reported (Appendix B.1).
  • domain assumption Refusal direction learned from text/image-text data transfers to video-conditioned hidden states
    The central representation analysis assumes the external safety data (XSTest, MM-SafetyBench) produces a refusal direction meaningful for video inputs; the transfer audit (Appendix D.2) partially supports this.
  • ad hoc to paper Structured attributes and summaries are a sufficient proxy for harmful-content understanding
    The understanding stage equates high accuracy on four scene attributes and LLM-judged summary similarity with 'recognizing harmful video content' (Section 4.2).
  • domain assumption Uniform sampling of 8 frames per video is sufficient to capture the harmful content
    All experiments use N=8 uniformly sampled frames (Appendix A.4); if critical harmful events are missed, understanding scores could be inflated.
  • domain assumption External datasets provide suitable REFUSAL/OTHER supervision for the latent direction
    The refusal direction relies on the class labels in XSTest and MM-SafetyBench; any label noise or domain mismatch affects the probe.

pith-pipeline@v1.3.0-alltime-deepseek · 29177 in / 13021 out tokens · 119004 ms · 2026-08-01T08:17:43.079268+00:00 · methodology

0 comments
read the original abstract

As Video Large Language Models are increasingly deployed in real-world applications, ensuring their safety alignment has become critical. Counterintuitively, we find that harmful videos paired with benign queries achieve higher attack success rates than the same videos paired with explicitly harmful queries. To understand the underlying mechanism of this vulnerability, we present V-DEAL, a three-level diagnostic framework that jointly analyzes this failure across model behaviour, understanding, and internal representations. By progressively ruling out perception failure and quantifying the model's internal refusal tendency, V-DEAL provides a new diagnostic perspective for analyzing the underlying mechanism of the observed vulnerability. We tested six Video LLMs on three public benchmarks and observed that models correctly recognize harmful video content with over 81\% accuracy, yet the average attack success rate still reaches 48.33\% under the condition pairing harmful videos with benign queries. Hidden-state analysis further shows that visual understanding activates a weaker refusal tendency than textual understanding. Furthermore, we introduce a prompt injection intervention method that reduces attack success rates by an average of 48.24 percentage points and achieves performance comparable to prior fine-tuning-based methods, providing an effective and practical means to address such safety risks in Video LLMs.

Figures

Figures reproduced from arXiv: 2607.21151 by Honghao Fu, Miao Xu, Yiwei Wang, Yujun Cai, Zhetong Zhang.

Figure 1
Figure 1. Figure 1: Motivation for studying video safety de-calibration in Video LLMs. A harmful [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the proposed V-DEAL framework. The framework consists of three [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Attack success rate (ASR) of six target models under four attack conditions. The [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Relationship between video understanding capability and ASR under VH-BQ. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Condition-level mean refusal-alignment score for six target models. Error bars [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Behaviour rates across conditions and risk bins (Low/Mid/High terciles of [PITH_FULL_IMAGE:figures/full_fig_p022_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

65 extracted references · 23 linked inside Pith

  1. [1]

    Qwen3-vl technical report,

    S. Bai, Y. Cai, R. Chen, K. Chen, X. Chen, Z. Cheng, L. Deng, W. Ding, C. Gao, C. Ge et al., “Qwen3-vl technical report,” 2025

    2025

  2. [2]

    Internvl3. 5: Advancing open-source multimodal models in versatility, reasoning, and efficiency,

    W. Wang, Z. Gao, L. Gu, H. Pu, L. Cui, X. Wei, Z. Liu, L. Jing, S. Ye, J. Shaoet al., “Internvl3. 5: Advancing open-source multimodal models in versatility, reasoning, and efficiency,”arXiv preprint arXiv:2508.18265, 2025

    Pith/arXiv arXiv 2025

  3. [3]

    Internvideo2. 5: Empowering video mllms with long and rich context modeling,

    Y. Wang, X. Li, Z. Yan, Y. He, J. Yu, X. Zeng, C. Wang, C. Ma, H. Huang, J. Gaoet al., “Internvideo2. 5: Empowering video mllms with long and rich context modeling,”arXiv preprint arXiv:2501.12386, 2025

    Pith/arXiv arXiv 2025

  4. [4]

    Vision-language models for vision tasks: A survey,

    J. Zhang, J. Huang, S. Jin, and S. Lu, “Vision-language models for vision tasks: A survey,”IEEE transactions on pattern analysis and machine intelligence, vol. 46, no. 8, pp. 5625–5644, 2024

    2024

  5. [5]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution,

    P . Wang, S. Bai, S. Tan, S. Wang, Z. Fan, J. Bai, K. Chen, X. Liu, J. Wang, W. Ge et al., “Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution,”arXiv preprint arXiv:2409.12191, 2024

    Pith/arXiv arXiv 2024

  6. [6]

    Safety of multimodal large language models on images and texts,

    X. Liu, Y. Zhu, Y. Lan, C. Yang, and Y. Qiao, “Safety of multimodal large language models on images and texts,”arXiv preprint arXiv:2402.00357, 2024

    Pith/arXiv arXiv 2024

  7. [7]

    Jailbreak attacks and defenses against multimodal generative models: A survey,

    X. Liu, X. Cui, P . Li, Z. Li, H. Huang, S. Xia, M. Zhang, Y. Zou, and R. He, “Jailbreak attacks and defenses against multimodal generative models: A survey,”arXiv preprint arXiv:2411.09259, 2024

    Pith/arXiv arXiv 2024

  8. [8]

    Vlsbench: Unveiling visual leakage in multimodal safety,

    X. Hu, D. Liu, H. Li, X.-J. Huang, and J. Shao, “Vlsbench: Unveiling visual leakage in multimodal safety,” inProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2025, pp. 8285–8316

    2025

  9. [9]

    Video-safetybench: A benchmark for safety evaluation of video lvlms,

    X. Liu, Z. Li, Z. He, P . Li, X. Cui, H. Huang, X. Yang, R. Heet al., “Video-safetybench: A benchmark for safety evaluation of video lvlms,”Advances in Neural Information Processing Systems, vol. 38, 2026

    2026

  10. [10]

    Sea: Low-resource safety alignment for multimodal large language models via synthetic embeddings,

    W. Lu, H. Peng, H. Zhuang, C. Chen, and Z. Zeng, “Sea: Low-resource safety alignment for multimodal large language models via synthetic embeddings,” inProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2025, pp. 24 894–24 913

    2025

  11. [11]

    Figstep: Jail- breaking large vision-language models via typographic visual prompts,

    Y. Gong, D. Ran, J. Liu, C. Wang, T. Cong, A. Wang, S. Duan, and X. Wang, “Figstep: Jail- breaking large vision-language models via typographic visual prompts,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 39, no. 22, 2025, pp. 23 951–23 959

    2025

  12. [12]

    Visual-roleplay: Universal jailbreak attack on multimodal large language models via role-playing image character,

    S. Ma, W. Luo, Y. Wang, and X. Liu, “Visual-roleplay: Universal jailbreak attack on multimodal large language models via role-playing image character,”arXiv preprint arXiv:2405.20773, 2024

    Pith/arXiv arXiv 2024

  13. [13]

    Ideator: Jailbreaking and benchmarking large vision-language models using them- selves,

    R. Wang, J. Li, Y. Wang, B. Wang, X. Wang, Y. Teng, Y. Wang, X. Ma, and Y.-G. Jiang, “Ideator: Jailbreaking and benchmarking large vision-language models using them- selves,” inProceedings of the IEEE/CVF International Conference on Computer Vision, 2025, pp. 8875–8884

    2025

  14. [14]

    Jailbreakv: A benchmark for assessing the robustness of multimodal large language models against jailbreak attacks,

    W. Luo, S. Ma, X. Liu, X. Guo, and C. Xiao, “Jailbreakv: A benchmark for assessing the robustness of multimodal large language models against jailbreak attacks,”arXiv preprint arXiv:2404.03027, 2024

    Pith/arXiv arXiv 2024

  15. [15]

    Mmj-bench: A comprehensive study on jail- break attacks and defenses for multimodal large language models,

    F. Weng, Y. Xu, C. Fu, and W. Wang, “Mmj-bench: A comprehensive study on jail- break attacks and defenses for multimodal large language models,”arXiv preprint arXiv:2408.08464, 2024

    Pith/arXiv arXiv 2024

  16. [16]

    Omni-SafetyBench: A benchmark for safety evaluation of audio-visual large language models,

    L. Pan, Z. Fu, Y. Zhai, S. Tao, S. Guan, S. Huang, L. Zhang, Z. Liu, B. Ding, F. Henry et al., “Omni-SafetyBench: A benchmark for safety evaluation of audio-visual large language models,”arXiv preprint arXiv:2508.07173, 2025. 10 Preprint. Under review

    arXiv 2025

  17. [17]

    Videostir: Understanding long videos via spatio-temporally structured and intent-aware rag,

    H. Fu, M. Xu, Y. Wang, D. Zhang, J. Liu, and Y. Cai, “Videostir: Understanding long videos via spatio-temporally structured and intent-aware rag,” inProceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2026, pp. 35 793–35 805

    2026

  18. [18]

    Camreasoner: Reinforcing camera movement understanding via structured spatial reasoning,

    H. Wu, Y. Cai, Z. Li, H. Ge, B. Sun, J. Yuan, and Y. Wang, “Camreasoner: Reinforcing camera movement understanding via structured spatial reasoning,”arXiv preprint arXiv:2602.00181, 2026

    Pith/arXiv arXiv 2026

  19. [19]

    Mm-safetybench: A benchmark for safety evaluation of multimodal large language models,

    X. Liu, Y. Zhu, J. Gu, Y. Lan, C. Yang, and Y. Qiao, “Mm-safetybench: A benchmark for safety evaluation of multimodal large language models,” inEuropean Conference on Computer Vision. Springer, 2024, pp. 386–403

    2024

  20. [20]

    Xstest: A test suite for identifying exaggerated safety behaviours in large language models,

    P . Röttger, H. Kirk, B. Vidgen, G. Attanasio, F. Bianchi, and D. Hovy, “Xstest: A test suite for identifying exaggerated safety behaviours in large language models,” inProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), 2024, pp. 5377–5400

    2024

  21. [21]

    Hiddenguard: Fine-grained safe generation with specialized representation router,

    L. Mei, S. Liu, Y. Wang, B. Bi, R. Yuan, and X. Cheng, “Hiddenguard: Fine-grained safe generation with specialized representation router,”arXiv preprint arXiv:2410.02684, 2024

    Pith/arXiv arXiv 2024

  22. [22]

    You know what i’m saying: Jailbreak attack via implicit reference,

    T. Wu, L. Mei, R. Yuan, L. Li, W. Xue, and Y. Guo, “You know what i’m saying: Jailbreak attack via implicit reference,”arXiv preprint arXiv:2410.03857, 2024

    Pith/arXiv arXiv 2024

  23. [23]

    Spa-vl: A comprehensive safety preference alignment dataset for vision language models,

    Y. Zhang, L. Chen, G. Zheng, Y. Gao, R. Zheng, J. Fu, Z. Yin, S. Jin, Y. Qiao, X. Huang et al., “Spa-vl: A comprehensive safety preference alignment dataset for vision language models,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025, pp. 19 867–19 878

    2025

  24. [24]

    Msr-align: Policy-grounded multimodal alignment for safety-aware reasoning in vision-language models,

    Y. Xia, Y. Jiang, Y. Tan, X. Zhu, X. Yue, and B. Zheng, “Msr-align: Policy-grounded multimodal alignment for safety-aware reasoning in vision-language models,”arXiv preprint arXiv:2506.19257, 2025

    arXiv 2025

  25. [25]

    Safety fine-tuning at (almost) no cost: A baseline for vision large language models,

    Y. Zong, O. Bohdal, T. Yu, Y. Yang, and T. Hospedales, “Safety fine-tuning at (almost) no cost: A baseline for vision large language models,”arXiv preprint arXiv:2402.02207, 2024

    Pith/arXiv arXiv 2024

  26. [26]

    Adashield: Safeguarding multimodal large language models from structure-based attack via adaptive shield prompting,

    Y. Wang, X. Liu, Y. Li, M. Chen, and C. Xiao, “Adashield: Safeguarding multimodal large language models from structure-based attack via adaptive shield prompting,” in European Conference on Computer Vision. Springer, 2024, pp. 77–94

    2024

  27. [27]

    Bluesuffix: Reinforced blue teaming for vision-language models against jailbreak attacks,

    Y. Zhao, X. Zheng, L. Luo, Y. Li, X. Ma, and Y.-G. Jiang, “Bluesuffix: Reinforced blue teaming for vision-language models against jailbreak attacks,” inInternational Conference on Learning Representations, vol. 2025, 2025, pp. 35 443–35 462

    2025

  28. [28]

    E2AT: Multimodal Jailbreak Defense via Dynamic Joint Optimization for Multimodal Large Language Models,

    L. Lu, X. Gu, S. Pang, S. Liang, H. Zhu, X. Zeng, X. Zheng, and Y. Zhou, “E2AT: Multimodal Jailbreak Defense via Dynamic Joint Optimization for Multimodal Large Language Models,”arXiv preprint arXiv:2503.04833, 2025

    arXiv 2025

  29. [29]

    Eta: Evaluating then aligning safety of vision language models at inference time,

    Y. Ding, B. Li, and R. Zhang, “Eta: Evaluating then aligning safety of vision language models at inference time,” inInternational Conference on Learning Representations, vol. 2025, 2025, pp. 2403–2431

    2025

  30. [30]

    Safedecoding: Defending against jailbreak attacks via safety-aware decoding,

    Z. Xu, F. Jiang, L. Niu, J. Jia, B. Y. Lin, and R. Poovendran, “Safedecoding: Defending against jailbreak attacks via safety-aware decoding,” inProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2024, pp. 5587–5605

    2024

  31. [31]

    Defending multimodal backdoored models by repulsive visual prompt tuning,

    Z. Zhang, S. He, H. Wang, B. Shen, and L. Feng, “Defending multimodal backdoored models by repulsive visual prompt tuning,”NeurIPS, 2025. 11 Preprint. Under review

    2025

  32. [32]

    Gated differentiable working memory for long-context language modeling,

    L. Mei, S. Liu, Y. Wang, Y. Ge, B. Bi, J. Yao, J. Wan, Z. Yin, J. Guo, and X. Cheng, “Gated differentiable working memory for long-context language modeling,”arXiv preprint arXiv:2601.12906, 2026

    arXiv 2026

  33. [33]

    Test-time attention purification for backdoored large vision language models,

    Z. Zhang, B. Yang, S. He, W. Chen, W. E. Zhang, O. Maennel, L. Feng, and M. Xu, “Test-time attention purification for backdoored large vision language models,” in CVPR, 2026

    2026

  34. [34]

    Mrfd: Multi-region fusion decoding with self-consistency for mitigating hallucinations in lvlms,

    H. Ge, Y. Wang, M.-H. Yang, and Y. Cai, “Mrfd: Multi-region fusion decoding with self-consistency for mitigating hallucinations in lvlms,”arXiv preprint arXiv:2508.10264, 2025

    arXiv 2025

  35. [35]

    Tokenswap: Backdoor attack on the compositional understanding of large vision-language models,

    Z. Zhang, Q. Tao, J. Lv, N. Zhao, L. Feng, and J. T. Zhou, “Tokenswap: Backdoor attack on the compositional understanding of large vision-language models,”arXiv preprint arXiv:2509.24566, 2025

    Pith/arXiv arXiv 2025

  36. [36]

    Dimo-gui: Advancing test-time scaling in gui grounding via modality-aware visual reasoning,

    H. Wu, H. Chen, Y. Cai, C. Liu, Q. Ye, M.-H. Yang, and Y. Wang, “Dimo-gui: Advancing test-time scaling in gui grounding via modality-aware visual reasoning,” inEMNLP 2025, 2025

    2025

  37. [37]

    Improving generaliz- ability and undetectability for targeted adversarial attacks on multimodal pre-trained models,

    Z. Zhang, J. Zhang, S. Zhou, Q. Wei, S. He, F. Liu, and L. Feng, “Improving generaliz- ability and undetectability for targeted adversarial attacks on multimodal pre-trained models,”arXiv preprint arXiv:2509.19994, 2025

    arXiv 2025

  38. [38]

    Representation engineering: A top-down approach to ai transparency,

    A. Zou, L. Phan, S. Chen, J. Campbell, P . Guo, R. Ren, A. Pan, X. Yin, M. Mazeika, A.-K. Dombrowskiet al., “Representation engineering: A top-down approach to ai transparency,”arXiv preprint arXiv:2310.01405, 2023

    Pith/arXiv arXiv 2023

  39. [39]

    Refusal in language models is mediated by a single direction,

    A. Arditi, O. Obeso, A. Syed, D. Paleka, N. Panickssery, W. Gurnee, and N. Nanda, “Refusal in language models is mediated by a single direction,”Advances in Neural Information Processing Systems, vol. 37, pp. 136 037–136 083, 2024

    2024

  40. [40]

    Guardreasoner-vl: Safeguarding vlms via reinforced reasoning,

    Y. Liu, S. Zhai, M. Du, Y. Chen, T. Cao, H. Gao, C. Wang, X. Li, K. Wang, J. Fanget al., “Guardreasoner-vl: Safeguarding vlms via reinforced reasoning,”Advances in Neural Information Processing Systems, vol. 38, pp. 29 131–29 161, 2026

    2026

  41. [41]

    Spot risks before speaking! unraveling safety attention heads in large vision-language models,

    Z. Zheng, J. Zhao, L. Yang, L. He, and F. Li, “Spot risks before speaking! unraveling safety attention heads in large vision-language models,”arXiv preprint arXiv:2501.02029, 2025

    Pith/arXiv arXiv 2025

  42. [42]

    Jail- breaklens: Visual analysis of jailbreak attacks against large language models,

    Y. Feng, Z. Chen, Z. Kang, S. Wang, H. Tian, W. Zhang, M. Zhu, and W. Chen, “Jail- breaklens: Visual analysis of jailbreak attacks against large language models,”IEEE Transactions on Visualization and Computer Graphics, 2025

    2025

  43. [43]

    "not aligned

    L. Mei, S. Liu, Y. Wang, B. Bi, J. Mao, and X. Cheng, “"not aligned" is not" malicious": Being careful about hallucinations of large language models’ jailbreak,”COLING 2025, 2024

    2025

  44. [44]

    Fo- cusing by contrastive attention: Enhancing vlms’ visual reasoning,

    Y. Ge, S. Liu, Y. Wang, L. Mei, B. Bi, X. Zhou, J. Yao, J. Guo, and X. Cheng, “Fo- cusing by contrastive attention: Enhancing vlms’ visual reasoning,”arXiv preprint arXiv:2509.06461, 2025

    Pith/arXiv arXiv 2025

  45. [45]

    Effective and efficient adversarial detection for vision-language models via a single vector,

    Y. Huang, F. Zhu, J. Tang, P . Zhou, W. Lei, J. Lv, and T.-S. Chua, “Effective and efficient adversarial detection for vision-language models via a single vector,”arXiv preprint arXiv:2410.22888, 2024

    Pith/arXiv arXiv 2024

  46. [46]

    VLM-Guard: Safeguarding vision-language models via fulfilling safety alignment gap,

    Q. Liu, F. Wang, C. Xiao, and M. Chen, “VLM-Guard: Safeguarding vision-language models via fulfilling safety alignment gap,”arXiv preprint arXiv:2502.10486, 2025

    Pith/arXiv arXiv 2025

  47. [47]

    Safety Subspaces are Not Linearly Distinct: A Fine-Tuning Case Study,

    K. Ponkshe, S. Shah, R. Singhal, and P . Vepakomma, “Safety Subspaces are Not Linearly Distinct: A Fine-Tuning Case Study,”arXiv preprint arXiv:2505.14185, 2025. 12 Preprint. Under review

    arXiv 2025

  48. [48]

    Hiddendetect: Detecting jailbreak attacks against large vision-language models via monitoring hidden states,

    Y. Jiang, X. Gao, T. Peng, Y. Tan, X. Zhu, B. Zheng, and X. Yue, “Hiddendetect: Detecting jailbreak attacks against large vision-language models via monitoring hidden states,” arXiv preprint arXiv:2502.14744, vol. 3, no. 5, 2025

    Pith/arXiv arXiv 2025

  49. [49]

    Brainvis: Exploring the bridge between brain and visual signals via image reconstruction,

    H. Fu, H. Wang, J. J. Chin, and Z. Shen, “Brainvis: Exploring the bridge between brain and visual signals via image reconstruction,” inICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2025, pp. 1–5

    2025

  50. [50]

    Tuning vision-language models with candidate labels by prompt alignment,

    Z. Zhang, Y. Niu, X. Liu, and B. Li, “Tuning vision-language models with candidate labels by prompt alignment,” inDASFAA, 2025

    2025

  51. [51]

    Refineshot: Rethinking cinematography understanding with foundational skill evaluation,

    H. Wu, Y. Cai, H. Ge, H. Chen, M.-H. Yang, and Y. Wang, “Refineshot: Rethinking cinematography understanding with foundational skill evaluation,”arXiv preprint arXiv:2510.02423, 2025

    arXiv 2025

  52. [52]

    What should a streaming video model remember?

    H. Ge, Y. Wang, H. Wu, and Y. Cai, “What should a streaming video model remember?” arXiv preprint arXiv:2606.16353, 2026

    arXiv 2026

  53. [53]

    VLSU: Mapping the Limits of Joint Multimodal Understanding for AI Safety,

    S. Palaskar, L. Gatys, M. Abdelrahman, M. Jacobo, L. Lindsey, R. Moharir, G. Lund, Y. Xu, N. Shiee, J. Bighamet al., “VLSU: Mapping the Limits of Joint Multimodal Understanding for AI Safety,”arXiv preprint arXiv:2510.18214, 2025

    arXiv 2025

  54. [54]

    Vistawise: Building cost-effective agent with cross-modal knowledge graph for minecraft,

    H. Fu, J. Ren, Q. Chai, D. Ye, Y. Cai, and H. Wang, “Vistawise: Building cost-effective agent with cross-modal knowledge graph for minecraft,” inProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, 2025, pp. 21 895–21 909

    2025

  55. [55]

    Safe inputs but unsafe output: Benchmarking cross-modality safety alignment of large vision-language models,

    S. Wang, X. Ye, Q. Cheng, J. Duan, S. Li, J. Fu, X. Qiu, and X.-J. Huang, “Safe inputs but unsafe output: Benchmarking cross-modality safety alignment of large vision-language models,” inFindings of the Association for Computational Linguistics: NAACL 2025, 2025, pp. 3563–3605

    2025

  56. [56]

    Contextnav: Towards agentic multimodal in-context learning,

    H. Fu, Y. Ouyang, K.-W. Chang, Y. Wang, and Y. Cai, “Contextnav: Towards agentic multimodal in-context learning,”arXiv preprint arXiv:2510.04560, 2025

    arXiv 2025

  57. [57]

    Innate reasoning is not enough: In-context learning enhances reasoning large language models with less overthinking,

    Y. Ge, S. Liu, Y. Wang, L. Mei, L. Chen, B. Bi, and X. Cheng, “Innate reasoning is not enough: In-context learning enhances reasoning large language models with less overthinking,”arXiv preprint arXiv:2503.19602, 2025

    Pith/arXiv arXiv 2025

  58. [58]

    MSTS: A Multimodal Safety Test Suite for Vision-Language Models,

    P . Röttger, G. Attanasio, F. Friedrich, J. Goldzycher, A. Parrish, R. Bhardwaj, C. Di Bonaventura, R. Eng, G. E. K. Geagea, S. Goswamiet al., “MSTS: A Multimodal Safety Test Suite for Vision-Language Models,”arXiv preprint arXiv:2501.10057, 2025

    Pith/arXiv arXiv 2025

  59. [59]

    Enhanced partially relevant video retrieval through inter-and intra-sample analysis with coherence prediction,

    J. Ren, G. Zhang, Y. Hu, J. Shu, H. Xiong, and H. Wang, “Enhanced partially relevant video retrieval through inter-and intra-sample analysis with coherence prediction,” arXiv preprint arXiv:2504.19637, 2025

    arXiv 2025

  60. [60]

    Slang: New concept comprehension of large language models,

    L. Mei, S. Liu, Y. Wang, B. Bi, and X. Cheng, “Slang: New concept comprehension of large language models,”EMNLP 2024, 2024

    2024

  61. [61]

    Wamo: Wavelet-enhanced multi- frequency trajectory analysis for fine-grained text-motion retrieval,

    J. Ren, G. Zhang, H. Fu, P . Wu, and H. Wang, “Wamo: Wavelet-enhanced multi- frequency trajectory analysis for fine-grained text-motion retrieval,”arXiv preprint arXiv:2508.03343, 2025

    Pith/arXiv arXiv 2025

  62. [62]

    Sdr-gain: A high real-time occluded pedestrian pose completion method for autonomous driving,

    H. Fu, Y. Gu, Y. Yan, Y. Shen, Y. Wu, and L. Sun, “Sdr-gain: A high real-time occluded pedestrian pose completion method for autonomous driving,”IEEE Transactions on Intelligent Transportation Systems, vol. 27, no. 1, pp. 972–982, 2025

    2025

  63. [63]

    Minicpm-o 4.5: Towards real-time full-duplex omni-modal interaction,

    J. Cui, B. Xu, C. Wang, T. Yu, W. Sun, Y. Xu, T. Wang, Z. He, W. Ma, T. Caiet al., “Minicpm-o 4.5: Towards real-time full-duplex omni-modal interaction,”arXiv preprint arXiv:2604.27393, 2026

    Pith/arXiv arXiv 2026

  64. [64]

    Enhanced cross-modal 3d retrieval via tri-modal reconstruction,

    J. Ren and H. Wang, “Enhanced cross-modal 3d retrieval via tri-modal reconstruction,” in2025 IEEE International Conference on Multimedia and Expo (ICME). IEEE, 2025, pp. 1–6. 13 Preprint. Under review

    2025

  65. [65]

    Minicpm-v 4.5: Cooking efficient mllms via architecture, data, and training recipe,

    T. Yu, Z. Wang, C. Wang, F. Huang, W. Ma, Z. He, T. Cai, W. Chen, Y. Huang, R. Zhao et al., “Minicpm-v 4.5: Cooking efficient mllms via architecture, data, and training recipe,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, 2026, pp. 11 704–11 715. 14 Preprint. Under review. A Dataset Setup and Attack Construction ...

    2026