Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

SafeVid: Toward Safety Aligned Video Large Multimodal Models

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Video-language models can be made substantially safer by preference training on video-grounded synthetic data, the paper argues.

desk verdict Useful resource paper: the 350K video safety dataset is the real contribution, but the headline safety gains are partly an artifact of the GPT-4/GPT-4o judge loop and need human evaluation before being taken at face value. read the letter →

arxiv 2505.11926 v1 pith:XPWF4ZI4 submitted 2025-05-17 cs.CV cs.AI

Yixu Wang , Jiaxin Song , Yifeng Gao , Xin Wang , Yang Yao , Yan Teng , Xingjun Ma , Yingchun Wang
show 1 more author
Yu-Gang Jiang
This is my paper · ORCID
classification cs.CVcs.AI
keywords videolargemultimodalmodelssafetyalignmentdirectpreferenceoptimizationdatasetmismatchedgeneralizationsaferefusalbenchmarktextualdescriptionbridge
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Video-language models often accept a harmful request when it is accompanied by a relevant video, even though the same model would refuse the identical request as text; the paper names this 'mismatched generalization' and proposes to fix it by training on safety preferences grounded in video context. It builds SafeVid-350K, 350,000 video-specific question-response preference pairs, by turning each video into a detailed textual description and using that description as a bridge for text-based safety reasoning: a strong LLM writes adversarial questions and both a safe and an unsafe answer for each video. Fine-tuning open video-language models with Direct Preference Optimization on this data raises AI-judged safety rates by tens of percentage points—on the base benchmark, LLaVA-NeXT-Video goes from 53.99% to 96.38% safe—and the paper reports that the gains transfer to image and text safety benchmarks without large capability loss. The result matters because it offers a data-driven route to video-specific safety alignment without waiting for expensive human red-teaming at scale.

What carries the argument

The device that carries the argument is the interpretive bridge: a video is converted into a detailed textual description by several video-language models and condensed by an LLM, so that a text-only safety reasoner can apply rule-based judgment to the dynamic scene. All downstream components—adversarial question generation, chosen-response construction, and benchmark question design—operate on this text rather than on the raw video. The alignment step is Direct Preference Optimization, which trains the policy to increase the likelihood of chosen responses and decrease the likelihood of rejected responses: $$L_{\mathrm{DPO}}(\pi_\$\theta$;\pi_{\mathrm{ref}}) = -\mathbb{E}_{(v,q,y_c,y_r)\sim\mathcal{D}}\log\$\sigma$\!\left(\$\beta$\log\frac{\pi_\$\theta$(y_c\mid v,q)}{\pi_{\mathrm{ref}}(y_c\mid v,q)} - \$\beta$\log\frac{\pi_\$\theta$(y_r\mid v,q)}{\pi_{\mathrm{ref}}(y_r\mid v,q)}\right).$$ SafeVidBench fixes the target quantity as Safety Rate over seven harmful categories, in an automated base set and a human-red-teamed challenge set that share no videos or questions with the training data.

What would settle it

Have human annotators blind to model identity score a random sample of aligned-model responses from SafeVidBench-Challenge with the same seven-category rubric; if the human-rated safety rate is close to the pre-alignment model's rate rather than the reported 85.94%, the headline improvement is an artifact of the AI judge.

Watch

Extended reading notes

Core claim

The paper's central claim is that the safety gap between text and video in large multimodal models is not irreducible: textual descriptions can carry enough of the video's meaning for an LLM to reason about harm, generate adversarial queries, and write rule-following responses. The authors operationalize this as SafeVid, a closed loop of data construction, alignment, and evaluation. They curate 12,377 videos into 30 scene categories, describe each video through multiple video-language models refined by an LLM, use those descriptions to prompt an instruction-following generator for candidate adversarial questions per safety subcategory, then produce a rejected response from a vulnerable video-language model and a chosen response from an LLM armed with safety guidelines. DPO alignment on the resulting 350K preference pairs lifts LLaVA-NeXT-Video from 53.99% to 96.38% average safety on SafeVidBench-Base and from 46.23% to 85.94% on SafeVidBench-Challenge, with similarly large gains for Qwen2.5-VL-7B. The authors read this as evidence that video-specific safety principles can be instilled through preference learning on description-grounded data.

Load-bearing premise

The load-bearing premise is that the AI model used as judge for every reported safety score agrees with human safety judgments; if the judge merely rewards the answer style of the same model family that wrote the preferred training responses, the headline gains would be alignment to the judge rather than real safety.

Editorial extensions

If this is right

  • On the paper's measurement, LLaVA-NeXT-Video's average safety on SafeVidBench-Base rises from 53.99% to 96.38% after DPO on SafeVid-350K, and on the human-red-teamed Challenge set from 46.23% to 85.94%.
  • Qwen2.5-VL-7B improves from 77.03% to 95.87% on the Base set and from 59.78% to 89.78% on the Challenge set, so the effect is not architecture-specific.
  • The gains appear to transfer out of distribution: the aligned LLaVA-NeXT-Video cuts VLBreakBench attack success from 68.00% to 16.48% and raises MM-SafetyBench safety from 42.02% to 74.31%, while Helpful Rate stays high.
  • The alignment tax is small: on MMBench-Video, perception scores hold roughly steady or improve and hallucination scores improve, while reasoning scores dip only modestly.
  • Data-scaling results indicate that basic refusal behavior appears with a small fraction of the 350K pairs, but the harder Challenge set and out-of-distribution benchmarks keep improving as more pairs are added, so dataset scale buys robustness rather than just refusal.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the description-bridge mechanism is what carries the effect, the same recipe could generate safety alignment pairs for any captionable scene or language without collecting new videos.
  • A direct stress test this suggests is to corrupt or temporally truncate the textual descriptions and watch SafeVidBench safety fall; the drop would localize which video information the bridge needs.
  • The judge-writer alignment loop implies a test the paper does not run: re-train with chosen responses produced by a different family of models and re-score with the same judge; if gains collapse, part of the measured improvement is stylistic conformity.
  • Because the evaluation reads through the same textual bridge that generated the training labels, probing the description step itself with misleading captions would be a natural next attack surface.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces SafeVid, a framework for safety alignment of video large multimodal models (VLMMs). The framework has three components: (1) SafeVid-350K, a large-scale preference dataset of 350,000 video-specific query–response pairs, constructed using textual video descriptions as an interpretive bridge and GPT-4-generated chosen responses; (2) Direct Preference Optimization (DPO) fine-tuning of VLMMs on this dataset; and (3) SafeVidBench, a new video-safety benchmark with Base and Challenge subsets. The main empirical claim is that DPO on SafeVid-350K substantially improves safety rates on SafeVidBench, e.g., a 42.39 percentage-point increase for LLaVA-NeXT-Video on SafeVidBench-Base, while producing only a minimal alignment tax on general video understanding (MMBench-Video). The paper also reports out-of-distribution evaluations on image-text and text-only safety benchmarks.

Significance. If the central claim is valid, the paper makes a useful contribution: it releases a large public preference dataset, a benchmark for video safety, and evidence that safety reasoning can be transferred from text to video through DPO with detailed textual descriptions. The breadth of evaluated models and the inclusion of OOD benchmarks are strengths. However, the validity of the quantitative claim currently rests on the assumption that GPT-4o adjudication provides an unbiased measure of safety; because the preferred training responses are also GPT-4-generated, the reported gains may partly reflect imitation of GPT-4 response style rather than improved safety. No human evaluation or alternative judge is provided. The core methodology is sound and the concerns are addressable with additional experiments, but the current evidence does not yet establish the magnitude of the safety improvement.

major comments (4)
  1. [Section 4.1 (Evaluation Metrics) and Section 3.1 (Preference Pair Synthesis)] The evaluation loop is circular: all safety and helpfulness scores in Tables 2 and 3 are adjudicated by GPT-4o (Section 4.1), while the chosen training responses are synthesized by GPT-4 (Section 3.1). DPO (Eq. 1) directly increases the likelihood of these GPT-4-written responses. If GPT-4o's judgments correlate with stylistic resemblance to GPT-4, then the reported deltas (e.g., +42.39 in Table 2) may reflect alignment to the judge rather than to safety. Please provide a human-annotation agreement study (e.g., Cohen's kappa between GPT-4o and human annotators on a random sample of responses) and/or validate the same comparisons with a second, non-GPT judge (e.g., a dedicated safety classifier or Llama-Guard). Without such evidence, the magnitude of the claimed safety improvement is not established.
  2. [Section 4.1 and Section 4.2] The paper reports only DPO training and does not include a supervised fine-tuning (SFT) control on the same chosen responses. Since all chosen responses are GPT-4-generated, a model fine-tuned with SFT on the chosen responses alone could recover much of the reported improvement by learning GPT-4's refusal style, independent of the preference loss. Please add an SFT-on-chosen-only baseline for both base models (LLaVA-NeXT-Video and Qwen2.5-VL-7B) and compare its SafeVidBench safety rates with the DPO results. This would isolate the contribution of the preference optimization itself.
  3. [Abstract, Section 4.2, and Table 2] There is a numerical inconsistency in the headline Challenge-set improvement: the abstract reports 39.17% for LLaVA-NeXT-Video + SafeVid-350K on SafeVidBench-Challenge, while Section 4.2 and Table 2 report 39.71%. Additionally, Table 2's headers use the name "VidSafeBench" while the rest of the paper uses "SafeVidBench"; the same inconsistent name appears in Section 2 (Related Work). Please correct the number and harmonize the benchmark name throughout the manuscript.
  4. [Section 4.1 and Figure 3] All reported safety rates are single-run point estimates without confidence intervals, significance tests, or error bars. This is particularly important for the smaller OOD deltas in Table 3 (e.g., the +7.31 safety-rate gain on miniJailBreakV-28K for LLaVA-NeXT-Video, or the +8.37 gain for Qwen2.5-VL-7B on the same benchmark), where the improvement is much smaller than the headline SafeVidBench numbers. Figure 3, which varies the training-data fraction, also lacks error bars across random subsets. Please report variance across at least three random seeds or, at minimum, provide bootstrap confidence intervals for the key comparisons.
minor comments (5)
  1. [Section 3.1 and Section 3.3] The number of safety subcategories is never stated explicitly. The benchmark construction in Section 3.3 counts 1,380 questions as 30 scene categories × 2 questions × 23 subcategories, but the text only lists 7 aggregate categories in Table 2. The same number of subcategories is needed to reconcile the 350K-pair dataset size with 12,377 curated videos (12,377 × 28 ≈ 350K, which requires 28 queries per video). Please state the exact number of subcategories and the number of preference pairs generated per video so the dataset size is reproducible.
  2. [Section 4.1] The sentence "We adapte LLaMA-Factory training framework" is missing a word; it should read "We adapt the LLaMA-Factory training framework" or similar. Please also verify the reference to LLaMA-Factory [52] is correctly cited.
  3. [Figure 3] The caption reads "Impact of SafeVid-350K data scale on alignment effectiveness" but the figure contains two panels (SafeVidBench and MM-SafetyBench) with different y-axis ranges. Please update the caption to describe both panels and label them explicitly.
  4. [References] Reference [8] is cited as "HarmEval" in Table 1, but the reference title is "SafeInfer: Context adaptive decoding time safety alignment for large language models." Please verify that this is the correct source for the HarmEval benchmark and, if so, clarify the relationship in the text.
  5. [Section 5 (Limitations)] The Limitations paragraph acknowledges the reliance on textual video descriptions but does not mention the potential bias from using GPT-family models both as the source of preferred responses and as the judge. Please add a sentence acknowledging this limitation and the need for human-validated evaluation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the GPT-4/GPT-4o evaluation loop is a validity threat, not a by-construction reduction.

full rationale

The paper's derivation chain is: construct SafeVid-350K from InternVid videos, Gemini-generated adversarial questions, LLaVA-NeXT-Video rejected responses, and GPT-4 chosen responses (Sec 3.1); DPO-tune VLMMs with Eq. (1); evaluate on disjoint SafeVidBench with GPT-4o-adjudicated Safety Rate (Sec 4.1). The claimed improvement (Table 2) is an empirical outcome, not an identity: the evaluation set has no overlap with training videos/questions, and the DPO objective does not optimize GPT-4o's score directly. The skeptic's judge-loop concern—that GPT-4o may reward GPT-4-like refusal style—is conditional and requires evidence of judge bias; it does not make the safety delta equivalent to the training data by construction. The paper's self-citations (FLAMES, Fake Alignment, Safety at Scale, SPA-VL, Ideator) are contextual and not load-bearing; no uniqueness theorem or result-restricting ansatz is imported from authors' prior work. The Limitations section honestly notes the reliance on textual descriptions is linked to description fidelity, which is a scope limitation, not a circular step. Minor inconsistencies (abstract 39.17% vs Sec 4.2/Table 2 39.71%; 'VidSafeBench' vs 'SafeVidBench') are reporting errors and do not affect circularity. Hence, no circular step meets the required standard of a quoted equation-level reduction; score 0.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim rests on unvalidated modeling choices: text descriptions as a proxy for video safety content, model-generated preferences without human validation, and an LLM judge for all outcome metrics. The only numerically chosen quantities are training hyperparameters, which are not ablated.

free parameters (3)
  • DPO beta = 0.1
    Chosen by hand; controls deviation from the reference policy. No ablation or sensitivity analysis is reported.
  • DPO learning rate = 1.0e-6
    Chosen for training; no sensitivity analysis or learning-rate sweep is reported.
  • DPO training epochs = 1
    Chosen; no ablation on number of epochs is reported.
assumptions (5)
  • domain assumption Textual video descriptions are a sufficient proxy for video content for safety reasoning.
    Used throughout SafeVid-350K construction and acknowledged in the Limitations as bounding the framework's efficacy.
  • domain assumption GPT-4-generated chosen responses are genuinely safer than the LLaVA-NeXT-Video rejected responses.
    The preference pairs are synthesized with no human validation of preferability or safety.
  • domain assumption GPT-4o adjudication provides a valid measure of safety and helpfulness.
    All Safety Rate and Helpful Rate scores in Tables 2 and 3 are assigned by GPT-4o; no human agreement or calibration is reported.
  • domain assumption The 3H-derived taxonomy covers the safety dimensions relevant to video contexts.
    The taxonomy guides question generation and benchmark categories; no completeness argument for video-specific risks is provided.
  • domain assumption SafeVidBench videos and questions do not overlap with SafeVid-350K.
    Stated in Sec 3.3 but no verification procedure, hashes, or deduplication evidence is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SafeVid: Toward Safety Aligned Video Large Multimodal Models." pith.science (2026). https://pith.science/paper/XPWF4ZI4

@misc{pith2026250511926,
  author       = {Pith},
  title        = {Pith review of: SafeVid: Toward Safety Aligned Video Large Multimodal Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XPWF4ZI4}},
  note         = {Machine review of arXiv:2505.11926}
}
read the original abstract

As Video Large Multimodal Models (VLMMs) rapidly advance, their inherent complexity introduces significant safety challenges, particularly the issue of mismatched generalization where static safety alignments fail to transfer to dynamic video contexts. We introduce SafeVid, a framework designed to instill video-specific safety principles in VLMMs. SafeVid uniquely transfers robust textual safety alignment capabilities to the video domain by employing detailed textual video descriptions as an interpretive bridge, facilitating LLM-based rule-driven safety reasoning. This is achieved through a closed-loop system comprising: 1) generation of SafeVid-350K, a novel 350,000-pair video-specific safety preference dataset; 2) targeted alignment of VLMMs using Direct Preference Optimization (DPO); and 3) comprehensive evaluation via our new SafeVidBench benchmark. Alignment with SafeVid-350K significantly enhances VLMM safety, with models like LLaVA-NeXT-Video demonstrating substantial improvements (e.g., up to 42.39%) on SafeVidBench. SafeVid provides critical resources and a structured approach, demonstrating that leveraging textual descriptions as a conduit for safety reasoning markedly improves the safety alignment of VLMMs. We have made SafeVid-350K dataset (https://huggingface.co/datasets/yxwang/SafeVid-350K) publicly available.

Figures

Figures reproduced from arXiv: 2505.11926 by the authors.

Figure 1
Figure 1. Illustration of mismatched generalization in VLMMs (left) and the improvement of [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of the SafeVid-350K construction framework. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 4
Figure 4. Impact of SAFEVID alignment on general VLMM capabilities, evaluated on MMBench￾Video. Performance scores (higher is better) on perception (CP: Coarse Perception, FP-S: Fine-grained Perception [Single-Instance], FP-C: Fine-grained Perception [Cross-Instance], HL: Hallucination) and reasoning (LR: Logic Reasoning, AR: Attribute Reasoning, RR: Relation Reasoning, CSR: Common Sense Reasoning, TR: Temporal Reasoning) cat… view at source ↗
Figures from the paper (1 more)
Figure 3
Figure 3. Figure 3: Impact of SafeVid-350K data scale on alignment effectiveness. Safety Rate is evaluated on SafeVidBench and MM-SafetyBench. Data Scale. To understand the relationship be￾tween the volume of preference data and align￾ment effectiveness, we conduct experiments varying the…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Child-Oriented AIGC Video Risk Reviewing: A Benchmark and Knowledge-Supported Iterative Reasoning Framework

    cs.CV 2026-07 reject novelty 6.0 of 10

    A multi-agent iterative-questioning framework plus a 605-video benchmark for detecting developmentally inappropriate risks in AI-generated children's videos.

Reference graph

Works this paper leans on

55 extracted references · 26 canonical work pages · cited by 1 Pith paper

  1. [1]

    URLhttps://huggingface.co/datasets/ gretelai/gretel-safety-alignment-en-v1

    Gretel synthetic safety alignment dataset, 2024. URLhttps://huggingface.co/datasets/ gretelai/gretel-safety-alignment-en-v1

    work page 2024

  2. [2]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  3. [3]

    Tuning large multimodal models for videos using reinforcement learning from ai feedback.arXiv preprint arXiv:2402.03746, 2024

    Daechul Ahn, Yura Choi, Youngjae Yu, Dongyeop Kang, and Jonghyun Choi. Tuning large multimodal models for videos using reinforcement learning from ai feedback.arXiv preprint arXiv:2402.03746, 2024

    arXiv 2024

  4. [4]

    Claude.https://claude.ai/chats, 2023

    Anthropic. Claude.https://claude.ai/chats, 2023

    work page 2023

  5. [5]

    Qwen technical report.arXiv preprint arXiv:2309.16609, 2023

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng X...

    arXiv 2023

  6. [6]

    Qwen2.5-vl technical report

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. a...

    arXiv 2025

  7. [7]

    Constitutional ai: Harmlessness from ai feedback.arXiv preprint arXiv:2212.08073, 2022

    Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback.arXiv preprint arXiv:2212.08073, 2022

    arXiv 2022

  8. [8]

    Safeinfer: Context adaptive decoding time safety alignment for large language models

    Somnath Banerjee, Sayan Layek, Soham Tripathy, Shanu Kumar, Animesh Mukherjee, and Rima Hazra. Safeinfer: Context adaptive decoding time safety alignment for large language models. InAAAI, 2025

    work page 2025

Show all 55 references
  1. [9]

    Movieclip: Visual scene recognition in movies

    Digbalay Bose, Rajat Hebbar, Krishna Somandepalli, Haoyang Zhang, Yin Cui, Kree Cole- McLaughlin, Huisheng Wang, and Shrikanth Narayanan. Movieclip: Visual scene recognition in movies. InWACV, 2023

    2023

  2. [10]

    Internlm2 technical report, 2024

    Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, Xiaoyi Dong, Haodong Duan, Qi Fan, Zhaoye Fei, Yang Gao, Jiaye Ge, Chenya Gu, Yuzhe Gu, Tao Gui, Aijia Guo, Qipeng Guo, Conghui He, Yingfan Hu, Ting Huang, Tao Jiang,...

    2024

  3. [11]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. InCVPR, 2024

    2024

  4. [12]

    Bypassing safety guardrails in llms using humor.arXiv preprint arXiv:2504.06577, 2025

    Pedro Cisneros-Velarde. Bypassing safety guardrails in llms using humor.arXiv preprint arXiv:2504.06577, 2025

    2025 arXiv

  5. [13]

    Large scale holistic video understanding

    Ali Diba, Mohsen Fayyaz, Vivek Sharma, Manohar Paluri, Jürgen Gall, Rainer Stiefelhagen, and Luc Van Gool. Large scale holistic video understanding. InECCV, 2020

    2020

  6. [14]

    Mmbench-video: A long-form multi-shot benchmark for holistic video understanding

    Xinyu Fang, Kangrui Mao, Haodong Duan, Xiangyu Zhao, Yining Li, Dahua Lin, and Kai Chen. Mmbench-video: A long-form multi-shot benchmark for holistic video understanding. NeurIPS, 2024

    2024

  7. [15]

    Videojail: Exploiting video-modality vulnerabilities for jailbreak attacks on multimodal large language models

    Wenbo Hu, Shishen Gu, Youze Wang, and Richang Hong. Videojail: Exploiting video-modality vulnerabilities for jailbreak attacks on multimodal large language models. InICLR Workshop on Building Trust in Language Models and Applications, 2025

    2025

  8. [16]

    Flames: Benchmarking value alignment of llms in chinese

    Kexin Huang, Xiangyang Liu, Qianyu Guo, Tianxiang Sun, Jiawei Sun, Yaru Wang, Zeyang Zhou, Yixu Wang, Yan Teng, Xipeng Qiu, et al. Flames: Benchmarking value alignment of llms in chinese. InNAACL, 2024

    2024

  9. [17]

    Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  10. [18]

    Beavertails: Towards improved safety alignment of llm via a human-preference dataset.NeurIPS, 2023

    Jiaming Ji, Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. Beavertails: Towards improved safety alignment of llm via a human-preference dataset.NeurIPS, 2023

    2023

  11. [19]

    Pku-saferlhf: A safety alignment preference dataset for llama family models.arXiv e-prints, pages arXiv–2406, 2024

    Jiaming Ji, Donghai Hong, Borong Zhang, Boyuan Chen, Josef Dai, Boren Zheng, Tianyi Qiu, Boxun Li, and Yaodong Yang. Pku-saferlhf: A safety alignment preference dataset for llama family models.arXiv e-prints, pages arXiv–2406, 2024

    2024

  12. [20]

    Safe rlhf-v: Safe reinforcement learning from human feedback in multimodal large language models.arXiv preprint arXiv:2503.17682, 2025

    Jiaming Ji, Xinyu Chen, Rui Pan, Han Zhu, Conghui Zhang, Jiahao Li, Donghai Hong, Boyuan Chen, Jiayi Zhou, Kaile Wang, et al. Safe rlhf-v: Safe reinforcement learning from human feedback in multimodal large language models.arXiv preprint arXiv:2503.17682, 2025

    2025 arXiv

  13. [21]

    Awesome-llm-robotics, 2022

    Zsolt Kira. Awesome-llm-robotics, 2022. URL https://github.com/GT-RIPL/ Awesome-LLM-Robotics

    2022

  14. [22]

    Rlaif vs

    Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, et al. Rlaif vs. rlhf: Scaling rein- forcement learning from human feedback with ai feedback.arXiv preprint arXiv:2309.00267, 2023

    2023 arXiv

  15. [23]

    Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    2024 arXiv

  16. [24]

    Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  17. [25]

    Harnessing llms for automated video content analysis: An exploratory workflow of short videos on depression

    Jiaying Liu, Yunlong Wang, Yao Lyu, Yiheng Su, Shuo Niu, Xuhai" Orson" Xu, and Yan Zhang. Harnessing llms for automated video content analysis: An exploratory workflow of short videos on depression. InCSCW, 2024

    2024

  18. [26]

    Mm-safetybench: A benchmark for safety evaluation of multimodal large language models

    Xin Liu, Yichen Zhu, Jindong Gu, Yunshi Lan, Chao Yang, and Yu Qiao. Mm-safetybench: A benchmark for safety evaluation of multimodal large language models. InECCV, 2024

    2024

  19. [27]

    Trustworthy llms: a survey and guideline for evaluating large language models’ alignment.arXiv preprint arXiv:2308.05374, 2023

    Yang Liu, Yuanshun Yao, Jean-Francois Ton, Xiaoying Zhang, Ruocheng Guo, Hao Cheng, Yegor Klochkov, Muhammad Faaiz Taufiq, and Hang Li. Trustworthy llms: a survey and guideline for evaluating large language models’ alignment.arXiv preprint arXiv:2308.05374, 2023

    2023 arXiv

  20. [28]

    Jailbreakv-28k: A benchmark for assessing the robustness of multimodal large language models against jailbreak attacks.arXiv e-prints, 2024

    Weidi Luo, Siyuan Ma, Xiaogeng Liu, Xiaoyu Guo, and Chaowei Xiao. Jailbreakv-28k: A benchmark for assessing the robustness of multimodal large language models against jailbreak attacks.arXiv e-prints, 2024. 11

    2024

  21. [29]

    Safety at scale: A comprehensive survey of large model safety.arXiv preprint arXiv:2502.05206, 2025

    Xingjun Ma, Yifeng Gao, Yixu Wang, Ruofan Wang, Xin Wang, Ye Sun, Yifan Ding, Hengyuan Xu, Yunhao Chen, Yunhan Zhao, et al. Safety at scale: A comprehensive survey of large model safety.arXiv preprint arXiv:2502.05206, 2025

    2025 arXiv

  22. [30]

    Sg-bench: Evaluating llm safety generalization across diverse tasks and prompt types.NeurIPS, 2024

    Yutao Mou, Shikun Zhang, and Wei Ye. Sg-bench: Evaluating llm safety generalization across diverse tasks and prompt types.NeurIPS, 2024

    2024

  23. [31]

    Chatgpt.https://chat.openai.com/chat, 2023

    OpenAI. Chatgpt.https://chat.openai.com/chat, 2023

    2023

  24. [32]

    Training language models to follow instructions with human feedback.NeurIPS, 2022

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback.NeurIPS, 2022

    2022

  25. [33]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. NeurIPS, 2023

    2023

  26. [34]

    Safetywashing: Do ai safety benchmarks actually measure safety progress?NeurIPS, 2024

    Richard Ren, Steven Basart, Adam Khoja, Alice Gatti, Long Phan, Xuwang Yin, Mantas Mazeika, Alexander Pan, Gabriel Mukobi, Ryan Kim, et al. Safetywashing: Do ai safety benchmarks actually measure safety progress?NeurIPS, 2024

    2024

  27. [35]

    Large language model safety: A holistic survey.arXiv preprint arXiv:2412.17686, 2024

    Dan Shi, Tianhao Shen, Yufei Huang, Zhigen Li, Yongqi Leng, Renren Jin, Chuang Liu, Xinwei Wu, Zishan Guo, Linhao Yu, et al. Large language model safety: A holistic survey.arXiv preprint arXiv:2412.17686, 2024

    2024 arXiv

  28. [36]

    A strongreject for empty jailbreaks.arXiv preprint arXiv:2402.10260, 2024

    Alexandra Souly, Qingyuan Lu, Dillon Bowen, Tu Trinh, Elvis Hsieh, Sana Pandey, Pieter Abbeel, Justin Svegliato, Scott Emmons, Olivia Watkins, et al. A strongreject for empty jailbreaks.arXiv preprint arXiv:2402.10260, 2024

    2024 arXiv

  29. [37]

    Aligning large multimodal models with factually augmented rlhf.arXiv preprint arXiv:2309.14525, 2023

    Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. Aligning large multimodal models with factually augmented rlhf.arXiv preprint arXiv:2309.14525, 2023

    2023 arXiv

  30. [38]

    Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  31. [39]

    Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  32. [40]

    Ideator: Jailbreaking large vision-language models using themselves.arXiv preprint arXiv:2411.00827, 2024

    Ruofan Wang, Bo Wang, Xiaosen Wang, Xingjun Ma, and Yu-Gang Jiang. Ideator: Jailbreaking large vision-language models using themselves.arXiv preprint arXiv:2411.00827, 2024

    2024

  33. [41]

    Internvid: A large-scale video-text dataset for multimodal understanding and generation.arXiv preprint arXiv:2307.06942, 2023

    Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, Xinyuan Chen, Yaohui Wang, Ping Luo, Ziwei Liu, Yali Wang, Limin Wang, and Yu Qiao. Internvid: A large-scale video-text dataset for multimodal understanding and generation.arXiv preprint arXiv:2307.06942, 2023

    2023 arXiv

  34. [42]

    Internvideo2: Scaling foundation models for multimodal video understanding

    Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Zun Wang, Yansong Shi, et al. Internvideo2: Scaling foundation models for multimodal video understanding. InECCV, 2024

    2024

  35. [43]

    Fake alignment: Are llms really aligned well? InNAACL, 2024

    Yixu Wang, Yan Teng, Kexin Huang, Chengqi Lyu, Songyang Zhang, Wenwei Zhang, Xingjun Ma, Yu-Gang Jiang, Yu Qiao, and Yingchun Wang. Fake alignment: Are llms really aligned well? InNAACL, 2024

    2024

  36. [44]

    Gpt4video: A unified multimodal large language model for lnstruction-followed understanding and safety-aware generation

    Zhanyu Wang, Longyue Wang, Zhen Zhao, Minghao Wu, Chenyang Lyu, Huayang Li, Deng Cai, Luping Zhou, Shuming Shi, and Zhaopeng Tu. Gpt4video: A unified multimodal large language model for lnstruction-followed understanding and safety-aware generation. InACM MM, 2024

    2024

  37. [45]

    Jailbroken: How does llm safety training fail?NeurIPS, 2023

    Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does llm safety training fail?NeurIPS, 2023

    2023

  38. [46]

    Videoclip: Contrastive pre-training for zero- shot video-text understanding

    Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. Videoclip: Contrastive pre-training for zero- shot video-text understanding. InEMNLP, 2021

    2021

  39. [47]

    Minicpm-v: A gpt-4v level mllm on your phone.arXiv preprint arXiv:2408.01800, 2024

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level mllm on your phone.arXiv preprint arXiv:2408.01800, 2024. 12

    2024 arXiv

  40. [48]

    Direct preference optimization of video large multimodal models from language model reward.arXiv preprint arXiv:2404.01258, 2024

    Ruohong Zhang, Liangke Gui, Zhiqing Sun, Yihao Feng, Keyang Xu, Yuanhan Zhang, Di Fu, Chunyuan Li, Alexander Hauptmann, Yonatan Bisk, et al. Direct preference optimization of video large multimodal models from language model reward.arXiv preprint arXiv:2404.01258, 2024

    2024 arXiv

  41. [49]

    Llava-next: A strong zero-shot video understanding model

    Y Zhang, B Li, H Liu, Y Lee, L Gui, D Fu, J Feng, Z Liu, and C Li. Llava-next: A strong zero-shot video understanding model. 2024

    2024

  42. [50]

    Spa-vl: A comprehensive safety preference alignment dataset for vision language model, 2024

    Yongting Zhang, Lu Chen, Guodong Zheng, Yifeng Gao, Rui Zheng, Jinlan Fu, Zhenfei Yin, Senjie Jin, Yu Qiao, Xuanjing Huang, Feng Zhao, Tao Gui, and Jing Shao. Spa-vl: A comprehensive safety preference alignment dataset for vision language model, 2024

    2024

  43. [51]

    Llava-next: A strong zero-shot video understanding model, 2024

    Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. Llava-next: A strong zero-shot video understanding model, 2024. URL https://llava-vl.github.io/blog/2024-04-30-llava-next-video/

    2024

  44. [52]

    Llamafactory: Unified efficient fine-tuning of 100+ language models

    Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. InACL,

  45. [53]

    Fine-tuning language models from human preferences

    Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019

    1909 arXiv

  46. [54]

    Universal and transferable adversarial attacks on aligned language models.arXiv preprint arXiv:2307.15043, 2023

    Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models.arXiv preprint arXiv:2307.15043, 2023. 13

    2023 arXiv

  47. [2024]

    URLhttp://arxiv.org/abs/2403.13372

    arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.