Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

Static bias tests miss how vision-language models fail; adaptive multi-turn probing exposes deeper stereotypes.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 06:08 UTC pith:II66LQIP

load-bearing objection Solid adaptive bias-evaluation pipeline for LVLMs; the main claim holds, with one definitional soft spot that does not sink the empirical results. the 3 major comments →

arxiv 2607.11228 v1 pith:II66LQIP submitted 2026-07-13 cs.CY cs.AI

DeepBias: Adaptive In-depth Probing of Social Biases in LVLMs

Anqi Li , Jie Zhang , Zhongqi Wang , Songkai Xue , Jiahao Wang , Shiguang Shan , Xilin Chen This is my paper
classification cs.CY cs.AI
keywords vision-language modelssocial bias evaluationadaptive benchmarkingmulti-turn probingDirect Preference OptimizationDeepBiasBenchagentic red-teaming
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Large vision-language models often look safe on fixed bias quizzes because a single surface question can be refused or answered carefully. This paper argues that those static tests systematically understate residual social bias. DeepBias instead runs a closed loop: one agent generates and reshapes image-question sets toward whatever fails a target model, while a second agent rewrites each question over several turns, conditioned on the previous answer, using a library of deepening and rewriting skills. The same process, run against an ensemble of five models, yields DeepBiasBench. On that benchmark, models that nearly saturate older static sets drop sharply in abstention accuracy, showing that adaptive, multi-turn probes uncover biases that fixed single-turn tests leave hidden. The practical claim is that bias evaluation itself should evolve with the model rather than remain a frozen checklist.

Core claim

When test data are adapted at the distribution level by preference optimization on a target model's biased responses, and then each instance is further rewritten over multiple response-conditioned turns, the resulting probes expose substantially more social bias than the original seed or existing static vision-language bias benchmarks, while still transferring across model families.

What carries the argument

The generation-evolution-probing loop: a ProposerAgent that expands and DPO-adapts candidate image-question distributions toward model failures, coupled with a skill-driven DiggerAgent that multi-turn rewrites each question using a curated library of rewriting and deepening strategies conditioned on prior answers.

Load-bearing premise

Any Yes or No answer on a carefully constrained three-way question is counted as stereotype-driven bias rather than visual error, format sensitivity, or simple instruction failure.

What would settle it

If the same models, after the full DeepBias pipeline, still abstain at rates comparable to the seed VLBBQ set on DeepBiasBench, or if manual audit shows most committed answers are visual misreads rather than stereotype judgments, the claim that adaptive probing uncovers deeper bias collapses.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes DeepBias, an adaptive two-agent framework for probing social biases in LVLMs. A ProposerAgent expands and DPO-adapts a multimodal seed set (VLBBQ, derived from BBQ Age/Race/Gender) toward target-model failure modes; a DiggerAgent then rewrites each instance over multiple turns using a curated skill library (deepening and rewriting families), conditioned on prior model responses. Using an ensemble of five anchor LVLMs, the authors construct DeepBiasBench (~55k instances) and report full-pipeline trajectories, a DPO ablation (Table III), cross-model transfer (Table IV), a 500-sample manual quality audit (95.2% pass), and comparisons showing lower abstention accuracy and larger model separation than VLBiasBench and SB-Bench (Tables VI–VII). Bias is operationalized as any committed Yes/No under four constraints (§III-A), with abstention as the sole correct answer.

Significance. If the central claim holds, the work supplies a concrete evolutionary alternative to static VL bias benchmarks and a reusable construction pipeline (Proposer DPO + multi-turn skill probing + anchor voting). Strengths include a clean DPO ablation, transfer experiments without regeneration, explicit separation of anchor vs non-anchor results, comparison against two BBQ-derived VL benchmarks, and a documented manual audit. The contribution is timely for LVLM safety evaluation, where static suites risk saturation and leakage. The main scientific value is the demonstration that distribution-level adaptation plus instance-level multi-turn rewriting can substantially reduce abstention rates relative to seed and existing static sets, while remaining partly transferable across families.

major comments (3)
  1. [§III-A; Tables I, III, VI, VII] §III-A and all accuracy/bias-rate results: the protocol equates any committed Yes/No under the four constraints with stereotype-driven social bias (bias rate = 1 − abstention accuracy). This is load-bearing for Tables I, III, VI, and VII. The manuscript does not isolate this surface behavior from confounds that the adaptive pipeline is especially likely to elicit—visual misrecognition of SDXL faces, option-format sensitivity, multi-turn instruction drift, or generic refusal collapse under progressive rewriting. The §IV-E audit checks that questions measure harmful bias and that abstention is correct; it does not audit whether committed answers track demographic stereotypes (e.g., directionally consistent with Age/Race image variants) rather than non-bias failures. A targeted analysis—e.g., stereotype-direction consistency across the multi-image setup, single-turn vs multi-turn control re
  2. [§III-C; Table I] §III-C and Table I (Deep 1–3): the largest accuracy reductions come from DiggerAgent multi-turn rewriting, yet there is no ablation of the skill library (Deepening vs Rewriting families, or individual skills) and no control that rewrites questions for length/complexity without bias-oriented skills. Without this, it remains unclear how much of the Deep-stage drop is due to the curated bias-probing skills versus generic multi-turn pressure or question difficulty. A minimal skill-ablation or non-bias rewrite control on Align 2 would make the instance-level contribution more interpretable and support the claim that the skill library specifically deepens social bias.
  3. [§III-D; Table VI] §III-D and Table VI (anchor block): DeepBiasBench is optimized via DPO preference voting and Digger feedback against the same five anchors later scored in the lower block; anchors therefore score lower by construction. The authors acknowledge this and report non-anchor models, which is appropriate. Still, the main claim that DeepBiasBench is a general challenging benchmark would be clearer if primary headline numbers and rank analyses emphasized non-anchor models only (or held-out construction anchors), and if at least one fully held-out construction run (different anchor set) were reported to quantify how much of the difficulty is ensemble-specific versus shared. Table IV transfer helps, but does not fully replace a held-out construction check for the released benchmark.
minor comments (6)
  1. [Fig. 3; Table II] Fig. 3 and Table II are helpful; consider adding one full multi-image (Age/Race) trajectory with model answers per demographic variant so readers can see whether committed answers align with stereotype direction.
  2. [§IV-A] §IV-A: free parameters (2 DPO rounds, T=3, 2000 candidates, ≥3/5 anchor vote, dedup threshold) are deferred to the supplement; a short sensitivity summary in the main text would improve reproducibility for readers who only see the main paper.
  3. [Table V] Table V non-monotonic accuracy within Proposer/Digger stages is explained, but a brief note in the table caption would prevent misreading as instability of the method.
  4. [§II-B] Related work on adaptive red-teaming is solid; a clearer one-paragraph contrast with RedHit (preference + iterative refinement for jailbreaks) already present could be tightened to stress the bias-specific metric and multi-image demographic controls.
  5. [§IV-H] Limitations (§IV-H) correctly flag compute cost and synthetic images; stating approximate GPU-hours for a single-target run in the main experimental protocol would help practitioners.
  6. [Throughout; Table VI] Minor consistency: “LLaV A” spacing/encoding and “Behaviour” vs “Behavior” in skill names should be normalized; check “seperately” → “separately” in Table VI caption.

Circularity Check

1 steps flagged

Partial construction–evaluation overlap on the five anchor models; central claims still rest on non-anchors, transfer, and external benchmarks.

specific steps
  1. fitted input called prediction [§III-D Anchor Ensemble Optimization; §IV-E Table V; §IV-F Table VI lower block]
    "During the ProposerAgent’s DPO adaptation, preference labels are determined by voting across the anchor ensemble. Specifically, a candidate is treated as a positive preference if it receives biased responses from at least three anchor models... As expected, these models generally obtain lower accuracy than other models, since the data optimization process is aimed at exposing the biases of these models."

    DeepBiasBench is optimized so that candidates that bias ≥3 of the five anchors are preferred; the same five anchors are then reported on DeepBiasBench with lower accuracy. That lower block is partly forced by the construction objective rather than an independent measurement of shared vulnerability. The authors acknowledge the design effect and separate anchors from non-anchors, so the circularity is partial and does not alone underwrite the non-anchor or external-benchmark claims.

full rationale

DeepBias is an adaptive evaluation/red-teaming framework, not a first-principles derivation of a physical or mathematical quantity. The only clear circularity-adjacent step is that DeepBiasBench is built by DPO preference voting and multi-turn probing against five anchor LVLMs that are later scored on that same benchmark; those anchors are expected to score lower by design, which the authors state explicitly and segregate in Table VI. That is a mild fitted-construction / evaluation overlap, not a self-definitional identity of equations. Single-target trajectories (Table I) and DPO ablations (Table III) measure the intended effect of adapting probes to a target’s responses—standard for adaptive probing—and are not predictions forced from fitted parameters renamed as independent forecasts. Preference pairs are used to update the ProposerAgent, which then generates new candidates; evaluation is not on the same preference pairs. Non-anchor models, cross-model transfer (Table IV), and comparisons to VLBiasBench/SB-Bench (Table VII) supply independent content for the claim that the pipeline yields harder probes. The §III-A definition (committed Yes/No under the four constraints = bias) is a load-bearing evaluation convention, not a circular derivation step under this analyzer’s criteria. No self-citation uniqueness theorems, smuggled ansatzes, or renaming of known results carry the central claim. Score 3 reflects one partial construction–evaluation loop with independent support remaining.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 5 invented entities

The central empirical claim rests on a small set of free design choices (iteration counts, pool size, voting threshold), domain assumptions that define what counts as bias, and several invented methodological entities (the two agents, the skill library, the ensemble-derived benchmark). No physical constants or pure-math free parameters appear; the free parameters are engineering knobs that control how aggressively the test distribution is shifted.

free parameters (5)
  • DPO adaptation rounds = 2
    Fixed at 2; controls how far the Proposer distribution is shifted toward target vulnerabilities.
  • Digger probing turns T = 3
    Fixed at 3; directly determines depth of instance-level rewriting and the reported Deep 1/2/3 accuracies.
  • Candidate pool size per stage = 2000
    Fixed at 2000 candidates per demographic category per generation stage; chosen for scale/diversity trade-off.
  • Anchor voting threshold for positive preference = ≥3 of 5
    A candidate is positive only if ≥3 of 5 anchors produce a biased response; intermediate cases are discarded.
  • Semantic deduplication cosine threshold
    Used to retain later-stage samples when questions are near-duplicates; exact value given only in supplementary material.
axioms (4)
  • domain assumption Under the four constraints of §III-A, abstention is the sole evidence-supported answer; any committed Yes/No is stereotype-driven bias.
    Load-bearing definition that converts every accuracy number into a bias rate (1 − accuracy).
  • ad hoc to paper Preference pairs formed by whether a candidate elicits at least one biased response (or ≥3 anchors) supply a useful DPO signal for shifting the test distribution toward vulnerabilities.
    Justifies the Proposer update rule; not derived from prior theory of evaluation data.
  • ad hoc to paper The curated skill library (Attribute Refining, Scenario Escalating, Contrasting, Contextualizing, Cognitive Projecting, Behaviour Deciding, Cognitive Attributing) is sufficient to elicit and deepen social bias without introducing new confounds.
    Skills were retained after internal trials; no external validation that the library is complete or unbiased.
  • domain assumption Synthetic images rendered by SDXL from controlled demographic prompts adequately convey the latent demographic cues needed for bias evaluation.
    All visual stimuli are synthetic; real-image distribution shift is acknowledged only as a limitation.
invented entities (5)
  • ProposerAgent no independent evidence
    purpose: Generate and DPO-adapt candidate image-question distributions toward target-model failure modes.
    Methodological construct; no independent existence outside the pipeline.
  • DiggerAgent no independent evidence
    purpose: Perform multi-turn, skill-conditioned rewriting of individual questions conditioned on prior model responses.
    Methodological construct whose behavior depends on the frozen Qwen3-32B backbone and the skill library.
  • DeepBias skill library (Deepening + Rewriting families) no independent evidence
    purpose: Provide discrete, selectable strategies for eliciting or deepening bias.
    Hand-curated set retained after internal trials; not independently validated.
  • DeepBiasBench no independent evidence
    purpose: Shared, ensemble-derived benchmark of ~55 k instances capturing cross-architecture bias vulnerabilities.
    Constructed artifact of the pipeline; its difficulty is partly by construction against the anchors.
  • VLBBQ seed set no independent evidence
    purpose: Multimodal conversion of selected BBQ Age/Race/Gender items via SDXL for initialization.
    Derived from BBQ but newly rendered; used only as seed, not as the final claim.

pith-pipeline@v1.1.0-grok45 · 27317 in / 3609 out tokens · 50714 ms · 2026-07-14T06:08:37.175730+00:00 · methodology

0 comments
read the original abstract

While Large Vision-Language Models (LVLMs) demonstrate remarkable capabilities, they remain highly susceptible to embedded social biases. Existing bias evaluation protocols predominantly rely on static datasets, which provide only a superficial assessment, as their fixed test cases cannot adaptively evolve to measure the true depth and limits of model vulnerabilities. We introduce DeepBias, an adaptive framework for the in-depth probing of social biases in LVLMs with carefully designed agents. Our approach operates through a dynamic ''generation-evolution-probing'' loop. First, a generative ProposerAgent synthesizes test data and is iteratively updated via Direct Preference Optimization (DPO) based on the target LVLM's responses, exploring model-specific failure modes. Second, an autonomous skill-driven DiggerAgent rewrites each test data across multiple probing turns, adaptively selecting from a curated skill library of deepening and rewriting strategies. At each turn, this process is conditioned on the model's previous response, enabling progressively deeper biases to be exposed. Furthermore, we build a benchmark named DeepBiasBench using our framework. By employing an ensemble of five diverse state-of-the-art LVLMs as anchors, the benchmark captures vulnerabilities shared across architectures. Comprehensive experiments demonstrate the effectiveness of our framework and show that DeepBias provides a challenging benchmark for in-depth bias evaluation, establishing an evolutionary paradigm for LVLM safety assessment.

Figures

Figures reproduced from arXiv: 2607.11228 by Anqi Li, Jiahao Wang, Jie Zhang, Shiguang Shan, Songkai Xue, Xilin Chen, Zhongqi Wang.

Figure 1
Figure 1. Figure 1: Comparison between traditional static evaluation and in-depth bias probing. The left panel illustrates conventional static evaluation, where a single-turn query gets a safe response and the model is therefore judged as unbiased. The right panel shows the iterative probing process in DeepBias, where follow-up queries progressively refine the original question while preserving its underlying intent, ultimate… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the DeepBias framework. The ProposerAgent generates candidate image-question pairs and iteratively adapts to target-model vulnerabilities through DPO updates derived from model responses. The resulting candidate pool is then passed to the DiggerAgent, which performs instance level in-depth probing by selecting skills from a curated library to deepen or rewrite the question based on previous res… view at source ↗
Figure 3
Figure 3. Figure 3: Question distributions before and after ProposerAgent’s DPO adaptation. Panel (a) compares topic coverage between the Seed pool and the three models’ Align 2 data. Panels (b-e) show word clouds for the same four groups, where larger words indicate higher frequency. we choose these numbers in the Supplementary Material §II-C. We render images with Stable Diffusion XL [45], and conduct experiments on 8× NVID… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Zing: Social Mind for LLMs

    cs.CL 2026-07 conditional novelty 6.0

    Coordinated evaluation (SoMBench), staged training (Zing), and typed inference supports (Actio) measurably improve LLM social reasoning, with large remaining headroom on a 3,481-item benchmark.

Reference graph

Works this paper leans on

63 extracted references · 18 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Vision-language models for vision tasks: A survey,

    J. Zhang, J. Huang, S. Jin, and S. Lu, “Vision-language models for vision tasks: A survey,”IEEE Trans. Pattern Anal. Mach. Intell., vol. 46, no. 8, pp. 5625–5644, 2024

    2024

  2. [2]

    Gpt-4o system card,

    A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radfordet al., “Gpt-4o system card,”arXiv preprint arXiv:2410.21276, 2024

    Pith/arXiv arXiv 2024

  3. [3]

    ShowUI: One vision-language-action model for GUI visual agent,

    K. Q. Lin, L. Li, D. Gao, Z. Yang, S. Wu, Z. Bai, S. W. Lei, L. Wang, and M. Z. Shou, “ShowUI: One vision-language-action model for GUI visual agent,” inIEEE Conf. Comput. Vis. Pattern Recog., 2025, pp. 19 498–19 508

    2025

  4. [4]

    Prox- imal policy optimization algorithms,

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Prox- imal policy optimization algorithms,”arXiv preprint arXiv:1707.06347, 2017

    Pith/arXiv arXiv 2017

  5. [5]

    Defying distractions in multimodal tasks: A novel benchmark for large vision-language models,

    J. Yang, M. Jiang, and Q. Zhao, “Defying distractions in multimodal tasks: A novel benchmark for large vision-language models,”IEEE Trans. Pattern Anal. Mach. Intell., 2026

    2026

  6. [6]

    Lvlm-ehub: A comprehensive evaluation bench- mark for large vision-language models,

    P. Xu, W. Shao, K. Zhang, P. Gao, S. Liu, M. Lei, F. Meng, S. Huang, Y . Qiao, and P. Luo, “Lvlm-ehub: A comprehensive evaluation bench- mark for large vision-language models,”IEEE Trans. Pattern Anal. Mach. Intell., vol. 47, no. 3, pp. 1877–1893, 2024

    2024

  7. [8]

    Visogender: A dataset for benchmarking gender bias in image-text pronoun resolution,

    S. M. Hall, F. Gonc ¸alves Abrantes, H. Zhu, G. Sodunke, A. Shtedritski, and H. R. Kirk, “Visogender: A dataset for benchmarking gender bias in image-text pronoun resolution,”Adv. Neural Inform. Process. Syst., vol. 36, pp. 63 687–63 723, 2023

    2023

  8. [9]

    Counterfactually measuring and eliminating social bias in vision-language pre-training models,

    Y . Zhang, J. Wang, and J. Sang, “Counterfactually measuring and eliminating social bias in vision-language pre-training models,” inACM Int. Conf. Multimedia, 2022, pp. 4996–5004

    2022

  9. [10]

    GPQA: A graduate-level google-proof q&a benchmark,

    D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y . Pang, J. Dirani, J. Michael, and S. R. Bowman, “GPQA: A graduate-level google-proof q&a benchmark,”arXiv preprint arXiv:2311.12022, 2023

    Pith/arXiv arXiv 2023

  10. [11]

    Gemini 3 pro model card,

    Google DeepMind, “Gemini 3 pro model card,” https: //storage.googleapis.com/deepmind-media/Model-Cards/ Gemini-3-Pro-Model-Card.pdf, 2025

    2025

  11. [12]

    Direct preference optimization: Your language model is secretly a reward model,

    R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn, “Direct preference optimization: Your language model is secretly a reward model,”Adv. Neural Inform. Process. Syst., vol. 36, pp. 53 728–53 741, 2023

    2023

  12. [13]

    Man is to computer programmer as woman is to homemaker? debiasing word embeddings,

    T. Bolukbasi, K.-W. Chang, J. Y . Zou, V . Saligrama, and A. T. Kalai, “Man is to computer programmer as woman is to homemaker? debiasing word embeddings,”Adv. Neural Inform. Process. Syst., vol. 29, 2016

    2016

  13. [14]

    Semantics derived au- tomatically from language corpora contain human-like biases,

    A. Caliskan, J. J. Bryson, and A. Narayanan, “Semantics derived au- tomatically from language corpora contain human-like biases,”Science, vol. 356, no. 6334, pp. 183–186, 2017

    2017

  14. [15]

    Crows-pairs: A challenge dataset for measuring social biases in masked language models,

    N. Nangia, C. Vania, R. Bhalerao, and S. Bowman, “Crows-pairs: A challenge dataset for measuring social biases in masked language models,” inConf. Empir. Methods Nat. Lang. Process., 2020, pp. 1953– 1967

    2020

  15. [16]

    Stereoset: Measuring stereo- typical bias in pretrained language models,

    M. Nadeem, A. Bethke, and S. Reddy, “Stereoset: Measuring stereo- typical bias in pretrained language models,” inAnnu. Meeting Assoc. Comput. Linguist., 2021, pp. 5356–5371

    2021

  16. [17]

    Bbq: A hand-built bias benchmark for question answering,

    A. Parrish, A. Chen, N. Nangia, V . Padmakumar, J. Phang, J. Thompson, P. M. Htut, and S. Bowman, “Bbq: A hand-built bias benchmark for question answering,” inAnnu. Meeting Assoc. Comput. Linguist., 2022, pp. 2086–2105

    2022

  17. [18]

    Genderbias-vl: Benchmarking gender bias in vision language JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 11 models via counterfactual probing,

    Y . Xiao, A. Liu, Q. Cheng, Z. Yin, S. Liang, J. Li, J. Shao, X. Liu, and D. Tao, “Genderbias-vl: Benchmarking gender bias in vision language JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 11 models via counterfactual probing,”Int. J. Comput. Vis., vol. 133, no. 12, pp. 8332–8355, 2025

    2021

  18. [19]

    Vignette: Socially grounded bias evaluation for vision-language models,

    C. Raj, B. Wei, A. Caliskan, A. Anastasopoulos, and Z. Zhu, “Vignette: Socially grounded bias evaluation for vision-language models,” inAnnu. Meeting Assoc. Comput. Linguist., 2026, pp. 15 645–15 673

    2026

  19. [20]

    Vlbiasbench: A comprehensive benchmark for evaluating bias in large vision-language model,

    S. Wang, X. Cao, J. Zhang, Z. Yuan, S. Shan, X. Chen, and W. Gao, “Vlbiasbench: A comprehensive benchmark for evaluating bias in large vision-language model,”IEEE Trans. Pattern Anal. Mach. Intell., 2026

    2026

  20. [21]

    SB- Bench: Stereotype bias benchmark for large multimodal models,

    V . Narnaware, A. Vayani, R. Gupta, S. Sirnam, and M. Shah, “SB- Bench: Stereotype bias benchmark for large multimodal models,”arXiv preprint arXiv:2502.08779, 2025

    arXiv 2025

  21. [22]

    Nlp evaluation in trouble: On the need to measure llm data contamination for each benchmark,

    O. Sainz, J. A. Campos, I. Garc ´ıa-Ferrero, J. Etxaniz, O. Lopez de Lacalle, and E. Agirre, “Nlp evaluation in trouble: On the need to measure llm data contamination for each benchmark,” inConf. Empir. Methods Nat. Lang. Process., 2023, pp. 10 776–10 787

    2023

  22. [23]

    Dynabench: Rethinking bench- marking in nlp,

    D. Kiela, M. Bartolo, Y . Nie, D. Kaushik, A. Geiger, Z. Wu, B. Vidgen, G. Prasad, A. Singh, P. Ringshiaet al., “Dynabench: Rethinking bench- marking in nlp,” inAnnu. Meeting Assoc. Comput. Linguist., 2021, pp. 4110–4124

    2021

  23. [24]

    Robust visual question answering: Datasets, methods, and future challenges,

    J. Ma, P. Wang, D. Kong, Z. Wang, J. Liu, H. Pei, and J. Zhao, “Robust visual question answering: Datasets, methods, and future challenges,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 46, no. 8, pp. 5575–5594, 2024

    2024

  24. [25]

    Red teaming language models with language models,

    E. Perez, S. Huang, F. Song, T. Cai, R. Ring, J. Aslanides, A. Glaese, N. McAleese, and G. Irving, “Red teaming language models with language models,” inConf. Empir. Methods Nat. Lang. Process., 2022, pp. 3419–3448

    2022

  25. [26]

    Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned,

    D. Ganguli, L. Lovitt, J. Kernion, A. Askell, Y . Bai, S. Kadavath, B. Mann, E. Perez, N. Schiefer, K. Ndousseet al., “Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned,”arXiv preprint arXiv:2209.07858, 2022

    Pith/arXiv arXiv 2022

  26. [27]

    Universal and transferable adversarial attacks on aligned language models,

    A. Zou, Z. Wang, N. Carlini, M. Nasr, J. Z. Kolter, and M. Fredrikson, “Universal and transferable adversarial attacks on aligned language models,”arXiv preprint arXiv:2307.15043, 2023

    Pith/arXiv arXiv 2023

  27. [28]

    Automatically auditing large language models via discrete optimization,

    E. Jones, A. Dragan, A. Raghunathan, and J. Steinhardt, “Automatically auditing large language models via discrete optimization,” inInt. Conf. Mach. Learn., 2023, pp. 15 307–15 329

    2023

  28. [29]

    Figstep: Jailbreaking large vision-language models via typographic visual prompts,

    Y . Gong, D. Ran, J. Liu, C. Wang, T. Cong, A. Wang, S. Duan, and X. Wang, “Figstep: Jailbreaking large vision-language models via typographic visual prompts,” inAAAI Conf. Artif. Intell., vol. 39, no. 22, 2025, pp. 23 951–23 959

    2025

  29. [30]

    Jailbreaking attack against multimodal large language model,

    Z. Niu, H. Ren, X. Gao, G. Hua, and R. Jin, “Jailbreaking attack against multimodal large language model,”arXiv preprint arXiv:2402.02309, 2024

    Pith/arXiv arXiv 2024

  30. [31]

    Jailbreak in pieces: Compositional adversarial attacks on multi-modal language models,

    E. Shayegani, Y . Dong, and N. Abu-Ghazaleh, “Jailbreak in pieces: Compositional adversarial attacks on multi-modal language models,” in Int. Conf. Learn. Represent., vol. 2024, 2024, pp. 30 853–30 885

    2024

  31. [32]

    Red-teaming the multimodal reasoning: Jailbreaking vision-language models via cross- modal entanglement attacks,

    Y . Yan, S. Sun, S. Cheng, T. Liu, M. Li, and M. Liu, “Red-teaming the multimodal reasoning: Jailbreaking vision-language models via cross- modal entanglement attacks,” inIEEE Conf. Comput. Vis. Pattern Recog., 2026, pp. 837–846

    2026

  32. [33]

    Semantic-aligned adversarial evolution triangle for high- transferability vision-language attack,

    X. Jia, S. Gao, Q. Guo, S. Qin, K. Ma, Y . Huang, Y . Liu, I. Tsang, and X. Cao, “Semantic-aligned adversarial evolution triangle for high- transferability vision-language attack,”IEEE Trans. Pattern Anal. Mach. Intell., 2025

    2025

  33. [34]

    Jailbreaking black box large language models in twenty queries,

    P. Chao, A. Robey, E. Dobriban, H. Hassani, G. J. Pappas, and E. Wong, “Jailbreaking black box large language models in twenty queries,” in 2025 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML). IEEE, 2025, pp. 23–42

    2025

  34. [35]

    Tree of attacks: Jailbreaking black-box llms automatically,

    A. Mehrotra, M. Zampetakis, P. Kassianik, B. Nelson, H. Anderson, Y . Singer, and A. Karbasi, “Tree of attacks: Jailbreaking black-box llms automatically,”Adv. Neural Inform. Process. Syst., vol. 37, pp. 61 065– 61 105, 2024

    2024

  35. [36]

    Autodan: Generating stealthy jailbreak prompts on aligned large language models,

    X. Liu, N. Xu, M. Chen, and C. Xiao, “Autodan: Generating stealthy jailbreak prompts on aligned large language models,” inInt. Conf. Learn. Represent., vol. 2024, 2024, pp. 56 174–56 194

    2024

  36. [37]

    Treeteaming: Autonomous red-teaming of vision-language models via hierarchical strategy exploration,

    C. Li, L. Li, and J. Shao, “Treeteaming: Autonomous red-teaming of vision-language models via hierarchical strategy exploration,” inIEEE Conf. Comput. Vis. Pattern Recog., 2026, pp. 37 684–37 693

    2026

  37. [38]

    ARMs: Adaptive red-teaming agent against multimodal models with plug-and-play attacks,

    Z. Chen, X. Liu, M. Kang, J. Zhang, M. Pan, S. Yang, and B. Li, “ARMs: Adaptive red-teaming agent against multimodal models with plug-and-play attacks,”arXiv preprint arXiv:2510.02677, 2025

    arXiv 2025

  38. [39]

    ProbeLLM: Automating principled diagnosis of LLM failures,

    Y . Huang, Z. Jiang, Y . Ma, Y . Jiang, X. Wang, Y . Zhou, Y . Hao, K. Guo, P.-Y . Chen, S. Feuerriegel, and X. Zhang, “ProbeLLM: Automating principled diagnosis of LLM failures,”arXiv preprint arXiv:2602.12966, 2026

    Pith/arXiv arXiv 2026

  39. [40]

    RedHit: Adaptive red-teaming of large language models via search, reasoning, and pref- erence optimization,

    M. Sorkhpour, A. Yazdinejad, and A. Dehghantanha, “RedHit: Adaptive red-teaming of large language models via search, reasoning, and pref- erence optimization,” inProceedings of The First Workshop on LLM Security (LLMSEC), 2025, pp. 7–16

    2025

  40. [41]

    Training language models to follow instructions with human feedback,

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Rayet al., “Training language models to follow instructions with human feedback,” inAdv. Neural Inform. Process. Syst., 2022

    2022

  41. [42]

    Self-instruct: Aligning language models with self- generated instructions,

    Y . Wang, Y . Kordi, S. Mishra, A. Liu, N. A. Smith, D. Khashabi, and H. Hajishirzi, “Self-instruct: Aligning language models with self- generated instructions,” inAnnu. Meeting Assoc. Comput. Linguist., 2023, pp. 13 484–13 508

    2023

  42. [43]

    WizardLM: Empowering large pre-trained language models to follow complex instructions,

    C. Xu, Q. Sun, K. Zheng, X. Geng, P. Zhao, J. Feng, C. Tao, Q. Lin, and D. Jiang, “WizardLM: Empowering large pre-trained language models to follow complex instructions,” inInt. Conf. Learn. Represent., vol. 2024, 2024, pp. 30 745–30 766

    2024

  43. [44]

    Self- rewarding language models,

    W. Yuan, R. Y . Pang, K. Cho, S. Sukhbaatar, J. Xu, and J. Weston, “Self- rewarding language models,”arXiv preprint arXiv:2401.10020, 2024

    Pith/arXiv arXiv 2024

  44. [45]

    Sdxl: Improving latent diffusion models for high-resolution image synthesis,

    D. Podell, Z. English, K. Lacey, A. Blattmann, T. Dockhorn, J. M ¨uller, J. Penna, and R. Rombach, “Sdxl: Improving latent diffusion models for high-resolution image synthesis,” inInt. Conf. Learn. Represent., vol. 2024, 2024, pp. 1862–1874

    2024

  45. [46]

    LoRA: Low-rank adaptation of large language models,

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “LoRA: Low-rank adaptation of large language models,” inInt. Conf. Learn. Represent., 2022

    2022

  46. [47]

    Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models,

    J. Zhu, W. Wang, Z. Chen, Z. Liu, S. Ye, L. Gu, H. Tian, Y . Duan, W. Su, J. Shaoet al., “Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models,”arXiv preprint arXiv:2504.10479, 2025

    Pith/arXiv arXiv 2025

  47. [48]

    Qwen2.5-vl technical report,

    S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, H. Zhong, Y . Zhu, M. Yang, Z. Li, J. Wan, P. Wang, W. Ding, Z. Fu, Y . Xu, J. Ye, X. Zhang, T. Xie, Z. Cheng, H. Zhang, Z. Yang, H. Xu, and J. Lin, “Qwen2.5-vl technical report,”arXiv preprint arXiv:2502.13923, 2025

    Pith/arXiv arXiv 2025

  48. [49]

    Gemma 3 technical report,

    Gemma Team, “Gemma 3 technical report,”arXiv preprint arXiv:2503.19786, 2025

    Pith/arXiv arXiv 2025

  49. [50]

    Qwen3 technical report,

    A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, C. Zheng, D. Liu, F. Zhou, F. Huang, F. Hu, H. Ge, H. Wei, H. Lin, J. Tang, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Zhou, J. Lin, K. Dang, K. Bao, K. Yang, L. Yu, L. Deng, M. Li, M. Xue, M. Li, P. Zhang, P. Wang, Q. Zhu, R. Men, R. Gao, S. Liu, S. Luo, T. ...

    Pith/arXiv arXiv 2025

  50. [51]

    Internvl3.5: Advancing open-source multi- modal models in versatility, reasoning, and efficiency,

    W. Wang, Z. Gao, L. Gu, H. Pu, L. Cui, X. Wei, Z. Liu, L. Jing, S. Ye, J. Shaoet al., “Internvl3.5: Advancing open-source multi- modal models in versatility, reasoning, and efficiency,”arXiv preprint arXiv:2508.18265, 2025

    Pith/arXiv arXiv 2025

  51. [52]

    Qwen3-vl technical report,

    S. Bai, Y . Cai, R. Chen, K. Chen, X. Chen, Z. Cheng, L. Deng, W. Ding, C. Gao, C. Ge, W. Ge, Z. Guo, Q. Huang, J. Huang, F. Huang, B. Hui, S. Jiang, Z. Li, M. Li, M. Li, K. Li, Z. Lin, J. Lin, X. Liu, J. Liu, C. Liu, Y . Liu, D. Liu, S. Liu, D. Lu, R. Luo, C. Lv, R. Men, L. Meng, X. Ren, X. Ren, S. Song, Y . Sun, J. Tang, J. Tu, J. Wan, P. Wang, P. Wang,...

    Pith/arXiv arXiv 2025

  52. [53]

    Deepseek-vl2: Mixture-of-experts vision-language models for advanced multimodal understanding,

    Z. Wu, X. Chen, Z. Pan, X. Liu, W. Liu, D. Dai, H. Gao, Y . Ma, C. Wu, B. Wanget al., “Deepseek-vl2: Mixture-of-experts vision-language models for advanced multimodal understanding,”arXiv preprint arXiv:2412.10302, 2024

    Pith/arXiv arXiv 2024

  53. [54]

    LLaV A-OneVision: Easy visual task transfer,

    B. Li, Y . Zhang, D. Guo, R. Zhang, F. Li, H. Zhang, K. Zhang, Y . Li, Z. Liu, and C. Li, “LLaV A-OneVision: Easy visual task transfer,”arXiv preprint arXiv:2408.03326, 2024

    Pith/arXiv arXiv 2024

  54. [55]

    Are we done with mmlu?

    A. P. Gema, J. O. J. Leang, G. Hong, A. Devoto, A. C. M. Mancino, R. Saxena, X. He, Y . Zhao, X. Du, M. R. G. Madaniet al., “Are we done with mmlu?” inAnnu. Meeting Assoc. Comput. Linguist., 2025, pp. 5069–5096

    2025

  55. [56]

    Introducing GPT-5.5,

    OpenAI, “Introducing GPT-5.5,” https://openai.com/index/ introducing-gpt-5-5/, 2026

    2026

  56. [57]

    The Claude family of models,

    Anthropic, “The Claude family of models,” https://www.anthropic.com/ claude, 2025, anthropic model card

    2025

  57. [58]

    Gemini 3 flash model card,

    Google DeepMind, “Gemini 3 flash model card,” https://storage.googleapis.com/deepmind-media/Model-Cards/ Gemini-3-Flash-Model-Card.pdf, 2025. JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 12

    2025

  58. [59]

    Gemini 2.5 Flash Model Card,

    ——, “Gemini 2.5 Flash Model Card,” https://storage.googleapis.com/ deepmind-media/Model-Cards/Gemini-2-5-Flash-Model-Card.pdf, 2025

    2025

  59. [60]

    Glm-4.1 v-thinking: Towards versatile multimodal reasoning with scalable reinforcement learning,

    W. Hong, W. Yu, X. Gu, G. Wang, G. Gan, H. Tang, J. Cheng, J. Qi, J. Ji, L. Panet al., “Glm-4.1 v-thinking: Towards versatile multimodal reasoning with scalable reinforcement learning,”arXiv e-prints, pp. arXiv–2507, 2025

    2025

  60. [61]

    Minicpm-v: A gpt-4v level mllm on your phone,

    Y . Yao, T. Yu, A. Zhang, C. Wang, J. Cui, H. Zhu, T. Cai, H. Li, W. Zhao, Z. Heet al., “Minicpm-v: A gpt-4v level mllm on your phone,”arXiv preprint arXiv:2408.01800, 2024

    Pith/arXiv arXiv 2024

  61. [62]

    Pixtral 12b,

    P. Agrawal, S. Antoniak, E. B. Hanna, B. Bout, D. Chaplot, J. Chud- novsky, D. Costa, B. De Monicault, S. Garg, T. Gervetet al., “Pixtral 12b,”arXiv preprint arXiv:2410.07073, 2024

    Pith/arXiv arXiv 2024

  62. [63]

    Visual instruction tuning,

    H. Liu, C. Li, Q. Wu, and Y . J. Lee, “Visual instruction tuning,”Adv. Neural Inform. Process. Syst., vol. 36, pp. 34 892–34 916, 2023

    2023

  63. [64]

    Visionllama: A unified llama backbone for vision tasks,

    X. Chu, J. Su, B. Zhang, and C. Shen, “Visionllama: A unified llama backbone for vision tasks,” inEur. Conf. Comput. Vis.Springer, 2024, pp. 1–18. Anqi Lireceived the B.S. and M.S. degrees from Beijing Institute of Technology, in 2022 and 2025, respectively. She is currently working toward the Ph.D. degree with the Institute of Computing Tech- nology (ICT...

    2024