Pith. sign in

REVIEW 3 major objections 4 minor 33 references

Multimodal jailbreaks leave a trace in how text-only, image-only, and fused predictions disagree, and this paper claims that trace alone can cut attack success from over 90% to under 15%.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 14:56 UTC pith:I275UCZJ

load-bearing objection A genuinely useful inference-time defense with a solid empirical core, but the first-token detection premise is under-tested against adaptive attacks, and the headline numbers overstate the text-attack case. the 3 major comments →

arxiv 2607.21600 v1 pith:I275UCZJ submitted 2026-05-03 cs.AI

Securing Multimodal AI through Internal Information Decomposition

Jehyeok Yeon , Hyeonjeong Ha , Qiusi Zhan , Heng Ji This is my paper
classification cs.AI
keywords multimodal large language modelsjailbreak detectioncross-modal consistencyPartial Information Decompositioninference-time defenseone-class classificationattack-agnostic detectionFlowVectors
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

FlowGuard tries to establish that harmful multimodal inputs can be detected without knowing any attack patterns, by watching a model's internal fusion process rather than its inputs or outputs. The central observation is that on benign image-text pairs, the model's first-token predictions from text alone and image alone are compatible, and fusing them reduces uncertainty; adversarial jailbreaks shatter that compatibility, even when each modality looks innocent. FlowGuard encodes this as a four-dimensional FlowVector—redundancy, two directional divergence terms, and entropy-reduction synergy—and trains a one-class anomaly detector on benign data only. The paper's headline evidence is that unseen attacks whose success rate exceeds 90% on the undefended model fall below 15% under FlowGuard, with under 3% utility loss and up to 6 times lower latency than diffusion-based verification. A cautious reader would care because this suggests multimodal safety can be monitored cheaply and attack-agnostically, without modifying the model or collecting adversarial examples.

Core claim

The paper's claim is that cross-modal consistency at the first decoding step is a sufficient safety signal for multimodal LLMs. The author's key assertion: benign inputs induce text-only, vision-only, and joint multimodal predictive distributions that align with each other and stabilize under fusion, whereas adversarial manipulation—textual, visual, or compositional—produces a fused posterior that diverges asymmetrically from one or both unimodal posteriors. FlowGuard measures this with PID-inspired FlowVectors and frames detection as one-class classification trained solely on benign data, reporting that Attack Success Rates drop from over 90% to under 15% on unseen attacks with under 3% uti

What carries the argument

The FlowVector φ(x) = (U_v, U_t, R, S) ∈ R^4. With P_t^(1), P_v^(1), P_mm^(1) the first-token predictive distributions under text-only, vision-only (neutral prompt), and joint prompting: U_v = D_KL(P_mm || P_t), U_t = D_KL(P_mm || P_v), R = 1 − JSD(P_t || P_v), S = (H(P_t)+H(P_v))/2 − H(P_mm). These PID-inspired proxies quantify vision dominance, text dominance, semantic agreement, and fusion stability; an Isolation Forest, fit on benign FlowVectors only, flags inputs whose fused prediction is an outlier relative to benign geometry. This 4D projection is what carries the argument: it compresses the full next-token distribution into the relational structure between modalities that the paper c

Load-bearing premise

The load-bearing premise is that harmful intent always disturbs the relationship between the model's first-token predictions from text alone, image alone, and the fused pair—so an input that keeps that first-token relationship looking benign while producing harmful content in later tokens would slip through the detector.

What would settle it

A jailbreak that consistently produces a harmful response while its first-token FlowVector lies inside the benign region (Isolation Forest anomaly score below the 0.5 threshold, with positive synergy and moderate redundancy) would falsify the central claim. The paper reports 2.1% of adversarial samples are caught only at k=3, so the decisive experiment is to scale up exactly that evasion—delaying harmful semantics beyond the first decoding step—and measure whether FlowGuard's AUROC collapses.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Deployment without attack supervision: a detection model trained once on ordinary image-question pairs can be dropped in front of an MLLM and suppress attacks it has never seen, because it models benign fusion rather than known attack signatures.
  • API compatibility: because only next-token logits or top-k logprobs are needed, the defense extends to hosted models where weights and activations are unavailable, with a modest accuracy penalty from truncated distributions.
  • Architecture transfer: the same benign-trained detector maintains high AUROC across models from 4B to 70B parameters, suggesting consistency-of-fusion is a model-scale-invariant property rather than a quirk of one backbone.
  • Complementarity with existing defenses: FlowGuard is a pre-generation filter that can be stacked with input purification or output verification; its residual weakness is text-only attacks (roughly 13-14% ASR), so pairing with a text-specific guard would close the largest remaining gap.
  • Latency headroom for adaptive probing: since k=1 already saturates the signal, the remaining compute budget can be spent conditionally—probing k=3 only near the decision boundary catches the 2.1% of attacks hidden beyond the first token at almost no average cost.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial: if the fusion-consistency signal is as general as the paper suggests, the same 4D FlowVector could serve as a general fusion-health monitor beyond safety—flagging hallucinations, image corruption, or distribution shift, since those also change the alignment between unimodal and fused posteriors.
  • Editorial: the vision-only prior is elicited with a fixed neutral prompt, so it is not the model's visual belief under the actual query; this leaves open whether attacks that anchor the image semantics to the specific query could compress the signal, despite the paper's robustness check on prompt phrasing.
  • Editorial: the first-token dependence sets up a concrete arms race—an adversary could try to keep the first-token FlowVector benign while deferring harmful semantics to later tokens; the paper's own k=3 margin (2.1% of adversarial samples) suggests this headroom is real and worth probing as a dedicated attack.
  • Editorial: the 6 times latency gain over generative verification implies consistency-based detection could be run per-turn inside interactive agents, where diffusion-based checks are too slow; the multi-turn and multi-image extensions the paper lists are the natural tests of that claim.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes FlowGuard, an inference-time detector for multimodal LLM jailbreaks. FlowGuard runs three first-token forward passes (text-only, vision-only, joint), computes four PID-inspired statistics (two KL divergences, one JSD-based redundancy, one entropy-reduction synergy), and feeds the resulting 4D FlowVector into a one-class Isolation Forest trained only on benign VQAv2 data. On three safety benchmarks and four model families, the paper reports ASR reductions from >90% to <15% across ten attack types, a 2.4% benign FPR, <3% utility drop, and roughly a 6x latency reduction relative to diffusion-based verification. The central design choice is that detection relies exclusively on the first decoding step's predictive distributions.

Significance. If the first-token consistency premise holds, FlowGuard is a lightweight, attack-agnostic, architecture-agnostic defense requiring no attack supervision, no gradient access, and no model modification. The paper's strengths are its extensive empirical coverage (three benchmarks, ten attack types, four model families), threshold-free AUROC reporting, and thorough ablations (feature importance, decoding steps, sample efficiency, neutral-prompt sensitivity, partial-logprob access). The one-class formulation is clean and reproducible. The main risk is that the detection signal is restricted to the first decoding step, and the adaptive evaluations do not include the most direct counter-strategy: forcing a benign first token and deferring harmful content to later tokens.

major comments (3)
  1. [§3.2, §5.5, §B.2] Detection uses only P^(1). §5.5 itself reports that 2.1% of adversarial samples that evade k=1 are only caught at k=3, so the first-token premise is not universal. The adaptive attacks in §B.1–B.2 target the first-token completion 'Sure, here is', which is atypical for benign VQAv2 answers; an adversary could instead make P_t^(1), P_v^(1), and P_mm^(1) all concentrate on a benign first token (e.g., 'The' or 'Yes') and defer harmful content to tokens 2..T. Then R≈1, U≈0, S≈0, which lies inside the benign cluster. The §B.1 claim that stealth and success are mutually exclusive is shown only for the tested target geometry, not for all harmful continuations. This is load-bearing for the 'unseen attacks' generalization; the paper should evaluate such deferred-harm attacks or explicitly restrict the claim.
  2. [§4.1, §5.2, Table 2] The one-class detector is trained on 10,000 pairs from the VQAv2 validation split, and the same benchmark is used for the VQAv2 utility column and contributes to the 2.4% FPR. This is an in-distribution evaluation of a detector trained on that distribution, so the benign FPR and utility drop are optimistic. Report per-benchmark FPR for the out-of-distribution sets (VizWiz, MOSSBench) and ideally train on the VQAv2 train split or use a strictly disjoint subset.
  3. [§B.1, Eq. (6)] The adaptive attacker minimizes ||φ(x')-μ_benign||^2, a Euclidean proxy, rather than the actual Isolation Forest anomaly score s(x). The conclusion that no λ achieves both high ASR and benign s is therefore limited to this proxy; direct optimization of s (e.g., through a differentiable surrogate of path lengths) might find different operating points. Since §5.6 uses this experiment to argue a 'structural tension', the claim should be verified against the actual decision rule.
minor comments (4)
  1. [References] VLSafe is cited as (Chen et al., 2024), but the Chen et al. entry in the references is the DRESS paper, not a VLSafe benchmark paper. Please add the correct reference for VLSafe.
  2. [§5.2] '2.4% FPR under the default threshold' — clarify whether this is pooled across VQAv2/VizWiz/MOSSBench or only VQAv2; report FPR per benchmark in Table 2.
  3. [Table 8] The text says Isolation Forest maintains AUC ≥0.88 across all settings, but VLSafe shows 0.878; reconcile the statement with the table.
  4. [§5.3, Figure 2] Define whether the reported 1.3s latency includes the three first-token forward passes or only the feature computation on top of them; 'excluding base model generation' is ambiguous and affects the 6x comparison.

Circularity Check

0 steps flagged

No significant circularity: FlowGuard's features are parameter-free functions of model distributions, the one-class detector is trained only on benign data, and the headline ASR results are measured on disjoint attack benchmarks.

full rationale

The derivation chain is self-contained. FlowVectors are defined in §3.4 as closed-form functions (Eqs. 3–5) of three first-token predictive distributions P_t^(1), P_v^(1), P_mm^(1), with no fitted constants and no attack labels. The detector is an Isolation Forest trained only on benign VQAv2 samples (§3.5, §4.1), and safety/utility are evaluated on separate benchmarks (§4.3) with ASR measured by actually suppressing generation and checking outputs against benchmark annotations (§4.6). The paper explicitly states it does not compute PID exactly (§3.3), so no imported uniqueness result is load-bearing. Self-citations (e.g., Fung et al. 2021 for cross-media consistency, Ji et al. 2025 as a survey) are contextual, not used to justify the central reduction. The first-token premise is an assumption, not a definitional equivalence: the paper acknowledges its limits by reporting residual text-only ASR of 13–14% and that only 2.1% of adversarial samples caught at k=3 evaded k=1 (§5.5, §6). Ablations (§5.4) analyze feature importance post hoc rather than fitting the detector on attack outcomes. No quoted equation or parameter choice reduces the claimed prediction to its inputs by construction, so no circular step is established.

Axiom & Free-Parameter Ledger

2 free parameters · 5 axioms · 1 invented entities

The central claim rests on two domain assumptions: (1) first-token predictive distributions carry the safety-relevant fusion state; (2) benign fusion is consistent and adversarial fusion is not. The feature set is an ad hoc proxy for PID, selected with the help of ablations on the evaluation benchmarks. No fitted constants enter the features; the Isolation Forest threshold is the only tunable operating point.

free parameters (2)
  • Isolation Forest threshold (contamination) = 'auto' (s=0.5)
    Determines the operating point of the ASR/FPR tradeoff. Not fitted to attack data, but a tunable decision boundary chosen by the authors.
  • Number of benign training samples N = 10000
    Training set size for the one-class detector; chosen because it saturates performance at N≈2000 (Appendix B.8).
axioms (5)
  • domain assumption The first-token predictive distribution P^(1) captures the model's safety-relevant decision state (refusal/compliance).
    Stated in §3.2; relies on attacks altering the onset of generation. The paper's own k-step ablation (§5.5) shows the signal is mostly saturated at k=1 but not fully.
  • domain assumption Benign inputs induce compatible unimodal priors and stabilizing fusion; adversarial inputs disrupt this consistency.
    Core hypothesis of the paper, stated in §3.1 and used to motivate every feature. Validated only on the specific attack benchmarks tested; residual text-attack ASR of 13-14% shows boundary cases.
  • domain assumption Text-only conditioning (empty image) and vision-only conditioning with fixed neutral prompt Q_∅='Describe this image' are valid proxies for unimodal reasoning.
    Introduced in §3.2. Robustness to prompt phrasing tested on 1,000 VQAv2 images (Appendix B.7), but the empty-image text prior is not equivalent to true unimodal reasoning.
  • ad hoc to paper The four scalar statistics (two KL divergences, 1-JSD, entropy reduction) instantiate PID notions of uniqueness, redundancy, and synergy.
    Defined in §3.4. The paper explicitly says it does not compute PID but uses it as conceptual inspiration; the feature set was selected and justified via ablations on the evaluation benchmarks (§5.4).
  • domain assumption Isolation Forest with contamination='auto' yields a well-calibrated decision boundary for benign FlowVectors.
    Used in §A.1.1. The auto threshold fixes the operating point and is not derived from first principles; the paper explores the tradeoff in Table 6.
invented entities (1)
  • FlowVector independent evidence
    purpose: 4D summary of cross-modal consistency used as the detection signal for FlowGuard.
    Computable from the model's own next-token distributions, so any practitioner can reproduce it; it has an external measurable handle, not a latent unobservable.

pith-pipeline@v1.3.0-alltime-deepseek · 27698 in / 14159 out tokens · 128305 ms · 2026-08-02T14:56:54.089964+00:00 · methodology

0 comments
read the original abstract

Multimodal large language models introduce attack surfaces absent in unimodal systems: adversaries can distribute malicious intent across modalities to evade unimodal safeguards. This motivates using cross-modal consistency as a detection signal rather than inspecting each modality in isolation. Our key observation is that benign inputs induce compatible predictive behavior from text-only and vision-only reasoning that stabilizes when fused, whereas adversarial manipulation disrupts this consistency, causing abnormal multimodal behavior. Existing defenses that examine raw inputs or outputs overlook this internal fusion process, rendering them brittle and computationally expensive. We propose FlowGuard, a lightweight inference-time framework that detects harmful inputs by monitoring internal multimodal consistency. Unlike approaches that rely on scalar confidence metrics, FlowGuard derives FlowVectors inspired by Partial Information Decomposition that quantify cross-modal redundancy, synergy, and modality-specific dominance, capturing whether fused multimodal predictions remain aligned with unimodal semantic evidence. In a one-class classification problem trained solely on benign data, FlowGuard reduces Attack Success Rates from >90% to <15% on unseen attacks, with <3% utility loss and up to a 6 times latency reduction. Our results demonstrate that monitoring cross-modal consistency offers an efficient and effective defense for multimodal reasoning.

Figures

Figures reproduced from arXiv: 2607.21600 by Heng Ji, Hyeonjeong Ha, Jehyeok Yeon, Qiusi Zhan.

Figure 1
Figure 1. Figure 1: Overview of FlowGuard. Given an image–text pair, FlowGuard first probes the MLLM under three configurations: vision-only, text-only, and joint multimodal, and obtains first-token predictive distributions. FlowGuard then derives PID-inspired features that capture uniqueness, redundancy, and synergy, forming a 4D FlowVector. A one-class Isolation Forest detects anomalous fusion patterns, labeling benign inpu… view at source ↗
Figure 3
Figure 3. Figure 3: Feature Necessity (Ablation). Negative values indicate the feature is essential. The exclusion of Synergy (S) results in the largest performance drops on attack datasets (Right), while Visual Uniqueness (Uv) is critical specifically for OOD VizWiz data. Standalone Predictive Power [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 2
Figure 2. Figure 2: Latency vs. Detection Performance. FlowGuard achieves a favorable trade-off between detection F1-score and inference latency compared to existing defenses. Role of Synergy (S) [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: t-SNE visualization of benign and adversarial sam￾ples using raw first-step logits versus FlowVectors. Raw logits exhibit substantial overlap between inputs, while FlowVectors pro￾duce clearer separation, supporting the need for relational cross￾modal features rather than direct logit-space anomaly detection. sion boundary, retains the efficiency of single-step detection while offering a built-in defense a… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

33 extracted references · 23 linked inside Pith

  1. [2]

    Burns, C., Ye, H., Klein, D., and Steinhardt, J

    URLhttps://arxiv.org/abs/2502.13923. Burns, C., Ye, H., Klein, D., and Steinhardt, J. Discov- ering latent knowledge in language models without su- pervision,

    Pith/arXiv arXiv

  2. [4]

    Chen, Y ., Sikka, K., Cogswell, M., Ji, H., and Divakaran, A

    URL https: //arxiv.org/abs/2310.08419. Chen, Y ., Sikka, K., Cogswell, M., Ji, H., and Divakaran, A. Dress: Instructing large vision-language models to align and interact with humans via natural language feedback,

    Pith/arXiv arXiv

  3. [5]

    Croce, F

    URL https://arxiv.org/abs/ 2311.10081. Croce, F. and Hein, M. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks,

    Pith/arXiv arXiv

  4. [7]

    Fares, S., Ziu, K., Aremu, T., Durasov, N., Tak´aˇc, M., Fua, P., Nandakumar, K., and Laptev, I

    URL https://arxiv.org/abs/2407.21783. Fares, S., Ziu, K., Aremu, T., Durasov, N., Tak´aˇc, M., Fua, P., Nandakumar, K., and Laptev, I. Mirrorcheck: Efficient adversarial defense for vision-language models,

    Pith/arXiv arXiv

  5. [8]

    Fung, Y ., Thomas, C., Gangi Reddy, R., Polisetty, S., Ji, H., Chang, S.-F., McKeown, K., Bansal, M., and Sil, A

    URLhttps://arxiv.org/abs/2406.09250. Fung, Y ., Thomas, C., Gangi Reddy, R., Polisetty, S., Ji, H., Chang, S.-F., McKeown, K., Bansal, M., and Sil, A. InfoSurgeon: Cross-media fine-grained informa- tion consistency checking for fake news detection. In Zong, C., Xia, F., Li, W., and Navigli, R. (eds.),Pro- ceedings of the 59th Annual Meeting of the Associa...

    Pith/arXiv arXiv

  6. [10]

    Gou, Y ., Chen, K., Liu, Z., Hong, L., Xu, H., Li, Z., Yeung, D.-Y ., Kwok, J

    URLhttps://arxiv.org/abs/2311.05608. Gou, Y ., Chen, K., Liu, Z., Hong, L., Xu, H., Li, Z., Yeung, D.-Y ., Kwok, J. T., and Zhang, Y . Eyes closed, safety on: Protecting multimodal llms via image-to-text transforma- tion,

    Pith/arXiv arXiv

  7. [13]

    Ji, J., Qiu, T., Chen, B., Zhang, B., Lou, H., Wang, K., Duan, Y ., He, Z., Vierling, L., Hong, D., Zhou, J., Zhang, Z., Zeng, F., Dai, J., Pan, X., Ng, K

    URLhttps://arxiv.org/abs/2309.00614. Ji, J., Qiu, T., Chen, B., Zhang, B., Lou, H., Wang, K., Duan, Y ., He, Z., Vierling, L., Hong, D., Zhou, J., Zhang, Z., Zeng, F., Dai, J., Pan, X., Ng, K. Y ., O’Gara, A., Xu, H., Tse, B., Fu, J., McAleer, S., Yang, Y ., Wang, Y ., Zhu, S.-C., Guo, Y ., and Gao, W. Ai alignment: A comprehensive survey,

    Pith/arXiv arXiv

  8. [14]

    org/abs/2310.19852

    URL https://arxiv. org/abs/2310.19852. Jiang, F., Xu, Z., Niu, L., Xiang, Z., Ramasubramanian, B., Li, B., and Poovendran, R. Artprompt: Ascii art- based jailbreak attacks against aligned llms,

    Pith/arXiv arXiv

  9. [15]

    Kamath, A., Ferret, J., Pathak, S., et al

    URL https://arxiv.org/abs/2402.11753. Kamath, A., Ferret, J., Pathak, S., et al. Gemma 3 techni- cal report,

    Pith/arXiv arXiv

  10. [16]

    Kullback, S

    URL https://arxiv.org/abs/ 2503.19786. Kullback, S. and Leibler, R. A. On Information and Suffi- ciency.The Annals of Mathematical Statistics, 22(1):79 – 86,

    Pith/arXiv arXiv

  11. [18]

    URLhttps: //arxiv.org/abs/2406.17806. Lin, J. Divergence measures based on the shannon entropy. IEEE Transactions on Information Theory, 37(1):145– 151,

    Pith/arXiv arXiv

  12. [21]

    Niu, Z., Ren, H., Gao, X., Hua, G., and Jin, R

    URL https: //arxiv.org/abs/2311.17600. Niu, Z., Ren, H., Gao, X., Hua, G., and Jin, R. Jailbreaking attack against multimodal large language model,

    Pith/arXiv arXiv

  13. [22]

    Oh, S., Jin, Y ., Sharma, M., Kim, D., Ma, E., Verma, G., and Kumar, S

    URLhttps://arxiv.org/abs/2402.02309. Oh, S., Jin, Y ., Sharma, M., Kim, D., Ma, E., Verma, G., and Kumar, S. Uniguard: Towards universal safety guardrails for jailbreak attacks on multimodal large language mod- els,

    Pith/arXiv arXiv

  14. [23]

    Qi, X., Huang, K., Panda, A., Henderson, P., Wang, M., and Mittal, P

    URLhttps://arxiv.org/abs/2510.18214. Qi, X., Huang, K., Panda, A., Henderson, P., Wang, M., and Mittal, P. Visual adversarial examples jailbreak aligned large language models,

    arXiv

  15. [24]

    org/abs/2306.13213

    URL https://arxiv. org/abs/2306.13213. Russinovich, M., Salem, A., and Eldan, R. Great, now write an article about that: The crescendo multi-turn llm jailbreak attack,

    Pith/arXiv arXiv

  16. [25]

    Shannon, C

    URL https://arxiv.org/ abs/2404.01833. Shannon, C. E. A mathematical theory of communication. The Bell System Technical Journal, 27(3):379–423,

    Pith/arXiv arXiv

  17. [27]

    Williams, P

    URL https://arxiv.org/ abs/2405.17894. Williams, P. L. and Beer, R. D. Nonnegative decomposi- tion of multivariate information,

    Pith/arXiv arXiv

  18. [29]

    Zou, A., Wang, Z., Carlini, N., Nasr, M., Kolter, J

    URL https: //arxiv.org/abs/2407.21659. Zou, A., Wang, Z., Carlini, N., Nasr, M., Kolter, J. Z., and Fredrikson, M. Universal and transferable adversarial attacks on aligned language models,

    Pith/arXiv arXiv

  19. [30]

    11 Securing Multimodal AI through Internal Information Decomposition A

    URL https: //arxiv.org/abs/2307.15043. 11 Securing Multimodal AI through Internal Information Decomposition A. Appendix A.1. Detailed Experimental Configuration A.1.1. FLOWGUARDIMPLEMENTATIONDETAILS FlowGuard is implemented as a post-hoc detector requiring access to next-token logits or top-k logprobs, but no gradient updates or internal activations. Anom...

    Pith/arXiv arXiv

  20. [33]

    Sure, here is

    optimizes a textual suffix, with the image held to its clean instance, to maximize the likelihood of the target completion “Sure, here is”; (ii) the PGD-based image attack of Croce & Hein (2020) optimizes a pixel perturbation, with the textual query held to its clean form, toward the same target. At evaluation time we pair the two optimized modalities for...

    arXiv 2020

  21. [64]

    DO NOT include harmful content in your response

    A.2.4. UNIGUARD(PRE-DEFINED) Method:Adversarial suffix appending (Text-based guardrail). • Implementation:We evaluate theUniGuard (Pre-defined)baseline variant (denoted as UniGuard (P)), which uses a pre-defined, human-written safety suffix rather than a learned adversarial suffix. This provides a direct comparison for inference-time, training-free defens...

    2023

  22. [1948]

    Wang, R., Ma, X., Zhou, H., Ji, C., Ye, G., and Jiang, Y .-G

    doi: 10.1002/j.1538-7305.1948.tb01338.x. Wang, R., Ma, X., Zhou, H., Ji, C., Ye, G., and Jiang, Y .-G. White-box multimodal jailbreaks against large vision- language models,

    arXiv 1948

  23. [1951]

    URL https: //doi.org/10.1214/aoms/1177729694

    doi: 10.1214/aoms/1177729694. URL https: //doi.org/10.1214/aoms/1177729694. Li, X., Zhou, H., Wang, R., Zhou, T., Cheng, M., and Hsieh, C.-J. Mossbench: Is your multimodal language model oversensitive to safe queries?,

    arXiv

  24. [1991]

    doi: 10.1109/18.61115. Liu, F. T., Ting, K. M., and Zhou, Z.-H. Isolation forest. In2008 Eighth IEEE International Conference on Data Mining, pp. 413–422,

    doi:10.1109/18.61115

  25. [2008]

    Liu, H., Li, C., Wu, Q., and Lee, Y

    doi: 10.1109/ICDM.2008.17. Liu, H., Li, C., Wu, Q., and Lee, Y . J. Visual instruction tun- ing,

    doi:10.1109/icdm.2008.17 2008

  26. [2010]

    Xu, Y ., Qi, X., Qin, Z., and Wang, W

    URL https: //arxiv.org/abs/1004.2515. Xu, Y ., Qi, X., Qin, Z., and Wang, W. Cross-modality information check for detecting jailbreaking in multi- modal large language models,

    Pith/arXiv arXiv

  27. [2017]

    Sure, here is

    ( N= 10,000 ) to fit the One-Class Classifier. This dataset represents standard, aligned multimodal behavior. For Out-Of-Distribution (OOD) testing, we employ the validation split ofVizWiz-VQA(Gurari et al., 2018), which contains images taken by blind users characterized by blur, poor framing, and occlusion. To evaluate utility on semantically complex inp...

    2018

  28. [2018]

    Inan, H., Upasani, K., Chi, J., Rungta, R., Iyer, K., Mao, Y ., Tontchev, M., Hu, Q., Fuller, B., Testuggine, D., and Khabsa, M

    URL https://arxiv.org/abs/1802.08218. Inan, H., Upasani, K., Chi, J., Rungta, R., Iyer, K., Mao, Y ., Tontchev, M., Hu, Q., Fuller, B., Testuggine, D., and Khabsa, M. Llama guard: Llm-based input-output safe- guard for human-ai conversations,

    Pith/arXiv arXiv

  29. [2020]

    Dubey, A

    URL https://arxiv.org/abs/ 2003.01690. Dubey, A. et al. The llama 3 herd of models,

    Pith/arXiv arXiv 2003

  30. [2021]

    doi: 10.18653/v1/2021.acl-long.133

    Association for Computational Linguis- tics. doi: 10.18653/v1/2021.acl-long.133. URL https: //aclanthology.org/2021.acl-long.133/. Gong, Y ., Ran, D., Liu, J., Wang, C., Cong, T., Wang, A., Duan, S., and Wang, X. Figstep: Jailbreaking large vision- language models via typographic visual prompts,

    doi:10.18653/v1/2021.acl-long.133 2021

  31. [2023]

    Jain, N., Schwarzschild, A., Wen, Y ., Somepalli, G., Kirchenbauer, J., yeh Chiang, P., Goldblum, M., Saha, A., Geiping, J., and Goldstein, T

    URL https: //arxiv.org/abs/2312.06674. Jain, N., Schwarzschild, A., Wen, Y ., Somepalli, G., Kirchenbauer, J., yeh Chiang, P., Goldblum, M., Saha, A., Geiping, J., and Goldstein, T. Baseline defenses for ad- versarial attacks against aligned language models,

    Pith/arXiv arXiv

  32. [2024]

    Chao, P., Robey, A., Dobriban, E., Hassani, H., Pappas, G

    URL https://arxiv.org/abs/ 2212.03827. Chao, P., Robey, A., Dobriban, E., Hassani, H., Pappas, G. J., and Wong, E. Jailbreaking black box large lan- guage models in twenty queries,

    Pith/arXiv arXiv

  33. [2025]

    URL https://arxiv.org/ abs/2504.09604. Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., Zhong, H., Zhu, Y ., Yang, M., Li, Z., Wan, J., Wang, P., Ding, W., Fu, Z., Xu, Y ., Ye, J., Zhang, X., Xie, T., Cheng, Z., Zhang, H., Yang, Z., Xu, H., and Lin, J. Qwen2.5-vl technical report,

    arXiv