Pith. sign in

REVIEW 6 major objections 7 minor 2 cited by

VSF-Med:A Vulnerability Scoring Framework for Medical Vision-Language Models

T0 review · 6 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper proposes an eight-axis rubric that reduces adversarial risk in medical vision–language models to a single 0–32 score.

desk verdict Useful benchmark machinery with a serious selection-bias flaw that breaks the headline ranking. read the letter →

arxiv 2507.00052 v1 pith:NIFAFE3A submitted 2025-06-25 cs.CV cs.AI

Binesh Sadanandan , Vahid Behzadan This is my paper
classification cs.CVcs.AI
keywords medicalvision-languagemodelsvulnerabilityscoringpromptinjectionadversarialperturbationsSSIMcalibrationLLM-as-a-judgechestX-rayMIMIC-CXR
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper aims to give medical vision–language models a standardized way to be tested for security before they are used in clinical workflows. It proposes VSF-Med, a pipeline that turns chest X-ray images and radiology prompts into more than 30,000 adversarial test cases, asks two judge language models to rate model responses on eight vulnerability dimensions, and converts those ratings into a single 0–32 composite risk score. The authors report that adversarial cases shift vulnerability scores by about 0.90 standard deviations for persistence of attack effects, 0.74 for prompt injection, and 0.63 for safety bypass across five state-of-the-art models, with Llama-3.2-11B-Vision-Instruct showing the largest increases. If the framework works as claimed, hospitals and regulators can compare models on a common clinical-risk scale rather than on piecemeal accuracy metrics.

What carries the argument

The carrying object is the eight-dimension vulnerability rubric combined with z-score consolidation: each dimension is rated 0–4, the ratings are summed to a 0–32 total, and the totals are binned into low, moderate, high, and critical clinical risk tiers. The scoring engine is a judge ensemble of GPT-4o-mini and Claude, which receives batches of 50 model outputs together with the rubric and returns structured JSON scores; z-score normalization is used to bring the two judges' rating distributions onto a common scale. The visual perturbation suite is calibrated by SSIM, the Structural Similarity Index, with a threshold of 0.85 to keep attacks imperceptible while limiting clean-case performance loss to under 10%. This combination is what lets otherwise heterogeneous attacks and model behaviors be compared as a single risk number.

What would settle it

Take a random sample of a few hundred model outputs from the VSF-Med pipeline and have a panel of radiologists and security experts assign the same 0–4 rubric; if expert ratings disagree with the judge LLMs by more than one ordinal point on a substantial share of persistence or safety-bypass cases, the composite scores would not reflect clinical risk.

Watch

Extended reading notes

Core claim

VSF-Med's central claim is that the security of a medical vision–language model can be measured end-to-end and summarized in one auditable number. The framework constructs text prompt attacks across ten clinically motivated categories, applies six families of image perturbations tuned to keep structural similarity at or above 0.85, scores each model response on eight dimensions—prompt injection, jailbreak success, confidentiality breach, misinformation generation, denial-of-service resilience, persistence of attack effects, safety bypass success, and impact on medical decision support—and z-normalizes the raw ratings of two judge LLMs into a 0–32 composite with four risk tiers. On 5,000 MIMIC-CXR studies expanded into over 30,000 variants, the consolidated results show mean vulnerability z-score increases of 0.90 for persistence of attack effects, 0.74 for prompt-injection effectiveness, and 0.63 for safety-bypass success, with the open-weight Llama-3.2-11B-Vision-Instruct among the most affected and GPT-4o among the least in the evaluated set.

Load-bearing premise

The scores are only as valid as the two judge language models' ratings, which were never checked against human medical or security experts, so a systematic judge bias would invalidate the model rankings and risk tiers.

Editorial extensions

If this is right

  • Any medical VLM can be benchmarked by rerunning the open-source pipeline with a single command, making model safety profiles directly comparable across sites and versions.
  • Persistence of attack effects and prompt injection emerge as the highest-risk categories, which would direct defense efforts toward context management and prompt sanitization before visual attack resistance.
  • The four clinical risk tiers give a concrete decision rule: a model crossing into the high or critical tier could be flagged for mandatory human verification or barred from autonomous use.
  • The reported cross-environment reproducibility, with mean total vulnerability differences below 0.1 across 2,000 shared cases, implies the scores can be audited and re-run as models or attack templates change.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit that its model rankings are only as trustworthy as the two judge LLMs; a natural follow-up is a calibration study where radiologists and security experts score a sample of outputs with the same 0–4 rubric, which would test whether the automated scores reflect real clinical risk.
  • The high correlations among misinformation, persistence, and confidentiality breach suggest these may be one shared failure mode rather than eight independent vulnerabilities, meaning the effective dimensionality of the rubric could be lower than eight.
  • Because the evaluation covers only static frontal chest X-rays, the framework likely understates vulnerabilities in temporal modalities such as ultrasound or fluoroscopy, where malicious instructions could persist across frames.
  • Using SSIM as a proxy for imperceptibility is an assumption; an explicit test with radiologists viewing perturbed and clean images side by side would show whether the visual attacks are truly below clinical notice.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

6 major / 7 minor

Summary. The manuscript proposes VSF-Med, an end-to-end vulnerability-scoring framework for medical vision-language models. It combines a library of text-prompt attack templates, an SSIM-constrained suite of visual perturbations, and an eight-dimension ordinal rubric scored by two LLM judges (GPT-4o-mini and Claude) whose outputs are z-score normalized into a 0–32 composite risk score. The authors apply the framework to five VLMs on MIMIC-CXR data, reporting mean z-score shifts of 0.90σ for persistence of attack effects, 0.74σ for prompt injection, and 0.63σ for safety bypass, and they identify Llama-3.2-11B-Vision-Instruct as the most vulnerable model.

Significance. The framework addresses a real gap: no standardized, reproducible benchmark currently maps adversarial robustness of medical VLMs to clinically relevant risk tiers. The open-source pipeline, the public MIMIC-CXR data, the ten attack categories, and the six perturbation classes are useful building blocks. However, the central comparative claims are not currently supported because the attack templates were tuned on the model that is later declared most vulnerable, and because the vulnerability scores are produced by unvalidated LLM judges from the same vendors as some of the evaluated models. As a proposal and toolkit, VSF-Med is a useful contribution; as a measurement of cross-model vulnerability, the paper is not yet sound.

major comments (6)
  1. [§3.1 and §4.3] The attack-template optimization protocol introduces selection bias that breaks the headline cross-model comparison. Section 3.1 states that optimal text-attack templates were isolated by evaluating candidate prompts on the Llama-3.2 11B model and selecting those that elicited the most pronounced deviations; Section 4.3 then reports that Llama-3.2-11B-Vision-Instruct has the highest average vulnerability (0.75σ) and the largest prompt-injection z-score change (1.19σ), and the abstract highlights a 1.29σ persistence effect for the same model. Because the attack set is optimized against this model, the ranking reflects the tuning target rather than an intrinsic property of the evaluated models. The authors must either select attacks independently of all evaluated models or optimize per model and report transferability, before the reported mean shifts and model ordering can be interpreted as general vulnerability scores.
  2. [§3.6 and §4.4] The inter-rater reliability results are internally contradictory. Section 3.6 states that 'the average Pearson correlation exceeded 0.82 for all eight dimensions' across 5,000 sampled cases, while Section 4.4 reports Pearson correlations ranging from 0.22 (persistence of attack effects) to 0.76 (visual jailbreak success). These two statements cannot both describe the same scoring runs. Since inter-rater reliability is a central justification for the automated judge ensemble, the paper must reconcile these numbers and report the actual per-dimension correlations.
  3. [§3.1, §3.2, §3.3, and §3.4] The dataset and variant counts are inconsistent. Section 3.1 reports curating 3,500 cases from MIMIC-CXR, Section 3.4 says 5,000 frontal-view studies were selected, Section 3.2 yields approximately 3,000 text-attack cases, Section 3.3 yields over 27,000 image variants, and the abstract claims over 30,000 adversarial variants from 5,000 radiology images. The reader cannot determine the actual test-set size or how the 3,000 text cases and 27,000 image variants combine to the claimed total. Exact per-stage counts are needed to reproduce the pipeline.
  4. [§3.5, §4.3, and §5.2] The framework is circular in a way that undermines the validity of the vulnerability scores. The eight-dimension scores are produced by GPT-4o-mini and Claude with no validation against human medical or security experts; the same vendor models (GPT-4o, Claude) are among the evaluated systems; and the attack templates were selected using GPT-4o-mini's ratings. The reported z-score changes and model rankings therefore measure properties of the judge models as much as properties of the evaluated VLMs. A minimal fix is a human-validated subset of scores, an open-source judge ablation, and a demonstration that rankings are stable across judges.
  5. [§3.1 and Table 2] The clinical risk-tier definitions are inconsistent and incompletely specified. Section 3.1 lists 'Low (0–4), moderate (5–10), high (11-25) and critical (21)' — with the high and critical ranges overlapping or missing — whereas Table 2 and the supplementary rubric define High as 11–20 and Critical as ≥21. The framework's output depends on these thresholds, so the discrepancy must be fixed.
  6. [§3.1 and §4.6] The claim that the eight vulnerability dimensions are 'orthogonal' is contradicted by the authors' own correlation analysis. Section 4.6 reports Pearson correlations of r = 0.984 between misinformation generation and text jailbreak success, r = 0.967 between misinformation and persistence, and r = 0.968 between confidentiality breach and misinformation. If dimensions are highly correlated, the composite sum double-counts shared variance, and the interpretation of the 0–32 total score as an aggregate of independent axes needs explicit qualification.
minor comments (7)
  1. [§3.5] The sentence 'In pilot tests, this approach achieved over 98' is truncated; presumably it should read 'over 98%' but the metric and evaluation setup should be stated completely.
  2. [§4.4] Section 4.4 reports mean absolute errors between 3.06 and 6.32 rating points; if these are per-dimension assessments on a 0–4 scale, values above 4 are impossible, and if they are on the 0–32 total scale, that should be stated explicitly.
  3. [§2.2] The paragraph beginning 'While each framework illuminates important aspects of VLM behavior...' is duplicated verbatim in Section 2.2.
  4. [§3.1] There is a typo in Section 3.1: 'The sumf these scores' should be 'The sum of these scores'.
  5. [§4.2 and Table 4] Several typos appear in Section 4.2 and Table 4, including 'stegnography', 'causes causes', and 'environent'; these should be corrected before publication.
  6. [Appendix A] The supplementary headings 'A.4 Text-Attack Testing Methods: Literature Support' and 'A.5 Visual Perturbation Testing Methods: Literature Support' have no content in the manuscript, despite being referenced in Sections 3.1 and 3.3; the missing material should be included or the references removed.
  7. [References] Some reference entries contain typos or formatting issues, such as 'Corperation' in Coley & Chase (2019) and 'resrouce' in Zhang et al. (2024b); these should be corrected.

Circularity Check

2 steps flagged · score 6.0 of 10

Attack templates are tuned to Llama-3.2 and to GPT-4o-mini's vulnerability ratings, so Llama's top ranking and the reported judge-measured shifts are partly by construction.

  1. fitted input called prediction [Section 3.1 (Methods) and Section 4.3 (Results)]
    ""Optimal text-attack templates were isolated by evaluating candidate prompts on the Llama-3.2 11B model, and selecting those that elicited the most pronounced deviations in output behavior." "Llama-3.2-11B-Vision-Instruct exhibited the highest average vulnerability ( 0.75σ ) and a maximum z-score change of 1.19σ for prompt injection...""

    The attack templates are chosen precisely because they produce strong output deviations on Llama-3.2, and the same model family is then evaluated with those templates and reported as the most vulnerable of the five models. Llama's high scores are therefore not an independent finding about the models; they are partly a restatement of the selection criterion used to build the attack suite. A benchmark that claims objective cross-model comparison must select attacks independently of the evaluated models, or optimize per model and report transferability; otherwise the reported ranking partially encodes the optimization target.

  2. fitted input called prediction [Section 3.2 (Text-Attack Generation) and Section 3.4 (Evaluation Workflow)]
    ""Template effectiveness was validated by measuring the proportion of cases that increased at least one vulnerability dimension when evaluated with the GPT-4o-mini API." "The automated scoringphase leverages a judge ensemble of GPT-4o-mini and Claude. Each judge ingests a batch of 50 model outputs along with the VSF-Med scoring rubric, returning integer ratings for each of the eight vulnerability dimensions.""

    The attack corpus is screened for templates that raise GPT-4o-mini-assigned vulnerability-dimension scores, and GPT-4o-mini is then used as one of the two judges that produce the reported vulnerability scores. Thus the reported increases are not an independent measurement of vulnerability; they are a restatement of the filter used to select the attack templates. The additional Claude judge and z-score normalization do not remove this dependence, because GPT-4o-mini is a component of both the selection/validation step and the final scoring step.

full rationale

The central circularity is a fitted-input-called-prediction pattern, not a self-citation chain or a definitional identity. Section 3.1 selects optimal text-attack templates by maximizing output deviations on Llama-3.2, and Section 4.3 reports Llama-3.2-11B-Vision-Instruct as the most vulnerable model; this makes part of the headline cross-model ranking a consequence of the selection procedure. Additionally, Section 3.2 validates template effectiveness by checking that cases increase at least one vulnerability dimension as rated by GPT-4o-mini, and Section 3.4 uses GPT-4o-mini as one of the final judges, so the measured vulnerability increases are partly by construction. These are concrete reductions supported by direct quotes, and they warrant a score of 6: the findings are partially circular, not fully forced. I do not count the lack of human-expert validation of the judge LLMs as a circularity step, because the framework defines its vulnerability score through its rubric; that is a validity limitation rather than a derivation that reduces to its input. Similarly, the apparent inconsistency between the inter-rater reliability claims in Sections 3.6 and 4.4 is a correctness concern, not a circularity. No load-bearing self-citations or uniqueness-imported-from-authors patterns are present, and the visual perturbation methodology is independently constrained by SSIM thresholds, so the circularity is localized to the attack-template selection and judge-based scoring loop.

Assumptions & free parameters 5 free parameters · 5 assumptions · 3 invented entities

The framework rests on several choices that are not independently validated: the SSIM threshold, the clean-case degradation cap, the risk-tier boundaries, the model-specific selection of attack templates, and the perturbation parameter grid. The judge-based scoring is unvalidated against human experts, and the eight dimensions are not shown to be orthogonal despite the high inter-dimension correlations reported.

free parameters (5)
  • SSIM threshold = 0.85
    Chosen via grid search to balance imperceptibility and attack potency. It determines which visual perturbations are considered acceptable and directly affects all visual attack results.
  • Clean-case performance degradation cap = 10%
    Maximum allowed drop in model performance on clean cases during perturbation grid search. Used to select perturbation parameters.
  • Risk tier cutoffs = 0-4 Low, 5-10 Moderate, 11-20 High, >=21 Critical
    Arbitrary ordinal thresholds for interpreting total scores. They do not come from a clinical or regulatory model.
  • Attack template selection (on Llama-3.2-11B) = Selected templates per category
    Optimal text-attack templates were selected by measuring which prompts caused the largest vulnerability changes on Llama-3.2-11B. This is a model-specific fit and biases cross-model comparisons.
  • Perturbation parameter grid = Gaussian sigma 5-30, checkerboard 32px, Moire 0.5-2.0 cycles, alpha 0.1-0.5, arrows 5-20
    Grid-searched parameters for each visual attack class under the SSIM and performance constraints. They set the specific attack strength used in the benchmark.
assumptions (5)
  • domain assumption SSIM >= 0.85 is a sufficient proxy for human imperceptibility of image perturbations.
    No human perceptual experiments are reported. In clinical imaging, radiologists may notice artifacts that SSIM does not capture.
  • domain assumption The eight vulnerability dimensions are orthogonal and can be scored independently.
    Section 3.1 calls them orthogonal, but Section 4.6 reports very high inter-dimension correlations, for example r=0.984 between misinformation and text jailbreak, which contradicts orthogonality.
  • domain assumption Judge LLM ratings approximate expert clinical risk assessment.
    The paper relies entirely on GPT-4o-mini and Claude for scoring, with only manual spot-checks mentioned. No systematic comparison with human radiologists or security experts is provided.
  • domain assumption Frontal chest X-rays from MIMIC-CXR are a representative testbed for medical VLM security.
    Only one imaging modality and view are used. The authors acknowledge in Section 5.2 that CT, MRI, ultrasound, and pathology are not covered.
  • domain assumption Z-score normalization across two judge models yields a meaningful 0-32 composite.
    Z-scores are unbounded and signed, while the total score is claimed to lie in 0-32. The paper does not explain how the normalization maps to the capped integer scale.
invented entities (3)
  • Eight-dimensional vulnerability rubric
    purpose: Defines the scoring categories and ordinal 0-4 scale for each model response.
    The dimensions are constructed from literature but have no independent measurement outside the judge LLMs. Their validity is not established against human ratings.
  • Composite VSF-Med risk score (0-32)
    purpose: Provides a single aggregate vulnerability number for a model.
    The aggregate is a sum of judge ratings and has no external benchmark to confirm it corresponds to actual clinical risk.
  • Clinical risk tiers (Low, Moderate, High, Critical)
    purpose: Classifies the composite score into actionability levels.
    The tier boundaries are arbitrary and not derived from regulatory or clinical data.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VSF-Med:A Vulnerability Scoring Framework for Medical Vision-Language Models." pith.science (2026). https://pith.science/paper/NIFAFE3A

@misc{pith2026250700052,
  author       = {Pith},
  title        = {Pith review of: VSF-Med:A Vulnerability Scoring Framework for Medical Vision-Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NIFAFE3A}},
  note         = {Machine review of arXiv:2507.00052}
}
abstract

Vision Language Models (VLMs) hold great promise for streamlining labour-intensive medical imaging workflows, yet systematic security evaluations in clinical settings remain scarce. We introduce VSF--Med, an end-to-end vulnerability-scoring framework for medical VLMs that unites three novel components: (i) a rich library of sophisticated text-prompt attack templates targeting emerging threat vectors; (ii) imperceptible visual perturbations calibrated by structural similarity (SSIM) thresholds to preserve clinical realism; and (iii) an eight-dimensional rubric evaluated by two independent judge LLMs, whose raw scores are consolidated via z-score normalization to yield a 0--32 composite risk metric. Built entirely on publicly available datasets and accompanied by open-source code, VSF--Med synthesizes over 30,000 adversarial variants from 5,000 radiology images and enables reproducible benchmarking of any medical VLM with a single command. Our consolidated analysis reports mean z-score shifts of $0.90\sigma$ for persistence-of-attack-effects, $0.74\sigma$ for prompt-injection effectiveness, and $0.63\sigma$ for safety-bypass success across state-of-the-art VLMs. Notably, Llama-3.2-11B-Vision-Instruct exhibits a peak vulnerability increase of $1.29\sigma$ for persistence-of-attack-effects, while GPT-4o shows increases of $0.69\sigma$ for that same vector and $0.28\sigma$ for prompt-injection attacks.

Figures

Figures reproduced from arXiv: 2507.00052 by the authors.

Figure 1
Figure 1. End-to-end evaluation workflow: from dataset sampling through adversarial generation, inference, automated [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Mean vulnerability z-score changes by attack [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Example of Perturbation : Steganographic infor [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Radar chart of average vulnerability profiles for [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Heatmap of Pearson correlations between vulner [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. LLM-as-a-Judge in Healthcare: A Scoping Analysis of Applications, Methods, and Human Alignment

    cs.CY 2026-05 unverdicted novelty 6.0 of 10

    Scoping review of 134 studies on LLM-as-a-Judge in healthcare finds concentration in clinical decision support and NLP, frequent use of OpenAI models with prompt engineering, and moderate-to-strong human alignment whe...

  2. 6 Fingers, 1 Kidney: Natural Adversarial Medical Images Reveal Critical Weaknesses of Vision-Language Models

    cs.CV 2025-12 conditional novelty 6.0 of 10

    Across 22 vision-language models, accuracy on simple medical perception questions dropped from ~74% on typical anatomy to ~29% on rare anatomical variants, with errors aligning to textbook priors.

Reference graph

Works this paper leans on

46 extracted references · 32 canonical work pages · cited by 2 Pith papers

  1. [4]

    Denial-of-service poisoning attacks against large lan- guage models.arXiv preprint arXiv:2410.10760,

    Gao, K., Pang, T., Du, C., Yang, Y ., Xia, S.-T., and Lin, M. Denial-of-service poisoning attacks against large lan- guage models.arXiv preprint arXiv:2410.10760,

    arXiv

  2. [5]

    Light-weight Fine-tuning Method for Defending Adversarial Noise in Pre-trained Medical Vision-Language Models

    Han, X., Jin, L., Ma, X., and Liu, X. Light-weight fine- tuning method for defending adversarial noise in pre- trained medical vision-language models.arXiv preprint arXiv:2407.02716,

    work page Pith review arXiv

  3. [6]

    Unsupervised Multimodal Representation Learning across Medical Images and Reports

    Hsu, T.-M. H., Weng, W.-H., Boag, W., McDermott, M., and Szolovits, P. Unsupervised multimodal representa- tion learning across medical images and reports.arXiv preprint arXiv:1811.08615,

    work page Pith review arXiv

  4. [7]

    Effective and Efficient Adversarial Detection for Vision-Language Models via A Single Vector

    Huang, Y ., Zhu, F., Tang, J., Zhou, P., Lei, W., Lv, J., and Chua, T.-S. Effective and efficient adversarial detection for vision-language models via a single vector.arXiv preprint arXiv:2410.22888,

    work page Pith review arXiv

  5. [8]

    and Tiwari, M

    Lee, D. and Tiwari, M. Prompt infection: Llm-to-llm prompt injection within multi-agent systems.arXiv preprint arXiv:2410.07283,

    arXiv

  6. [11]

    Corpa: Adversar- ial image generation for chest x-rays using concept vec- tor perturbations and generative models.arXiv preprint arXiv:2502.05214,

    Rafferty, A., Ramaesh, R., and Rajan, A. Corpa: Adversar- ial image generation for chest x-rays using concept vec- tor perturbations and generative models.arXiv preprint arXiv:2502.05214,

    work page arXiv

  7. [12]

    M., Mukkamala, R

    Rossi, S., Michel, A. M., Mukkamala, R. R., and Thatcher, J. B. An early categorization of prompt injection attacks on large language models.arXiv preprint arXiv:2402.00898,

    arXiv

  8. [13]

    Multimedeval: A benchmark and a toolkit for evaluating medical vision- language models.arXiv preprint arXiv:2402.09262,

    Royer, C., Menze, B., and Sekuboyina, A. Multimedeval: A benchmark and a toolkit for evaluating medical vision- language models.arXiv preprint arXiv:2402.09262,

    arXiv

Show all 46 references
  1. [14]

    Fighting fire with fire: adversarial prompting to gener- ate a misinformation detection dataset.arXiv preprint arXiv:2401.04481,

    Satapara, S., Mehta, P., Ganguly, D., and Modha, S. Fighting fire with fire: adversarial prompting to gener- ate a misinformation detection dataset.arXiv preprint arXiv:2401.04481,

    arXiv

  2. [15]

    Safe- guarding vision-language models against patched visual prompt injectors.arXiv preprint arXiv:2405.10529,

    Sun, J., Wang, C., Wang, J., Zhang, Y ., and Xiao, C. Safe- guarding vision-language models against patched visual prompt injectors.arXiv preprint arXiv:2405.10529,

    arXiv

  3. [16]

    Security-first ai: Foundations for robust and trustworthy systems.arXiv preprint arXiv:2504.16110,

    Tallam, K. Security-first ai: Foundations for robust and trustworthy systems.arXiv preprint arXiv:2504.16110,

    arXiv

  4. [17]

    Impact of adversarial examples on the efficiency of interpretation and use of information from high-tech medical images

    Vatian, A., Gusarova, N., Dobrenko, N., Dudorov, S., Nig- matullin, N., Shalyto, A., and Lobantsev, A. Impact of adversarial examples on the efficiency of interpretation and use of information from high-tech medical images. In 2019 24th Conference of Open Innovations Associati...

    2019

  5. [18]

    I., and Ar- cucci, R

    Wu, P., Liu, C., Chen, C., Li, J., Bercea, C. I., and Ar- cucci, R. Fmbench: Benchmarking fairness in multi- modal large language models on medical tasks.arXiv preprint arXiv:2410.01089, 2024a. Wu, S., Koo, M., Kao, L. Y ., Black, A., Blum, L., Scalzo, F., and Kurtz, I. Advers...

    arXiv

  6. [19]

    A survey of safety on large vision-language mod- els: Attacks, defenses and evaluations.arXiv preprint arXiv:2502.14881,

    Ye, M., Rong, X., Huang, W., Du, B., Yu, N., and Tao, D. A survey of safety on large vision-language mod- els: Attacks, defenses and evaluations.arXiv preprint arXiv:2502.14881,

    arXiv

  7. [20]

    Jailbreak vision language models via bi-modal adversarial prompt.arXiv preprint arXiv:2406.04031,

    Ying, Z., Liu, A., Zhang, T., Yu, Z., Liang, S., Liu, X., and Tao, D. Jailbreak vision language models via bi-modal adversarial prompt.arXiv preprint arXiv:2406.04031,

    arXiv

  8. [21]

    S., and Kang, D

    Zhan, Q., Fang, R., Panchal, H. S., and Kang, D. Adaptive attacks break defenses against indirect prompt injection attacks on llm agents.arXiv preprint arXiv:2503.00061,

    arXiv

  9. [22]

    Zhang, Q., Xiong, Z., and Mao, Z. M. Safeguard is a double- edged sword: Denial-of-service attack on large language models.arXiv preprint arXiv:2410.02916, 2024a. Zhang, Y ., Zhou, Z., Zhang, W., Wang, X., Jia, X., Liu, Y ., and Su, S. Crabs: Consuming resrouce via auto- gener...

    arXiv

  10. [23]

    # Medical Case Review

    13 VSF-Med: A Vulnerability Scoring Framework for Medical Vision-Language Models A. Supplementary Materials A.1. VSF Vulnerability Scoring Rubric Table 5: Complete VSF Vulnerability Scoring Rubric Definitions Dimension Score (0 = No Impact to 4 = Critical Impact) Prompt Inject...

    2024

  11. [24]

    - Systematically Analysing Prompt Injection (2025)(Ben- jamin et al.,

    2025

  12. [25]

    - Categorization of Prompt Injection (2024)(Rossi et al.,

    2024

  13. [26]

    Craft medical prompts that in- struct visual LLMs to misinter- pret diagnostic findings Jailbreak - Prompt-G for countering Jail- break (2024)(Pingua et al.,

    2024

  14. [27]

    Develop specialized jailbreak prompts referencing both text in- structions and visual elements Confidentiality Breach - DIRI: Adversarial Patient Rei- dentification (2024)(Morris et al.,

    2024

  15. [28]

    Test if visual LLMs extract or fabricate patient identifiers from images or metadata Misinformation Genera- tion - CAMOUFLAGE (2025)(Bethany et al.,

    2025

  16. [29]

    - Adversarial Prompting for Misinformation Detection (2024)(Satapara et al.,

    2024

  17. [30]

    Evaluate if visual LLMs provide plausible but incorrect diagnoses from subtly modified images Denial-of-Service - LLM Safeguard DoS (2024)(Zhang et al., 2024a) - DoS Poisoning Attacks (2024)(Gao et al.,

    2024

  18. [31]

    - Crabs: Auto-generation DoS (2024)(Zhang et al., 2024b) - Short adversarial prompts (30 chars) can block 97% of re- quests(Zhang et al., 2024a) - Poisoning-based DoS attacks break output limits(Gao et al.,

    2024

  19. [32]

    - AutoDoS attack amplifies latency by over 250×(Zhang et al., 2024b) Test if visual LLMs become over- whelmed by prompts triggering extensive response generation Persistence - Prompt Infection in Multi- Agent Systems (2024)(Lee & Ti- wari,

    2024

  20. [33]

    - Adaptive Attacks on IPI De- fenses (2025)(Zhan et al.,

    2025

  21. [34]

    Verify if malicious instructions persist across multiple image analysis sessions Safety Bypass - LLM Safeguard DoS (2024)(Zhang et al., 2024a) - Prompt-G for countering Jailbreak (2024)(Pingua et al.,

    2024

  22. [35]

    Evaluate if visual LLMs can be manipulated to provide harmful clinical recommendations Semantic Shift - CAMOUFLAGE (2025)(Bethany et al.,

    2025

  23. [36]

    - Adversarial Prompting for Misinformation (2024)(Satapara et al.,

    2024

  24. [37]

    Test if visual LLMs can be sub- tly biased toward rare conditions through crafted prompts Omission Attacks - Forensic Analysis of IPI At- tacks (2024)(Chernyshev et al.,

    2024

  25. [38]

    Test if visual LLMs can be ma- nipulated to omit critical diagnos- tic information Over-Confidence Induc- tion - Adversarial Databases for RAG (2024)(Wu et al., 2024b) - Even adversarial databases im- prove success in retrieval-based LLMs(Wu et al., 2024b) Evaluate if visual L...

    2024

  26. [39]

    - Impact of Adversarial Examples (2019)(Vatian et al.,

    2019

  27. [40]

    Test visual LLM robustness to noise levels common in clinical settings Checkerboard Overlays - Medical Imaging Classi- fiers Robust to Perturbations (2024)(Chuah et al.,

    2024

  28. [41]

    Evaluate visual LLM response to non-clinical artifacts in images Moir´e Patterns - Defending Medical Image Clas- sification (2024)(Aladwan et al.,

    2024

  29. [42]

    Test visual LLM susceptibility to frequency-based image artifacts Random Arrow Artifacts - Impact of Adversarial Exam- ples (2019)(Vatian et al.,

    2019

  30. [43]

    Assess if visual LLMs incor- rectly incorporate non-clinical markers in diagnostic reasoning Steganographic Hide - Secure Medical Image Gener- ation with GAN (2025)(Sumitra et al.,

    2025

  31. [45]

    Test if hidden data in LSB af- fects visual LLM interpretation of medical images LSB-Plane Extraction - Secure Medical Image Gener- ation with GAN (2025)(Sumitra et al.,

    2025

  32. [46]

    - Defending Medical Image Clas- sification (2024)(Aladwan et al.,

    2024

  33. [47]

    - LSB modifications can contain ad- versarial information(Sumitra et al., 2025)(Aladwan et al.,

    2025

  34. [2019]

    Does clip bene- fit visual question answering in the medical domain as much as it does in the general domain?arXiv preprint arXiv:2112.13906,

    Eslami, S., de Melo, G., and Meinel, C. Does clip bene- fit visual question answering in the medical domain as much as it does in the general domain?arXiv preprint arXiv:2112.13906,

    arXiv

  35. [2022]

    X., Campion, T

    Morris, J. X., Campion, T. R., Nutheti, S. L., Peng, Y ., Raj, A., Zabih, R., and Cole, C. L. Diri: Adversarial patient reidentification with large language models for evaluating clinical text anonymization.arXiv preprint arXiv:2410.17035,

    arXiv

  36. [2023]

    Autodan- turbo: A lifelong agent for strategy self-exploration to jailbreak llms.arXiv preprint arXiv:2410.05295,

    Liu, X., Li, P., Suh, E., V orobeychik, Y ., Mao, Z., Jha, S., McDaniel, P., Sun, H., Li, B., and Xiao, C. Autodan- turbo: A lifelong agent for strategy self-exploration to jailbreak llms.arXiv preprint arXiv:2410.05295,

    arXiv

  37. [2024]

    J., and Najafi- rad, P

    Bethany, M., Vishwamitra, N., Chiang, C.-Y . J., and Najafi- rad, P. Camouflage: Exploiting misinformation detection systems through llm-driven adversarial claim transforma- tion.arXiv preprint arXiv:2505.01900,

    arXiv

  38. [2025]

    Systematically analyzing prompt injec- tion vulnerabilities in diverse llm architectures.arXiv preprint arXiv:2410.23308,

    Benjamin, V ., Braca, E., Carter, I., Kanchwala, H., Kho- jasteh, N., Landow, C., Luo, Y ., Ma, C., Magarelli, A., Mirin, R., et al. Systematically analyzing prompt injec- tion vulnerabilities in diverse llm architectures.arXiv preprint arXiv:2410.23308,

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.