{"id":"8e77ad1e-7c90-4ed5-91f8-e11076192ad4","arxiv_id":"2505.23793","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"USB-SafeBench is a unified MLLM safety benchmark with 61 risk categories, 4 modality combinations, and dual-language vulnerability and oversensitivity tests.","lead":"A new safety benchmark for multimodal AI combines 61 risk categories with four image-text modality combinations and tests models in English and Chinese. It evaluates 17 commercial and open-source models and finds that most are weakest when the risk is hidden in the image or appears only when image and text are combined.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Every ASR and ARR result, and the resulting safety rankings, is produced by an unvalidated GPT-4o judge that is itself one of the 17 evaluated models; without a human-agreement check the empirical conclusions are only conditional.","rationale":"The reader's CONDITIONAL verdict and my read converge on the same load-bearing point: the empirical evaluation is only as good as the GPT-4o judge, and no validation of that judge is provided. I add one concrete aggravator—GPT-4o is simultaneously the judge and a subject, so even random judge noise cannot be assumed to be symmetric across models. I also considered the internal tension between the abstract's '4 distinct modality combinations for each of the 61 risk sub-categories' and Section 2.2's statement that oversensitivity evaluation uses only MOSSBench, and the circularity of USB-Hard's selection by attack success on the same model families that are later evaluated; both are worth inspecting in the released data and analysis scripts, but neither is as immediately important to the headline empirical claims as an unvalidated judge. The dataset/code release, the human annotation protocol in Appendix C, and the reuse of established judge prompts are genuine strengths; they make the artifact valuable but do not independently establish the validity of the reported metrics. A human-judge agreement study is the single check that would settle whether the concern lands. Since the reader already conditioned the verdict on resolving this issue, my stress test does not change the verdict.","tokens_in":22891,"tokens_out":10195,"duration_ms":117699,"concrete_test":"Take a stratified random sample of at least 600 response triples (300 vulnerability, 300 oversensitivity), selected to span all 17 models, the 61 risk categories, the 4 modality combinations, and both languages. Have at least 3 trained annotators independently label each response with the exact rubrics in Appendix E.6/E.7, and compute Fleiss' kappa among annotators and Cohen's kappa between majority human labels and GPT-4o labels. Then re-derive Table 2 ASR/ARR and Figure 6 rankings using human majority labels, with per-model confidence intervals and a separate analysis restricted to GPT-4o's own outputs. If human-judge kappa is below 0.8, or if any model's ASR/ARR moves by more than 5 percentage points relative to GPT-4o's labels, the empirical conclusions are not supported and should be re-reported or qualified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's empirical contribution—Table 2 ASR/ARR, Figure 6 trade-off analysis, the modality-combination finding in §3.2, and the USB-Base vs USB-Hard comparison in Figure 8—is entirely mediated by GPT-4o as judge, using the prompts in Appendix E.6 (vulnerability) and E.7 (oversensitivity). No human inter-annotator agreement or judge-vs-human agreement is reported anywhere. Appendix C describes human cross-assessment only for data attributes (risk category, modality, style), not for the safety labels that generate ASR/ARR. The conflict is compounded because GPT-4o is also one of the 17 evaluated models in Table 2; a judge that systematically recognizes its own refusal style as 'safe' would lower GPT-4o's ASR/ARR relative to other models without reflecting true robustness. If the judge is biased on even a modest fraction of responses, the model rankings, the claimed cross-modal vulnerability gap, and the 'consistent relative rankings on USB-Hard' conclusion all lose support. This concern does not attack the existence or coverage of the released dataset itself, but the paper's central claim of a 'trustworthy safety assessment' rests on the unverified assumption that GPT-4o's labels are accurate.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces USB-SafeBench, a large-scale multimodal safety benchmark covering 61 risk subcategories, 4 image-text modality combinations (RIRT, SIRT, RIST, SIST), both English and Chinese, and both vulnerability (ASR) and oversensitivity (ARR). The dataset combines curated samples from 13 existing benchmarks with synthetic data generated through a pipeline that uses GPT-4o/Gemini-1.5-Pro for scenario and query generation and Stable-Diffusion-3.5/Flux for image synthesis. The final benchmark is split into USB-Base (14,126 samples) and USB-Hard (3,935 samples). The authors evaluate 17 MLLMs (5 closed-source, 12 open-source) and report ASR/ARR values, a safety-refusal trade-off analysis, a modality-combination analysis, a model-size analysis, and a comparison between USB-Base and USB-Hard. The central empirical claims are that USB achieves 98.3% coverage of the category-modality space, yields substantially higher ASR than existing benchmarks, reveals that models are more vulnerable to visual-only and cross-modal risks, and produces consistent relative model rankings across difficulty levels.","tokens_in":23190,"tokens_out":6832,"duration_ms":65371,"significance":"If validated, this would be a genuinely valuable contribution: the first benchmark to cover 61 risk subcategories across 4 modality combinations, 2 languages, and 2 safety dimensions at this scale, with a transparent construction pipeline and public release. The taxonomy is detailed, the coverage analysis of existing benchmarks is useful, and the simultaneous treatment of vulnerability and oversensitivity is a step forward. The dataset artifact and its supporting human annotation for data attributes are strengths. However, the empirical conclusions are conditional on an unvalidated GPT-4o judge and on selection procedures that introduce circularity; these issues must be resolved before the benchmark can support the paper's claim of enabling 'a trustworthy safety assessment'.","major_comments":[{"comment":"All ASR and ARR numbers in Table 2, Figures 6 and 8, and the modality-combination analysis in §3.2 are produced by a single GPT-4o judge using the prompts in Appendix E.6 (vulnerability) and E.7 (oversensitivity). No human agreement or judge-vs-human agreement is reported; Appendix C describes human annotation only for data attributes (risk category, modality, style), not for the safety labels that generate ASR/ARR. Because GPT-4o is itself one of the 17 evaluated models, the judge may systematically recognize its own refusal style as 'safe', biasing relative rankings. The paper must report judge-human agreement statistics (e.g., Cohen's kappa on a representative sample) or replace the judge with a validated, independently verified classifier.","section":"Section 3.1, Appendix E.6/E.7"},{"comment":"USB-Hard is constructed by selecting the 15 highest-ASR samples per viewpoint from the 10 filter MLLMs (Section 2.4), and USB-Base is filtered to keep only queries that successfully attack at least one of those 10 models (Section 2.3, Data Quality Test). Since these filter models are also among the 17 evaluated models in Table 2, the finding in §3.2 and Figure 8 that ASR on USB-Hard is higher for every model, and that relative rankings are consistent, is at least partly forced by construction for the filter models. The paper should either exclude the filter models from the evaluation set, construct USB-Hard using a disjoint model set, or explicitly separate the analysis for filter and non-filter models before drawing the 'consistent rankings' conclusion.","section":"Section 2.4 and Section 2.3"},{"comment":"The reported ASR/ARR values are single-run point estimates with no confidence intervals, standard errors, or significance tests. Several model differences are small (e.g., Qwen-VL-Max 56.87% vs GLM-4V-9B 56.73% vs Qwen2-VL-7B 55.33% in Table 2), and the 'consistent relative rankings' conclusion in Figure 8 is based on visual inspection of a scatter plot without any statistical measure. The paper should provide multiple runs or bootstrap confidence intervals and report a rank-correlation statistic (e.g., Spearman's rho) with a confidence interval for the Base-vs-Hard comparison.","section":"Table 2 and Section 3.2"},{"comment":"The coverage metric is defined as the percentage of cells with more than 20 samples out of 244 possible category-modality combinations. This threshold is arbitrary, and because USB was generated specifically to fill every cell while existing benchmarks were not constructed with this cell-filling objective, the 98.3% vs 59.8% comparison is not an apples-to-apples measure of benchmark quality. The paper should report coverage under multiple thresholds (e.g., 10, 20, 50 samples) and discuss the sensitivity of the comparison.","section":"Table 1 and Section 2.2"}],"minor_comments":[{"comment":"The word 'coverge' should be 'coverage', and the phrase 'modal risk combinations' is grammatically awkward; consider 'risk-modality combinations'.","section":"Abstract"},{"comment":"Typos: 'Appedix' should be 'Appendix' in §3.1; 'commerical' should be 'commercial' in §5; 'choosed' should be 'chose' in §3.1.","section":"Section 3.1 and Section 5"},{"comment":"The sentence 'Figure 4 illustrates the of our synthetic data' is missing a word; it should be 'illustrates examples of our synthetic data'.","section":"Section 2.4"},{"comment":"Typos in headings/text: 'Iterattion' should be 'Iteration' (E.5), 'calssification' should be 'classification' (E.8), 'instrutions' should be 'instructions' (E.4).","section":"Appendix E.5, E.8, E.4"},{"comment":"The paper refers to '10 mainstream open-source MLLMs' used as filter models but never lists them. Specify the exact model names and versions for reproducibility, especially because these models also appear in the main evaluation.","section":"Section 2.3"},{"comment":"Clarify whether the coverage calculation applies the same 'low-quality data filtering' to the existing benchmarks as to USB; if not, the comparison may be unfair to benchmarks that were not filtered in the same way.","section":"Table 1 note (i)"}],"recommendation":"major_revision","confidential_remarks":"The dataset construction and taxonomy are solid, and the release is a service to the community. However, the empirical claims rest on two load-bearing issues that need substantial work: the unvalidated GPT-4o judge (which is also an evaluated model) and the circular selection of USB-Hard and the filtered base set. I would ask the authors to either add a human-agreement study, use a panel of judges, or clearly restrict the scope of their empirical claims. The paper should also disclose the exact list of filter models and provide uncertainty estimates. With those additions, the contribution could be appropriate for the journal."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing: the dataset and taxonomy are the contribution, not the empirical results. They assembled 61 risk categories × 4 modality combinations, English and Chinese, vulnerability and oversensitivity, and they released the data and code. That is real and useful in a fragmented area. The coverage analysis of existing benchmarks is a handy reference, and the human annotation pipeline for dataset labels, with cross-assessment and adjudication, is more careful than most. Give credit: this is a serious benchmark artifact.\n\nThe soft spots are in the evaluation. Every ASR and ARR number comes from a single GPT-4o judge with no human agreement check, and GPT-4o is also one of the 17 evaluated models. That puts all the model rankings, the cross-modal vulnerability finding, and the trade-off analysis on one unvalidated assumption. The paper should report a sample of judge-vs-human agreement or soften the claim of 'trustworthy safety assessment.' This is fixable but load-bearing.\n\nSecond, USB-Hard is partly circular: it picks the 15 highest-ASR samples per viewpoint from the 10 filter MLLMs, so higher ASR on USB-Hard for those models is forced. It is still a reasonable stress set, but they should present it as adversarial selection rather than evidence of consistent relative rankings. Also, single runs with no confidence intervals make fine-grained comparisons shaky.\n\nFor a benchmark paper these are not fatal. The artifact deserves serious refereeing. I'd send it to review with the request that they validate the judge and report error bars. If you work on MLLM safety evaluation, cite it as a resource and maybe bring it to reading group to debate the judge issue.","headline":"The dataset and taxonomy are a real contribution; the empirical rankings rest on an unvalidated GPT-4o judge, so treat them as provisional.","tokens_in":682,"tokens_out":1213,"would_cite":true,"duration_ms":32267,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a single benchmark grid crossing 61 risk categories with four image-text combinations can replace the patchwork of existing MLLM safety tests, in English and Chinese, for both vulnerability and oversensitivity.","keywords":["MLLM safety evaluation","safety benchmark","attack success rate","oversensitivity","modality combinations","risk taxonomy","data synthesis pipeline","safety alignment"],"falsifier":"Take a stratified sample of model responses spanning all 17 models, the four modality combinations, and both risky and safe queries; have independent human annotators apply the paper's own three-level rubric (safe with refusal, safe with warning, unsafe) and its refusal rubric from Appendices E.6 and E.7, and compute agreement with GPT-4o's labels. If agreement is low, or if disagreement concentrates in particular risk categories or modality combinations — for instance if GPT-4o systematically marks cautious refusals as safe or misses subtly unsafe answers — then the reported ASR/ARR values and the model rankings would not survive re-scoring, even though the dataset itself would remain usable.","tokens_in":22701,"feed_emoji":"🛡️","tokens_out":10590,"duration_ms":80306,"temperature":0.7,"pith_summary":"The paper claims that existing safety benchmarks for multimodal large language models are too narrow to be trusted: most test only text-based risk paired with harmless images, skip cross-modal risks, and contain easy questions that inflate safety scores. To fix this, it introduces USB-SafeBench, built on a three-level taxonomy of 61 risk sub-categories crossed with four image-text risk combinations, in English and Chinese, testing both vulnerability (attack success rate) and oversensitivity (refusal rate on harmless inputs). The paper argues that this reaches 98.3% coverage of the 244 category-modality cells, whereas even the union of the collected open-source benchmarks reaches only about 60%, so a single benchmark can replace assembling many. Evaluations on 17 models show that no model is both safe and non-refusing, and that visual-only and cross-modal risks defeat even the strongest commercial models. If true, this gives the field a common measuring stick for MLLM safety and a concrete map of where alignment fails.","feed_headline":"One benchmark covers 98.3% of the MLLM risk space","feed_subtitle":"It pairs 61 risk categories with four image-text mixes and scores both attacks and over-refusals in two languages.","key_machinery":"The load-bearing object is the modality composition matrix, a $2 \\times 2$ crossing of image risk (risky vs safe) with text risk (risky vs safe), giving the four configurations RIRT, RIST, SIRT, and SIST. This matrix defines what the paper means by 'comprehensive': coverage is the fraction of the $61 \\times 4 = 244$ cells that contain more than twenty usable samples after filtering. To fill the empty cells, a data synthesis pipeline generates risk scenarios with auxiliary models, renders images with text-to-image diffusion models, pairs them with risky or safe questions, filters candidate samples through ten open-source MLLMs, iteratively rewrites failed queries to get past refusals, and finishes with dual human annotation under a cross-assessment protocol. On top of this matrix sit the two metrics that carry the empirical argument, the attack success rate $\\mathrm{ASR}=\\frac{1}{N_h}\\sum_{i=1}^{N_h} f_h(i)$ and the average refusal rate $\\mathrm{ARR}=\\frac{1}{N_r}\\sum_{j=1}^{N_r} f_r(j)$, which measure the two failure directions separately.","core_discovery":"On its own terms, the paper's central claim is that MLLM safety should be measured as a grid, not a list: every risk category should be tested in all four modality combinations — risky image with risky text (RIRT), risky image with safe text (RIST), safe image with risky text (SIRT), and safe image with safe text (SIST) — and both directions of failure (answering harmful queries, and refusing harmless ones) should be recorded at once. USB-SafeBench is the claimed embodiment: 61 tertiary risk categories under three primary topics (public safety, ethical safety, national safety), four modality combinations per category, English and Chinese, with 14,126 base samples and 3,935 hard samples. The paper reports 98.3% coverage of the 244 category-modality cells after quality filtering, against roughly 59.8% for the union of existing datasets, and an average attack success rate of 53.25% on the base set and 72.71% on the hard set across ten models, showing USB is substantially harder as well as broader. The empirical payload is that text-only safety alignment is misleadingly flattering: models are far more attackable when the risk lives in the image alone (RIST) or emerges from the image-text combination (SIST), and no evaluated model simultaneously achieves a low attack success rate and a low refusal rate.","pith_inferences":["The 98.3% coverage figure counts a cell as covered only if it has more than twenty samples; re-auditing the grid at higher sample-size thresholds would thin it, so fine-grained per-category conclusions need per-cell size checks.","The paper's own appendix notes that politically sensitive national-security cases and video content were excluded, so the modality matrix's generality to the highest-severity cases and to dynamic visual input is untested; carrying the same risky/safe crossing to video is a direct next step.","Because every ASR and ARR label comes from the GPT-4o judge, the model rankings read as conditional on that judge's standards; an obvious validation is human re-scoring of a stratified sample, and until that exists the dataset itself and the rankings should be treated separately.","The strong linear correlation between USB-Base and USB-Hard, with USB-Hard amplifying gaps, suggests the hard set could double as a cheap screening stage in a two-tier evaluation pipeline, with the base set reserved for detailed diagnostics."],"forward_implications":["Safety results from older text-centric benchmarks need re-reading: models reported as safe look substantially weaker once the image carries the risk, so scores should be reported per modality combination rather than as one number.","Evaluators can replace a patchwork of datasets with one suite: USB-Base gives a broad baseline and USB-Hard (the 15 hardest samples per cell) differentiates robust models while preserving the same relative ranking.","Model developers should treat the safety-helpfulness trade-off as a two-axis constraint: optimizing attack success rate alone is gameable by refusing everything, so releases should report the refusal rate alongside it.","The synthesis pipeline offers a template for generating hard adversarial samples in under-covered risk categories without manual red-teaming, which could be reused for new risk categories or new modalities."],"supporting_citations":[{"why":"Supplies the vulnerability-evaluation judge prompt (Appendix E.6) that defines the paper's ASR metric, and contributes data to the collected benchmark union.","marker":"[17]"},{"why":"Supplies the oversensitivity-evaluation judge prompt (Appendix E.7) that defines the paper's ARR metric, and is the only oversensitivity dataset the paper uses.","marker":"[18]"},{"why":"Quoted as the judge model (GPT-4o) behind every safe/unsafe and refusal label in the evaluation pipeline.","marker":"[8]"},{"why":"Source dataset used in the collection and union-coverage analysis; its limited modality combinations motivate USB's four-configuration grid.","marker":"[13]"},{"why":"Collected benchmark on visual (typographic) vulnerabilities that grounds the risky-image configurations RIST and RIRT.","marker":"[14]"},{"why":"Collected benchmark that synthesizes query-specific images with text-to-image models, the precedent for USB's image synthesis step.","marker":"[23]"},{"why":"Collected jailbreak-robustness benchmark included in the union analysis and compared in Table 1.","marker":"[12]"},{"why":"Collected trustworthiness benchmark included in the coverage analysis and the comparison against existing suites.","marker":"[16]"}],"fun_headline_variants":["MLLM safety grid: 61 risks, 4 modalities, 2 languages","Image-only attacks slip past text-aligned MLLMs","98.3% risk coverage: USB maps MLLM safety grid","Text-only alignment flatters MLLMs: image risks hidden"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Every number in the paper — every attack success rate, refusal rate, model ranking, and trade-off claim — is produced by GPT-4o acting as the judge of whether a model output is unsafe or over-refused, and the paper supplies no comparison of that judge against human raters, so a systematic bias in the judge would shift all the empirical conclusions.","fun_headline_variants_meta":{"raw":{"variants":["MLLM safety grid: 61 risks, 4 modalities, 2 languages","Image-only attacks slip past text-aligned MLLMs","98.3% risk coverage: USB maps MLLM safety grid","Text-only alignment flatters MLLMs: image risks hidden"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000768,"raw_usage":{"total_tokens":3471,"prompt_tokens":1080,"completion_tokens":2391,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":696,"completion_tokens_details":{"reasoning_tokens":2314}},"tokens_in":696,"tokens_out":2391,"duration_ms":15784,"temperature":1.0,"reasoning_tokens":2314,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:08:40.553665+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a stratified sample of model responses spanning all 17 models, the four modality combinations, and both risky and safe queries; have independent human annotators apply the paper's own three-level rubric (safe with refusal, safe with warning, unsafe) and its refusal rubric from Appendices E.6 and E.7, and compute agreement with GPT-4o's labels. If agreement is low, or if disagreement concentrates in particular risk categories or modality combinations — for instance if GPT-4o systematically marks cautious refusals as safe or misses subtly unsafe answers — then the reported ASR/ARR values and the model rankings would not survive re-scoring, even though the dataset itself would remain usable.","supporting_citations":[{"cited_title":"MM-SafetyBench: A benchmark for safety evaluation of multimodal large language models,","cited_arxiv_id":null,"evidence_quote":"Source dataset used in the collection and union-coverage analysis; its limited modality combinations motivate USB's four-configuration grid."},{"cited_title":"Images are achilles’ heel of alignment: Exploiting visual vulnerabilities for jailbreaking multimodal large language models,","cited_arxiv_id":null,"evidence_quote":"Collected benchmark on visual (typographic) vulnerabilities that grounds the risky-image configurations RIST and RIRT."}],"review_version":1}