Pith. sign in

REVIEW 4 major objections 7 minor 47 references

Omni LLMs extract safety cues from image, audio, and speech but often fail to combine them when judging risk.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 19:27 UTC pith:7BC6W4AO

load-bearing objection Solid first omni multicontext safety benchmark with clean minimal-pair design and useful perception-vs-reasoning diagnostics; synthetic-data fidelity is the real soft spot, not a load-bearing collapse. the 4 major comments →

arxiv 2606.05177 v1 pith:7BC6W4AO submitted 2026-04-17 cs.CL cs.AIeess.AS

MCBench: A Multicontext Safety Assessment Benchmark for Omni Large Language Models

Manh Luong , Tamas Abraham , Junae Kim , Amar Kaur , Rollin Omari , Gholamreza Haffari , Trang Vu , Lizhen Qu
show 1 more author
Dinh Phung
This is my paper
classification cs.CL cs.AIeess.AS
keywords omni large language modelsmultimodal safetycross-modal reasoningsafety benchmarkvision-audio-speechoversensitivityMCBench
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Current multimodal safety tests only look at pictures, so they cannot measure models that also hear sound and speech. This paper introduces MCBench, a set of 1,196 paired safe/unsafe situations across physical harm, social harm, illegal harm, and property damage. Every situation must be judged from an image, an audio clip, and a spoken utterance together; a tiny change in one of those inputs flips the correct label from unsafe to safe. When today's best open and proprietary omni models are run on the set, they do better when danger is visually or acoustically obvious, yet they systematically miss or invent risk for social and legal situations. Looking inside their reasoning shows they can name the right pieces of evidence yet still fail to weigh those pieces against one another. The practical message is that simply feeding three modalities into a model does not produce reliable safety judgment; the models need better ways to integrate what they see and hear.

Core claim

State-of-the-art Omni LLMs lack robust cross-modal reasoning for safety: they can extract modality-specific information that matches ground-truth predicates, yet they frequently fail to integrate those cues into correct safe/unsafe decisions, especially for subtle or non-physical risks, while performing better when salient visual or acoustic signals are present.

What carries the argument

MCBench: 1,196 multicontext scenarios, each an image–audio–speech triple paired with a minimally different safe counterpart and an explicit If-Then predicate that lists the conditions that determine the safety label.

Load-bearing premise

The synthetically generated images and audio clips, after human filtering, plus the model-written If-Then rules, faithfully stand in for the real multicontext safety situations these systems will face.

What would settle it

A controlled human study in which people rate the same generated image–audio–speech triples for realism and safety label, or a re-run of the benchmark using only real-world recordings, that produces accuracy rankings or oversensitivity patterns that reverse the paper’s reported model ordering.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Safety evaluations for omni models must require joint vision–audio–speech input rather than image-only probes.
  • Architectures and training will need explicit mechanisms that force balanced fusion of modalities instead of letting one cue dominate.
  • Models that look strong on physical or property damage may still be unreliable for social and legal risk.
  • Oversensitivity on safe cases will remain a dominant failure mode until models learn to weigh contradictory evidence rather than latch onto single concerning signals.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the integration failure is architectural rather than data-driven, simply scaling model size or adding more multimodal pre-training may not close the gap.
  • The same oversensitivity pattern already documented for text-only and vision-only models appears to generalize to three-modality inputs, suggesting a deeper reasoning bias rather than a modality-specific defect.
  • Benchmarks that pair near-identical safe and unsafe counterparts could become a standard stress test for any future multimodal safety claim.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper introduces MCBench, a 1196-scenario multicontext safety benchmark for Omni LLMs that process vision, audio, and speech jointly. Scenarios span four coarse categories (physical harm, social harm, illegal harm, property damage), each with fine-grained subcategories, and are constructed as minimally differing unsafe–safe pairs with If–Then ground-truth predicates. Multimodal inputs are synthesized (Gemini-Flash-2.5 images, Stable Audio clips) from Claude-generated scenario descriptions, then human-filtered. Evaluations of open-source and proprietary Omni LLMs (Table 3) show above-chance but modest accuracy (~64.5% best), with particular weakness on social and illegal harm and relative strength on physical harm and property damage. Ablations with textual alternatives (Figs. 4–6), perception-alignment scoring against predicates (Fig. 7), and predicate-only inference (Tables 4–5) support the claim that models can extract modality-specific cues yet fail to integrate them, and that they exhibit oversensitivity on safe scenarios.

Significance. If the findings hold under the intended real-world distribution, MCBench fills a clear gap: existing multimodal safety benchmarks are vision-only, and existing omnimodal benchmarks are not safety-focused. The paired safe/unsafe design, explicit predicates, and perception-vs-reasoning diagnosis are useful methodological contributions for diagnosing cross-modal safety failures. The empirical pattern—better performance when cues are salient and visual/acoustic, worse on subtle social/legal risks, plus oversensitivity when given rich context—is actionable for architecture and training work on multimodal safety. The paper does not ship code, human IAA statistics, or machine-checked proofs; its value is the benchmark construction and the diagnostic analysis of reasoning traces.

major comments (4)
  1. §2.2 (Phases 1–2) and the central claim in the abstract/§5: the claim that Omni LLMs lack robust cross-modal safety reasoning rests on synthetic media (Gemini-Flash-2.5 images, Stable Audio 1.0 event-decomposed clips) and Claude-generated If–Then predicates, with only manual filtering for “unrealistic” or “low-quality” items. There is no quantitative alignment check between generated (I, A) and the predicates L, nor human inter-annotator agreement on final safety labels R. Without such validation, accuracy gaps in Table 3 and the perception-vs-reasoning diagnosis (Figs. 7–8, Tables 4–5) may partly reflect under-/over-specified synthetic cues rather than a genuine architectural deficit. A human study measuring agreement on labels and predicate–media match (even on a stratified subset) is needed to underwrite the main claim.
  2. §3.1 Evaluation metric and Table 3: free-form responses are mapped to {safe, unsafe} solely by GPT-4o as LLM-as-a-judge, with five runs averaged but no human agreement statistics, no reported judge–human concordance, and no sensitivity analysis to the judge model. Because all headline accuracies and category-specific conclusions depend on this mapping, the absence of human validation of the judge is load-bearing. Report human agreement on a sample of model responses, or at least agreement between multiple judge models, and quantify how often the judge’s binary label is ambiguous.
  3. §2.2 Phase 2 and §3.2 / Fig. 7: Gemini-Flash-2.5 is both the image generator for the benchmark and an evaluated model that achieves the highest perception-alignment score. This dual role risks distributional favoritism (the model is scored on images drawn from its own generative prior). The manuscript should either (i) re-generate or re-sample images with a held-out generator for the Gemini evaluation, or (ii) explicitly quantify and discuss contamination risk and show that Gemini’s relative ranking is stable under alternative image sources.
  4. §3.4.2 Setting 2 and Tables 4–5: when models receive ground-truth predicates as text, safe-scenario accuracy collapses (e.g., −46% for Qwen-Omni-2.5-3B) while unsafe accuracy rises. The interpretation that this demonstrates a lack of “balanced multicontext reasoning” is plausible but under-specified: the input format changes from multimodal perception to pure textual entailment over an If–Then formula, so the oversensitivity may partly be a prompt/format effect rather than the same integration failure diagnosed under Setting 1. Clarify the exact prompt for Setting 2, and add a control that supplies the same information as unstructured multimodal captions (not as logical predicates) to separate format bias from integration failure.
minor comments (7)
  1. Table 1: “Ground-truth predicate” columns for other benchmarks are marked ×; a short footnote clarifying that some prior work has rationales or explanations (even if not formal If–Then predicates) would avoid overstating uniqueness.
  2. §2.1 taxonomy and Table 2: fine-grained category counts are given, but the paper never reports per-fine-grained accuracy. Even a brief appendix table would help readers see whether failures concentrate in specific subcategories (e.g., misinformation vs. harassment).
  3. §3.1: GPT-4o-mini is listed among evaluated proprietary models, but Table 3 only reports Gemini-Flash-2.5 among proprietary systems. Either add GPT-4o-mini results or remove it from the evaluated list.
  4. Figure 1 and the garage/car example: the speech utterance is truncated (“Feeling so tired …”); full utterances should be shown or provided in an appendix so readers can reproduce the intended multicontext condition.
  5. §3.4.1 perception-alignment: the three-way entailment discretization (entail=1, likely entail=0.5, not entail=0) via Claude-Sonnet-4.5 is reasonable but unvalidated; a small human calibration of the scoring rubric would strengthen Fig. 7.
  6. Affiliation line: “@companyA.com” appears to be a redaction artifact; restore the correct institutional domain for camera-ready.
  7. Typos/consistency: “V oice” (Table 5), “Qwen2-Omni” vs “Qwen-Omni” naming, and “ratio-nales” line break in Fig. 7 caption should be cleaned.

Circularity Check

0 steps flagged

Empirical benchmark paper with no circular derivation: safety claims rest on model evaluations against human-refined ground truth, not on fitted parameters or self-defining constructions.

full rationale

MCBench is a dataset-and-evaluation paper. Its central claims (Omni LLMs extract modality-specific cues yet fail to integrate them for safety judgments; worse on social/illegal harm than on physical/property categories) are empirical accuracy and perception-alignment measurements on 1196 held-out scenarios. The If-Then predicates L are human-refined ground-truth rationales used only for post-hoc diagnosis (entailment scoring of reasoning traces; Setting-2 ablation that feeds predicates as input). They are not fitted parameters that the evaluated models were trained on, nor quantities the paper claims to 'predict' from a fit. Image/audio synthesis (Gemini-Flash-2.5, Stable Audio) and Claude scenario drafting are construction steps for the benchmark; they do not make any reported accuracy or alignment score equal its inputs by construction. Using Gemini both as a generator and as an evaluated model is a possible contamination confound, not a circular derivation. No uniqueness theorem, ansatz, or self-citation chain forces the results. The derivation chain is therefore self-contained against external model outputs and human labels; circularity score is 0.

Axiom & Free-Parameter Ledger

0 free parameters · 3 axioms · 1 invented entities

Benchmark construction rests on a small set of domain assumptions about what constitutes multicontext safety and on the fidelity of commercial generative models after human filtering. No free parameters are fitted to produce the central accuracy numbers; the invented taxonomy and predicate format are definitional scaffolding rather than physical entities.

axioms (3)
  • domain assumption Safety of a situation is fully determined by the conjunction of visual, acoustic and linguistic conditions expressible as an If-Then predicate.
    Stated in the problem definition (Section 2.1) and used to generate every scenario pair.
  • domain assumption Synthetic images from Gemini-Flash-2.5 and audio from Stable Audio 1.0, after human verification, are sufficiently realistic proxies for real-world multicontext scenes.
    Phase 2 generation and quality-control steps; no quantitative fidelity metric is supplied.
  • domain assumption GPT-4o as LLM-as-a-judge correctly maps free-form model responses onto the binary safe/unsafe label.
    Evaluation metric section; five runs are averaged but no human inter-annotator agreement is reported.
invented entities (1)
  • MCBench multicontext safety taxonomy (four coarse + fine-grained categories plus If-Then predicates) no independent evidence
    purpose: Organise scenarios so that safety requires cross-modal integration and enable perception-alignment diagnosis.
    Newly defined in Section 2.1 and Figure 2; no independent external validation beyond the authors’ construction process.

pith-pipeline@v1.1.0-grok45 · 19584 in / 2440 out tokens · 25034 ms · 2026-07-12T19:27:36.466083+00:00 · methodology

0 comments
read the original abstract

Existing multimodal safety benchmarks focus solely on visual inputs and cannot assess Omni Large Language Models (LLMs) that process vision, audio, and text. We introduce MCBench, a benchmark with 1196 scenarios spanning four safety categories that require integrating multiple modalities for accurate safety assessment. Each unsafe scenario is paired with a minimally different safe counterpart to assess model sensitivity. Our evaluations of state-of-the-art models reveal significant challenges. Omni LLMs struggle with subtle or non-physical risks but perform better when salient visual or acoustic cues are present. Analysis of reasoning traces shows that, although models can extract modality-specific information, they often fail to integrate these cues effectively for safety judgments. Our findings reveal that current Omni LLMs lack robust cross-modal reasoning in safety-critical settings, underscoring the need for improved architectures and training strategies for multimodal safety.

Figures

Figures reproduced from arXiv: 2606.05177 by Amar Kaur, Dinh Phung, Gholamreza Haffari, Junae Kim, Lizhen Qu, Manh Luong, Rollin Omari, Tamas Abraham, Trang Vu.

Figure 2
Figure 2. Figure 2 [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: The overall pipeline of benchmark construction which consists of two phases: the multicontext safety scenario generation and multicontext data generation. descriptions for the image and audio context, S is a speech ut￾terance, R ∈ {safe, unsafe} is the ground-truth safety label, and L is the If-Then predicate for the safety assessment. In the final step, we manually verify and refine the gener￾ated scenari… view at source ↗
Figure 4
Figure 4. Figure 4: Performance Comparison between Image-Audio Con￾text and Textual Image Alternatives. Textual Image Alternatives. We replace image context with an image description to understand the impact of visual context in safety assessment [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Performance Comparison between Image-Audio Con￾text and Textual Audio Alternatives. of audio context, we replace actual audio with textual descrip￾tions while keeping images unchanged [PITH_FULL_IMAGE:figures/full_fig_p005_5.png] view at source ↗
Figure 7
Figure 7. Figure 7: Perception alignment scores, ranging from 0 to 1, for Gemini-Flash-2.5 and Qwen-omni-2.5 (3B, 7B) across the four coarse￾grained categories in MCBench. A score of 1 indicates complete alignment between model reasoning traces and ground-truth ratio￾nales, while 0 indicates complete misalignment. provides insight into state-of-the-art proprietary model perfor￾mance. Evaluation metric for perception-alignment… view at source ↗
Figure 8
Figure 8. Figure 8: An oversensitive safety assessment example of Omni LLMs for the Illegal Harm category in MCBench. Although all three models are capable of generating ground-truth premises, they failed to combine the extracted information cues from multimodal inputs. Therefore, they make a wrong safety assessment prediction. conditions: models making safety assessments (1) using only multimodal inputs versus (2) using grou… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

47 extracted references · 6 linked inside Pith

  1. [1]

    Re- cent work has developed Omni LLMs capable of processing vi- sion, audio, and speech simultaneously [7, 8]

    Introduction Large language models (LLMs) have expanded beyond text- only processing through recent advances in multimodal train- ing data [1, 2, 3] and architectural innovations [4, 5, 6]. Re- cent work has developed Omni LLMs capable of processing vi- sion, audio, and speech simultaneously [7, 8]. These models demonstrate strong performance on complex m...

    Pith/arXiv arXiv 2026

  2. [2]

    Multimodal Multicontext Safety Benchmark 2.1. Overview Problem definition.We define the problem of multimodal multicontext safety as follows: Given a triplet of omnimodal situational dataT={I, A, S}, where I, A, and S denote an im- age, an audio clip, and a speech utterance, respectively, a model must classify the safety labelR(I, A, S)∈ {safe,unsafe} bas...

  3. [3]

    General Prompt

    Experiments 3.1. Experimental Setup Evaluated MLLMs.We evaluate both open-source and pro- prietary models on MCBench. The evaluated open-source MLLMs include the Qwen-Omni2.5 [8] (3B and 7B models), AnyGPT [19], InternOmni [20], Baichuan-Omni1.5 [21], and OmniVinci [22]. For proprietary models, we conduct experi- ments with Gemini-Flash-2.5 [7] and GPT-4o...

  4. [4]

    Related Work Multimodal Large Language Models.Recent advances in multimodal training data [1, 2, 3] and architectural innova- tions [4, 5, 6] have enabled large language models to extend be- yond text-only processing. Visual language models [29, 30, 31] can interpret images and videos alongside textual queries, while audio language models [32, 33, 34, 35]...

  5. [5]

    Conclusion We introduce a new multicontext multimodal safety bench- mark, called MCBench, to examine the safety-awareness of Omni LLMs. Unlike existing benchmarks that rely on single- modality cues, MCBench scenarios require integrating infor- mation across vision, audio, and speech to determine a safety assessment, reflecting the complexity of real-world...

  6. [6]

    All research design, methodology, anal- ysis, and scientific content are the original work of the authors, who take full responsibility for the publication

    Generative AI Use Disclosure The authors used Generative AI for grammar checking and pol- ishing the manuscript. All research design, methodology, anal- ysis, and scientific content are the original work of the authors, who take full responsibility for the publication

  7. [7]

    LAION-5b: An open large-scale dataset for training next generation image-text models,

    C. Schuhmann, R. Beaumont, R. Vencu, C. W. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, P. Schramowski, S. R. Kundurthy, K. Crowson, L. Schmidt, R. Kaczmarczyk, and J. Jitsev, “LAION-5b: An open large-scale dataset for training next generation image-text models,” inThirty-sixth Conference on Neural Information Processing S...

  8. [8]

    Available: https://openreview.net/forum?id= M3Y74vmsMcY

    [Online]. Available: https://openreview.net/forum?id= M3Y74vmsMcY

  9. [9]

    Frozen in time: A joint video and image encoder for end-to-end retrieval,

    M. Bain, A. Nagrani, G. Varol, and A. Zisserman, “Frozen in time: A joint video and image encoder for end-to-end retrieval,” inIEEE International Conference on Computer Vision, 2021

    2021

  10. [10]

    WavCaps: A ChatGPT-assisted weakly-labelled audio captioning dataset for audio-language mul- timodal research,

    X. Mei, C. Meng, H. Liu, Q. Kong, T. Ko, C. Zhao, M. D. Plumbley, Y . Zou, and W. Wang, “WavCaps: A ChatGPT-assisted weakly-labelled audio captioning dataset for audio-language mul- timodal research,”IEEE/ACM Transactions on Audio, Speech, and Language Processing, pp. 1–15, 2024

    2024

  11. [11]

    Flamingo: a visual language model for few-shot learning,

    J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y . Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, R. Ring, E. Rutherford, S. Cabi, T. Han, Z. Gong, S. Samangooei, M. Monteiro, J. Menick, S. Borgeaud, A. Brock, A. Nematzadeh, S. Sharifzadeh, M. Binkowski, R. Barreira, O. Vinyals, A. Zisserman, and K. Simonyan, “Flamingo: a visual language mod...

    2022

  12. [12]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,

    J. Li, D. Li, S. Savarese, and S. Hoi, “Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,” inInternational conference on machine learning. PMLR, 2023, pp. 19 730–19 742

    2023

  13. [13]

    AST: Audio Spectrogram Transformer,

    Y . Gong, Y .-A. Chung, and J. Glass, “AST: Audio Spectrogram Transformer,” inProc. Interspeech 2021, 2021, pp. 571–575

    2021

  14. [14]

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabil- ities,

    G. Comanici, E. Bieber, M. Schaekermann, I. Pasupat, N. Sachdeva, I. Dhillon, M. Blistein, O. Ram, D. Zhang, E. Rosen et al., “Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabil- ities,”arXiv preprint arXiv:2507.06261, 2025

    Pith/arXiv arXiv 2025

  15. [15]

    Qwen2. 5-omni technical report,

    J. Xu, Z. Guo, J. He, H. Hu, T. He, S. Bai, K. Chen, J. Wang, Y . Fan, K. Danget al., “Qwen2. 5-omni technical report,”arXiv preprint arXiv:2503.20215, 2025

    Pith/arXiv arXiv 2025

  16. [16]

    Introducing gemini 2.0: Our new ai model for the agentic era,

    S. Pichai, D. Hassabis, and K. Kavukcuoglu, “Introducing gemini 2.0: Our new ai model for the agentic era,” The Keyword (Google Blog), December 2024, accessed: 2026-01-14. [Online]. Available: https://blog.google/technology/ google-deepmind/google-gemini-ai-update-december-2024/

    2024

  17. [17]

    Multimodal medical ai,

    G. Corrado and Y . Matias, “Multimodal medical ai,” Google Research Blog, August 2023, accessed: 2026-01-14. [Online]. Available: https://research.google/blog/multimodal-medical-ai/

    2023

  18. [18]

    Multimodal situational safety,

    K. Zhou, C. Liu, X. Zhao, A. Compalas, D. Song, and X. E. Wang, “Multimodal situational safety,” inThe Thirteenth International Conference on Learning Representations, 2025. [Online]. Available: https://openreview.net/forum?id=I9bEi6LNgt

    2025

  19. [19]

    Self-adaptive dataset construction for real-world multimodal safety scenarios,

    J. Qu, L. Li, B. Zhang, Y . Yan, and J. Shao, “Self-adaptive dataset construction for real-world multimodal safety scenarios,” inFindings of the Association for Computational Linguistics: EMNLP 2025, C. Christodoulopoulos, T. Chakraborty, C. Rose, and V . Peng, Eds. Suzhou, China: Association for Computational Linguistics, Nov. 2025, pp. 16 805–16 829. [O...

    2025

  20. [20]

    Can’t see the forest for the trees: Benchmarking multimodal safety awareness for multimodal LLMs,

    W. Wang, X. Liu, K. Gao, J.-t. Huang, Y . Yuan, P. He, S. Wang, and Z. Tu, “Can’t see the forest for the trees: Benchmarking multimodal safety awareness for multimodal LLMs,” inProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar, Eds. Vienna, Au...

    2025

  21. [21]

    Omnibench: Towards the future of universal omni-language models,

    Y . LI, G. Zhang, Y . Ma, R. Yuan, K. Zhu, H. Guo, Y . Liang, J. Liu, Z. M. Wang, J. Yang, S. Wu, X. Qu, J. Shi, X. Zhang, Z. Yang, Y . WEN, Y . Wang, S. Li, Z. Zhang, R. Liu, E. Benetos, W. Huang, and C. Lin, “Omnibench: Towards the future of universal omni-language models,” inThe Thirty-ninth Annual Conference on Neural Information Processing Systems Da...

    2025

  22. [22]

    SA VVY: Spatial awareness via audio- visual LLMs through seeing and hearing,

    M. Chen, Z. Cui, X. Liu, J. Xiang, C. Zheng, J. Li, and E. Shlizerman, “SA VVY: Spatial awareness via audio- visual LLMs through seeing and hearing,” inThe Thirty-ninth Annual Conference on Neural Information Processing Systems,

  23. [23]

    Available: https://openreview.net/forum?id= zwCb9cKHpd

    [Online]. Available: https://openreview.net/forum?id= zwCb9cKHpd

  24. [24]

    Is your multimodal language model oversensitive to safe queries?

    X. Li, H. Zhou, R. Wang, T. Zhou, M. Cheng, and C.-J. Hsieh, “Is your multimodal language model oversensitive to safe queries?” inThe Thirteenth International Conference on Learning Representations, 2025. [Online]. Available: https: //openreview.net/forum?id=QsA3YzNUxA

    2025

  25. [25]

    Claude 4.5,

    Anthropic, “Claude 4.5,” 2025. [Online]. Available: https: //www.anthropic.com/news/claude-sonnet-4-5

    2025

  26. [26]

    Stable audio open,

    Z. Evans, J. D. Parker, C. Carr, Z. Zukowski, J. Taylor, and J. Pons, “Stable audio open,” inICASSP 2025-2025 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2025, pp. 1–5

    2025

  27. [27]

    Anygpt: Unified multimodal llm with discrete sequence modeling,

    J. Zhan, J. Dai, J. Ye, Y . Zhou, D. Zhang, Z. Liu, X. Zhang, R. Yuan, G. Zhang, L. Liet al., “Anygpt: Unified multimodal llm with discrete sequence modeling,” inProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2024, pp. 9637–9662

    2024

  28. [28]

    Internomni: Extending internvl with audio modality,

    L. Chenet al., “Internomni: Extending internvl with audio modality,”CoRR, vol. abs/2409.18042, 2024. [Online]. Available: arxiv.org

    Pith/arXiv arXiv 2024

  29. [29]

    Baichuan-omni-1.5 technical report,

    Y . Li, J. Liu, T. Zhang, S. Chen, T. Li, Z. Li, L. Liu, L. Ming, G. Dong, D. Panet al., “Baichuan-omni-1.5 technical report,” arXiv preprint arXiv:2501.15368, 2025

    arXiv 2025

  30. [30]

    Omnivinci: Enhancing architecture and data for omni-modal understanding llm,

    H. Ye, C.-H. H. Yang, A. Goel, W. Huang, L. Zhu, Y . Su, S. Lin, A.-C. Cheng, Z. Wan, J. Tianet al., “Omnivinci: Enhancing architecture and data for omni-modal understanding llm,”arXiv preprint arXiv:2510.15870, 2025

    arXiv 2025

  31. [31]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Ale- man, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkatet al., “Gpt-4 technical report,”arXiv preprint arXiv:2303.08774, 2023

    Pith/arXiv arXiv 2023

  32. [32]

    Judging LLM-as-a-judge with MT- bench and chatbot arena,

    L. Zheng, W.-L. Chiang, Y . Sheng, S. Zhuang, Z. Wu, Y . Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica, “Judging LLM-as-a-judge with MT- bench and chatbot arena,” inThirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track,

  33. [33]

    Available: https://openreview.net/forum?id= uccHPGDlao

    [Online]. Available: https://openreview.net/forum?id= uccHPGDlao

  34. [34]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V . Le, D. Zhouet al., “Chain-of-thought prompting elicits reasoning in large language models,”Advances in neural information pro- cessing systems, vol. 35, pp. 24 824–24 837, 2022

    2022

  35. [35]

    CASE- Bench: Context-Aware SafEty Benchmark for Large Language Models,

    G. Sun, X. Zhan, S. Feng, P. C. Woodland, and J. Such, “CASE- Bench: Context-Aware SafEty Benchmark for Large Language Models,” inICML, 2025

    2025

  36. [36]

    XSTest: A test suite for identifying exaggerated safety behaviours in large language models,

    P. R ¨ottger, H. Kirk, B. Vidgen, G. Attanasio, F. Bianchi, and D. Hovy, “XSTest: A test suite for identifying exaggerated safety behaviours in large language models,” inProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), K. Duh, H. Gomez, a...

    2024

  37. [37]

    OR-bench: An over-refusal benchmark for large language models,

    J. Cui, W.-L. Chiang, I. Stoica, and C.-J. Hsieh, “OR-bench: An over-refusal benchmark for large language models,” in Proceedings of the 42nd International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, A. Singh, M. Fazel, D. Hsu, S. Lacoste-Julien, F. Berkenkamp, T. Maharaj, K. Wagstaff, and J. Zhu, Eds., vol. 267. PMLR, 1...

    2025

  38. [38]

    Visual instruction tuning,

    H. Liu, C. Li, Q. Wu, and Y . J. Lee, “Visual instruction tuning,” inThirty-seventh Conference on Neural Information Processing Systems, 2023. [Online]. Available: https://openreview.net/ forum?id=w0H2xGHlkw

    2023

  39. [39]

    Video-xl: Extra-long vision language model for hour-scale video understanding,

    Y . Shu, Z. Liu, P. Zhang, M. Qin, J. Zhou, Z. Liang, T. Huang, and B. Zhao, “Video-xl: Extra-long vision language model for hour-scale video understanding,” inProceedings of the Computer Vision and Pattern Recognition Conference, 2025, pp. 26 160– 26 169

    2025

  40. [40]

    Internvl: Scaling up vision foun- dation models and aligning for generic visual-linguistic tasks,

    Z. Chen, J. Wu, W. Wang, W. Su, G. Chen, S. Xing, M. Zhong, Q. Zhang, X. Zhu, L. Luet al., “Internvl: Scaling up vision foun- dation models and aligning for generic visual-linguistic tasks,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2024, pp. 24 185–24 198

    2024

  41. [41]

    Gama: A large audio-language model with advanced audio understanding and complex reasoning abilities,

    S. Ghosh, S. Kumar, A. Seth, C. K. R. Evuru, U. Tyagi, S. Sak- shi, O. Nieto, R. Duraiswami, and D. Manocha, “Gama: A large audio-language model with advanced audio understanding and complex reasoning abilities,”arXiv preprint arXiv:2406.11768, 2024

    Pith/arXiv arXiv 2024

  42. [42]

    Unbiased sliced wasserstein kernels for high-quality audio captioning,

    M. Luong, K. Nguyen, D. Phung, G. Haffari, and L. Qu, “Unbiased sliced wasserstein kernels for high-quality audio captioning,” inThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. [Online]. Available: https://openreview.net/forum?id=83HlJrZigZ

    2025

  43. [43]

    Audio flamingo 3: Advancing audio intelligence with fully open large audio language models,

    S. Ghosh, A. Goel, J. Kim, S. Kumar, Z. Kong, S. gil Lee, C.-H. H. Yang, R. Duraiswami, D. Manocha, R. Valle, and B. Catanzaro, “Audio flamingo 3: Advancing audio intelligence with fully open large audio language models,” inThe Thirty-ninth Annual Conference on Neural Information Processing Systems,

  44. [44]

    Available: https://openreview.net/forum?id= FjByDpDVIO

    [Online]. Available: https://openreview.net/forum?id= FjByDpDVIO

  45. [45]

    Listen, think, and understand,

    Y . Gong, H. Luo, A. H. Liu, L. Karlinsky, and J. R. Glass, “Listen, think, and understand,” inThe Twelfth International Conference on Learning Representations, 2024. [Online]. Available: https://openreview.net/forum?id=nBZBPXdJlC

    2024

  46. [46]

    Onellm: One framework to align all modali- ties with language,

    J. Han, K. Gong, Y . Zhang, J. Wang, K. Zhang, D. Lin, Y . Qiao, P. Gao, and X. Yue, “Onellm: One framework to align all modali- ties with language,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024

    2024

  47. [47]

    SORRY-bench: Systematically evaluating large language model safety refusal,

    T. Xie, X. Qi, Y . Zeng, Y . Huang, U. M. Sehwag, K. Huang, L. He, B. Wei, D. Li, Y . Sheng, R. Jia, B. Li, K. Li, D. Chen, P. Henderson, and P. Mittal, “SORRY-bench: Systematically evaluating large language model safety refusal,” inThe Thirteenth International Conference on Learning Representations, 2025. [Online]. Available: https: //openreview.net/foru...

    2025