{"id":"03cd1aae-5446-40d4-8db5-5b258ac18d00","arxiv_id":"2505.19598","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Five leading audio-language models show substantial, scenario-dependent vulnerability to injected audio instructions that can override user requests and sway evaluations.","lead":"Large audio-language AI models that listen to user uploads can be tricked by hidden voice commands mixed into the audio. This study tests five leading models and finds that all can be manipulated, with outcomes depending on the attack type and the position of the malicious audio.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The abstract-level correlation claims (instruction-following negatively, safety positively correlated with robustness) rest on only five models with no reported ability scores, no significance tests, and no jackknife; a single model can flip the trend.","rationale":"I read the paper as an empirical benchmark rather than a formal proof. The qualitative finding—audio injections can reliably flip outputs, such as a toilet flush changing 'office' to 'bathroom' or an injected 'benchmark for score 5' turning a rating of 1 into 5—is concrete and credible, and the reported human/LLM agreement rates (AR > 0.8, PCC > 0.7) give some support to the automatic scoring, although only on 100 sampled pairs per task. However, the paper's most generalizable and novel claims are the ability-robustness correlations in Section 4.2. Those claims depend on five data points per attack type, text-based capability benchmarks validated for LLMs rather than LALMs, and no inferential statistics. This is not an internal inconsistency; it is an unsupported generalization that the reader's conditional verdict already flags. The concrete test I propose—recomputing correlations with reported ability scores, permutation tests, and jackknife—would settle whether the correlations are real or artifacts of small n. If the correlations fail, the abstract and conclusion should be softened to describe trends observed only in the five evaluated models. Because the reader's verdict already conditions acceptance on addressing these concerns, I do not move the verdict; it remains CONDITIONAL/UNCHANGED.","tokens_in":12820,"tokens_out":4440,"duration_ms":48658,"concrete_test":"Obtain from the authors the per-model IFEval, BBH, and AdvBench scores and the raw per-scenario DSR/CRS/JRI values behind Figure 3. Recompute Pearson and Spearman correlations with 95% bootstrap confidence intervals and a permutation test under H0: rho=0, then perform leave-one-model-out jackknife. If removing any single model flips the sign of the instruction-following or safety correlation, or if the permutation p-value exceeds 0.05, the paper should downgrade these correlational claims to observations about five specific models rather than general relationships. As a second check, compare the text-based capability scores against a small audio instruction-following probe (e.g., an AIR-Bench audio subset) to test whether the text benchmark is the relevant capability measure for LALMs.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing claim is the abstract and conclusion assertion that robustness correlates negatively with instruction-following capability and positively with safety alignment (Sections 1 and 4.2, Figure 3). This claim is derived from scatter plots with exactly five models per attack type; no correlation coefficients, p-values, confidence intervals, or per-model IFEval/BBH/AdvBench scores are reported anywhere in the paper. With n=5, both Pearson and Spearman correlations are highly unstable: removing one model (e.g., GPT-4o-audio or Qwen-omni-turbo) can change the sign or magnitude of the trend, and no permutation or leave-one-out analysis is provided. In addition, the 'ability' values come from text-based LLM benchmarks applied to audio-language models; it is not established that these measure the same capability in the audio modality, and they are confounded with model family, parameter count, and safety-training status. Because the paper's most generalizable conclusions—'enhancing functional capabilities can introduce vulnerabilities' and 'safety alignment helps'—rest on these correlations, the empirical basis is currently too thin. The core qualitative finding (injections can flip outputs; no model is robust everywhere) is better supported by concrete examples and DSR values, so this objection targets the correlational generalization rather than the existence of audio injection vulnerabilities.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents an empirical study of audio injection robustness for five large audio-language models (GPT-4o-audio-preview, Qwen-omni-turbo, Qwen2-audio, Salmonn-7B, and Phi-4-multimodal) under four attack scenarios: Audio Interference, Instruction Following, Context Injection, and Judgment Hijacking. Robustness is quantified with three proposed metrics: Defense Success Rate, Context Robustness Score, and Judgment Robustness Index. The main findings are that no model is robust across all attack types, that the position of the injected audio matters, that speed and volume have scenario-dependent effects, that system prompts provide inconsistent protection, and that robustness appears negatively correlated with instruction-following capability and positively correlated with safety alignment. The paper also includes a limitations section acknowledging the use of TTS-synthesized audio and only five models.","tokens_in":13250,"tokens_out":5671,"duration_ms":55683,"significance":"The attack taxonomy and the concrete qualitative demonstrations in Appendix A are useful and likely to be of interest to the security and multimodal LLM communities. If the quantitative framework is made reproducible, the benchmark could be a solid contribution: this appears to be one of the first systematic studies of audio injection attacks against LALMs. The qualitative result that injected audio can flip model outputs is supported by the examples and by the reported DSR ranges. Credit is due for including an explicit limitations section and for reporting a human/LLM agreement check. However, the broader generalization that capability and safety alignment are systematically correlated with robustness rests on a five-model scatter plot with no reported statistics, and several metric definitions are incomplete. These issues need to be addressed before the paper's central claims can be accepted.","major_comments":[{"comment":"The Context Robustness Score formula is not well-formed as printed: \"s = ω1 · sh + ω2 · sc sM\" lacks the necessary parentheses, and the values of the weights ω1 and ω2 are never stated. Since CRS is the sole quantitative evidence for the Context Injection Attack results in Figures 2 and 4, the authors must specify the exact calculation and the weight values, or the metric is not reproducible.","section":"Section 3.3, Eq. (2)"},{"comment":"The Judgment Robustness Index depends on a Bonferroni-corrected p-value, p_corr,A, but the underlying statistical test, sample size, and comparison groups are not described anywhere in the paper. The constant ε is arbitrary, and for the audio quality judgment task the definition of μ_N is unclear. Without these details, the JHA results cannot be reproduced or meaningfully compared across models.","section":"Section 3.3, Eq. (4)"},{"comment":"The claims of a negative correlation between instruction-following ability and robustness, and a positive correlation between safety alignment and robustness, are based on scatter plots with exactly five points per attack type. No correlation coefficients, confidence intervals, p-values, or per-model IFEval/BBH/AdvBench scores are reported, so a single model can drive or invert the apparent trend. The authors should either provide the underlying per-model data with leave-one-out or permutation analyses, or soften the conclusion to describe these as observational hypotheses rather than established correlations.","section":"Section 4.2, Figure 3, and Conclusion"},{"comment":"The trustworthiness evaluation uses 100 randomly sampled question-answer pairs per task, but it is unclear whether the DSR and CRS values in the main experiments are computed from those 100 samples or from the full 10,000 pairs, and whether the CRS scores used in Eq. (2) come from human raters or from the Gemini 2.5 Flash judge. The paper must state the evaluation pipeline explicitly, including the number of samples behind every reported number in Figures 2, 4, and 5.","section":"Section 4.5"}],"minor_comments":[{"comment":"DSR is defined as u/v × 100%, but the text says that \"the closer DSR approaches 1, the higher the robustness\"; please use a 0-1 scale or percent consistently throughout.","section":"Section 3.3, Eq. (1)"},{"comment":"The scoring method in Appendix D is internally inconsistent: each dimension is scored 0-2, averaging gives a 0-2 total, but the text claims a 0-5 range and uses result bands 0-4, 5-8, and 9-12, which imply a sum over six dimensions. Please correct the scoring description.","section":"Appendix D"},{"comment":"The axes in Figure 3 are not labeled with the actual ability scores, and it is not stated how CRS and JRI are scaled to align with DSR; without this information, the scatter plots are difficult to interpret.","section":"Figure 3"},{"comment":"Several dataset access dates are listed as September 8, 2025, while the arXiv version is dated July 10, 2025; these dates should be corrected.","section":"References and dataset access dates"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the journal's scope and addresses a timely topic. For a benchmark paper, the absence of released code, data, and evaluation scripts is a significant replicability concern; I would encourage the editor to ask for these as part of the revision. The four attack scenarios are somewhat ad hoc, and situating them within the broader adversarial-audio literature would strengthen the contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The first thing to know: this is the first systematic look at audio injection attacks on LALMs, and the basic finding holds up. Crafted audio in user uploads can flip outputs — office becomes bathroom, a fearful voice becomes a 5, a competent engineer gets judged by gender stereotypes. The four-attack taxonomy is sensible and probably reusable. I also credit the authors for checking their evaluation against human judgment (AR >0.8, PCC >0.7) and for stating plainly that the audio is mostly TTS-synthetic and limited to five models.\n\nThe soft spots are real but localized. The correlations in the abstract — instruction-following hurts robustness, safety alignment helps — come from scatter plots over exactly five models. No ability scores, no correlation coefficients, no significance tests. A single model can plausibly flip the trend. That is not enough support for the generalized claim that 'enhancing functional capabilities can introduce vulnerabilities.' It is a hypothesis, not a result.\n\nSecond, the metric formulas are under-specified. CRS in Eq. 2 has weights ω1 and ω2 that are never defined. JRI's ε is chosen as 0.01 without sensitivity analysis. That makes reproduction impossible until the appendix is cleaned up. No code or data release either. The human-evaluation sample size (100 pairs) is fine for agreement checking but not for strong claims.\n\nThird, the positional result (front is worse) is plausible and supported by the examples; the speed/volume/jailbreak analyses are exploratory and modestly useful.\n\nOn balance: the qualitative core is solid enough to deserve a serious referee; the correlational section needs a major rewrite or removal, and the formulas need full specification. Since this is a benchmark paper, releasing the evaluation harness would substantially increase its value.\n\nWho is it for? Anyone working on multimodal security, especially audio. It establishes the vocabulary for a new attack surface. I would take it to a reading group, mainly to discuss what counts as evidence for a robustness benchmark, and I would cite it as the first systematic audio injection study, with the caveat on the correlations.\n\nRecommendation: send to peer review. A good referee will ask for the missing numbers, the weights, and the artifact, but the topic and the core demonstrations are worth referee time.","headline":"First systematic audio-injection benchmark for LALMs — qualitative finding holds, but the capability correlations rest on five points and need a rewrite.","tokens_in":13628,"tokens_out":2540,"would_cite":true,"duration_ms":14129,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that large audio-language models are vulnerable to four audio injection attacks, that none of the five tested models resists all of them, and that stronger instruction-following predicts weaker resistance while stronger…","keywords":["large audio-language models","audio injection attack","prompt injection","robustness benchmark","defense success rate","instruction following","safety alignment","multi-modal security"],"falsifier":"Re-run the four attack scenarios on a sixth LALM with known high instruction-following ability; the paper's negative-correlation claim predicts it should be among the least robust, so a single model with both high instruction-following and near-perfect DSR, CRS, and JRI across all attacks would directly contradict the claimed trade-off.","tokens_in":12643,"feed_emoji":"🎧","tokens_out":9993,"duration_ms":71739,"temperature":0.7,"pith_summary":"The paper argues that large audio-language models (LALMs) can be hijacked by malicious audio hidden inside a user's upload, and that this is a systematic weakness, not an edge case. Evaluating five leading LALMs against four attack types, it finds no model that is robust across all attacks, and it reports that the position, speed, and volume of the injected audio change how much damage it does. The paper also reports a trade-off: models that follow instructions more faithfully tend to be less robust, while models with stronger safety alignment tend to be more robust. This matters because LALMs are already deployed in applications that accept user audio, so an injection could produce biased, incorrect, or attacker-controlled responses from a system users trust.","feed_headline":"No voice-AI model resists all four audio injection attacks","feed_subtitle":"A five-model benchmark ties weak defenses to strong instruction-following and strong defenses to safety alignment.","key_machinery":"The benchmark framework itself is the load-bearing machinery. It defines four attack scenarios: Audio Interference Attack (irrelevant sounds that corrupt understanding), Instruction Following Attack (a competing spoken instruction), Context Injection Attack (harmful contextual claims such as gender stereotypes), and Judgment Hijacking Attack (a fake reference that re-anchors scoring). Robustness is scored by three self-defined metrics: Defense Success Rate (fraction of failed attacks), Context Robustness Score (a function of the mean and variance shift in harmlessness-and-consistency scores under injection), and Judgment Robustness Index (a normalized ratio of rating changes with a statistical-significance penalty). These metrics are computed from model outputs graded by an LLM judge checked against human raters, and the attack suite is run with variations in injection position, playback speed, volume, jailbreak speech, and defensive system prompts.","core_discovery":"Across four attack scenarios (audio interference, instruction following, context injection, and judgment hijacking) and three quantitative metrics, every one of the five evaluated models showed at least one attack type that degraded its responses, and no model defended all four evenly. The strongest reported pattern is a negative correlation between instruction-following ability and robustness, most pronounced under instruction-following attacks, and a consistently positive correlation between safety alignment and robustness across all scenarios. The paper also finds that malicious audio placed at the start of the input is often more effective, that faster playback usually increases robustness while louder injection often decreases it, and that defensive system prompts produce mixed, scenario-specific gains.","pith_inferences":["The reported negative correlation suggests a cheap screening test: a model's instruction-following score may predict its likely audio-injection vulnerability before a full attack suite is run.","Because the benchmark relies mostly on text-to-speech audio, a natural-speech version with real background noise and accents would be a direct test of whether the vulnerability pattern generalizes beyond synthetic inputs.","The judgment-hijacking results imply a broader risk: any LALM used as an automated judge (for moderation, hiring, or scoring) can have its criteria re-anchored by a single injected reference audio.","The three metrics could be repurposed as a continuous robustness objective for training, connecting this benchmark to future defense work."],"forward_implications":["Deployed LALMs that accept user-uploaded audio can be steered to answer attacker-chosen questions or output attacker-chosen judgments.","Instruction-following strength is a vulnerability signal: models optimized for obedience may need explicit training to distinguish the user's intent from injected instructions.","Safety alignment appears to double as injection defense, so robustness should be measured alongside alignment rather than treated as a separate goal.","Defensive system prompts are an unreliable mitigation on their own; defenses must live in the architecture or training data.","Evaluation of audio-model security should treat injection position, speed, and volume as first-class variables, not confounds."],"supporting_citations":[{"why":"Qwen2-Audio technical report; one of the five evaluated open-source models and the reference for its architecture.","marker":"Chu et al., 2024"},{"why":"Salmonn; another evaluated open-source LALM and source of the LoRA-based adapter design.","marker":"Tang et al., 2023"},{"why":"Phi-4-multimodal; the third open-source model evaluated.","marker":"Abouelenin et al., 2025"},{"why":"GPT-4o system card; the proprietary model whose audio interface is attacked.","marker":"Hurst et al., 2024"},{"why":"ESC50; the environmental-sound dataset used as malicious audio in Audio Interference Attack.","marker":"DynamicSuperb, 2024b"},{"why":"RAVDESS; the emotion-labeled audio dataset used in Judgment Hijacking (emotion task).","marker":"DynamicSuperb, 2024a"},{"why":"Content-Articles; the paper-abstract collection that supplies benign and malicious audio for Instruction Following and audio quality judgment.","marker":"Sakthi, 2025"},{"why":"IFEval; the instruction-following benchmark whose scores anchor the negative-correlation finding.","marker":"Kovalevskyi, 2024"},{"why":"BBH; the reasoning benchmark used to derive the reasoning-robustness correlations.","marker":"Suzgun et al., 2022"},{"why":"AdvBench; the safety benchmark behind the positive safety-robustness correlation.","marker":"Chen et al., 2022"}],"fun_headline_variants":["No voice AI resists all four audio injection attacks","Instruction-following voice AIs more vulnerable to injection","Safety alignment boosts voice AI resistance to injection","Audio injection at start fools voice AIs more easily","Five voice AIs, no full defense against audio injection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire study rests on the assumption that the four attack scenarios and the three new metrics (DSR, CRS, JRI) genuinely capture how robust a large audio-language model is, and that results from five models on mostly synthetic, text-to-speech audio carry over to real-world attacks.","fun_headline_variants_meta":{"raw":{"variants":["No voice AI resists all four audio injection attacks","Instruction-following voice AIs more vulnerable to injection","Safety alignment boosts voice AI resistance to injection","Audio injection at start fools voice AIs more easily","Five voice AIs, no full defense against audio injection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000142,"raw_usage":{"total_tokens":1129,"prompt_tokens":864,"completion_tokens":265,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":480,"completion_tokens_details":{"reasoning_tokens":189}},"tokens_in":480,"tokens_out":265,"duration_ms":2949,"temperature":1.0,"reasoning_tokens":189,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:10:21.232924+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the four attack scenarios on a sixth LALM with known high instruction-following ability; the paper's negative-correlation claim predicts it should be among the least robust, so a single model with both high instruction-following and near-perfect DSR, CRS, and JRI across all attacks would directly contradict the claimed trade-off.","supporting_citations":[],"review_version":1}