{"id":"20ad8992-d725-4a0a-b896-0713835f228d","arxiv_id":"2505.18864","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"By appending optimized token sequences to harmful speech, the authors achieve up to 89% attack success rate on SpeechGPT across six forbidden categories.","lead":"This paper shows that voice-enabled AI assistants can be tricked into breaking their safety rules by adding specially crafted noise to the audio. The attack exploits the way models convert speech into abstract tokens, and the authors report success rates up to 89 percent on forbidden questions.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The token-level mechanism is unvalidated: Algorithm 2's cluster-matching loss is the same objective being optimized, so the synthesized audio may not actually deliver the intended token sequence to SpeechGPT.","rationale":"The reader's weakest_assumption already identifies the audio-to-token reconstruction as load-bearing, and I agree. The greedy token search in Algorithm 1 operates entirely in token space; the only bridge from those tokens back to actual model input is Algorithm 2's vocoder synthesis plus noise optimization. That bridge is validated solely by the reverse loss, which is the same objective being minimized. Because SpeechGPT's input pipeline may differ from the HuBERT cluster model used in Algorithm 2, or because vocoder artifacts can shift cluster assignments, the final audio may not carry the intended token sequence. Without an independent verification, the high ASR could be an artifact of generic adversarial noise—the random-noise baseline already reaches 0.83 ASR. The paper's evaluation weaknesses (manual labeling, small sample, the 89% vs. 93% inconsistency) further support caution, but the missing reconstruction validation is the most direct threat to the paper's central mechanistic claim. A conditional verdict remains appropriate: the paper should not be treated as definitively establishing a token-level attack until this check is performed, but the empirical phenomenon of high ASR on SpeechGPT is still noteworthy and does not warrant rejection on this basis alone.","tokens_in":9810,"tokens_out":5748,"duration_ms":59301,"concrete_test":"Run an independent transcription/tokenization of the final attack audio (e.g., Whisper large-v3 ASR, or a separately trained HuBERT k-means checkpoint not used in Algorithm 2) and compare the result to the intended harmful prompt plus adversarial tokens. If the independent transcript does not preserve the harmful content, or the token error rate relative to the target sequence is high, then the reconstructed audio does not reliably deliver the crafted token sequence, and the reported ASR cannot be attributed to the token-level attack. A stronger variant: ask SpeechGPT itself to repeat or transcribe the audio and compare that output to the intended prompt.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that a greedy token-level search (Algorithm 1) produces adversarial token sequences that bypass SpeechGPT's alignment—depends on the audio reconstructed in Algorithm 2 actually reaching the model as the intended token sequence. Algorithm 2 optimizes a global noise perturbation to minimize L = D(hat y, y), the cross-entropy between HuBERT cluster predictions on the perturbed audio and the target cluster sequence. This is a cycle-consistency objective, not an independent validation: the same HuBERT model is used both to craft the noise and to score it, and the only reported check is the 'reverse loss' in Figure 4, which is exactly the loss being minimized. If the final waveform, after vocoder synthesis and noise optimization, maps to different tokens when processed by SpeechGPT's actual discrete unit extractor, then the 89% ASR in Table II may be caused by generic audio artifacts rather than the crafted token sequence. This is load-bearing because the paper's novelty is the token-level white-box mechanism; the random-noise baseline already attains 0.83 ASR, so the claim that semantic adversarial audio 'significantly outperforms' baselines requires that the semantic content actually survives the reconstruction.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a white-box, token-level adversarial attack against the speech input of SpeechGPT, a multimodal LLM that represents audio via discrete HuBERT units. The attack pipeline (Algorithm 1) appends a short adversarial token sequence to the tokens of a harmful speech prompt and optimizes that sequence greedily by querying a scalar loss between the model output and a target response, without using gradients. Algorithm 2 then reconstructs a waveform from the optimized token sequence with a vocoder and adds a global noise perturbation, optimized to make the HuBERT cluster predictions on the reconstructed audio match the intended token sequence. The authors evaluate the attack on six forbidden-policy categories (illegal activity, hate speech, physical harm, fraud, pornography, privacy violation) using ten hand-picked questions per category, comparing against random noise, harmful speech, and two prior voice-jailbreak baselines. They report an average attack success rate (ASR) of 0.89, claim a significant advantage over baselines, and include additional experiments on audio quality (NISQA), noise budget, voice variation, and optimization iterations.","tokens_in":10068,"tokens_out":3212,"duration_ms":30462,"significance":"If the central claim holds, the paper would be one of the first systematic white-box studies of audio-token-level jailbreaks against speech-enabled LLMs, and it would demonstrate that alignment trained primarily on text can be bypassed through discrete audio perturbations without access to model gradients. The paper has several concrete strengths: it targets a timely and underexplored attack surface (voice inputs in MLLMs); it provides a completely automated, non-interactive attack pipeline; it releases code; it compares against reasonable baselines (random noise, direct harmful speech, and two prior voice-jailbreak approaches); and it examines auxiliary factors such as noise budget, voice identity, and required iterations. The significance is, however, conditional on two things that the manuscript does not yet establish: that the reconstructed audio actually reaches the model as the intended token sequence (rather than as generic noise artifacts), and that the reported performance differences are statistically meaningful despite the small manual evaluation set. If those points are resolved, the work would be a useful contribution to the security evaluation of multimodal LLMs.","major_comments":[{"comment":"The reported success rates are internally inconsistent: the abstract and Section IV-B state that the method achieves 'up to 89%' attack success rate, yet Figure 4 states 'achieving over 93% ASR at a noise level of 0.1' for the semantic audio method. Since Table II reports an average ASR of 0.89 for the same method, the reader cannot tell whether Figure 4 uses a different experimental setting (e.g., a different subset of questions or a different noise budget) or whether the numbers are simply irreconcilable. Please clarify the exact experimental configuration behind Figure 4 and ensure all reported ASR values are consistent across the abstract, Section IV-B, and the figures.","section":"Section IV-B and Figure 4"},{"comment":"The validation that the synthesized audio actually delivers the intended adversarial token sequence to SpeechGPT is circular. Algorithm 2 optimizes a global noise perturbation to minimize L = D(ŷ, y), where ŷ is the HuBERT cluster prediction on the perturbed waveform and y is the target cluster sequence. The only reported check, the 'reverse loss' in Figure 4, is exactly this same objective. Since the same HuBERT model is used both to craft the perturbation and to score it, the optimization can succeed even if the actual discrete unit extractor used by SpeechGPT maps the waveform to different tokens. This is load-bearing because the paper's novelty is the token-level mechanism, and the random-noise baseline already achieves 0.83 ASR; the claimed advantage of semantic adversarial audio requires evidence that the semantic content survives reconstruction. Please provide an independent verification, for example by feeding the final waveform through SpeechGPT's own discrete unit extractor and reporting token-level agreement, or by showing that the attack fails when the token sequence is not preserved.","section":"Section III-C, Algorithm 2"},{"comment":"The evaluation is too thin to support the claim that the proposed method 'significantly outperforms' baselines. Only ten questions per category (60 total) are used, and the success criteria are not defined; the paper says the evaluation is 'primarily manual' but provides no rubric, no inter-annotator agreement, and no confidence intervals. With 60 samples, the difference between an ASR of 0.89 and the random-noise baseline of 0.83 corresponds to just four additional successes, which is within the range of random fluctuation. Please report per-sample success/failure data, define the exact judging criterion, and provide confidence intervals or a statistical test (e.g., a binomial test or paired bootstrap) to demonstrate that the improvement over random noise is significant.","section":"Section IV-A, Table II"},{"comment":"The threat model and Algorithm 1 rely on a loss function L(Model(xtemp), y_target) that is never defined. The paper states that the adversary can 'observe the scalar loss value associated with a target decoding,' but it does not specify how this loss is computed for SpeechGPT's text output — for instance, whether it is the cross-entropy of the target response under the model's output distribution, and how the target response is constructed for each forbidden question. Without this definition, the optimization procedure in Algorithm 1 is not reproducible. Please specify the loss function, the target response construction, and the values of n (adversarial token length) and k (candidate sampling size) used in the experiments.","section":"Section II-C and Algorithm 1"}],"minor_comments":[{"comment":"The abstract contains grammatical fragments: 'which effectively bypass alignment safeguards and to induce prohibited outputs' and 'and to help guide the development' should be revised.","section":"Abstract and Section I"},{"comment":"The column header reads 'Violence' while the text and Table III use 'Privacy Violence' or 'Privacy Violation'; please standardize the category name.","section":"Table II"},{"comment":"The sentence 'For Voice Jailbreak [9], Plot [9] improves ASR over Harmful Speech through prompt design' is unclear and likely missing a verb or comparison; please rewrite.","section":"Section IV-B"},{"comment":"Figure 3 shows NISQA scores but the figure caption is incomplete; it states 'NISQA Score Comparison of Adversarial Speech for Jailbreak Attacks' and does not explain what the boxes/whiskers represent or how many samples per question.","section":"Section IV-B and Figure 3"},{"comment":"The paper says ten questions were 'randomly selected' from ForbiddenQuestionSet, but the selection method is not described (e.g., were they uniformly sampled, manually curated to avoid trivial cases?). Please state the selection procedure and whether the same questions were used across all methods.","section":"Section IV, Dataset Setup"},{"comment":"Several hyperparameters are introduced but not reported, including the number of optimization iterations T in Algorithm 2, the learning rate for the noise update, and the exact adversarial token length n (the experiments mention n=200 for one table, but it is unclear if this is used everywhere).","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper was accepted for a DSN 2025 workshop, and the topic is timely. My main concern is the circular validation of the token-level mechanism; if the authors cannot provide an independent measure that the reconstructed audio maps to the intended tokens, the central novelty collapses into a generic noise-perturbation result. The small manual evaluation and the inconsistency between the 89% and 93% figures are also blocking issues. I believe these are fixable within the manuscript's scope, so I recommend major revision rather than rejection, but the revision must include a genuine validation of the audio-to-token pipeline."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this paper. The genuinely new thing is the attack setup: white-box, token-level, no gradients, greedy search over HuBERT discrete tokens to append an adversarial suffix to harmful speech tokens, then synthesize and add noise. That specific pipeline doesn't appear in the prior voice jailbreak work I know, which is mostly black-box or text-to-speech. The code is public, the pipeline is described in enough detail to reproduce, and they are careful to state that the adversary only needs the loss oracle, not the weights. Credit where due: if the claimed attack success rate is real, this matters for voice-enabled MLLMs.\n\nBut I would not take 89% at face value. The evaluation has three soft spots. First, the numbers don't reconcile: Table II says the average is 0.89, while Figure 4 claims \"over 93% ASR at a noise level of 0.1.\" That could be a different noise budget setting, but the paper does not explain it and it reads as inconsistent. Second, the manual labeling has no rubric, no inter-annotator agreement, no confidence intervals, and only 10 questions per category. With a random-noise baseline at 0.83, the claimed marginal advantage of semantic adversarial audio (0.89) is within a range that a small-sample manual eval cannot reliably resolve. Third, the stress-test point is fair: Algorithm 2 optimizes waveform noise so that HuBERT cluster predictions match the target token sequence, and the only check is the \"reverse loss\"—the same objective being optimized. Since SpeechGPT supposedly uses the same HuBERT, this is not necessarily fatal, but an independent check, such as re-tokenizing the final waveform with the actual speech encoder and comparing token sequences, or measuring ASR conditioned on observed tokens, would be the load-bearing missing experiment.\n\nThe threat model's loss-oracle assumption is strong, but they state it clearly in Section II-C, so it is not a hidden flaw.\n\nVerdict: plausible first result, weak evidence. The mechanism is new and the code is out, so it deserves a serious referee to force a better evaluation rather than a desk reject. The realistic contribution is \"here is a concrete white-box attack surface and a candidate search method,\" not \"semantic audio beats random noise by 6 points.\" If I cite it, I would cite it as the first token-level audio jailbreak, not as a validated effectiveness number. Recommendation: send to peer review with a request for an independent token-delivery check, a proper annotation protocol, and a clear reconciliation of the 89% and 93% numbers.","headline":"First white-box token-level audio jailbreak against SpeechGPT, but the evaluation is too loose to say how much of the 89% is the mechanism versus generic audio artifacts.","tokens_in":10587,"tokens_out":2022,"would_cite":true,"duration_ms":19189,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A white-box, token-level adversarial audio attack jailbreaks SpeechGPT in up to 89% of attempts across six policy-violating categories.","keywords":["audio jailbreak attacks","multimodal large language models","speech tokenization","white-box adversarial attack","greedy search","alignment bypass","SpeechGPT","discrete speech tokens"],"falsifier":"Run the same greedy search while withholding the scalar loss value and see whether attack success collapses; separately, take the reconstructed adversarial audio, transcribe it with an independent speech recognizer that does not use the HuBERT clusterer, and check whether the intended harmful phrase is recovered. If a separate recognizer hears different words, the attack may be exploiting an artifact of the optimized reverse-loss loop rather than the claimed token sequence.","tokens_in":9631,"feed_emoji":"🔊","tokens_out":6123,"duration_ms":48987,"temperature":0.7,"pith_summary":"This paper argues that the voice interface of an aligned multimodal language model, SpeechGPT, can be broken by a fully automated, token-level attack that needs no handcrafted prompts and no gradient access. The attacker starts from a harmful spoken question, converts it to discrete speech tokens via the model's own HuBERT extractor, appends a short random token suffix, and greedily replaces those suffix tokens to minimize the loss between the model's output and a desired target response. The optimized token sequence is turned back into audio with a vocoder, and that audio is played to the model. Across six prohibited categories defined by the model's usage policy, the paper reports up to 89% attack success, higher than existing voice jailbreak baselines, with semantically meaningful audio also sounding more natural than pure noise. If correct, this means text-based alignment does not transfer to the audio token space, and voice-enabled assistants inherit a largely unexamined attack surface.","feed_headline":"Adversarial audio tokens jailbreak SpeechGPT in 89% of attempts","feed_subtitle":"Greedy search over discrete speech tokens beats voice jailbreak baselines across six prohibited categories.","key_machinery":"The mechanism that carries the argument is a three-stage pipeline. A Discrete Unit Extractor (HuBERT) converts harmful speech into discrete tokens; a greedy search (Algorithm 1) samples candidate replacement tokens for each position of an appended adversarial suffix and keeps the candidate that minimizes the loss between the model's decoding and a predefined target response, iterating until jailbreak behavior appears; and a cluster-matching reconstruction step (Algorithm 2) synthesizes the final token sequence into a waveform with a vocoder (HiFi-GAN), then optimizes a global noise perturbation so that re-extracted clusters match the intended token sequence. The key property is that optimization happens entirely in the discrete token space using only observable loss values, with no gradients or internal parameters.","core_discovery":"On its own terms, the paper's central claim is that a discrete-token adversarial attack can bypass the safety alignment of SpeechGPT without modifying the harmful speech tokens themselves. The original harmful audio is tokenized and kept intact; only a short, randomly initialized adversarial token suffix is optimized, using repeated queries that return a scalar loss value. Because the original tokens are unchanged, the natural prosody of the harmful question survives, and the added perturbation is small. The paper further claims that this method outperforms direct harmful speech, random noise, and black-box voice jailbreak methods, reaching 0.95 on illegal activity and 0.90 on hate speech, physical harm, pornography, and privacy violations, for an average success rate of 0.89.","pith_inferences":["The attack's reliance on a scalar loss value suggests a black-box variant could be built by estimating that loss from confidence scores or a surrogate model, extending the threat beyond white-box settings.","Because the reconstruction step is validated only by the optimized reverse loss, the adversarial audio may be exploiting quirks of the HuBERT clusterer rather than carrying the intended spoken phrase to a human listener; that gap deserves a direct test.","The evaluation covers ten questions per category on one model, so the 89% figure is a proof of feasibility rather than a measured field rate; transfer to other token-based speech LLMs is plausible but not demonstrated."],"forward_implications":["Voice-enabled multimodal models that inherit text-based alignment are vulnerable to automated token-level audio attacks, so safety training must be evaluated directly on audio inputs.","The reported success rates imply that black-box voice jailbreak baselines are weaker than white-box token-space optimization, and speech-model security benchmarks should include token-space attackers.","Because the attack preserves the original harmful tokens, waveform-denoising defenses may not remove the adversarial suffix without also degrading the spoken content.","The paper's own defense directions--denoising in the discrete audio token space, adversarial training, and aligning audio tokens with semantic expectations--follow directly from the attack's mechanism."],"supporting_citations":[{"why":"Defines SpeechGPT and its discrete speech-token interface, the victim model the attack targets.","marker":"[3]"},{"why":"Supplies HuBERT, the Discrete Unit Extractor used both by SpeechGPT and by the attack's tokenization and cluster-matching steps.","marker":"[32]"},{"why":"Supplies HiFi-GAN, the vocoder used to synthesize waveforms from discrete tokens during audio reconstruction.","marker":"[33]"},{"why":"Provides the Voice Jailbreak and Plot baselines whose attack success rates the paper compares against.","marker":"[9]"},{"why":"Provides the ForbiddenQuestionSet dataset from which ten questions per restricted category are selected for evaluation.","marker":"[12]"},{"why":"Supplies the NISQA score used to compare the perceptual quality of adversarial audio versus pure noise audio.","marker":"[35]"}],"fun_headline_variants":["White-box audio tokens jailbreak SpeechGPT at 89% rate","Adversarial speech tokens bypass SpeechGPT safety 89% of time","Token-level attack fools SpeechGPT with 89% success","SpeechGPT exposed: token attack achieves 89% jailbreak rate","Audio jailbreak: discrete tokens break SpeechGPT alignment"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The attack assumes the adversary can observe a scalar loss value between the model's output and a desired target response for arbitrary audio-token queries, and that the synthesized audio actually reaches the model as the intended token sequence--the latter verified only by an optimized reverse loss rather than an independent transcription.","fun_headline_variants_meta":{"raw":{"variants":["White-box audio tokens jailbreak SpeechGPT at 89% rate","Adversarial speech tokens bypass SpeechGPT safety 89% of time","Token-level attack fools SpeechGPT with 89% success","SpeechGPT exposed: token attack achieves 89% jailbreak rate","Audio jailbreak: discrete tokens break SpeechGPT alignment"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000386,"raw_usage":{"total_tokens":2041,"prompt_tokens":948,"completion_tokens":1093,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":564,"completion_tokens_details":{"reasoning_tokens":1006}},"tokens_in":564,"tokens_out":1093,"duration_ms":6776,"temperature":1.0,"reasoning_tokens":1006,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:23:41.070658+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same greedy search while withholding the scalar loss value and see whether attack success collapses; separately, take the reconstructed adversarial audio, transcribe it with an independent speech recognizer that does not use the HuBERT clusterer, and check whether the intended harmful phrase is recovered. If a separate recognizer hears different words, the attack may be exploiting an artifact of the optimized reverse-loss loop rather than the claimed token sequence.","supporting_citations":[{"cited_title":"\" do anything now","cited_arxiv_id":null,"evidence_quote":"Provides the ForbiddenQuestionSet dataset from which ten questions per restricted category are selected for evaluation."}],"review_version":1}