{"id":"f182e259-4581-4e86-8036-b63d94c65a3b","arxiv_id":"2506.06756","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Zero-shot audio-language models are not reliable spoof detectors because they over-predict 'spoof', and FP16 quantization keeps this bias while INT8 worsens it.","lead":"This paper tests five large audio-language models as zero-shot detectors of fake audio and finds they are strongly biased toward saying 'spoofed', so their practical accuracy is near chance. It also finds that 16-bit quantization preserves this behavior while cutting memory in half, whereas 8-bit quantization makes the bias worse.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Zero-shot decision rule rests on an undisclosed prompt; observed spoof bias could be a prompt or token artifact rather than an intrinsic LALM limitation.","rationale":"The reader's verdict is CONDITIONAL and identifies the prompt-based decision rule as the load-bearing premise. My stress-test converges on the same point: the paper's central negative result—that LALMs are severely biased toward the spoof class and hence no better than random in balanced terms—rests on comparing token probabilities from one unspecified prompt. Since the manuscript never reveals the prompt, the tokenization of 'spoof' and 'bonafide', or any prompt-robustness check, the observed bias could be an artifact of the prompt or token choice rather than an intrinsic LALM limitation. This is not a fatal flaw in the sense that the underlying experiment may still be correct, but it is a required condition for the central claim to hold. Without prompt/token ablations or at least full disclosure, the paper cannot support its generalization across 'all models' and 'all datasets.' The internal inconsistency in Table 2 (GAMA In-the-Wild Bonafide Accuracy FP16 = 0.170) further undermines confidence in the quantitative claims, but that is secondary to the decision-rule issue. My proposed test—varying prompts and token pairs—would settle whether the bias is intrinsic or an artifact. The reader already conditioned acceptance on addressing this class of concern, so the verdict remains CONDITIONAL; no change is needed, but the revision must include prompt and token details and ideally the ablation.","tokens_in":9319,"tokens_out":3545,"duration_ms":41910,"concrete_test":"Repeat the zero-shot protocol with at least three distinct prompts (e.g., 'Is this audio spoofed or bonafide?', 'Classify the audio as genuine or fake', 'Which label applies: spoof, bonafide?') and at least two token-pair choices (spoof/bonafide and fake/genuine) on one representative model (e.g., MERaLiON) across all three datasets at FP32. If balanced accuracy varies by more than 0.1 across prompt/token settings, or if the spoof bias reverses under any prompt, the reported 'severe predictive bias' is a prompt artifact rather than an intrinsic model property. The exact prompt strings and token surface forms must also be disclosed in the paper or an appendix.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that LALMs are intrinsically biased toward the spoof class—is established entirely through the decision rule in Section 2, which compares P(t_spoof | X, p) and P(t_bonafide | X, p). The paper never specifies the prompt p, the exact surface forms of tspoof and tbonafide, or the decoding/tokenization used to extract these probabilities. For autoregressive audio-language models, next-token probabilities are highly sensitive to prompt phrasing and to the prior frequency of the target words in the model's vocabulary. If p is phrased as, e.g., 'Is this audio spoofed?' or if the bonafide token is rare or subword-tokenized differently, the model can show a systematic spoof bias that has nothing to do with audio content. The fact that all five models exhibit the same bias is consistent with a shared prompt/token artifact, not necessarily with an intrinsic architectural limitation. The paper also reports FP16 'negligible degradation,' but the In-the-Wild GAMA row in Table 2 (Bonafide Accuracy: 0.849 / 0.170 / 0.858) directly contradicts that qualitative claim, suggesting additional data-reporting issues. For the headline result, however, the load-bearing problem is the under-specified decision rule: without prompt and token details, the empirical finding cannot be interpreted as evidence about LALMs in general.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reports an empirical evaluation of five large audio-language models (GAMA, LTU-AS, MERaLiON, Qwen-Audio, SALMONN) as zero-shot binary spoof/bonafide classifiers on ASVspoof2019, In-the-Wild, and WaveFake, under FP32, FP16, and INT8 quantization. The decision rule compares the probabilities of two class tokens for each audio sample, as described in Section 2. The authors report that high overall accuracy/F1 can be misleading because classwise metrics reveal a strong bias toward the spoof class, that FP16 is essentially lossless relative to FP32, and that INT8 intensifies the bias and reduces balanced accuracy. They also report memory and inference-time measurements (Table 3) and conclude with deployment guidance favoring FP16 and suggestions for architectural refinements.","tokens_in":9554,"tokens_out":5346,"duration_ms":57868,"significance":"The paper addresses a timely and practically important question: whether quantized audio-language models can serve as zero-shot spoofing countermeasures. A well-documented negative result of this form would be useful to the community, and the deployment recommendation (FP16 over INT8, with caution) is concrete and falsifiable. The efficiency measurements in Table 3 are also potentially valuable. However, the central finding cannot currently be interpreted as a property of the models rather than of the experimental protocol, because the prompt, class tokens, and tokenization are not disclosed, and Table 2 contains arithmetic inconsistencies that undermine the quantitative claims. The paper would be strengthened materially by releasing the exact protocol and corrected, code-verifiable tables.","major_comments":[{"comment":"The binary decision rule is never instantiated concretely. The manuscript does not state the prompt p, the exact strings tspoof and tbonafide, whether the probabilities are taken from the first generated token, from forced decoding, or from marginalization, or how the tokenizer handles capitalization, whitespace, and subword splits. Since every quantitative result in Table 2 is a consequence of this rule, the observed spoof bias could be caused by a prompt or token artifact rather than an intrinsic model limitation. Please specify the full input template, the exact class-token surface forms and token IDs, and the probability extraction procedure; in addition, run an ablation with at least two alternative prompt phrasings and report the token-level probabilities. Without this, the manuscript's central claim that LALMs are intrinsically spoof-biased cannot be evaluated.","section":"Section 2, decision rule paragraph"},{"comment":"Several reported aggregate metrics are inconsistent with the classwise accuracies in the same row. For In-the-Wild LTU-AS at FP16, Balanced Accuracy is listed as 0.490 while the adjacent classwise values give (0.866 + 0.013) / 2 = 0.4395; the same pattern appears for Qwen-Audio at FP32 and FP16 and for LTU-AS at FP32. In addition, the GAMA In-the-Wild FP16 Bonafide Accuracy of 0.170 is implausible relative to 0.849 at FP32 and 0.858 at INT8 and is inconsistent with the reported Balanced Accuracy of 0.433; this cell appears to be a typo or a swapped value. These errors directly affect the paper's FP16 'negligible degradation' claim and prevent the reader from trusting the table. Please recompute all rows, provide per-class confusion matrices or the code that generates them, and reconcile every aggregate metric with the classwise rates.","section":"Table 2"},{"comment":"The paper defines zero-shot as 'without explicit fine-tuning on spoof-specific data,' but this is insufficient for the strong conclusion that the observed bias is an intrinsic architectural limitation. The three evaluation corpora are public and widely used, and the manuscript provides no check for overlap between the LALMs' pretraining data and the test sets, no statement of which ASVspoof2019 protocol (LA, PA, or DF) and which In-the-Wild split were used, and no details on audio preprocessing such as duration, resampling, or truncation. Please state the concrete evaluation subsets, report a contamination analysis (for example, exact or near-duplicate search) for each model if possible, and soften the 'intrinsic' language to 'under the tested decision rule and data distributions' if such evidence is not available.","section":"Sections 2 and 3, zero-shot premise"}],"minor_comments":[{"comment":"The model name MERaLiON is rendered as 'Meralion' in Tables 2 and 3 and in Figure 2; please use one canonical spelling throughout.","section":"Throughout, Tables 2 and 3, Figure 2"},{"comment":"The caption begins 'Illustrate Accuracy' (subject-verb disagreement) and contains repeated '(a)(a)(a)' labels; these formatting issues should be cleaned up.","section":"Figure 2 caption"},{"comment":"Memory and inference-time measurements are reported without stating the batch size, input length, number of warm-up iterations, or whether the times are medians over repeated runs; without these details, the comparison, especially the counterintuitive INT8 slowdown, is difficult to interpret.","section":"Table 3"},{"comment":"The text states that experiments 'leverag[ed] mixed-precision inferencing' while also reporting comparisons among FP32, FP16, and INT8; please clarify how mixed precision relates to the three nominal precision levels.","section":"Section 2, Implementation Details"},{"comment":"No code or configuration files are provided; given that the prompt and probability extraction are the entire methodology, the revised version should include this material or at least a detailed protocol appendix.","section":"General reproducibility"}],"recommendation":"major_revision","confidential_remarks":"The under-specified prompt is the main risk to the paper's central claim. I would ask the editor to request, along with the revision, the exact prompt template, token IDs, and the code or confusion-matrix files needed to reproduce Table 2 before considering the paper further."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core finding here is worth knowing: five different large audio language models, tested zero-shot on three spoof datasets, all collapse toward a spoof-everything bias. MERaLiON and LTU-AS hit 100% spoof recall and 0% bonafide recall in several conditions. That pattern is visible in the printed classwise accuracies, and it is the kind of result that should stop someone from deploying LALMs as zero-shot spoof detectors without serious recalibration or fine-tuning.\n\nThe paper does several things right. It covers a reasonable spread of architectures, includes datasets with different class priors, reports classwise accuracy alongside F1 and MCC (which is what makes the bias visible), and measures memory and inference time across FP32/FP16/INT8. The FP16-as-a-deployment-default suggestion is sensible, and the INT8 inference-time regression is a nice practical observation.\n\nBut there are real problems. The decision rule in Section 2 compares P(t_spoof) and P(t_bonafide) after prompting the model, yet the prompt is never disclosed, nor are the exact token strings or the decoding procedure. For autoregressive models, next-token probabilities are highly sensitive to prompt phrasing and token frequency. The identical bias across all five models could be a shared artifact of a badly chosen prompt rather than an intrinsic architectural limitation. The zero-shot claim is also unverified against possible pretraining overlap with these datasets.\n\nWorse, Table 2 contains clear arithmetic inconsistencies. For example, In-the-Wild LTU-AS FP16 shows Spoof Accuracy 0.866 and Bonafide Accuracy 0.013, which averages to 0.440, but balanced accuracy is printed as 0.490. GAMA In-the-Wild FP16 similarly shows 0.009 and 0.170 averaging to 0.089, yet balanced accuracy is 0.433. The FP16 'negligible degradation' claim is directly contradicted by GAMA's bonafide accuracy dropping from 0.849 to 0.170 in that same row. These errors make the quantitative results unusable until corrected. There are also no error bars or repeats, which is a lesser but still relevant gap.\n\nMy read: the qualitative bias story is probably real, because it shows up consistently across many model/dataset cells even if the exact numbers are off. But the paper as posted cannot be used as a reliable benchmark. The fix is straightforward: release the prompt, tokens, and code; correct the tables; and rerun with a couple of prompt variations to show the bias is not prompt-induced.\n\nI would send this to peer review because the question is practically important and a corrected version would be a useful negative result. But I would not cite the current arXiv version in my own work, and the revision needs to be substantive, not cosmetic.","headline":"Useful negative result about LALM spoof bias, but the under-specified decision rule and inconsistent tables mean the paper needs a careful revision before the numbers can be trusted.","tokens_in":10101,"tokens_out":1353,"would_cite":false,"duration_ms":16898,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Large audio language models are biased toward 'spoof' in zero-shot detection, making their practical performance equivalent to random classification.","keywords":["large audio language models","audio spoofing detection","zero-shot evaluation","model quantization","FP16","INT8","classification bias","balanced accuracy"],"falsifier":"Measure the area under the ROC curve using P(t_spoof) - P(t_bonafide) with several prompts and with the spoof/bonafide label order flipped; if any model's AUC is close to 1, or if balanced accuracy rises well above random for a different prompt, the claim that LALMs are no better than random classification is false.","tokens_in":9115,"feed_emoji":"🎧","tokens_out":5378,"duration_ms":52291,"temperature":0.7,"pith_summary":"This paper asks whether five large audio language models can detect spoofed audio without any task-specific training, and whether quantization changes that ability. The authors find that the models' high accuracy on spoofed samples is an illusion: across ASVspoof2019, In-the-Wild, and WaveFake, all five models strongly prefer the 'spoof' label, so balanced accuracy hovers near random even when raw accuracy looks good. Quantization results are mixed: FP16 preserves the models' behavior while roughly halving memory, whereas INT8 intensifies the spoof bias and degrades balanced accuracy further. The practical upshot is that current LALMs are not ready for zero-shot spoofing detection, and that FP16 is the safest efficiency-precision trade-off. This matters because deployers need to know whether lightweight audio models can be trusted for security tasks.","feed_headline":"Spoof bias renders audio language models no better than random","feed_subtitle":"High accuracy hides an all-spoof bias; FP16 halves memory safely, INT8 makes detection worse.","key_machinery":"The load-bearing object is the token-probability decision rule: given audio frames and a prompt, the model assigns probabilities to vocabulary tokens, and the sample is called spoof if P(t_spoof) > P(t_bonafide). Around this rule the paper wraps systematic evaluation with balanced accuracy, classwise accuracy, and MCC, plus the quantization function Qp that converts FP32 weights to FP16 or INT8. The rule explains both the headline result and the quantization story: the spoof token has a persistent probability advantage that inflates raw accuracy, FP16 leaves the advantage intact, and INT8 amplifies it enough to erase bonafide detection entirely.","core_discovery":"The central discovery is that large audio language models are systematically biased toward the spoof class in zero-shot audio spoofing detection. The decision rule compares the probability of the token 'spoof' with that of 'bonafide' after a prompt; because the spoof token almost always wins, metrics like F1 and accuracy overstate real capability while balanced accuracy and MCC show performance equivalent to random classification. For example, MERaLiON detects every spoof and no bonafide sample, and LTU-AS's INT8 'improvement' from 0.958 to 1.000 spoof accuracy is actually a strengthening of the same bias. FP16 quantization changes almost nothing, while INT8 pushes several models to zero bonafide accuracy and, paradoxically, slows inference despite shrinking memory.","pith_inferences":["A natural extension is to test whether the spoof bias is a decision-rule artifact: if temperature scaling, a different prompt, or an ROC threshold chosen from P(t_spoof) - P(t_bonafide) lifts balanced accuracy, the models may contain usable signal that the fixed token comparison throws away.","The zero-shot claim would be stronger if the authors showed that the evaluation datasets were absent from the pretraining data; without that check, some of the apparent behavior could come from memorized examples rather than generalization.","The INT8 inference-time penalty suggests that the comparison should be repeated with quantization-aware training or optimized integer kernels, which might restore the speed benefit the paper expected."],"forward_implications":["Accuracy and F1 alone cannot validate a zero-shot spoof detector; balanced metrics must be reported, since a model that labels everything spoofed can score well on both.","None of the five architectures is ready for unsupervised spoof detection in real systems; calibration or task-specific adaptation will be needed before deployment.","FP16 quantization is a practical default for LALMs in memory-constrained settings, because it halves memory without materially changing detection behavior.","INT8 quantization should not be adopted without per-model validation, since it can intensify class bias and even increase inference time relative to FP16."],"supporting_citations":[{"why":"Supplies the spoof-heavy ASVspoof2019 evaluation set that exposes the models' bias.","marker":"[20]"},{"why":"Supplies the bonafide-heavy In-the-Wild evaluation set that tests sensitivity to genuine audio.","marker":"[21]"},{"why":"Supplies the balanced WaveFake evaluation set for controlled generalization assessment.","marker":"[22]"},{"why":"Defines the GAMA architecture, one of the five LALMs evaluated.","marker":"[11]"},{"why":"Defines the LTU-AS architecture, one of the five LALMs evaluated.","marker":"[13]"},{"why":"Defines the MERaLiON architecture, one of the five LALMs evaluated.","marker":"[15]"},{"why":"Defines the Qwen-Audio architecture, one of the five LALMs evaluated.","marker":"[17]"},{"why":"Defines the SALMONN architecture, one of the five LALMs evaluated.","marker":"[18]"},{"why":"Provides the quantization framework and precision definitions used in the experiments.","marker":"[7]"}],"fun_headline_variants":["Audio LALMs: zero-shot spoof detection fails due to bias","Spoof bias makes quantized audio models equivalent to random","FP16 quantization preserves bias, INT8 amplifies it in audio spoofing","Zero-shot audio spoofing: bias rules over accuracy","Quantized audio LLMs: FP16 safe, INT8 worsens spoof detection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The result assumes that one undisclosed prompt and a raw comparison of the model's probabilities for the words 'spoof' and 'bonafide' is a fair zero-shot decision rule; if that rule or prompt is miscalibrated, the observed bias could be an experimental artifact rather than an intrinsic model limitation.","fun_headline_variants_meta":{"raw":{"variants":["Audio LALMs: zero-shot spoof detection fails due to bias","Spoof bias makes quantized audio models equivalent to random","FP16 quantization preserves bias, INT8 amplifies it in audio spoofing","Zero-shot audio spoofing: bias rules over accuracy","Quantized audio LLMs: FP16 safe, INT8 worsens spoof detection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000591,"raw_usage":{"total_tokens":2749,"prompt_tokens":897,"completion_tokens":1852,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":513,"completion_tokens_details":{"reasoning_tokens":1756}},"tokens_in":513,"tokens_out":1852,"duration_ms":17955,"temperature":1.0,"reasoning_tokens":1756,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:50:01.584593+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the area under the ROC curve using P(t_spoof) - P(t_bonafide) with several prompts and with the spoof/bonafide label order flipped; if any model's AUC is close to 1, or if balanced accuracy rises well above random for a different prompt, the claim that LALMs are no better than random classification is false.","supporting_citations":[{"cited_title":"Sea-lion (southeast asian languages in one net- work): A family of large language models for southeast asia,","cited_arxiv_id":null,"evidence_quote":"Supplies the bonafide-heavy In-the-Wild evaluation set that tests sensitivity to genuine audio."},{"cited_title":"Can rag- driven enhancements amplify audio llms for low-resource lan- guages?","cited_arxiv_id":null,"evidence_quote":"Defines the GAMA architecture, one of the five LALMs evaluated."},{"cited_title":"Sv-deit: Speaker verifica- tion with deitcap spoofing detection,","cited_arxiv_id":null,"evidence_quote":"Defines the LTU-AS architecture, one of the five LALMs evaluated."},{"cited_title":"Context encoded multi-modal attention network for detecting audio spoofing,","cited_arxiv_id":null,"evidence_quote":"Defines the MERaLiON architecture, one of the five LALMs evaluated."},{"cited_title":"Joint audio and speech understanding,","cited_arxiv_id":null,"evidence_quote":"Defines the SALMONN architecture, one of the five LALMs evaluated."},{"cited_title":"Pengi: An audio language model for audio tasks,","cited_arxiv_id":null,"evidence_quote":"Provides the quantization framework and precision definitions used in the experiments."}],"review_version":1}