{"id":"c310b4bd-39c1-496d-9997-db3487c61e82","arxiv_id":"2507.12595","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Multilingual speech foundation models, fused with a Tucker-Hadamard module, are reported to reach about 1% equal error rate on emotion fake audio detection, a large drop from prior benchmarks.","lead":"This paper tests whether multilingual speech foundation models detect emotionally manipulated audio better than monolingual, speaker, or music models, and proposes a fusion method called THAMA. It reports large error-rate drops on the only public EmoFake benchmark, but the comparison has several methodological caveats.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Claimed SOTA EERs are the best of 78 test-set fusion results with no validation split or repeated runs, so the headline numbers are not yet unbiased estimates.","rationale":"The reader's weakest_assumption is correct and load-bearing. The paper does provide useful supporting evidence: the multilingual models consistently outperform monolingual, speaker, and music models across English, Chinese, and both cross-lingual directions, and THAMA improves over concatenation in many rows of Table 2. But the central SOTA claim is empirical, and the two headline EERs are produced by a test-set-selection procedure. This is a standard correctness risk, not an accusation: a validation-based rerun could confirm the claim, in which case no substantive objection remains. I also note a secondary reproducibility issue in Section 3.1: the expanded sum for Z over i,j,k gives a scalar, whereas the compact form F1^T T F2 denotes a vector, so the THAMA layer is notated inconsistently. That ambiguity is worth fixing, but the primary risk is the evaluation protocol. The verdict remains CONDITIONAL, unchanged from the reader.","tokens_in":9840,"tokens_out":11505,"duration_ms":133277,"concrete_test":"Use the provided development split (or a held-out portion of the training split) to select the fusion pair and downstream head, then evaluate the chosen model exactly once on the official test set. Repeat the full protocol with at least 5 random seeds and report mean and 95% CI of EER. If XLS-R+MMS remains the best pair and the CI upper bound stays below the [10] EERs (3.65%/8.34%), the SOTA claim holds; otherwise, report the validation-selected numbers as the supported claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.2, Tables 1–2 report EERs for 13 individual FMs and 78 fusion pairs, all evaluated on the official test split of Zhao et al. [10]. The headline THAMA result with XLS-R+MMS (0.89%/1.03% EER) is the minimum of the fusion table; the test labels therefore entered model selection. The same test set also appears to have guided the choice of downstream head (CNN vs FCN) and the best concatenation baseline. No development-set numbers, repeated runs, or confidence intervals are reported. Selecting the minimum of 78 correlated estimates biases the reported EER downward relative to the true EER of any fixed model, and the magnitude of that bias is unknown. Since the 'new SOTA' claim rests entirely on these two single-run numbers, the comparison against [10] (3.65%/8.34%) is not yet supported. No code or data are released, so the selection protocol cannot be audited.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates EmoFake Detection (EFD) using frozen speech foundation models. It compares 13 foundation models (monolingual, multilingual, speaker-recognition, and music models) on the EmoFake dataset in both same-language (English/Chinese) and cross-lingual settings, and proposes THAMA, a fusion method combining Tucker decomposition with a Hadamard product. The authors report that multilingual speech foundation models consistently outperform other families, and that THAMA fusing XLS-R and MMS achieves the best results, with 0.89% and 1.03% EER on English and Chinese, which they claim is a new state of the art compared with the 3.65% and 8.34% EER reported by prior work [10].","tokens_in":9999,"tokens_out":5043,"duration_ms":60422,"significance":"If validated, the paper would provide a useful empirical benchmark: it is, to my knowledge, the first systematic comparison of multiple speech foundation model families for EFD, the first cross-lingual EFD evaluation on this dataset, and it proposes a fusion method that improves over simple concatenation. The breadth of the comparison (13 individual models and 78 fusion pairs) and the consistent multilingual advantage across languages are genuine strengths. However, the central empirical claims currently rest on an evaluation protocol that selects the best model on the test set and on a cross-family comparison that is confounded by model scale and architecture; these issues must be resolved before the state-of-the-art claim can be accepted.","major_comments":[{"comment":"The reported state-of-the-art numbers are selected from the test set rather than estimated on held-out data. Table 2 reports EER for 78 fusion pairs evaluated on the official test split, and the headline XLS-R+MMS THAMA results (0.89%/1.03%) are the minimum of that table. No development-set results, repeated runs, or confidence intervals are reported, and the same test set also appears to have guided the choice between CNN and FCN downstream heads. Selecting the minimum of many correlated estimates biases the reported EER downward relative to the true EER of any fixed model, and the magnitude of this bias is unknown. Because the entire comparison to prior work [10] (3.65%/8.34%) depends on these two single-run numbers, the state-of-the-art claim is not yet supported. Please select the fusion pair and downstream architecture on a held-out validation split, and report mean and standard deviation over multiple training seeds, or use nested cross-validation.","section":"Section 4.2, Tables 1-2"},{"comment":"The core fusion equation is internally inconsistent as written. The text defines Z = sum_i sum_j sum_k T_ijk * F1_i * F2_j, where the sum runs over all three indices; since no output index appears, Z is a scalar. The subsequent operations H = Z ⊙ Z and the FCN block require a vector. The intended formula is presumably Z_k = sum_i sum_j T_ijk * F1_i * F2_j, with the tensor contraction leaving one free index. Please correct the equation and specify the tensor contraction explicitly; the current definition does not match Figure 2 or the downstream processing.","section":"Section 3.1, THAMA definition"},{"comment":"The main hypothesis, that multilingual pre-training causes the observed superiority, is confounded with model scale and architecture. The multilingual models are MMS (1B parameters), XLS-R (300M), and Whisper base (74M), while the monolingual baselines are all roughly 95M. The paper itself attributes MMS's top individual performance to its larger model size, which undercuts the claim that multilinguality per se is the cause. The Whisper result (74M multilingual outperforming 95M monolingual models) suggests scale is not the whole story, but a controlled comparison is needed, e.g., XLS-R 300M versus wav2vec2 300M, or MMS 1B versus a monolingual 1B model, before the attribution to multilingual pre-training can be accepted.","section":"Section 2/Table 1"},{"comment":"The out-domain evaluation is narrower than the claims suggest. Only the in-domain best pair (XLS-R+MMS) is carried to the cross-lingual setting, and that pair was selected from the same test split via Table 2, so the same selection-bias concern applies. In addition, 'out-domain' here means training on one language and testing on the other within the same EmoFake corpus; it does not include unseen datasets, unseen speakers, or unseen emotional voice conversion systems. The contribution statement claiming 'the first-ever out-domain assessment' should be scoped accordingly, and the selection protocol for the out-domain model should be described explicitly.","section":"Section 4.2, out-domain evaluation"}],"minor_comments":[{"comment":"There are several typographical and grammatical errors, e.g., 'Our results shows', 'To our end', 'tehcnique', 'resarch', and 'Hardmard' in Figure 2. These should be corrected.","section":"Abstract and throughout"},{"comment":"The table does not state which downstream head (CNN or FCN) was used for the fusion experiments. The text says CNN generally performed better for individual models, but it should be explicit for Table 2 and for the out-domain results.","section":"Table 2"},{"comment":"The statement 'Due to space constraint we are unable to present the comparison with SOTA in tabular format' is not acceptable in a full paper; the comparison to [10] should be shown in a table with the same evaluation protocol, including the training/development/test split details.","section":"Section 4.2, comparison to SOTA"},{"comment":"No ablation isolates the contributions of the Tucker core, the Hadamard product, and the learned shared projection. The comparison to concatenation is useful, but an ablation would strengthen the claim that each component of THAMA is necessary.","section":"Section 3.1"},{"comment":"The dataset description says training, development, and test contain 27300, 9100, and 17500 samples 'for both English and Chinese', but it is not clear whether these counts are per language or combined; please clarify.","section":"Section 4.1"},{"comment":"Reference [7] is cited in two consecutive sentences with the same description; the duplication should be removed. Also, the paper says models and code will be open-sourced after double-blind review, but the current submission contains no code or data release, which limits reproducibility checks of the reported EER values.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"I concur with the stress-test concern: the headline EERs are minima selected from the test set, so the state-of-the-art claim is not yet evidence. The paper is potentially publishable as an empirical benchmark after the authors add a validation-based selection protocol, repeated runs, controlled model comparisons to remove the scale/architecture confound, and a corrected THAMA equation. The fit to the journal is acceptable if the revised evaluation protocol is sound."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper's central empirical pattern—multilingual speech FMs beat monolingual, music, and speaker models for EmoFake detection—holds up across English, Chinese, and cross-lingual splits, and that is a useful contribution. But the headline claim of a new state of the art at 0.89%/1.03% EER rests on the single best entry picked from 78 fusion results all evaluated on the test split, with no validation set, no repeated runs, and no confidence intervals. As reported, those numbers are not unbiased estimates of what any fixed system achieves; they are the minimum of a correlated sweep. That is a load-bearing flaw, not a cosmetic one.\n\nWhat is actually new: first evaluation of frozen speech foundation models for EFD, first out-domain/cross-lingual EFD benchmark, and a fusion block (Tucker decomposition + Hadamard, called THAMA) that is a small but legitimate variant of existing tensor fusion. The comparison is broad—13 FMs, two downstream heads, 78 fusion pairs—and the consistency of the multilingual advantage is genuinely convincing even before fusion. The t-SNE plots are a nice sanity check. The authors also cite their own prior work [16] appropriately; it motivates, but does not construct, the result.\n\nSoft spots, in order: (1) The SOTA claim is unsupported as presented: the best pair XLS-R+MMS and the downstream head were selected on the same test set that produces the final scores. Selecting the minimum of 78 correlated EERs biases the result downward by an unknown amount, so the 0.89/1.03 vs 3.65/8.34 comparison is not yet meaningful. (2) No error bars or repeated runs—for a 1% EER claim, a single run is fragile. (3) Model-scale confound: MMS (1B) and XLS-R (300M) are bigger than the 95M monolingual bases; the multilingual conclusion is consistent but not fully controlled. The paper even credits MMS's \"larger model size\" for its performance, which undercuts the language-pretraining explanation. (4) No code released, and the comparison to [10] is only in text. (5) Minor: the fusion is not ablated into its Tucker and Hadamard parts, so we don't know what THAMA adds beyond concatenation. These are all addressable; the core pattern is probably real and worth publishing once the evaluation protocol is fixed.\n\nWho is this for? Anyone working on audio deepfake detection or speech foundation model evaluation for affective forensics. It deserves a serious referee, but the revision should require a held-out validation split for model selection, repeated runs with confidence intervals, parameter-matched fusion baselines, and code release.\n\nRecommendation: send to peer review with a strong request to fix the evaluation protocol. My own verdict on the current form is conditional, not an accept.","headline":"The multilingual-over-monolingual pattern for EmoFake detection looks real; the 0.89%/1.03% EER 'SOTA' does not, because the best fusion pair was picked on the test set.","tokens_in":10568,"tokens_out":2474,"would_cite":true,"duration_ms":25307,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Multilingual speech foundation models, fused by THAMA, outperform single-language models at detecting emotion-manipulated fake audio.","keywords":["EmoFake detection","speech foundation models","multilingual pre-training","Tucker decomposition","Hadamard product","fusion","cross-lingual detection","fake audio detection"],"falsifier":"Run the same pipeline with model selection done on a held-out development fold and report test EER for every fusion pair; if XLS-R+MMS is not the selected pair, or its EER no longer beats the 3.65%/8.34% baseline with a clear margin, the central SOTA claim is not supported. Reporting per-seed variability would let a reader see whether the top fusion is statistically distinct from the runner-ups.","tokens_in":9596,"feed_emoji":"🎭","tokens_out":5729,"duration_ms":56610,"temperature":0.7,"pith_summary":"This paper sets out to show that speech foundation models trained on many languages are better at detecting EmoFake audio—recordings whose emotional tone has been altered while the words stay the same—than models trained on one language or on music. It tests thirteen models in same-language and cross-lingual settings on the only public EmoFake dataset, covering English and Chinese, and reports that multilingual models consistently give the lowest error rates. The paper also introduces THAMA, a fusion method that combines Tucker decomposition with a Hadamard product to merge representations from two models. Fusing the two strongest multilingual models, XLS-R and MMS, with THAMA yields 0.89% and 1.03% equal-error rates on English and Chinese, against 3.65% and 8.34% for the prior benchmark. If these results hold, they suggest that broad linguistic pre-training captures the pitch, tone, and intensity cues that emotional manipulation changes.","feed_headline":"Two multilingual models catch emotion-fake audio at under 1% error","feed_subtitle":"Fusing XLS-R and MMS beats the prior EmoFake benchmark by a wide margin in both English and Chinese.","key_machinery":"THAMA (Tucker-HAdaMArd Conjunction) is the paper's fusion mechanism. It takes the flattened representations of two foundation models, projects them into a shared space, and combines them with a Tucker decomposition, in which a small core tensor models the high-order interactions between the two feature sets; the result is then squared element-wise with a Hadamard product, which sharpens the fused features before a fully connected classifier. The method is what turns two individually strong multilingual models into a single detector that outperforms either alone and outperforms simple concatenation-based fusion.","core_discovery":"The central claim is that multilingual speech foundation models are intrinsically better suited to EmoFake detection and that fusing them with THAMA sets a new state of the art. The paper argues that pre-training across languages gives the models a more refined grasp of prosodic variation, letting them spot the subtle acoustic edits left by emotion voice conversion. In the author's own comparison, the multilingual models XLS-R and MMS beat every monolingual, speaker-recognition, and music model in both in-domain and cross-lingual tests, and their THAMA fusion improves further still. The reported outcome is that the XLS-R + MMS pair with THAMA reaches 0.89% EER on English and 1.03% on Chinese, versus 3.65% and 8.34% for the previous best method on the same dataset.","pith_inferences":["The paper's comparison is limited to one dataset with two languages; a natural extension is to test whether the same THAMA fusion generalizes to other language pairs or to emotion-manipulated samples generated by unseen voice-conversion systems.","Because the official test split was used to select the best fusion pair, the reported SOTA numbers should be read as optimistic upper bounds; a held-out validation protocol would give a fairer estimate.","The THAMA fusion mechanism is task-agnostic in principle and could be evaluated for other audio deepfake tasks such as singing-voice or scene-fake detection, where complementary multilingual representations may also help.","Re-running with different random seeds and reporting variance would clarify whether the small EER differences between top fusion pairs are meaningful."],"forward_implications":["If correct, THAMA-style fusion of XLS-R and MMS becomes the default baseline to beat for future EmoFake detection research.","Cross-lingual EmoFake detection is feasible: a system trained on English transfers to Chinese and vice versa, with EERs in the 3-5% range, which the paper reports as a new benchmark for out-domain evaluation.","Multilingual pre-training matters more than model size alone for this task: MMS's advantage over smaller multilingual models is attributed partly to scale, but all multilingual models beat much larger monolingual and music alternatives.","Music foundation models are not useful for EmoFake detection; the paper's results show they systematically perform worse, so resources for this task should go to speech models."],"supporting_citations":[{"why":"Supplies the only public EmoFake dataset and the prior EFD benchmark that the paper compares against.","marker":"[10]"},{"why":"XLS-R, one of the two multilingual speech foundation models whose THAMA fusion produces the reported SOTA results.","marker":"[21]"},{"why":"MMS, the other model in the best fusion pair, a speech representation model scaled to 1,000+ languages.","marker":"[23]"},{"why":"WavLM, a monolingual speech foundation model used as a baseline in the comparative study.","marker":"[17]"},{"why":"wav2vec2, another monolingual baseline whose representations are compared in the FM evaluation.","marker":"[19]"},{"why":"MERT, the music foundation model family used to test whether music pre-training helps EmoFake detection.","marker":"[26]"},{"why":"Earlier work showing multilingual speech pre-trained models improve audio deepfake detection, motivating the paper's central hypothesis.","marker":"[16]"}],"fun_headline_variants":["Multilingual fusion cuts EmoFake error to under 1%","XLS-R + MMS with THAMA: new EmoFake state of the art","THAMA fusion of XLS-R and MMS outperforms prior EmoFake methods"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline error rates assume the test set was not used to choose which two models to fuse; if the best pair was selected by reading test results, the 0.89%/1.03% figures are optimistic.","fun_headline_variants_meta":{"raw":{"variants":["Multilingual fusion cuts EmoFake error to under 1%","XLS-R + MMS with THAMA: new EmoFake state of the art","THAMA fusion of XLS-R and MMS outperforms prior EmoFake methods"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000485,"raw_usage":{"total_tokens":2369,"prompt_tokens":897,"completion_tokens":1472,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":513,"completion_tokens_details":{"reasoning_tokens":1401}},"tokens_in":513,"tokens_out":1472,"duration_ms":12711,"temperature":1.0,"reasoning_tokens":1401,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T16:44:16.949728+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same pipeline with model selection done on a held-out development fold and report test EER for every fusion pair; if XLS-R+MMS is not the selected pair, or its EER no longer beats the 3.65%/8.34% baseline with a clear margin, the central SOTA claim is not supported. Reporting per-seed variability would let a reader see whether the top fusion is statistically distinct from the runner-ups.","supporting_citations":[{"cited_title":"The ASVspoof 2017 Challenge: Assessing the Limits of Replay Spoofing Attack Detection,","cited_arxiv_id":null,"evidence_quote":"Supplies the only public EmoFake dataset and the prior EFD benchmark that the paper compares against."},{"cited_title":"Investi- gation of ensemble features of self-supervised pretrained models for automatic speech recognition,","cited_arxiv_id":null,"evidence_quote":"XLS-R, one of the two multilingual speech foundation models whose THAMA fusion produces the reported SOTA results."},{"cited_title":"Heterogeneity over homogeneity: Investigating multilingual speech pre-trained models for detecting audio deepfake,","cited_arxiv_id":null,"evidence_quote":"MMS, the other model in the best fusion pair, a speech representation model scaled to 1,000+ languages."},{"cited_title":"Emofake: An initial dataset for emotion fake audio detection,","cited_arxiv_id":null,"evidence_quote":"WavLM, a monolingual speech foundation model used as a baseline in the comparative study."},{"cited_title":"Detection of deepfake environmental audio,","cited_arxiv_id":null,"evidence_quote":"wav2vec2, another monolingual baseline whose representations are compared in the FM evaluation."},{"cited_title":"wav2vec 2.0: A framework for self-supervised learning of speech representations,","cited_arxiv_id":null,"evidence_quote":"MERT, the music foundation model family used to test whether music pre-training helps EmoFake detection."},{"cited_title":"Scenefake: An initial dataset and benchmarks for scene fake audio detection,","cited_arxiv_id":null,"evidence_quote":"Earlier work showing multilingual speech pre-trained models improve audio deepfake detection, motivating the paper's central hypothesis."}],"review_version":1}