{"id":"ad635e5f-3afe-48a2-af37-5e3757856207","arxiv_id":"2508.04143","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"The first multilingual source-tracing benchmark for speech deepfakes, showing LFCC-ECAPA-TDNN generalizes best across languages.","lead":"This paper creates a benchmark for identifying which AI speech generator produced a deepfake audio clip, across six languages and three language families. It compares classical signal-processing and self-supervised learning models and measures how well they transfer when training and test languages differ.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"DSP-vs-SSL comparison is confounded by backend: all SSL models use AASIST, while LFCC's best score uses ECAPA-TDNN; no SSL-ECAPA run exists.","rationale":"The reader's weakest assumption—that all four TTS architectures appear in all six languages—is a documentation gap rather than a demonstrated flaw; the dataset statistics (1,200–2,100 samples per language, four architectures) make it likely the coverage exists, and the concern is addressable with one sentence. The more load-bearing issue is the front-end/back-end confound in the paper's central comparison. The reader's verdict of CONDITIONAL is appropriate, but for a different reason than the one highlighted. My proposed check directly tests whether the paper's main 'DSP superiority' conclusion holds when the backend is controlled. This does not invalidate the benchmark's contribution or the raw performance numbers, but it means the paper should either add the missing SSL-ECAPA condition or substantially soften the 'DSP inherently robust' interpretation. Since the reader already recommended conditional acceptance, my assessment does not change the verdict category—hence UNCHANGED—but it identifies a more fundamental condition for acceptance.","tokens_in":13860,"tokens_out":11156,"duration_ms":123493,"concrete_test":"Train the same SSL front-ends used in Table 4 (e.g., XLS-R-300M, wav2vec2.0 Large, and one language-fine-tuned variant) with an ECAPA-TDNN backend, using exactly the same training recipe as the LFCC-ECAPA-TDNN system (§3.2, including batch size, epochs, LR, Mixup usage, and number of seeds). Evaluate cross-lingual macro-F1 with the same leave-one-language-out protocol. If SSL-ECAPA-TDNN matches or exceeds the 88.40% of LFCC-ECAPA-TDNN, the claimed DSP feature advantage is refuted; if it falls clearly below, the claim is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central empirical insight (RQ4, §4.4, §5) is that DSP front-ends generalize better cross-lingually than SSL front-ends ('LFCC features combined with ResNet or ECAPA-TDNN backends demonstrate superior cross-lingual generalization'). But the comparison conflates front-end and back-end. Every SSL model in Table 3 uses an AASIST classifier, while the DSP family includes three different backends (AASIST, ResNet18, ECAPA-TDNN). Holding the backend fixed at AASIST, LFCC-AASIST achieves only 54.46% cross-lingual macro-F1, below every SSL-AASIST variant (XLSR 59.75%, W2V2EN 78.18%, W2V2(xx) 73.45%). The best DSP number (88.40%) is obtained with LFCC-ECAPA-TDNN, but no SSL-ECAPA-TDNN is ever trained. Therefore the observed advantage of the DSP pipeline could be entirely due to the stronger ECAPA-TDNN backend rather than any property of LFCC features or DSP front-ends. The paper acknowledges the backend difference in places but still concludes that 'DSP architectures inherently prioritize language-agnostic patterns,' an interpretation not supported by the experimental matrix. This weakens the headline insight about DSP vs SSL cross-lingual robustness, even though the benchmark and raw scores remain valid as a resource.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"Motivated by the lack of multilingual source tracing for speech deepfakes, the authors construct MCL-MLAAD from MLAAD, covering six languages across three families (Germanic, Romance, Slavic) and four TTS architectures (Griffin-Lim, Bark, XTTS v1.1, XTTS v2). They define four evaluation protocols: monolingual, cross-lingual, cross-family, leave-one-language-out (unseen language), and pseudo-speaker seen/unseen. They compare three LFCC-based DSP models (AASIST, ResNet18, ECAPA-TDNN backends) with SSL front-end models (XLS-R, wav2vec2 Large, and six language-fine-tuned XLS-R variants, all with AASIST). Reported results: W2V2(xx)-AASIST achieves the best average monolingual macro-F1 (97.91%), LFCC-ECAPA-TDNN the best cross-lingual average (88.40%), same-family transfers outperform cross-family, LFCC models generalize better than XLSR to unseen languages and pseudo-speakers, and no consistent seen/unseen speaker gap is found. Code, dataset, and protocol are released.","tokens_in":14207,"tokens_out":10513,"duration_ms":111664,"significance":"The paper provides a useful and reproducible benchmark resource: public code/data/protocols, a clean four-class design, and a broad comparison across feature types, language families, and pretraining strategies. If the confounds are resolved, it would be a solid first step for multilingual source tracing. However, the headline DSP-versus-SSL conclusion is not identified by the current experimental matrix because front-end, backend, and Mixup are compounded; the per-language TTS composition needs explicit verification; and single-run macro-F1 numbers lack statistical support.","major_comments":[{"comment":"The central claim that 'DSP architectures inherently prioritize language-agnostic patterns' is not supported. All SSL models use the AASIST backend and Mixup, while the LFCC family uses three different backends and no Mixup. In the backend-matched comparison, LFCC-AASIST is the worst cross-lingual model (54.46%), below XLSR-AASIST (59.75%), W2V2(xx)-AASIST (73.45%) and W2V2EN-AASIST (78.18%). The best DSP score (88.40%) comes from LFCC-ECAPA-TDNN, and no SSL-ECAPA-TDNN run is reported. The observed advantage may therefore be due to the ECAPA-TDNN backend or the absence/presence of Mixup rather than to LFCC as a DSP front-end. Please add an SSL-ECAPA-TDNN baseline, or rephrase the conclusion to describe the specific LFCC-ECAPA-TDNN configuration and remove the 'inherently' generalization.","section":"Section 4.4 / Table 4 (RQ4)"},{"comment":"The benchmark defines a 4-class source-tracing task, and all cross-lingual and unseen-language comparisons rely on the class structure being the same in every language. The text states that six languages and four TTS architectures were selected, but it never explicitly says that each language contains all four architectures, and Table 1 only gives per-language utterance totals. If, for example, one architecture is absent from Russian (which has fewer samples, 1,200 vs 2,100), cross-lingual scores could reflect language/class prior differences rather than source-tracing ability. Please state the coverage explicitly and provide a language × TTS contingency table.","section":"Section 2.1 / Table 1"},{"comment":"Every macro-F1 number is a single run. Tables 4-6 report 36 cells and averages with no confidence intervals, standard deviations, or significance tests. Some headline differences are small (e.g., 97.91% vs 97.78% monolingual average), while macro-F1 on roughly 1,000 test samples per language can vary by several points across seeds. Because several conclusions are rankings (best model, within-family advantage, SSL-versus-DSP), the absence of seed variability makes the comparisons unverifiable. Please report at least three seeds with mean±std, or apply significance tests.","section":"Sections 3.2.3 and 4, Tables 4-6"},{"comment":"The seen/unseen speaker protocol is based on pseudo-speaker labels from unsupervised ECAPA2 clustering with an elbow-selected k=18 and per-language thresholds θ. The authors themselves note the lack of ground-truth speaker labels and the low speaker count. Given this, the conclusion that models show 'potential robustness to speaker variation' is fragile and should not be presented as a key finding. Please mark Table 6 as exploratory only and either validate the clustering with a small labeled subset or soften the conclusion accordingly. Also clarify the exact rule for classifying a pseudo-speaker as 'seen' vs 'unseen' (e.g., what happens when the prior is between 0 and θ).","section":"Sections 2.2.4 and 4.7, Table 6"}],"minor_comments":[{"comment":"The macro-F1 formula, F1 = 2PR/(P+R) with macro-averaged P and R, is nonstandard; macro-F1 is typically the average of per-class F1 values. Please define and use the standard version or explain why this variant was chosen.","section":"Section 3.2.3"},{"comment":"'When backend architectures are comparable (top four subtables)' is confusing; Table 4 has three SSL subtables and three DSP subtables, not four. Please correct the cross-reference.","section":"Section 4.4"},{"comment":"There are typos: 'Multil-lingual' and 'Lingual-Balanced' (likely 'linguistically balanced').","section":"Section 2.1"},{"comment":"'unlike was hypothesized' should be 'contrary to the hypothesis'.","section":"Section 4.7"},{"comment":"Some cited references (e.g., [7], [8], [25], [30], [31], [34]-[37], [67]) are not clearly connected to source tracing or the benchmark; trimming would improve readability.","section":"References"},{"comment":"The meaning of '(×5)' should be explained in the caption; as written it is ambiguous whether the counts are utterances or audio files after perturbation.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"To the editor only: The paper’s novelty claims ('first benchmark') are reasonable relative to [26]-[28], but the bibliography contains a noticeable number of unrelated/self-citations that appear to pad the reference list (e.g., [7], [8], [25], [30], [31], [34]-[37], [67]). I would ask the authors to prune them during revision; this does not affect my technical recommendation. The main revision request is the backend/augmentation confound and the missing per-language TTS coverage table."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, this is a genuinely useful resource: the first multilingual source-tracing benchmark for speech deepfakes, with six languages from three families, four TTS architectures, and protocols covering mono/cross-lingual, leave-one-language-out, and seen/unseen pseudo-speakers. The dataset and code are public, so the benchmark itself is citable and reusable. Second, the paper's central interpretive claim—that DSP front-ends generalize better cross-lingually than SSL front-ends—is confounded by backend choice and is not supported by the paper's own numbers.\n\nWhat's actually new: prior source-tracing work was English-only; this extends to multilingual and cross-lingual settings. The protocols are sensible, and the authors are transparent about the pseudo-speaker limitations. The results on fine-tuned SSL and language-family effects are plausible and worth reporting.\n\nThe big soft spot is RQ4. All SSL models use AASIST; the DSP family includes AASIST, ResNet18, and ECAPA-TDNN. The best DSP cross-lingual number (88.4%) comes from LFCC-ECAPA-TDNN, but no SSL model was ever paired with ECAPA-TDNN. Hold the backend fixed at AASIST: LFCC-AASIST gets 54.5%, while XLSR gets 59.8%, W2V2EN 78.2%, fine-tuned XLSR 73.5%. That is the opposite of the paper's conclusion. So \"DSP architectures inherently prioritize language-agnostic patterns\" is not a finding; the data only support something like \"ECAPA-TDNN works better than AASIST for LFCC features here,\" and even that is speculative without an SSL-ECAPA condition. The paper half-acknowledges this but the abstract and conclusion overstate it. This needs reframing and ideally an extra experiment.\n\nMinor issues: no confidence intervals or multiple seeds, so single-run Macro-F1 numbers are noisy; the pseudo-speaker clusters are unvalidated (admitted); the TTS-model-per-language coverage should be stated explicitly; and the reference list contains a string of tangential self-citations that should be trimmed.\n\nBottom line: the benchmark deserves a serious referee. The flaws are fixable—add an SSL-ECAPA run or at least stop claiming DSP features are inherently better, report variance, and tighten the citations. I'd send it to peer review, but the reviewers should push on the backend confound and the missing coverage statement.","headline":"Useful first benchmark for multilingual source tracing, but the headline DSP-vs-SSL conclusion is undermined by a front-end/backend confound.","tokens_in":14677,"tokens_out":2935,"would_cite":true,"duration_ms":34311,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper establishes the first benchmark for tracing which text-to-speech model generated a deepfake speech sample across languages, and finds that a classic signal-processing front-end (LFCC with ECAPA-TDNN) generalizes across languages","keywords":["speech deepfake source tracing","cross-lingual generalization","multilingual benchmark","LFCC","ECAPA-TDNN","self-supervised learning","text-to-speech attribution","MLAAD"],"falsifier":"Check the MCL-MLAAD dataset metadata to confirm that all four TTS architectures (Griffin-Lim, Bark, XTTS v1.1, XTTS v2) appear in each of the six languages. If any language lacks a class, recompute the cross-lingual macro-F1 scores using only the present classes; a large drop in LFCC-ECAPA-TDNN's 88.40% average would indicate the result partly reflects language detection rather than source tracing.","tokens_in":13770,"feed_emoji":"🎙️","tokens_out":4672,"duration_ms":48369,"temperature":0.7,"pith_summary":"The paper introduces the first benchmark for multilingual speech deepfake source tracing: given a synthetic speech sample, identify which text-to-speech architecture produced it, in both monolingual and cross-lingual settings. It builds a linguistically balanced dataset from the MLAAD corpus covering six languages in three language families and four TTS architectures, with protocols for cross-lingual transfer, unseen languages, and unseen speakers. The central finding is a trade-off: language-fine-tuned self-supervised front-ends achieve the best monolingual accuracy (up to 97.91% macro-F1), while a DSP-based system—LFCC features with an ECAPA-TDNN backend—is the most robust cross-lingually (88.40% average macro-F1). The paper also shows that same-language-family transfer outperforms cross-family transfer, and that unseen-language generalization is strong for LFCC-based systems.","feed_headline":"Classic LFCC features beat SSL for cross-lingual deepfake tracing","feed_subtitle":"Monolingual accuracy peaks at 97.9% with tuned SSL; cross-lingual best is 88.4% with LFCC-ECAPA-TDNN.","key_machinery":"The benchmark's engine is the MCL-MLAAD dataset: a balanced subset of the MLAAD corpus with six languages across three language families (Germanic, Romance, Slavic) and four TTS architectures (Griffin-Lim, Bark, XTTS v1.1, XTTS v2), each utterance available clean and with four noise perturbations from MUSAN. The central comparison is between DSP front-ends (LFCCs fed to ResNet18, AASIST, or ECAPA-TDNN) and SSL front-ends (XLS-R, wav2vec2.0, and language-fine-tuned variants fed to AASIST), evaluated under four protocols: mono/cross-lingual, language-family, leave-one-language-out, and seen/unseen pseudo-speakers.","core_discovery":"On its own terms, the paper's central discovery is that cross-lingual deepfake source tracing is achievable with current architectures, but the optimal model differs by scenario. In monolingual settings, fine-tuning an SSL front-end (XLS-R) on the target language gives the highest accuracy (97.91% macro-F1). In cross-lingual settings—training on one language and testing on another—the hand-crafted LFCC front-end paired with an ECAPA-TDNN backend outperforms all SSL variants, including multilingual pre-trained ones, by a large margin (88.40% vs. 59.75% for XLSR-AASIST). This suggests that DSP features capture language-agnostic generation artifacts, whereas SSL representations are biased by th","pith_inferences":["Editorial inference: the strong leave-one-language-out results imply that a small set of languages may suffice to train a universal source tracer for these four architectures; testing on typologically more distant languages (e.g., Mandarin or Arabic) would stress this claim.","Editorial inference: the cross-lingual advantage of LFCC over XLS-R suggests that generation artifacts are partly channel-like and language-independent; future SSL fine-tuning could target artifact-discrimination directly rather than language-matched features.","Editorial inference: if the four TTS architectures are not all present in every language, the cross-lingual results could conflate language detection with source tracing; checking the MCL-MLAAD metadata is a simple way to settle this.","Editorial inference: the pseudo-speaker clustering relies on embeddings from an off-the-shelf speaker model, so the unseen-speaker robustness finding is an upper bound on true speaker invariance; a dataset with ground-truth speaker identities would be needed to confirm it."],"forward_implications":["Forensic attribution of deepfake speech can work across languages, not just within a single training language.","A model trained on one language family transfers better within that family, so deployment should consider linguistic typology when data are scarce.","DSP-based systems are a stronger baseline for cross-lingual source tracing than SSL-based systems in the tested conditions, contrary to the usual deep-learning advantage.","Unseen-language generalization is achievable with LFCC+ResNet18 or LFCC+ECAPA-TDNN (around 96–97% macro-F1), enabling tracing for languages absent from training data.","The lack of a consistent seen/unseen speaker gap suggests the models are not obviously shortcutting on speaker identity, though the pseudo-speaker labels are approximate."],"supporting_citations":[{"why":"Supplies the original MLAAD corpus from which the linguistically balanced MCL-MLAAD dataset is built.","marker":"[43]"},{"why":"Provides the source tracing system design and macro-F1 evaluation metric that the benchmark adopts.","marker":"[26]"},{"why":"Informs the language partitioning method used to split languages into training and testing groups.","marker":"[47]"},{"why":"AASIST is the backend classifier used for all SSL-based source tracing models.","marker":"[54]"},{"why":"ECAPA-TDNN is the backend that achieves the best cross-lingual performance in the DSP-based models.","marker":"[56]"},{"why":"XLS-R is the multilingual SSL front-end that serves as the base for language-fine-tuned variants.","marker":"[57]"},{"why":"wav2vec2.0 is the English-only pretrained SSL front-end used to show language-bias effects.","marker":"[63]"},{"why":"ECAPA2 supplies speaker embeddings for the pseudo-speaker clustering used in the unseen-speaker protocol.","marker":"[50]"},{"why":"MUSAN provides the noise, music, babble, and reverberation perturbations applied to each utterance.","marker":"[48]"}],"fun_headline_variants":["First multilingual benchmark for tracing speech deepfake sources","Cross-lingual deepfake tracing: classic features beat SSL by 29%","Deepfake source tracing: LFCC wins cross-lingual, SSL wins monolingual","New benchmark: classic features outdo SSL for cross-lingual deepfake ID","Monolingual vs cross-lingual: best deepfake source model differs"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The claim that cross-lingual performance reflects source tracing rather than language detection assumes that every one of the four TTS architectures is available in all six languages; if a language lacks one architecture, the 4-way classification task is not well-defined and results could be confounded.","fun_headline_variants_meta":{"raw":{"variants":["First multilingual benchmark for tracing speech deepfake sources","Cross-lingual deepfake tracing: classic features beat SSL by 29%","Deepfake source tracing: LFCC wins cross-lingual, SSL wins monolingual","New benchmark: classic features outdo SSL for cross-lingual deepfake ID","Monolingual vs cross-lingual: best deepfake source model differs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000286,"raw_usage":{"total_tokens":1505,"prompt_tokens":719,"completion_tokens":786,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":463,"completion_tokens_details":{"reasoning_tokens":686}},"tokens_in":463,"tokens_out":786,"duration_ms":7783,"temperature":1.0,"reasoning_tokens":686,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T00:50:07.806325+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Check the MCL-MLAAD dataset metadata to confirm that all four TTS architectures (Griffin-Lim, Bark, XTTS v1.1, XTTS v2) appear in each of the six languages. If any language lacks a class, recompute the cross-lingual macro-F1 scores using only the present classes; a large drop in LFCC-ECAPA-TDNN's 88.40% average would indicate the result partly reflects language detection rather than source tracing.","supporting_citations":[{"cited_title":"Research on speaker identifica- tion models based on cnn and additive angular margin loss,","cited_arxiv_id":null,"evidence_quote":"Supplies the original MLAAD corpus from which the linguistically balanced MCL-MLAAD dataset is built."},{"cited_title":"Primek-net: Multi- scale spectral learning via group prime-kernel convolutional neu- ral networks for single channel speech enhancement,","cited_arxiv_id":null,"evidence_quote":"Provides the source tracing system design and macro-F1 evaluation metric that the benchmark adopts."},{"cited_title":"Polyglotfake: A novel multilingual and multimodal deepfake dataset,","cited_arxiv_id":null,"evidence_quote":"Informs the language partitioning method used to split languages into training and testing groups."},{"cited_title":"Aasist: Audio anti-spoofing us- ing integrated spectro-temporal graph attention networks,","cited_arxiv_id":null,"evidence_quote":"AASIST is the backend classifier used for all SSL-based source tracing models."},{"cited_title":"Ecapa2: A hybrid neural net- work architecture and training strategy for robust speaker embed- dings,","cited_arxiv_id":null,"evidence_quote":"ECAPA-TDNN is the backend that achieves the best cross-lingual performance in the DSP-based models."},{"cited_title":"Concept decompositions for large sparse text data using clustering,","cited_arxiv_id":null,"evidence_quote":"XLS-R is the multilingual SSL front-end that serves as the base for language-fine-tuned variants."},{"cited_title":"Unsu- pervised cross-lingual representation learning for speech recogni- tion,","cited_arxiv_id":null,"evidence_quote":"wav2vec2.0 is the English-only pretrained SSL front-end used to show language-bias effects."},{"cited_title":"Heterogeneity over homogeneity: Investigating multilingual speech pre-trained models for detecting audio deepfake,","cited_arxiv_id":null,"evidence_quote":"ECAPA2 supplies speaker embeddings for the pseudo-speaker clustering used in the unseen-speaker protocol."}],"review_version":1}