REVIEW 4 major objections 5 minor 1 cited by
The paper introduces Echoes, a music deepfake dataset where each AI-generated track is semantically matched to a real song, and shows that detectors trained on it generalize better across datasets than those trained on any current benchmark
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-02 17:31 UTC pith:ZTM7SKQX
load-bearing objection Echoes is a genuinely useful new music-deepfake dataset with a solid baseline protocol, but the paper's central claim that semantic alignment drives the gains is not actually validated — the dataset is still worth publishing after fixes. the 4 major comments →
Echoes: A semantically-aligned music deepfake detection dataset
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The central claim is that dataset construction—semantic alignment plus provider diversity—drives both benchmark difficulty and transferability. Echoes is built by generating, for each bona fide track, LLM-derived stylistic descriptions (from title and genre) that condition text-to-audio generation, with reference-audio conditioning for models that support it. In cross-dataset experiments with a frozen Wav2Vec2 XLS-R 2B encoder and a linear classifier, Echoes is the hardest to classify in-domain (9.36% EER), and training on Echoes yields the lowest average out-of-domain EER (21.02%) versus 21.36% (AIME), 28.65% (SONICS), and 37.10% (FakeMusicCaps). The authors interpret the asymmetry—existing
What carries the argument
The semantic-alignment pipeline: for each bona fide track, a large language model (ChatGPT) turns the track's title and genre into a short comma-separated list of stylistic characteristics; that descriptor conditions the text-to-audio generators, and the original waveform is additionally passed as reference for audio-to-audio providers. This is the mechanism that equalizes content (style, genre, instrumentation, tempo) between real and fake tracks, so a detector that exploits only content differences cannot succeed. Provider diversity complements it: the ten generators (e.g., Suno, Udio, AudioLDM, Stable Audio, DiffRhythm) yield distinct architectures and artifacts, so the trained detector m
Load-bearing premise
The entire claim that Echoes is both harder and more transferable depends on the assumption that LLM-generated descriptors derived from title and genre successfully align each fake track's content with its bona fide reference; if the descriptors capture only coarse metadata, fake tracks may still differ systematically in content, letting detectors exploit content cues rather than generation artifacts.
What would settle it
Generate a control subset of Echoes where the descriptor conditioning is deliberately mismatched (each fake track is produced from a descriptor of a different genre or track). If in-domain EER on that subset drops substantially below the 9.36% of the aligned set, semantic alignment is confirmed as the cause of difficulty; if the EER stays near 9%, alignment is not doing the work claimed.
If this is right
- A detector trained on AIME, SONICS, or FakeMusicCaps degrades sharply on Echoes (28.6-41.7% EER), so reported in-domain numbers on older benchmarks likely overstate deployed performance.
- Training on Echoes gives the best average transfer (21.02% EER) to the other three datasets, suggesting it is a stronger default training corpus for general-purpose music deepfake detectors.
- Per-provider accuracy on Echoes spans 87-94%, so benchmark results should be disaggregated by generator rather than pooled.
- Text-to-audio and audio-to-audio generated segments are nearly equally detectable (90.6% vs 90.1% accuracy at the EER threshold), indicating LLM-descriptor alignment is on par with waveform conditioning.
- Echoes' full-track structure (up to 480 s, median 106 s) supports future sequence-level detection that clip-based datasets cannot.
Where Pith is reading between the lines
- A direct test of the alignment mechanism: regenerate Echoes fakes with descriptors randomly reassigned across tracks; if in-domain EER falls well below 9.36%, semantic alignment is the active ingredient, while a similar EER would mean the dataset's difficulty comes from provider diversity or generator artifacts instead.
- The 9.36% EER could serve as a lower bound for current commercial generator detectability under content-matched conditions, a number useful for platform moderation capacity planning.
- If descriptors leak any source-specific information (e.g., a distinctive lyric theme that identifies the reference track), fakes could be matched to their bona fide source, inflating transfer scores; an oracle probing for such leakage would clarify the result.
- Future work could extend the alignment protocol to partial/mixed content (AI vocals over real accompaniment), the harder setting named in the paper's conclusion.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces Echoes, a dataset for music deepfake detection built from Free Music Archive bona fide tracks. For each reference track, an LLM (ChatGPT-5.0 Thinking) generates stylistic descriptors from title and genre, and ten music-generation providers produce fake tracks; audio-to-audio conditioning is used where supported. The full text reports 3,577 generated tracks / 110 hours (the abstract states 4,468 tracks / 131 hours). The baseline is frozen Wav2Vec2 XLS-R 2B embeddings plus logistic regression on 10-second segments, evaluated with EER. In-domain results give Echoes 9.36% EER vs AIME 6.40%, SONICS 2.06%, FakeMusicCaps 8.61%. Cross-dataset results show that models trained on existing datasets transfer poorly to Echoes (28.6–41.7% EER), while training on Echoes yields the best average out-of-domain EER (21.02%). The paper attributes this to provider diversity and semantic alignment.
Significance. If the dataset details are corrected and the causal claims are supported, Echoes would be a useful public benchmark: it covers ten providers, includes both short and long synthetic tracks, uses a reproducible SSL+LR baseline, performs track-level splitting to avoid leakage, and releases the data under an MIT license. The per-provider and per-modality analyses are also valuable. However, the current evidence for semantic alignment is indirect, the cross-dataset comparison is confounded by multiple dataset attributes, and the headline transfer advantage is small relative to the apparent experimental noise. These issues need to be resolved before the central claims can be accepted.
major comments (4)
- [Abstract, §1, §2.3, Table 1] The abstract states that Echoes contains 4,468 tracks and 131 hours of audio, while the full text consistently reports 3,577 tracks and 110 hours (e.g., §1, §2.3, Table 1, Table 2, §4). This is not a formatting nit: the dataset size is a core factual claim and the discrepancy prevents verification of the resource description. Please reconcile the numbers and ensure all instances match.
- [§2.1, §3.3, §4] Semantic alignment is asserted but never directly validated. The generation pipeline conditions on LLM descriptors derived from title and genre, and on reference audio where supported, but no check is reported that a generated track actually resembles its paired bona fide track in content (melody, instrumentation, tempo, lyrics, production style). The only evidence offered in §3.3 is that TTA and ATA subsets have similar per-segment accuracy at the EER threshold; this compares two conditioning modalities and does not measure content similarity. Without a direct alignment check, the abstract's causal sentence—'These findings suggest that provider diversity and semantic alignment help learn more transferable detection cues'—is unsupported.
- [Table 3, §3.2, §4] The cross-dataset comparison is confounded. Echoes differs from AIME, SONICS, and FakeMusicCaps in total hours, provider count, track duration distributions, and source corpus simultaneously. The observed transfer advantage of training on Echoes cannot be attributed specifically to 'provider diversity and semantic alignment' as claimed in the Abstract and Conclusion. Controlled ablations or subset-matching experiments (e.g., matching provider count, total duration, or segment content) are needed, or the causal claim should be substantially softened.
- [Table 3, §3.2] No error bars, confidence intervals, or multiple-seed experiments are reported for any EER value. The headline 'training on Echoes yields the strongest generalization performance' rests on a 0.34-percentage-point gap (21.02% vs 21.36% for AIME), which may be within experimental noise. Additionally, the FakeMusicCaps row lists an out-of-domain average of 37.10, but (31.03 + 30.16 + 41.66) / 3 = 34.28; the average is arithmetically incorrect. Please provide repeated runs with seeds and variability estimates, and correct the table.
minor comments (5)
- [§1] Typo: 'post-processinw' should be 'post-processing'.
- [Table 4] The header contains 'ACEStep Producer'; the provider name elsewhere is 'AceStep' (Table 2). 'Producer' appears to be a stray label. Also, the accuracy figures at the EER threshold would benefit from being tied to the threshold definition (0.91) with a brief explanation.
- [§2.3, §4] The dataset size is typeset inconsistently as '3 577' and '3577'; choose one format throughout.
- [§3.3] The terminology jumps between 'high-level intent (style/theme)' in §4 and 'semantic-level alignment' in the Abstract. These are not equivalent; please align the terminology or define what 'semantic' means operationally.
- [Table 3] The note that 'higher is better' for in-domain EER expresses a dataset-centric desirability judgment, not a model-quality measure. This should be clearly framed as an interpretation, not a property of the metric.
Circularity Check
No circular derivation: Echoes results are empirical benchmarks; self-citations are motivational only.
full rationale
The paper's core claims are empirical measurements obtained with a fixed protocol: identical frozen Wav2Vec2 XLS-R embeddings and a logistic-regression classifier are applied to all datasets, and no parameter is fitted to the reported EER outcomes. The dataset construction (LLM descriptors + reference-audio conditioning) is described independently of the evaluation results, and the in-domain/out-of-domain comparisons do not reduce to the construction by definition. The weak point flagged by the skeptic—that semantic alignment is asserted rather than directly validated—is a validity concern, not circularity: even if the causal attribution 'provider diversity and semantic alignment help learn more transferable detection cues' is under-supported, the EER numbers themselves are not derived from that assumption. The paper cites the authors' own prior work ([6], [13]) for motivation (SSL representation choice and shortcut-avoidance in datasets), but these citations are not load-bearing for the central transfer comparisons, which would stand unchanged if those citations were removed. There is an internal inconsistency in dataset scale (abstract 4,468 tracks/131 h vs. body 3,577 tracks/110 h), but that is an editing error, not a circular step. Overall, no prediction reduces by construction to a fitted input or to a self-citation chain; the score reflects only the presence of minor, non-load-bearing self-citations.
Axiom & Free-Parameter Ledger
free parameters (2)
- Logistic regression regularization strength C =
1e6
- 10-second segment length =
10 s
axioms (4)
- domain assumption LLM-generated textual descriptions preserve enough stylistic information to make fake tracks semantically aligned with bona fide tracks.
- domain assumption Wav2Vec2 XLS-R 2B frozen embeddings plus logistic regression is a fair common baseline for comparing dataset difficulty and transfer.
- domain assumption Songs from the Free Music Archive (310 CC0/CC-BY/public-domain tracks) are representative bona fide music for pop/rock/electronic genres.
- domain assumption The 80:20 random split at track level is statistically adequate without multiple seeds.
read the original abstract
We introduce Echoes, a new dataset for music deepfake detection designed for training and benchmarking detectors under realistic and provider-diverse conditions. Echoes comprises 4,468 tracks (131 hours of audio) spanning multiple genres (pop, rock, electronic), and includes content generated by ten popular AI music generation systems. To prevent shortcut learning and promote robust generalization, the dataset is deliberately constructed to be challenging, enforcing semantic-level alignment between spoofed audio and bona fide references. This alignment is achieved by conditioning generated audio samples directly on bona-fide waveforms or song descriptors. We evaluate Echoes in a cross-dataset setting against three existing AI-generated music datasets using state-of-the-art Wav2Vec2 XLS-R 2B representations. Results show that (i) Echoes is the hardest in-domain dataset; (ii) detectors trained on existing datasets transfer poorly to Echoes; (iii) training on Echoes yields the strongest generalization performance. These findings suggest that provider diversity and semantic alignment help learn more transferable detection cues.
Figures
Forward citations
Cited by 1 Pith paper
-
Finding the noise: Zero-shot AI Music Detection
A zero-shot method based on fakeprints, NMF and a blur-based reconstruction error detects unknown AI-music generators in one-class and clustering setups, working for most services but missing Mubert and pre-v9 Mureka.
Reference graph
Works this paper leans on
-
[1]
Modern generators can pro- duce realistic music from short prompts, increasing the feasi- bility of large-scale synthetic music creation
Introduction 1 Recent advances in music generation have enabled the syn- thesis of convincing songs that emulate genres, production aes- thetics, and vocal styles at scale. Modern generators can pro- duce realistic music from short prompts, increasing the feasi- bility of large-scale synthetic music creation. This shift cre- ates integrity and provenance ...
2026
-
[2]
Generation pipeline To build our dataset, we start from bona fide music tracks from the Free Music Archive (FMA) [26]
Dataset description 2.1. Generation pipeline To build our dataset, we start from bona fide music tracks from the Free Music Archive (FMA) [26]. We select 310 tracks li- censed under CC0, CC-BY or the public domain. For each bona fide track, we generate a corresponding fake samples using models from ten providers (Table 2). To obtain aligned samples that p...
-
[3]
Table 2:AI-music providers used to create Echoes
Evaluation We provide initial results on our dataset, as well as cross- dataset generalization results using the three other major avail- able datasets (AIME, SONICS, FakeMusicCaps). Table 2:AI-music providers used to create Echoes. “Unknown” means the provider does not publicly specify the model. The two generator types are either audio-to-audio (A→A) or...
-
[4]
Echoes contains 3577tracks (110hours) across pop, rock, and electronic mu- sic genres
Conclusion We presentedEchoes, a semantically-aligned dataset for AI- generated music detection built to support training and bench- marking under realistic provider diversity. Echoes contains 3577tracks (110hours) across pop, rock, and electronic mu- sic genres. For each bona fide song from the Free Music Archive, we generate AI-produced counterparts fro...
-
[5]
Automatic speaker verification spoofing and deep- fake detection using wav2vec 2.0 and data augmentation,
H. Tak, M. Todisco, X. Wang, J.-w. Jung, J. Yamagishi, and N. Evans, “Automatic speaker verification spoofing and deep- fake detection using wav2vec 2.0 and data augmentation,” inThe Speaker and Language Recognition Workshop (Odyssey 2022). ISCA, 2022
2022
-
[6]
Towards generalisable and calibrated audio deepfake detection with self- supervised representations,
O. Pascu, A. Stan, D. Oneata, E. Oneata, and H. Cucu, “Towards generalisable and calibrated audio deepfake detection with self- supervised representations,” inInterspeech, vol. 2024, 2024, pp. 4828–4832
2024
-
[7]
Deezer: 28% of all delivered music is now fully AI-generated,
Deezer, “Deezer: 28% of all delivered music is now fully AI-generated,” Deezer Newsroom (press release), Sep. 2025, accessed: 2026-01-11. [Online]. Available: https: //newsroom-deezer.com/2025/09/28-fully-ai-generated-music/
2025
-
[8]
End-to-end anti-spoofing with RawNet2,
H. Tak, J. Patino, M. Todisco, A. Nautsch, N. Evans, and A. Larcher, “End-to-end anti-spoofing with RawNet2,” in ICASSP, 2021
2021
-
[9]
AASIST: Audio anti-spoofing using in- tegrated spectro-temporal graph attention networks,
J.-w. Jung, H.-S. Heo, H. Tak, H.-j. Shim, J. S. Chung, B.-J. Lee, H.-J. Yu, and N. Evans, “AASIST: Audio anti-spoofing using in- tegrated spectro-temporal graph attention networks,” inICASSP, 2022
2022
-
[10]
Singing voice graph modeling for singfake detection,
X. Chen, H. Wu, R. Jang, and H.-y. Lee, “Singing voice graph modeling for singfake detection,” inProc. Interspeech 2024, 2024, pp. 4843–4847
2024
-
[11]
Speech is silver, silence is golden: What do asvspoof-trained models really learn?
N. M ¨uller, F. Dieckmann, P. Czempin, R. Canals, K. B ¨ottinger, and J. Williams, “Speech is silver, silence is golden: What do asvspoof-trained models really learn?” inProc. ASVSPOOF 2021, 2021, pp. 55–60
2021
-
[12]
Is synthetic voice detection research going into the right direction?
S. Borz `ı, O. Giudice, F. Stanco, and D. Allegra, “Is synthetic voice detection research going into the right direction?” inCVPRW, 2022
2022
-
[13]
Detect- ing music deepfakes is easy but actually hard,
D. Afchar, G. Meseguer-Brocal, and R. Hennequin, “Detect- ing music deepfakes is easy but actually hard,”arXiv preprint arXiv:2405.04181, 2024
Pith/arXiv arXiv 2024
-
[14]
From audio deepfake detection to AI-generated music detection–A pathway and overview,
Y . Li, M. Milling, L. Specia, and B. W. Schuller, “From audio deepfake detection to AI-generated music detection–A pathway and overview,”arXiv preprint arXiv:2412.00571, 2024
Pith/arXiv arXiv 2024
-
[15]
Asvspoof 2019: Future horizons in spoofed and fake audio detec- tion,
M. Todisco, X. Wang, V . Vestman, M. Sahidullah, H. Delgado, A. Nautsch, J. Yamagishi, N. Evans, T. Kinnunen, and K. A. Lee, “Asvspoof 2019: Future horizons in spoofed and fake audio detec- tion,” inInterspeech 2019. International Speech Communication Association, 2019, pp. 1008–1012
2019
-
[16]
Does audio deepfake detection generalize?
N. M. M ¨uller, P. Czempin, F. Dieckmann, A. Froghyar, and K. B¨ottinger, “Does audio deepfake detection generalize?” inIn- terspeech, 2022
2022
-
[17]
Simple and controllable music gen- eration,
J. Copet, F. Kreuk, I. Gat, T. Remez, D. Kant, G. Synnaeve, Y . Adi, and A. D ´efossez, “Simple and controllable music gen- eration,”Advances in Neural Information Processing Systems, vol. 36, pp. 47 704–47 720, 2023
2023
-
[18]
Audioldm: Text-to-audio generation with la- tent diffusion models,
H. Liu, Z. Chen, Y . Yuan, X. Mei, X. Liu, D. Mandic, W. Wang, and M. Plumbley, “Audioldm: Text-to-audio generation with la- tent diffusion models,” inProceedings of the 40th International Conference on Machine Learning, PMLR 2023, vol. 202. In- ternational Machine Learning Society (IMLS), 2023, pp. 21 450– 21 474
2023
-
[19]
Circum- venting shortcuts in audio-visual deepfake detection datasets with unsupervised learning,
S. Smeu, D.-A. Boldisor, D. Oneata, and E. Oneata, “Circum- venting shortcuts in audio-visual deepfake detection datasets with unsupervised learning,” inCVPR, 2025
2025
-
[20]
Spoofed training data for speech spoofing countermeasure can be efficiently created using neural vocoders,
X. Wang and J. Yamagishi, “Spoofed training data for speech spoofing countermeasure can be efficiently created using neural vocoders,” inICASSP, 2023
2023
-
[21]
On the effec- tiveness of dataset alignment for fake image detection,
A. S. Rajan, U. Ojha, J. Schloesser, and Y . J. Lee, “On the effec- tiveness of dataset alignment for fake image detection,” inICLR, 2025
2025
-
[22]
MusicLM: Generating music from text,
A. Agostinelli, T. I. Denk, Z. Borsos, J. Engel, M. Verzetti, A. Caillon, Q. Huang, A. Jansen, A. Roberts, M. Tagliasacchi et al., “MusicLM: Generating music from text,”arXiv preprint arXiv:2301.11325, 2023
Pith/arXiv arXiv 2023
-
[23]
Benchmarking music generation models and metrics via human preference studies,
F. Gr ¨otschla, A. Solak, L. A. Lanzend ¨orfer, and R. Wattenhofer, “Benchmarking music generation models and metrics via human preference studies,” inICASSP, 2025
2025
-
[24]
supports detection and attribution but remains largely clip- oriented and does not contain long-form structure; SONICS
-
[25]
emphasizes long-duration counterfeit-song detection, yet covers only two popular providers. To address these gaps, we introduceEchoes, an evaluation- centric dataset for training and benchmarking AI-generated mu- sic detectors. Echoes is designed to be semantically aligned and covers a broad range of providers. The dataset contains3 577 tracks totaling 11...
Pith/arXiv arXiv 2026
-
[26]
Stable audio open,
Z. Evans, J. D. Parker, C. Carr, Z. Zukowski, J. Taylor, and J. Pons, “Stable audio open,” inICASSP, 2025
2025
-
[27]
The AI mu- sic arms race: On the detection of AI-generated music,
L. Cros Vila, B. Sturm, L. Casini, and D. Dalmazzo, “The AI mu- sic arms race: On the detection of AI-generated music,”Transac- tions of the International Society for Music Information Retrieval, vol. 8, no. 1, pp. 179–194, 2025
2025
-
[28]
Singfake: Singing voice deepfake detection,
Y . Zang, Y . Zhang, M. Heydari, and Z. Duan, “Singfake: Singing voice deepfake detection,” inICASSP, 2024
2024
-
[29]
SVDD 2024: The inaugural singing voice deepfake detection challenge,
Y . Zhang, Y . Zang, J. Shi, R. Yamamoto, T. Toda, and Z. Duan, “SVDD 2024: The inaugural singing voice deepfake detection challenge,” inSLT, 2024
2024
-
[30]
FakeMusicCaps: A dataset for detection and attribution of synthetic music generated via text-to-music models,
L. Comanducci, P. Bestagini, and S. Tubaro, “FakeMusicCaps: A dataset for detection and attribution of synthetic music generated via text-to-music models,”Journal of Imaging, vol. 11, no. 7, p. 242, 2025
2025
-
[31]
Sonics: Synthetic or not-identifying counterfeit songs,
M. A. Rahman, Z. I. A. Hakim, N. H. Sarker, B. Paul, and S. A. Fattah, “Sonics: Synthetic or not-identifying counterfeit songs,” inThe Thirteenth International Conference on Learning Repre- sentations, 2024
2024
-
[32]
Fma: A dataset for music analysis,
M. Defferrard, K. Benzi, P. Vandergheynst, and X. Bresson, “Fma: A dataset for music analysis,” in18th International Society for Music Information Retrieval Conference, 2017
2017
-
[33]
ACE-step: A step towards music generation foundation model,
J. Gong, S. Zhao, S. Wang, S. Xu, and J. Guo, “ACE-step: A step towards music generation foundation model,”arXiv preprint arXiv:2506.00045, 2025
Pith/arXiv arXiv 2025
-
[34]
Brev AI Music Generator,
Brev, “Brev AI Music Generator,” Website, Jan. 2026, Accessed: 2026-01-13. [Online]. Available: https://brev.ai/
2026
-
[35]
Z. Ning, H. Chen, Y . Jiang, C. Hao, G. Ma, S. Wang, J. Yao, and L. Xie, “DiffRhythm: Blazingly fast and embarrassingly sim- ple end-to-end full-length song generation with latent diffusion,” arXiv preprint arXiv:2503.01183, 2025
Pith/arXiv arXiv 2025
-
[36]
Mubert AI Music Generator,
Mubert, “Mubert AI Music Generator,” Website, Jan. 2026, Accessed: 2026-01-13. [Online]. Available: https://mubert.com/
2026
-
[37]
Stable Audio 2.5,
Stability AI, “Stable Audio 2.5,” Website, Jan. 2026, Accessed: 2026-01-13. [Online]. Available: https://stability.ai/stable-audio
2026
-
[38]
Songgen: A single stage auto-regressive transformer for text-to-song generation,
Z. Liu, S. Ding, Z. Zhang, X. Dong, P. Zhang, Y . Zang, Y . Cao, D. Lin, and J. Wang, “Songgen: A single stage auto-regressive transformer for text-to-song generation,” inInternational Confer- ence on Machine Learning. PMLR, 2025, pp. 38 351–38 364
2025
-
[39]
Suno: AI Music,
Suno, “Suno: AI Music,” Website, Jan. 2026, Accessed: 2026-01-13. [Online]. Available: https://suno.com/
2026
-
[40]
Riffusion – Stable Diffusion for Real-Time Music Generation,
Seth Forsgren and Hayk Martiros, “Riffusion – Stable Diffusion for Real-Time Music Generation,” GitHub repository, Dec. 2022, Accessed: 2026-01-13. [Online]. Available: https: //github.com/riffusion/riffusion-hobby
2022
-
[41]
Udio: AI Music Generator,
Udio, “Udio: AI Music Generator,” Website, Jan. 2026, Accessed: 2026-01-13. [Online]. Available: https://www.udio.com/
2026
-
[42]
wav2vec 2.0: A framework for self-supervised learning of speech representa- tions,
A. Baevski, Y . Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representa- tions,” inNeurIPS, 2020
2020
-
[43]
Xls-r: Self- supervised cross-lingual speech representation learning at scale,
A. Babu, C. Wang, A. Tjandra, K. Lakhotia, Q. Xu, N. Goyal, K. Singh, P. von Platen, Y . Saraf, J. Pinoet al., “Xls-r: Self- supervised cross-lingual speech representation learning at scale,” inProc. Interspeech 2022, 2022, pp. 2278–2282
2022
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.