REVIEW 3 major objections 4 minor 12 references
Evaluating Fake Music Detection Performance Under Audio Augmentations
T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read A two-semitone pitch shift is enough to make a fake-music detector call Suno real.
desk verdict A useful, honest early probe of SONICS robustness with released code and data, but the headline pitch-shift claim is a qualitative reading of a selected figure, not a reported number. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the SONICS SpecTTTra-α detector, the configuration its creators reported as most accurate, applied to 120-second audio samples. The evaluation machinery is a purpose-built dataset of 20 songs per source (real, Suno, Udio, YuE, MusicGen), generated from LLM-written prompts, plus a battery of audio augmentations (aliasing, bit crush, equalization, filtering, masking, mp3/ogg compression, pitch shift, speed changes, silence, reverb, vibrato, white noise) applied across parameter ranges kept within a reasonable range. The model's output is the mean probability of fakeness, and the paper's figure isolates augmentations with the most significant impact. This setup is what turns a robustness suspicion into the claim that light edits fool the detector.
What would settle it
Re-run the same SONICS configuration on a larger held-out sample (say, hundreds of songs per generator) with pitch shifts of −2 to +2 semitones and blind human perception ratings for the same shifts. If the model's accuracy drop vanishes or falls within the baseline variance, or if human listeners reliably notice the two-semitone change, the central claim is not supported.
Extended reading notes
Core claim
The central discovery is that SONICS (the SpecTTTra-α variant with 120-second samples), despite high accuracy on unmodified Suno tracks (96.24±5% fake probability), degrades sharply under light audio transformations. A pitch shift down by two semitones is enough for the model to label Suno as highly real, even though a human listener would barely perceive it. Silencing fragments raises the fake probability monotonically, with an empty file classified as fake. The paper also reports weak generalization to unseen generators: MusicGen tracks are mostly classified as real (34.83±31% fake probability) and Udio hovers near chance (50.51±43%). The authors infer that the model latches onto spectral artifacts in high frequencies rather than holistic musical content.
Load-bearing premise
The claim that 'light augmentations' fool the detector assumes the chosen augmentations and their parameter ranges fairly represent real-world light edits, and that 20 songs per generator and single-run probabilities without error bars are enough to estimate the effect.
Editorial extensions
If this is right
- SONICS cannot be assumed safe for deployment on user-uploaded or streaming audio, where pitch shifts and silent gaps are common.
- Training detectors with a diverse augmentation set, including pitch shifts and pauses, should harden them against this failure mode.
- Even an empty audio file is classified as fake, so silence handling (e.g., an ambiguous class or a loss term for silence) is needed before the detector can be used in realistic pipelines.
- The model's consistent reaction across sources suggests a shared spectral artifact cue, so explainability work on frequency bands may identify the root cause.
Reading between the lines
- The silence finding implies a practical gating mechanism, such as detecting non-silent segments before classification, which the paper does not spell out.
- If the model relies on absolute pitch, a simple fix—training with pitch-transposed copies of real music—could be tested; the paper does not propose this.
- The lack of confidence intervals means the headline drop may shrink or grow with more data; a larger benchmark would be needed before trusting the effect size.
- The same augmentation battery could be applied to other detectors, and comparing their response patterns would show whether pitch fragility is specific to SONICS or common to the approach.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This late-breaking extended abstract evaluates the robustness of the SONICS fake-music detector under a range of audio augmentations. The authors construct a dataset of 20 real songs and 20 songs from each of Suno, Udio, YuE, and MusicGen, using deepseek-r1 to generate independent prompts. They first measure baseline fake-probability scores (Table 1), then apply augmentations such as pitch shifting, filtering, compression, time stretching, silence insertion, and noise, reporting that the model's predictions degrade under supposedly light modifications. The central claims are that the model generalizes poorly to unseen generators, that a two-semitone downward pitch shift can make Suno tracks appear highly real, that silence pushes predictions toward fake, and that high-frequency corruption correlates with fake predictions. The paper concludes that SONICS is not robust and suggests augmentations during training and explainability as future work.
Significance. If the central claims hold, this is a timely and useful external evaluation of a state-of-the-art music deepfake detector, highlighting concrete failure modes that are perceptually minor for humans. The authors provide a public dataset and code, use independent prompt generation to avoid contamination, and test several generators beyond the detector's training distribution. However, the significance is limited by the small sample size (20 songs per generator), the use of a single detector model, the absence of statistical tests or confidence intervals on the augmentation results, and the selective presentation of only the most impactful augmentations in Figure 1. These issues leave the magnitude and generality of the reported degradation unestablished, despite the rhetorical strength of the abstract's claim that performance 'decreases significantly even with the introduction of light augmentations.'
major comments (3)
- [§3.2, Figure 1] The central claim that a two-semitone downward pitch shift 'already fools the model into classifying suno as highly real' is not numerically supported. The text refers to Figure 1, but the figure is explicitly said to show only 'augmentations with the most significant impact,' and no per-condition mean, per-track scores, confidence interval, or paired significance test is provided for the -2 semitone condition. With n=20 and a baseline standard deviation of 5 percentage points for Suno, the reported flip could be a selection artifact or an outlier rather than a representative light augmentation. This claim is load-bearing for the abstract's 'light augmentations' statement and should be backed by summary statistics or a scatter plot with paired comparisons.
- [§3.1, Table 1] The generalization results are reported only as means with parenthetical '±' values, with no statement of whether these are standard deviations, standard errors, or ranges, and no statistical tests against chance. For Udio, the mean is 50.51% with a spread of ±43 percentage points, making the baseline statistically indistinguishable from random guessing on 20 tracks. The conclusion that the model 'doesn't generalize well' is therefore not formally established, nor is the claim that only real and Suno songs are classified correctly. Confidence intervals or a paired test (e.g., Wilcoxon against 50%) should be reported for each generator.
- [§3.2] The parameter ranges that define 'light' augmentations are not stated in the paper; the authors only say the ranges are 'reasonable' and refer the reader to a GitHub repository. Because the central claim concerns the effect of 'light' modifications, the actual values (e.g., pitch-shift step sizes, filter cutoff frequencies, noise levels, bit-crush amounts) must be specified in the text or an appendix. Without this, the reader cannot assess whether the tested augmentations are indeed light or whether the reported effects are driven by severe settings that would be obvious to a human listener.
minor comments (4)
- [Table 1] The caption reads 'Mean Probability of fakeness' and parenthetical values like '6.1 (±4)%' are ambiguous; state explicitly whether the parenthetical is a standard deviation, standard error, or interquartile range.
- [References] Several references have formatting issues, such as 'R. Y . et al.' and 'D.-A. et al.'; these should be expanded or corrected for completeness.
- [§3.2] The statement that 'an empty file is classified as fake' is intriguing, but it is not explained how a silent file is presented to a model that expects 120-second audio; clarify the preprocessing.
- [Title and author block] The title and author names contain odd spacing and encoding artifacts; these should be cleaned up for publication.
Circularity Check
No circularity: the paper is an external evaluation of the SONICS detector against independently generated audio and augmentations, with no fitted parameters or self-citation chain doing load-bearing work.
full rationale
This paper does not derive any prediction from a fitted input. It takes SONICS, a model published by other authors, and measures its outputs on a dataset constructed from real music and four generative systems (Suno, Udio, YuE, MusicGen), with prompts generated by deepseek-r1. No parameter of SONICS is fitted in this work, and no quantity reported here is used to define SONICS's behavior. The central robustness claim ('a shift down of two semitones already fools the model') is an observed output of an external model under an augmentation, not a quantity constructed to equal its own input. The authors' own framework and augmentation code are independent of the detector's training procedure. No step reduces by definition to its own inputs, and no load-bearing argument rests on a self-citation: the cited SONICS paper is external prior work being tested, not an unverified premise imported to force a conclusion. Concerns about missing confidence intervals, selective figure presentation, and small sample size (n=20 per generator) are statistical-evidence weaknesses, not circularity, and therefore do not raise the circularity score.
Assumptions & free parameters
assumptions (3)
- domain assumption SONICS SpecTTTra-alpha is a representative state-of-the-art fake-music detector.
- domain assumption Twenty songs per generator are sufficient to estimate the model's classification behavior.
- domain assumption The augmentation parameter ranges are 'reasonable' and representative of real-world light edits.
Cite this review
Pith. "Pith review of Evaluating Fake Music Detection Performance Under Audio Augmentations." pith.science (2026). https://pith.science/paper/TMBBBII6
@misc{pith2026250710447,
author = {Pith},
title = {Pith review of: Evaluating Fake Music Detection Performance Under Audio Augmentations},
year = {2026},
howpublished = {\url{https://pith.science/paper/TMBBBII6}},
note = {Machine review of arXiv:2507.10447}
}
read the original abstract
With the rapid advancement of generative audio models, distinguishing between human-composed and generated music is becoming increasingly challenging. As a response, models for detecting fake music have been proposed. In this work, we explore the robustness of such systems under audio augmentations. To evaluate model generalization, we constructed a dataset consisting of both real and synthetic music generated using several systems. We then apply a range of audio transformations and analyze how they affect classification accuracy. We test the performance of a recent state-of-the-art musical deepfake detection model in the presence of audio augmentations. The performance of the model decreases significantly even with the introduction of light augmentations.
Reference graph
Works this paper leans on
-
[1]
Evaluating Fake Music Detection Per- formance Under Audio Augmentations
INTRODUCTION Generative music models have made significant progress in recent years, allowing for the creation of synthetic mu- sic that closely resembles one composed by humans. In response, machine learning-based detectors are being de- signed to automatically recognize synthetic music and dis- tinguish it from real compositions. Similar work has al- re...
work page 2025
-
[2]
DATASET The dataset consists of 20 songs from each model, includ- ing both the models used to train the SONICS model and new ones, to test its ability to generalize across differ- ent generative models. To generate the dataset, we used prompts generated with deepseek-r1 [7] independently for every model. Then all audio files were initially downsam- pled t...
-
[3]
Evaluating Fake Music Detection Performance Under Audio Augmentations
EXPERIMENTS The original work offers a few different subtypes of the model. We have chosen ”SpecTTTra-α” with 120 seconds audio samples, which was the configuration that authors described as reaching the highest classification accuracy. 3.1 Generalisation to different models Initially, we ran classification on our dataset as is, the re- sults are shown in...
work page Pith review arXiv 2025
-
[4]
CONCLUSION AND FURTHER WORK Firstly, we show that the SONICS model doesn’t perform well on music from unseen generators. This is to be ex- pected, since machine learning models are known for be- ing unable to handle distribution shift between training and test data. Secondly, the results can be easily skewed by cer- tain augmentations. The model almost al...
-
[5]
Xlsr-mamba: A dual- column bidirectional state space model for spoofing attack detection,
Y . Xiao and R. K. Das, “Xlsr-mamba: A dual- column bidirectional state space model for spoofing attack detection,” 2025. [Online]. Available: https: //arxiv.org/abs/2411.10027
arXiv 2025
-
[6]
H. Tak, M. Todisco, X. Wang, J. weon Jung, J. Yamagishi, and N. Evans, “Automatic speaker verification spoofing and deepfake detection using wav2vec 2.0 and data augmentation,” 2022. [Online]. Available: https://arxiv.org/abs/2202.12233
arXiv 2022
-
[7]
Sonics: Synthetic or not – identifying counterfeit songs,
M. A. Rahman, Z. I. A. Hakim, N. H. Sarker, B. Paul, and S. A. Fattah, “Sonics: Synthetic or not – identifying counterfeit songs,” 2025. [Online]. Available: https://arxiv.org/abs/2408.14080
arXiv 2025
-
[8]
Detecting music deepfakes is easy but actually hard,
D. Afchar, G. Meseguer-Brocal, and R. Hennequin, “Detecting music deepfakes is easy but actually hard,”
Show all 12 references
-
[10]
Yue: Scaling open foundation models for long-form music generation,
R. Y . et al., “Yue: Scaling open foundation models for long-form music generation,” 2025. [Online]. Available: https://arxiv.org/abs/2503.08638
2025
-
[11]
Simple and controllable music generation,
J. Copet, F. Kreuk, I. Gat, T. Remez, D. Kant, G. Synnaeve, Y . Adi, and A. Défossez, “Simple and controllable music generation,” 2024. [Online]. Available: https://arxiv.org/abs/2306.05284
2024 arXiv
-
[12]
Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,
D.-A. et al., “Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,” 2025. [Online]. Available: https://arxiv.org/abs/2501.12948
2025 arXiv
-
[2024]
Available: https://arxiv.org/abs/2405
[Online]. Available: https://arxiv.org/abs/2405. 04181
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.