REVIEW 3 major objections 6 minor 43 references
Detecting Musical Deepfakes
T0 review · 3 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read A standard image-classification network can identify AI-generated music even after pitch shifting and tempo stretching.
desk verdict A clean but incremental ResNet18 baseline on FakeMusicCaps; the claim about adversarial robustness is not supported because training and testing share the same manipulation distribution. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing machinery is the mel-spectrogram image representation plus a pretrained ResNet18 classifier. Mel spectrograms display time, frequency, and amplitude in one image, so audio classification becomes image classification; ResNet18, an 18-layer convolutional network pretrained on image data, is adapted by replacing its final layer with a two-class output. The adversarial-simulation machinery is the random manipulation: each clip gets its own pitch shift drawn from -2 to +2 semitones, tempo stretch from 0.8 to 1.2, or both, before spectrogram generation.
What would settle it
Train the same model on only unmodified clips, then test on clips manipulated at inference time with pitch shifts outside -2 to +2 semitones or tempo factors outside 0.8 to 1.2, such as a three-semitone shift or a 1.4x stretch; if accuracy drops toward chance, the not-insurmountable conclusion would fail for unmatched adversarial edits.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is that a transfer-learned convolutional network trained on spectrogram images provides a workable detector for synthetic music under light adversarial modification. Using a paired corpus of 5,373 human and 5,373 machine-generated ten-second clips, the author converts audio to mel spectrograms and trains ResNet18 with two output classes. The classifier reaches F1 0.878 and accuracy 0.885 on clean clips; with per-clip random pitch shifts between -2 and +2 semitones, tempo stretches between 0.8 and 1.2, or both, F1 remains above 0.83. The paper interprets these results as showing that adversarial evasion of musical deepfake detection is not insurmountable, and that detection systems can therefore be part of a protective response to generative music platforms.
Load-bearing premise
The load-bearing premise is that random pitch and tempo changes applied to the training and test sets the same way behave like a real adversary trying to evade detection.
Editorial extensions
If this is right
- A detector trained on clean clips generalizes to mild pitch and tempo changes, with F1 falling from 0.878 on clean clips to no lower than 0.837 when both manipulations are applied.
- Sequential training across manipulated datasets can raise recall for human clips to 0.889, but it doubles the false-positive rate relative to the single-dataset baseline.
- With only 20 training epochs, every configuration stays above 0.80 in F1, accuracy, recall, and precision, so the approach is inexpensive to reproduce.
- The best single configuration reaches 0.885 accuracy and 0.911 precision on unmodified clips, giving a plausible screening baseline for streaming platforms.
Reading between the lines
- A stronger adversarial test would train only on clean audio and then apply previously unseen manipulations at test time; the reported numbers leave that version of robustness open.
- False positives may matter more than false negatives in real deployment: human musicians who use MIDI, effects, or AI-assisted production could be flagged by the same detector.
- The same pipeline could be extended to other cheap manipulations such as reverb, echo, distortion, volume changes, and codec compression, and to deeper networks, as the paper's own future-work section suggests.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript reports experiments on detecting AI-generated music (deepfakes) using the FakeMusicCaps dataset. The author converts audio clips to mel spectrograms, applies pitch-shift and tempo-stretch modifications to simulate adversarial evasion, and trains a transfer-learned ResNet18 to classify clips as human or deepfake. Table I reports F1, accuracy, recall, false positive rate, false negative rate, precision, and specificity for five conditions: baseline, tempo stretch, pitch shift, pitch shift plus tempo stretch, and a continuous-learning condition. The paper concludes in Section IV.D that 'the problem of adversarial techniques to evade deepfake detection is not an insurmountable one.' A large portion of the paper is devoted to ethical, legal, and societal implications of musical deepfakes, including impacts on musicians, copyright and right-of-publicity issues, and positive uses of the technology.
Significance. If the robustness conclusion were supported, the paper would provide a useful baseline showing that a standard transfer-learned CNN with simple augmentation can resist common audio manipulations. The study has genuine strengths: the labels are external ground truth from FakeMusicCaps rather than the model's own predictions, so there is no definitional circularity; it uses a public dataset and standard tools (librosa, torchvision); and Table I reports a complete set of metrics that others can build on. However, the experimental design does not actually measure adversarial evasion: manipulated audio is present in the training distribution, so the results demonstrate in-distribution classification under augmentation, not robustness against an adversary who chooses transformations after the model is fixed. The significance of the paper is therefore limited unless the design is corrected; the ethical discussion, while broad, is not empirically load-bearing.
major comments (3)
- [Section IV.B / IV.C / IV.D, Table I] The conclusion in Section IV.D that 'the problem of adversarial techniques to evade deepfake detection is not an insurmountable one' is not supported by the experiments. The pitch-shifted and tempo-stretched datasets are created in Section III.C and used as training data in Section IV.B, and the test splits are drawn from the same modified pools; the 'Continuous Learning' condition additionally trains on every manipulated version before evaluation. An adversary, by contrast, applies transformations at inference time to a fixed model, possibly using magnitudes or types outside the training range. As designed, Table I measures the model's ability to classify audio perturbed by the same augmentation family seen in training; that is data augmentation, not adversarial robustness. The claim would require at least a model trained only on clean clips and evaluated on modified test clips, and preferably held-out manipulation ranges or types.
- [Section IV.B / Table I] The paper reports a single run with no random seeds, no standard deviations, and no confidence intervals. With 1,074 test tensors, the 2-4 percentage point differences between conditions (for example, pitch shift versus tempo stretch) may well be within run-to-run noise, so the qualitative claims in Section IV.C, such as 'it appears easier to detect rudimentary pitch shifts compared to temporal stretches,' are not statistically supported. Reporting multiple seeds with mean and confidence intervals is necessary before comparing conditions.
- [Section III.C] The choice of manipulation parameters (-2 to +2 semitones, 0.8 to 1.2 tempo stretch) is not justified as a proxy for a 'moderately skilled malicious actor,' and no evidence is provided that these ranges are representative of real-world evasion attempts. Even after adding a clean-trained evaluation, the model should be tested on out-of-range magnitudes (e.g., ±3 semitones, 0.7 and 1.3 stretch factors) and on manipulation types not seen in training (e.g., reverb, EQ, dynamic range compression) to support the general claim that evasion is 'not insurmountable.'
minor comments (6)
- [Section III.A] The sentence '5,373 tracks were selected from the 5 different deepfake platform subsets of the FakeMusicCaps dataset' is ambiguous: does it mean 5,373 deepfake tracks in total (about 1,075 per platform) or 5,373 per platform? The reported total of 10,746 tracks suggests the former, but the wording should be explicit.
- [Figure numbering] The manuscript contains two 'Fig. 1' labels; the second figure (Recall by Data Source Type) should be numbered Fig. 2, and the first figure should be referenced in the text.
- [References] The URL 'http://www.ctan.org/tex-archive/macros/latex/contrib/IEEEtran/' appears to be incorrectly embedded in reference [8]; it should be removed or placed in an appropriate entry.
- [Section IV.A] The preprocessing details for the mel spectrograms are missing: sample rate, number of mel bins, FFT and hop length, handling of clips of variable duration after stretching, and the input image size fed to ResNet18. These details are essential for reproducibility.
- [Abstract and Introduction] The wording 'To simulate a real-world adversarial entity' is awkward; consider 'To simulate real-world adversarial conditions.'
- [Section IV.C] The sentence 'The other models that were trained on one type of dataset were each within 3-4 percentage points of the Baseline experiment' is vague; specify which metric and which models are meant.
Circularity Check
No circularity: the empirical pipeline uses external ground truth; the robustness conclusion is an overgeneralization, not a construction.
full rationale
The paper's central result (Table I) is an empirical measurement, not a derived prediction. Labels are the external human/deepfake ground truth from FakeMusicCaps (Sec. III.A), and the model is a standard ResNet18 trained with cross-entropy; no parameter is fit to the test labels and no quantity in Table I is defined in terms of the model's own output. The manipulations in Sec. III.C (pitch shift in [-2,+2] semitones, tempo stretch in [0.8,1.2]) are stated independently of the model, and the same manipulation family appears in both the training and test splits, so the results honestly measure in-distribution classification under those augmentations. The Section IV.D conclusion that adversarial evasion is 'not an insurmountable' problem goes beyond what the design can support, because a real adversary would choose modifications after training, possibly outside the trained ranges; this is a limitation in external validity (and the paper itself lists more extreme manipulations as future work), not a circular reduction. No self-citation is load-bearing: the cited prior work [1],[3],[4] is external and the method is not justified by a self-citation chain.
Assumptions & free parameters
free parameters (7)
- Learning rate =
Not reported (Adam defaults assumed)
- Epoch count =
20
- Batch size =
32
- Pitch shift range =
-2 to +2 semitones
- Tempo stretch range =
0.8 to 1.2
- Mel spectrogram parameters =
Not reported
- Deepfake subsample size =
5,373 tracks
assumptions (6)
- domain assumption FakeMusicCaps ground-truth labels (human vs deepfake) are correct and representative.
- domain assumption Mel spectrograms preserve enough discriminative information for the binary task.
- ad hoc to paper Librosa pitch shift and time stretch are valid proxies for real-world adversarial manipulation.
- domain assumption ImageNet-pretrained ResNet18 features transfer to mel spectrogram inputs.
- domain assumption The random split is i.i.d. and does not leak between augmented variants.
- ad hoc to paper Training and testing on the same manipulation type estimates robustness.
Cite this review
Pith. "Pith review of Detecting Musical Deepfakes." pith.science (2026). https://pith.science/paper/34QTJ3DN
@misc{pith2026250509633,
author = {Pith},
title = {Pith review of: Detecting Musical Deepfakes},
year = {2026},
howpublished = {\url{https://pith.science/paper/34QTJ3DN}},
note = {Machine review of arXiv:2505.09633}
}
read the original abstract
The proliferation of Text-to-Music (TTM) platforms has democratized music creation, enabling users to effortlessly generate high-quality compositions. However, this innovation also presents new challenges to musicians and the broader music industry. This study investigates the detection of AI-generated songs using the FakeMusicCaps dataset by classifying audio as either deepfake or human. To simulate real-world adversarial conditions, tempo stretching and pitch shifting were applied to the dataset. Mel spectrograms were generated from the modified audio, then used to train and evaluate a convolutional neural network. In addition to presenting technical results, this work explores the ethical and societal implications of TTM platforms, arguing that carefully designed detection systems are essential to both protecting artists and unlocking the positive potential of generative AI in music.
Figures
Reference graph
Works this paper leans on
-
[1]
Comanducci, L., Bestagini, P., & Tubaro, S. (2024). FakeMusicCaps: a Dataset for Detection and Attribution of Synthetic Music Generated via Text-to-Music Models. arXiv preprint arXiv:2409.10684J
arXiv 2024
-
[2]
I., Borsos, Z., Engel, J., Verzetti, M., Caillon, A.,
Agostinelli, A., Denk, T. I., Borsos, Z., Engel, J., Verzetti, M., Caillon, A., ... & Frank, C. (2023). Musiclm: Generating music from text. arXiv preprint arXiv:2301.11325
arXiv 2023
-
[3]
Rahman, M. A., Hakim, Z. I. A., Sarker, N. H., Paul, B., & Fattah, S. A. (2024). SONICS: Synthetic Or Not --Identifying Counterfeit Songs. arXiv preprint arXiv:2408.14080
arXiv 2024
-
[4]
Afchar, D., Meseguer -Brocal, G., & Hennequin, R. (2024). Detecting music deepfakes is easy but actually hard. arXiv preprint arXiv:2405.04181
arXiv 2024
-
[5]
Using Librosa to Change the Pitch of an Audio File
Paley, J., (2022) “Using Librosa to Change the Pitch of an Audio File”. [Online]. https://medium.com/strategio/using-librosa-to-change-the- pitch-of-an-audio-file-49efdb2dd6c
work page 2022
-
[6]
GTZAN – Mel Spectrogram + Resnet18
Nippani, A., (2022) “GTZAN – Mel Spectrogram + Resnet18 ”, [Online]. https://www.kaggle.com/code/nippani/gtzan-mel- spectrogram-resnet18/notebook#ResNet18---Transfer-Learning
work page 2022
-
[7]
How to Create & Understand Mel - Spectrograms,
Lewis, C. , (7 April 2021) . “How to Create & Understand Mel - Spectrograms,” [Online]. https://importchris.medium.com/how-to- create-understand-mel-spectrograms-ff7634991056. http://www.ctan.org/tex-archive/macros/latex/contrib/IEEEtran/
work page 2021
-
[8]
Audio Deep Learning Made Simple – Why Mel Spectrograms Perform Better
Doshi, K., (19 February 2021). “Audio Deep Learning Made Simple – Why Mel Spectrograms Perform Better ”. [Online]. https://ketanhdoshi.github.io/Audio-Mel/
work page 2021
Show all 43 references
-
[9]
An AI Hit of Fake ‘Drake’ and ‘The Weeknd’ Rattles the Music World
Coscarelli, J., (24 April 2023). “An AI Hit of Fake ‘Drake’ and ‘The Weeknd’ Rattles the Music World” . [Online] The New York Times: https://www.nytimes.com/2023/04/19/arts/music/ai-drake-the-weeknd- fake.html
2023
-
[10]
The AI Effect: A New Era in Music and Its Unintended Consequences
Rick Beato, (4 May 2023). “The AI Effect: A New Era in Music and Its Unintended Consequences” . [Video]. YouTube. https://youtu.be/- eAQOhDNLt4?si=pDQXhnGB6Gqq90Rb
2023
-
[11]
How AI will slowly destroy the music business
Rick Beato, (11 June 2024). “How AI will slowly destroy the music business”. [Video]. YouTube. https://www.youtube.com/watch?v=zbo6SdyWGns
2024
-
[12]
Ingham, T. “Universal Music Group Responds to ‘Fake Drake’ AI Track: Streaming Platforms Have ‘A fundamental responsibility to prevent the use of their services in ways that harm artists’ 17 April 2023, [Online] . https://www.musicbusinessworldwide.com/universal- music-group-r...
2023
-
[13]
C., & Donahue, C
Feffer, M., Lipton, Z. C., & Donahue, C. (2023). DeepDrake ft. BTS - GAN and TayloRVC: An Exploratory Analysis of Musical Deepfakes and Hosting Platforms. In HCMIR@ ISMIR. https://ceur-ws.org/Vol- 3528/paper3.pdf
2023
-
[14]
Josan, H. H. S. (2024). AI and Deepfake Voice Cloning: Innovation, Copyright and Artists’ Rights. Artificial Intelligence . https://www.cigionline.org/static/documents/DPH-paper-Josan.pdf
2024
-
[15]
Why Drake Had to Take Down His Song That Featured AI -Tupac Vocals
Chung, M., ( 9 December 2022). Why Drake Had to Take Down His Song That Featured AI -Tupac Vocals . Time. https://time.com/6971720/drake-tupac-ai/
2022
-
[16]
How That Tupac Hologram At Coachella Worked
Ganz, J., (17 April 2012). How That Tupac Hologram At Coachella Worked. [Online]. NPR. https://www.npr.org/sections/therecord/2012/04/17/150820261/how- that-tupac-hologram-at-coachella-worked
2012
-
[17]
The strange legacy of Tupac’s ‘hologram’ lives on five years after its historic Coachella debut
Dodson, A., ( 14 April 2017). The strange legacy of Tupac’s ‘hologram’ lives on five years after its historic Coachella debut . [Online]. Andscape. https://andscape.com/features/the-strange-legacy- of-tupacs-hologram-after-coachella/
2017
-
[18]
P'ng, J. (2024). The Resurrection Will Not Be Televised: Legal Remedies for Posthumous Deepfakes. Geo. L. Tech. Rev., 8, 338
2024
-
[19]
Juzon, H. (2024). Fake Drake? AI Music Generation Implicates Copyright and the Right of Publicity. Washington Law Review , 99(3), 987
2024
-
[20]
M., (2024)
Koski, R. M., (2024). Warhol, Drake, and Deepfakes: Monetizing the Right of Publicity in the Generative AI Era, 40 GA. ST. U. L. R EV. 981 Available at: https://readingroom.law.gsu.edu/gsulr/vol40/iss4/11
2024
-
[21]
Shirish, A., & Komal, S. (2024). A Socio -Legal Inquiry on Deepfakes. California Western International Law Journal, 54(2), 6
2024
-
[22]
G., (2024) Music Producer Accused of Using AI Songs to Scam Streaming Platforms out of $10 Million in Royalties
Maddaus. G., (2024) Music Producer Accused of Using AI Songs to Scam Streaming Platforms out of $10 Million in Royalties . [Online] Variety. https://variety.com/2024/music/news/ai-songs-spotify-apple- michael-smith-fraud-charged-artificial-intelligence-1236130680/
2024
-
[23]
[Online] arstechnica
Belanger, A., (2024) Spotify criticized for letting fake albums appear on real artist pages. [Online] arstechnica. https://variety.com/2024/music/news/ai-songs-spotify-apple-michael- smith-fraud-charged-artificial-intelligence-1236130680/
2024
-
[24]
(2023, August)
Barnett, J. (2023, August). The ethical implications of generative audio models: A systematic literature review. In Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society (pp. 146-161)
2023
-
[25]
(2020, April)
Frid, E., Gomes, C., & Jin, Z. (2020, April). Music creation by example. In Proceedings of the 2020 CHI conference on human factors in computing systems (pp. 1-13)
2020
-
[26]
(2024) ChatGPT and similar generative artificial intelligence in art, music, and literature industries: Applications and ethical challenges ResearchGate
Patil, D. (2024) ChatGPT and similar generative artificial intelligence in art, music, and literature industries: Applications and ethical challenges ResearchGate
2024
-
[27]
Reje, A. (2022). Ethical Risk Analysis of the Use of AI in Music Production
2022
-
[28]
Morreale, Fabio. (2021). Where Does the Buck Stop? Ethical and Political Issues with AI in Music Creation. Transactions of the International Society for Music Information Retrieval. 4. 105 -113. 10.5334/tismir.86
2021 doi
-
[29]
Sturm, Bob & Coeckelbergh, Mark. (2018). Ethical Dimensions of Music Information Retrieval Technology. Transactions of the International Society for Music Information Retrieval. 1. 44 -55. 10.5334/tismir.13
2018 doi
-
[30]
(2019, May)
Roselli, D., Matthews, J., & Talagala, N. (2019, May). Managing bias in AI. In Companion proceedings of the 2019 world wide web conference (pp. 539-544)
2019
-
[31]
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2021). A survey on bias and fairness in machine learning. ACM computing surveys (CSUR), 54(6), 1-35
2021
-
[32]
Douwes, C., Esling, P., & Briot, J. P. (2021). Energy consumption of deep generative audio models. arXiv preprint arXiv:2107.02621
2021 arXiv
-
[33]
Bültemann, M., Rzepka, N., Junger, D., Simbeck, K., & Müller, H. G. (2023). Energy consumption of ai in education: A case study
2023
-
[34]
https://www.mightyhappycrew.com/artist-faq/appalachian-anarchy
Coffin, P., (2024) Appalachian Anarchy F.A.Q.: Is Appalachian Anarchy Artificial Intelligence (AI) Generated Music? [Online]. https://www.mightyhappycrew.com/artist-faq/appalachian-anarchy
2024
-
[35]
[Video] YouTube
Anchorsmashed, (2024) The Bluegrass Metal Band That LIED To you. [Video] YouTube. https://youtu.be/7sSDPoxg6Zk?si=qB_gUZbCt0Fy9zYG
2024
-
[36]
Appalachian Anarchy (2024) Banjo Bloodbath Walkthrough . [Video]. YouTube. https://youtu.be/2kQP_S9DM2E?si=OS9MIsDVsYNCeFxU
2024
-
[37]
Meskys, E., Kalpokiene, J., Jurcys, P., & Liaudanskas, A. (2020). Regulating deep fakes: legal and ethical considerations. Journal of Intellectual Property Law & Practice, 15(1), 24-31
2020
-
[38]
Randy Travis gets his voice back in a new Warner AI music experiment
Davis, W., (2024) . Randy Travis gets his voice back in a new Warner AI music experiment . [Online] The Verge. https://www.theverge.com/2024/5/5/24149685/randy-travis-ai-voice- clone-song-where-that-came-from
2024
-
[39]
Unveiling the Impacts and Disruption of AI on Music Industry Stakeholders [Online] Rolling Stone Culture Council
Rolling Stone Culture Council ( n.d.). Unveiling the Impacts and Disruption of AI on Music Industry Stakeholders [Online] Rolling Stone Culture Council . https://council.rollingstone.com/blog/the- impacts-and-disruption-of-ai-on-music-industry-stakeholders
-
[40]
BBL Drizzy Retrieved 29 November 2024 from https://en.wikipedia.org/wiki/BBL_Drizzy
Wikipedia (2024). BBL Drizzy Retrieved 29 November 2024 from https://en.wikipedia.org/wiki/BBL_Drizzy
2024
-
[41]
[Online] Loudwire
Rapp, A., (2024) Mee t the Man Ruining Your Favorite Rock Songs With Mashups. [Online] Loudwire. https://loudwire.com/interview- there-i-ruined-it-song-mashup/
2024
-
[42]
Available at https://pytorch.org/docs/stable/index.html
The Linux Founda tion, PyTorch Library Reference, version 2.5 . Available at https://pytorch.org/docs/stable/index.html
-
[43]
Available at https://librosa.org/doc/latest/index.html
Librosa Development Team, Librosa Library Reference, version 0.10.2 . Available at https://librosa.org/doc/latest/index.html
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.