REVIEW 4 major objections 5 minor 43 references
SmoothSinger: A Conditional Diffusion Model for Singing Voice Synthesis with Multi-Resolution Architecture
T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read A conditional diffusion model for singing voice synthesis, SmoothSinger, claims to replace the vocoder stage by refining low-quality baseline audio directly under reference guidance, and reports the best subjective scores among compared…
desk verdict Incremental but plausible engineering on top of the authors' own RDSinger; the SOTA claim needs significance testing and artifact release before it holds. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing machinery is a dual-branch reference-conditioned diffusion U-Net. A trainable copy of the encoder, initialized from the main model, processes the reference waveform and its outputs are fused into the main branch at matching resolutions by zero-initialized 1x1 convolutions (Eq. 4), and again into the LVC upsampling blocks (Eq. 5). A parallel Multi-Resolution low-frequency block upsamples with 1D convolutions and sliding-window self-attention of O(L) complexity, feeding every scale directly to the final output. The third piece is training-time degradation: in the last 400K of 1.6M steps, with 50% probability the reference is a randomly distorted copy of ground truth (noise, amplitude scaling, exponential distortion, frequency-band scaling), which sidesteps the temporal mismatch between baseline reference and target.
What would settle it
Train SmoothSinger exactly as described, then at inference feed reference audio produced by FastSpeech2+HiFi-GAN from a held-out set and compare the output against a variant trained only with clean references; if the degraded-ground-truth model's MOS advantage disappears outside its training-reference distribution, the central transfer claim fails.
Extended reading notes
Core claim
The discovery the paper argues for is that a diffusion U-Net can absorb the role of the vocoder if it is conditioned on reference audio from a baseline synthesizer. The reference branch, initialized as a trainable copy of the main branch, feeds multi-resolution features into both the downward and upward paths through zero-initialized 1x1 convolutions, letting the model use the baseline's acoustic context while learning to correct it. The added low-frequency branch upsampling in parallel with the standard LVC blocks preserves pitch contours and long-term spectral structure, and degrading ground-truth audio during the final training phase aligns the reference with the target so the denoising loss is stable. With this setup SmoothSinger reports state-of-the-art MOS, ranking, SIG MOS, and STOI on Opencpop, and its PESQ is close to but below VISinger2; the ablation studies attribute the largest single drop in quality to removing the degraded-data training.
Load-bearing premise
The load-bearing premise is that degraded ground-truth audio used during training behaves like the low-quality baseline reference audio used at inference, so the model learns to fix real baseline artifacts rather than artifacts specific to the degraded copies.
Editorial extensions
If this is right
- Because the reference can come from any baseline synthesizer, an existing SVS pipeline can be upgraded by appending SmoothSinger without retraining the acoustic model or vocoder.
- FastSpeech2 plus HiFi-GAN is sufficient as the reference source; using slower, higher-quality references such as DiffSinger or RDSinger does not improve quality, so the fast baseline remains the practical choice.
- The same pretrained weights transfer to TTS with light fine-tuning and match dedicated TTS models on LJSpeech, suggesting a unified vocoder-free refinement model for both speech and singing.
- The ablation study shows the degraded-ground-truth training is the largest single contributor: removing it drops MOS to 3.16, so the alignment trick is central to the reported naturalness.
Reading between the lines
- If the degraded-ground-truth trick is what carries the gains, then the headline comparison is sensitive to reference distribution: the model should be re-tested with references whose distortion statistics match FastSpeech2+HiFi-GAN rather than randomly degraded clean audio.
- Because SmoothSinger trails VISinger2 on PESQ while beating it on listening preference, the paper's notion of quality is listener-centric; a natural next step would be to add a distortion-oriented loss or discriminator to close the objective gap.
- The architecture's sliding-window attention and independent low-frequency branch should scale to longer clips and more singers than the single female Opencpop voice, which would stress the temporal-alignment assumption the degraded-data training is meant to patch.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes SmoothSinger, a conditional diffusion model for singing voice synthesis that operates directly on waveforms instead of using a separate vocoder stage. The model conditions on a reference waveform produced by a baseline system (FastSpeech2 + HiFi-GAN) through a duplicated reference branch, augments the U-Net with a parallel low-frequency upsampling path, and during part of training substitutes degraded ground-truth audio as the reference to mitigate temporal misalignment. The method is evaluated on Opencpop for SVS and on LJSpeech for TTS, with objective metrics, listening tests, and ablations. The central claim is that SmoothSinger achieves state-of-the-art results in both objective and subjective evaluations.
Significance. The architectural ideas are plausible and clearly described: a vocoder-free diffusion framework with reference conditioning is a useful direction, the 16M-parameter design is compact, and the ablations show that the proposed components matter (Table 2: removing the reference downsampling path drops MOS from 3.61 to 3.31, and removing degraded-data training drops it to 3.16). The transfer to TTS is an interesting generality check. However, the empirical support for the headline claim is currently weak: the reported subjective gains over RDSinger are small, their 95% confidence intervals overlap or touch, no significance tests are reported, and the objective results are not consistently state-of-the-art. The manuscript also overstates the TTS results. The core architecture may be sound, but the SOTA claim needs to be either statistically substantiated or substantially softened.
major comments (4)
- [Section 4.2.1-4.2.2, Table 1] The claim of state-of-the-art subjective performance is not supported by the reported statistics. SmoothSinger's MOS is 3.61 ± 0.06 versus RDSinger's 3.52 ± 0.07; these 95% confidence intervals overlap. The ranking difference is 2.36 ± 0.10 versus 2.56 ± 0.10, with intervals touching at 2.46. With only 16 participants and no pairwise significance tests, a 0.09 MOS gain and a 0.20 ranking gain are within the noise level. Please report significance tests (e.g., paired tests with multiple-comparison correction) or soften the SOTA claim to a statement about observed improvements.
- [Abstract and Section 4.2.2, Table 1] The abstract's 'state-of-the-art results in both objective and subjective evaluations' is contradicted by Table 1. SmoothSinger's PESQ is 3.25 ± 0.07, below VISinger2's 3.31 ± 0.07, and its BAK MOS is 4.08, below RDSinger's 4.12. Section 4.2.2 itself narrows the claim to SIG MOS and STOI and describes other metrics as 'on-par,' which is a more accurate characterization. The abstract and conclusion should be revised to reflect the actual pattern of results.
- [Section 4.4, Table 3] The conclusion states that SmoothSinger achieves 'superior performance in both objective and subjective evaluations compared to state-of-the-art models in SVS and TTS tasks,' but the TTS results do not support this. In Table 3, SmoothSinger's MOS (3.88) is below FastDiff (3.94), its PESQ (3.52) is below FastDiff (3.59), and its STOI (0.947) is below FastDiff (0.948). The TTS claim should be stated as 'comparable,' as the body text actually says, not 'superior.'
- [Section 3.2, Section 4.2.2] The comparison protocol is not fully specified with respect to reference audio. SmoothSinger receives an additional reference waveform produced by FastSpeech2+HiFi-GAN, while the baselines are described as acoustic models followed by HiFi-GAN. If the baselines do not receive equivalent side information, the comparison is not controlled. Please state explicitly what conditioning inputs each baseline received, and clarify whether any baseline also uses reference audio; if none does, this is a confound that should be discussed as a limitation.
minor comments (5)
- [Section 3.1] The section heading contains a typo: 'Preliminariy' should be 'Preliminary.'
- [Section 4.2.1] The claim that 'minor temporal discrepancies do not affect the reliability' of PESQ and STOI is overly strong; these metrics are alignment-sensitive, and a brief justification or caveat would be appropriate.
- [Section 3.3] The notation 'Ki ∈ RL for any i' is unclear, since the low-frequency blocks appear to operate at different internal resolutions before reshaping; please define the dimensionality more precisely.
- [Section 4.1.1] Reference [32] is cited for PyPinyin, but the entry points to Ren et al. (DeepSinger); the citation appears to be mismatched.
- [Figures 1 and 2] The figure lettering and the distinction between the black and yellow arrows in Figure 2 are difficult to read at printed size; please enlarge the annotations.
Circularity Check
No significant circularity: the paper's SOTA claim rests on independent Table 1 measurements and ablations; the sole self-citation (RDSinger) is a baseline and evaluation-protocol reference, not a load-bearing derivation.
full rationale
SmoothSinger is an empirical systems paper rather than a derivation paper. Its central claim of state-of-the-art performance is supported by directly measured metrics (MOS, Ranking, SIG/BAK MOS, PESQ, STOI) in Table 1 and by retraining-based ablations in Tables 2, 4, 5, and 6. These evaluations use the public Opencpop train/test split and are reported with 95% confidence intervals; they do not reduce to the training objective itself, and no fitted parameter is subsequently relabeled as a prediction. The only self-citation is the authors' prior RDSinger work [35], which is used as a baseline for comparison and as the source of the Ranking evaluation protocol; this is a normal related-work and benchmarking reference, not a load-bearing uniqueness claim or a smuggled ansatz. The degraded-ground-truth training strategy is tested through a real ablation ('W/o Degraded Data'), and any train/inference mismatch with baseline-generated reference audio is an empirical robustness concern, not a circularity. Overlapping confidence intervals between SmoothSinger and RDSinger on MOS/Ranking are a statistical-significance weakness, which belongs to correctness risk rather than circularity. Under the requirement to exhibit a specific reduction to the paper's own inputs, no such reduction exists, so no circular steps are identified.
Assumptions & free parameters
assumptions (3)
- domain assumption Opencpop's train/test split is representative of singing voice synthesis quality
- domain assumption PESQ, STOI, and DNSMOS, developed for speech, are valid proxies for singing audio quality
- domain assumption Reference audio from FastSpeech2+HiFi-GAN provides sufficient conditioning for enhancement
Cite this review
Pith. "Pith review of SmoothSinger: A Conditional Diffusion Model for Singing Voice Synthesis with Multi-Resolution Architecture." pith.science (2026). https://pith.science/paper/CQV5FXAG
@misc{pith2026250621478,
author = {Pith},
title = {Pith review of: SmoothSinger: A Conditional Diffusion Model for Singing Voice Synthesis with Multi-Resolution Architecture},
year = {2026},
howpublished = {\url{https://pith.science/paper/CQV5FXAG}},
note = {Machine review of arXiv:2506.21478}
}
read the original abstract
Singing voice synthesis (SVS) aims to generate expressive and high-quality vocals from musical scores, requiring precise modeling of pitch, duration, and articulation. While diffusion-based models have achieved remarkable success in image and video generation, their application to SVS remains challenging due to the complex acoustic and musical characteristics of singing, often resulting in artifacts that degrade naturalness. In this work, we propose SmoothSinger, a conditional diffusion model designed to synthesize high quality and natural singing voices. Unlike prior methods that depend on vocoders as a final stage and often introduce distortion, SmoothSinger refines low-quality synthesized audio directly in a unified framework, mitigating the degradation associated with two-stage pipelines. The model adopts a reference-guided dual-branch architecture, using low-quality audio from any baseline system as a reference to guide the denoising process, enabling more expressive and context-aware synthesis. Furthermore, it enhances the conventional U-Net with a parallel low-frequency upsampling path, allowing the model to better capture pitch contours and long term spectral dependencies. To improve alignment during training, we replace reference audio with degraded ground truth audio, addressing temporal mismatch between reference and target signals. Experiments on the Opencpop dataset, a large-scale Chinese singing corpus, demonstrate that SmoothSinger achieves state-of-the-art results in both objective and subjective evaluations. Extensive ablation studies confirm its effectiveness in reducing artifacts and improving the naturalness of synthesized voices.
Reference graph
Works this paper leans on
-
[1]
I. Beltagy, M. E. Peters, and A. Cohan. Longformer: The long- document transformer. arXiv preprint arXiv:2004.05150, 2020
arXiv 2004
-
[2]
P. Chandna, M. Blaauw, J. Bonada, and E. Gómez. Wgansing: A multi- voice singing voice synthesizer based on the wasserstein-gan. In 2019 27th European signal processing conference (EUSIPCO) , pages 1–5. IEEE, 2019
work page 2019
-
[3]
N. Chen, Y . Zhang, H. Zen, R. J. Weiss, M. Norouzi, N. Dehak, and W. Chan. Wavegrad 2: Iterative refinement for text-to-speech synthesis. arXiv preprint arXiv:2106.09660, 2021
arXiv 2021
-
[4]
Y .-R. Chien, H.-M. Wang, and S.-K. Jeng. Alignment of lyrics with ac- companied singing audio based on acoustic-phonetic vowel likelihood modeling. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 24(11):1998–2008, 2016
work page 1998
-
[5]
Dhariwal and A
P. Dhariwal and A. Nichol. Diffusion models beat gans on image syn- thesis. Advances in neural information processing systems , 34:8780– 8794, 2021
2021
-
[6]
Goodfellow, J
I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y . Bengio. Generative adversarial networks. Communications of the ACM, 63(11):139–144, 2020
2020
-
[7]
J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic mod- els. Advances in neural information processing systems, 33:6840–6851, 2020
2020
-
[8]
J. Ho, T. Salimans, A. Gritsenko, W. Chan, M. Norouzi, and D. J. Fleet. Video diffusion models. Advances in Neural Information Processing Systems, 35:8633–8646, 2022
2022
Show all 43 references
-
[9]
Y . Hono, K. Hashimoto, K. Oura, Y . Nankaku, and K. Tokuda. Singing voice synthesis based on generative adversarial networks. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6955–6959. IEEE, 2019
2019
-
[10]
L. Hu. Animate anyone: Consistent and controllable image-to-video synthesis for character animation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8153– 8163, 2024
2024
-
[11]
Huang, E
J. Huang, E. Benetos, and S. Ewert. Improving lyrics alignment through joint pitch detection. In ICASSP 2022-2022 IEEE International Con- ference on Acoustics, Speech and Signal Processing (ICASSP) , pages 451–455. IEEE, 2022
2022
-
[12]
Huang, M
R. Huang, M. W. Lam, J. Wang, D. Su, D. Yu, Y . Ren, and Z. Zhao. Fastdiff: A fast conditional diffusion model for high-quality speech syn- thesis. arXiv preprint arXiv:2204.09934, 2022
2022 arXiv
-
[13]
Huang, Z
R. Huang, Z. Zhao, H. Liu, J. Liu, C. Cui, and Y . Ren. Prodiff: Progres- sive fast diffusion model for high-quality text-to-speech. InProceedings of the 30th ACM International Conference on Multimedia, pages 2595– 2605, 2022
2022
-
[14]
Ito and L
K. Ito and L. Johnson. The lj speech dataset. https://keithito.com/ LJ-Speech-Dataset/, 2017
2017
-
[15]
Jadoul, B
Y . Jadoul, B. Thompson, and B. de Boer. Introducing Parselmouth: A Python interface to Praat. Journal of Phonetics , 71:1–15, 2018. doi: https://doi.org/10.1016/j.wocn.2018.07.001
2018 doi
-
[16]
Karras, M
T. Karras, M. Aittala, S. Laine, E. Härkönen, J. Hellsten, J. Lehtinen, and T. Aila. Alias-free generative adversarial networks. Advances in neural information processing systems, 34:852–863, 2021
2021
-
[17]
Kim, M.-S
T.-W. Kim, M.-S. Kang, and G.-H. Lee. Adversarial multi-task learning for disentangling timbre and pitch in singing voice synthesis. arXiv preprint arXiv:2206.11558, 2022
2022 arXiv
-
[18]
D. P. Kingma, M. Welling, et al. An introduction to variational autoen- coders. Foundations and Trends® in Machine Learning , 12(4):307– 392, 2019
2019
-
[19]
J. Kong, J. Kim, and J. Bae. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. Advances in neural in- formation processing systems, 33:17022–17033, 2020
2020
-
[20]
Z. Kong, W. Ping, J. Huang, K. Zhao, and B. Catanzaro. Dif- fwave: A versatile diffusion model for audio synthesis. arXiv preprint arXiv:2009.09761, 2020
2009 arXiv
-
[21]
S.-g. Lee, W. Ping, B. Ginsburg, B. Catanzaro, and S. Yoon. Bigvgan: A universal neural vocoder with large-scale training. arXiv preprint arXiv:2206.04658, 2022
2022 arXiv
-
[22]
J. Liu, C. Li, Y . Ren, F. Chen, and Z. Zhao. Diffsinger: Singing voice synthesis via shallow diffusion mechanism. In Proceedings of the AAAI conference on artificial intelligence , volume 36, pages 11020–11028, 2022
2022
-
[23]
R. Liu, J. Xiang, B. Zhao, R. Zhang, J. Yu, and C. Zheng. Neural im- postor: Editing neural radiance fields with explicit shape manipulation. In Computer Graphics Forum, volume 42, page e14981. Wiley Online Library, 2023
2023
-
[24]
P. Lu, J. Wu, J. Luan, X. Tan, and L. Zhou. Xiaoicesing: A high- quality and integrated singing voice synthesis system. arXiv preprint arXiv:2006.06261, 2020
2006 arXiv
-
[25]
Müller, A
T. Müller, A. Evans, C. Schied, and A. Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM transactions on graphics (TOG), 41(4):1–15, 2022
2022
-
[26]
Nakamura, K
K. Nakamura, K. Hashimoto, K. Oura, Y . Nankaku, and K. Tokuda. Singing voice synthesis based on convolutional neural networks. arXiv preprint arXiv:1904.06868, 2019
1904 arXiv
-
[27]
Popov, I
V . Popov, I. V ovk, V . Gogoryan, T. Sadekova, and M. Kudinov. Grad- tts: A diffusion probabilistic model for text-to-speech. In International conference on machine learning, pages 8599–8608. PMLR, 2021
2021
-
[28]
Prenger, R
R. Prenger, R. Valle, and B. Catanzaro. Waveglow: A flow-based gen- erative network for speech synthesis. In ICASSP 2019-2019 IEEE In- ternational Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3617–3621. IEEE, 2019
2019
-
[29]
Ramesh, P
A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen. Hierarchi- cal text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3, 2022
2022 arXiv
-
[30]
C. K. Reddy, V . Gopal, and R. Cutler. Dnsmos p. 835: A non-intrusive perceptual objective speech quality metric to evaluate noise suppres- sors. In ICASSP 2022-2022 IEEE International Conference on Acous- tics, Speech and Signal Processing (ICASSP) , pages 886–890. IEEE, 2022
2022
-
[31]
Y . Ren, C. Hu, X. Tan, T. Qin, S. Zhao, Z. Zhao, and T.-Y . Liu. Fast- speech 2: Fast and high-quality end-to-end text to speech.arXiv preprint arXiv:2006.04558, 2020
2006 arXiv
-
[32]
Y . Ren, X. Tan, T. Qin, J. Luan, Z. Zhao, and T.-Y . Liu. Deepsinger: Singing voice synthesis with data mined from the web. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Dis- covery & Data Mining, pages 1979–1989, 2020
1979
-
[33]
A. W. Rix, J. G. Beerends, M. P. Hollier, and A. P. Hekstra. Perceptual evaluation of speech quality (pesq)-a new method for speech quality assessment of telephone networks and codecs. In 2001 IEEE interna- tional conference on acoustics, speech, and signal processing. Proceed...
2001
-
[34]
Saharia, J
C. Saharia, J. Ho, W. Chan, T. Salimans, D. J. Fleet, and M. Norouzi. Image super-resolution via iterative refinement. IEEE transactions on pattern analysis and machine intelligence, 45(4):4713–4726, 2022
2022
-
[35]
K. Sui, J. Xiang, and F. Jin. Rdsinger: Reference-based diffusion net- work for singing voice synthesis. arXiv preprint arXiv:2410.21641 , 2024
2024 arXiv
-
[36]
C. H. Taal, R. C. Hendriks, R. Heusdens, and J. Jensen. An algorithm for intelligibility prediction of time–frequency weighted noisy speech. IEEE Transactions on audio, speech, and language processing , 19(7): 2125–2136, 2011
2011
-
[37]
Van Den Oord, S
A. Van Den Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. Senior, K. Kavukcuoglu, et al. Wavenet: A generative model for raw audio. arXiv preprint arXiv:1609.03499 , 12, 2016
2016 arXiv
-
[38]
Y . Wang, X. Wang, P. Zhu, J. Wu, H. Li, H. Xue, Y . Zhang, L. Xie, and M. Bi. Opencpop: A high-quality open source chinese popular song corpus for singing voice synthesis. arXiv preprint arXiv:2201.07429, 2022
2022 arXiv
-
[39]
Wu and J
J. Wu and J. Luan. Adversarially trained multi-singer sequence-to- sequence singing synthesizer. arXiv preprint arXiv:2006.10317, 2020
2006 arXiv
-
[40]
Zhang, A
L. Zhang, A. Rao, and M. Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF inter- national conference on computer vision, pages 3836–3847, 2023
2023
-
[41]
Zhang, H
Y . Zhang, H. Xue, H. Li, L. Xie, T. Guo, R. Zhang, and C. Gong. Visinger 2: High-fidelity end-to-end singing voice synthesis en- hanced by digital signal processing synthesizer. arXiv preprint arXiv:2211.02903, 2022
2022 arXiv
-
[42]
Zhang, R
Y . Zhang, R. Huang, R. Li, J. He, Y . Xia, F. Chen, X. Duan, B. Huai, and Z. Zhao. Stylesinger: Style transfer for out-of-domain singing voice synthesis. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 38, pages 19597–19605, 2024
2024
-
[43]
Zhang, Z
Y . Zhang, Z. Jiang, R. Li, C. Pan, J. He, R. Huang, C. Wang, and Z. Zhao. Tcsinger: Zero-shot singing voice synthesis with style transfer and multi-level style control. arXiv preprint arXiv:2409.15977, 2024
2024 arXiv
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.