Adversarially Trained End-to-end Korean Singing Voice Synthesis System

Chang-Bin Jeon; Hyeong-Seok Choi; Juheon Lee; Junghyun Koo; Kyogu Lee

REVIEW 3 major objections 4 minor 33 references

Adversarially Trained End-to-end Korean Singing Voice Synthesis System

T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read An end-to-end Korean singing-voice synthesizer trained from lyrics and melody produces realistic audio by combining a phonetic enhancement mask, reused text and pitch conditioning, and conditional adversarial training.

desk verdict A useful Korean SVS system with three sensible tricks, but the headline claim that adversarial training helps the super-resolution stage specifically is confounded by the joint generator. read the letter →

arxiv 1908.01919 v1 pith:OTVBOIQ5 submitted 2019-08-06 cs.SD eess.AS

Juheon Lee , Hyeong-Seok Choi , Chang-Bin Jeon , Junghyun Koo , Kyogu Lee This is my paper · ORCID

classification cs.SDeess.AS

keywords singingvoicesynthesisend-to-endneuralnetworkmel-spectrogramsuper-resolutionconditionaladversarialtrainingphoneticenhancementmaskKoreansource-filterdecomposition

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that an end-to-end Korean singing-voice synthesizer can be trained from lyrics and symbolic melody with only about two hours of studio audio, provided three mechanisms are combined. First, a phonetic enhancement mask, generated from text alone and multiplied onto the mel-decoder output, lets the network separate pronunciation-related spectral shaping from pitch and harmonic content. Second, the super-resolution network that turns the mel-spectrogram into a full linear-spectrogram reuses the aligned text and pitch as local conditioning. Third, a conditional adversarial loss trains the two modules jointly as one generator, which the authors show is what removes the blurry high-frequency detail and drives the largest jump in perceived sound quality. In ablations, the full model scores significantly higher than every partial version on pronunciation accuracy, sound quality, and naturalness.

What carries the argument

The load-bearing object is the multiplicative phonetic enhancement mask: a decoder branch that receives only encoded text and computes a mask that is element-wise multiplied by the mel decoder output so that the final mel-spectrogram is $\hat{M} = \mathrm{Mask} \odot D_M$. This mask acts as an implicit formant envelope that shapes pronunciation independently of pitch. Around it, the system's second mechanism is reuse: the same text and pitch encodings that drive the mel network are fed again into the super-resolution network via local conditioning. The third is a conditional adversarial loss in which a projection discriminator sees the generated mel-spectrogram alongside the upsampled linear-spectrogram, treating the two modules as one generator $G_\theta = SR(MS(\cdot),\cdot)$, with R1 regularization to stabilize training. Joint training of the two modules rests on the approximation $p(M) \simeq p(\hat{M})$.

What would settle it

Measure the distribution gap between generated and ground-truth mel-spectrograms on the test set while holding the rest fixed, and feed ground-truth mel-spectrograms into the trained super-resolution network at test time; a large quality jump or a large distribution gap would show that the claimed conditional adversarial mechanism depends on the joint-training approximation rather than being robust to it.

Watch

Extended reading notes

Core claim

Starting from a convolutional text-to-mel backbone, the authors build a two-module generator: a mel-synthesis network produces a mel-spectrogram from text, pitch, and previous mel frames, and a super-resolution network upsamples it to a linear-spectrogram that is converted to audio by phase reconstruction. They claim that three additions are responsible for realistic output: a phonetic enhancement mask produced only from text and multiplied element-wise onto the mel prediction, acting as an implicit formant and filter mask; local conditioning of the super-resolution network on the same aligned text and pitch encodings; and a conditional discriminator, using projection conditioning and R1 regularization, that checks both realism and mel-to-linear correspondence. The reported experiments show the complete model receives the highest mean opinion scores on pronunciation, sound quality, and naturalness, with adversarial training producing the clearest high-frequency structure in the generated spectrograms. The authors also observe that the mask and the decoder output separate into filter-like and source-like components without supervision, mirroring a classical source-filter view of voice production.

Load-bearing premise

The joint training works only if the mel-spectrogram produced by the first module closely matches the distribution of real mel-spectrograms, because the discriminator conditions on that generated mel; if the approximation is poor, the adversarial loss is conditioning on corrupted input and can distort the final audio.

Editorial extensions

If this is right

The complete model significantly outperforms every ablated variant on listener ratings for pronunciation accuracy, sound quality, and naturalness, so the three mechanisms are complementary rather than interchangeable.
Adversarial training is the main driver of sound quality: adding it produces visibly sharper high-frequency structure in the linear-spectrogram and a significant mean-opinion-score gain, while the other two methods mainly improve pronunciation.
The phonetic enhancement mask alone improves pronunciation accuracy in the listening test, indicating that text-derived masks can shape articulation even when the exact phoneme timing is only coarsely represented.
Because the mask and decoder output separate into filter-like and source-like components without supervision, the learned representation can be read as an implicit source-filter decomposition of the voice.
The model matches or exceeds the pitch precision of real recordings at generating correct notes, showing that the symbolic melody condition is honored in the output.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same mask mechanism should transfer to other languages or to expressive speech synthesis: since the mask learns to encode pronunciation from text alone, explicit control of the mask could exaggerate or soften consonants without retraining.
The conditional adversarial step is a generic cure for blurry high-frequency audio and could be grafted onto any mel-to-linear or mel-to-waveform upsampler, including bandwidth extension and speech enhancement pipelines.
A direct test of the source-filter reading would be to manipulate the learned mask independently, for example by scaling it in frequency, and check whether perceived vowels change while pitch stays fixed; the paper does not run this experiment.
With a neural vocoder replacing the phase-reconstruction backend, the same end-to-end objective could likely reach higher subjective quality and shorter audio lengths, though phase reconstruction currently caps the ceiling.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Fairly interesting paper. The authors build an end-to-end Korean singing voice synthesizer from text and MIDI, using a DCTTS-style mel network followed by a super-resolution net, and they add three things: a phonetic enhancement mask, local conditioning of text/pitch into the SR net, and conditional adversarial training with a projection discriminator. The mask idea is the most original: it learns an implicit formant/filter mask in the mel domain, and the qualitative separation of harmonic source and spectral envelope is worth a look.

The ablation is the core evidence. Model5 (all three methods) beats model4 on MOS, and the differences are described as significant, though no p-values are given. The phonetic mask indeed seems to help pronunciation (model4 vs model1). That part is defensible.

The soft spots are real, and one is load-bearing. The paper claims adversarial training is 'crucial for realistic generation ... in the super-resolution process.' But in Eq. (6) the discriminator loss also updates the mel-synthesis network, since Gθ = SR(MS(·),·). So model5 differs from model4 in both modules, and the spectrogram comparison in Fig. 3 inherits the same confound. The specific attribution to super-resolution is not supported by the experimental design. An ablation with MS frozen or with the adversarial gradient detached from MS would settle it. This is fixable, but it's not a detail.

Also, the evaluation is narrow: one female singer, 2 hours of audio, no external SVS comparison (not even the earlier LSTM Korean SVS they cite), and no code/data. Pitch F1 is a useful sanity check but doesn't tell you about pronunciation quality directly. The MOS gaps to the Griffin-Lim reconstruction of ground truth are large, so everything is relative.

The authors do state their mel distribution assumption explicitly, so no hidden circularity. The writing is clear.

I'd send this to peer review—it's a legitimate system paper with a new combination of known pieces and a genuinely novel mask idea. But I'd ask for the SR-isolation ablation, reported p-values or CIs, and a baseline outside the own ablations. With those, it'd be a solid accept.

For my own reading, maybe bring to group if we're doing a speech/audio session, but I wouldn't cite it in the next year.

Referee Report

3 major / 4 minor

Summary. The paper describes an end-to-end Korean singing voice synthesis system that takes lyrics and a symbolic melody as input and produces a linear-spectrogram, which is converted to audio with Griffin-Lim. The architecture combines a mel-spectrogram synthesis network with a super-resolution network, and the paper proposes three additions: phonetic enhancement masking in the mel-synthesis network, local conditioning of text and pitch in the super-resolution network, and conditional adversarial training with a projection discriminator and R1 regularization. The experiments compare five model variants on a private, single-singer, two-hour Korean pop dataset. The evaluation reports pitch-sequence precision, recall, and F1, as well as MOS for pronunciation accuracy, sound quality, and naturalness, together with a qualitative spectrogram analysis. The paper claims that all three proposed methods improve synthesis quality and that the adversarial training is crucial for realistic generation in the super-resolution process.

Significance. If the claims are supported, the paper is a useful engineering contribution: it extends efficient end-to-end text-to-speech ideas to singing voice synthesis, introduces a phonetic enhancement mask that separates source-like and filter-like information without supervision, and demonstrates that local conditioning and adversarial training can be applied at the mel-to-linear upsampling stage. The reported mean MOS gains for the full model over ablated versions are consistent and in the expected direction. However, the central attribution claim about the super-resolution stage is confounded by the joint optimization, and the statistical and external-validity evidence is thin. The contribution is plausible but needs a revised experimental design or a more careful statement of what is being claimed.

major comments (3)

[Section 3.3.2 and Eq. (6)] The adversarial loss is not confined to the super-resolution network. In Eq. (6), LadvG is added to LMS,SR and, as stated in Section 3.3.2, θ includes the parameters of both MS and SR so that Gθ = SR(MS(·),·). Therefore, model5 differs from model4 in both the mel-synthesis network and the super-resolution network. The paper's claim that adversarial training is 'crucial for a realistic generation of the human singing voice in the super-resolution process' is not supported by the reported ablation, because the MOS gain and the spectrogram comparison in Section 4.4 could equally be caused by adversarially improved mel-spectrograms. Please add an ablation in which LadvG gradients are detached from MS or MS is frozen, or restrict the claim to 'joint end-to-end adversarial training' and provide direct evidence specifically for the super-resolution stage.
[Section 4.3.2 and Table 1] The manuscript states that a paired t-test was conducted and that significant differences were obtained, but it reports no p-values, test statistics, degrees of freedom, or multiple-comparison corrections. With five models there are ten pairwise comparisons, and the qualitative conclusion that 'all of the proposed methods helped' rests entirely on this unreported statistical analysis. Please report the full statistical results, including effect sizes and a correction for multiple comparisons, so that the central claims can be verified.
[Section 4.1 and Section 4.3] The evaluation has limited external validity for the paper's broad claims. The dataset contains a single professional female singer and about two hours of audio, and there is no comparison with an existing singing voice synthesis system, such as the LSTM-based Korean SVS system cited in [5], or with a vocoder-feature baseline. In addition, the only pronunciation measure is a subjective MOS with lyrics provided to the listeners; the f0 precision/recall evaluation is a pitch-tracking metric, not an objective measure of phonetic accuracy. Please add an external baseline comparison or an objective pronunciation metric, and temper the 'accurate phonetic control' claim accordingly.

minor comments (4)

[Section 4.3] There is a typo in 'Noe that' before the description of Recons samples, and the table header 'Quantitative Qualititative' contains an extra 'i'; please correct both.
[Section 4.2] The variables δ and ζ are referred to as 'pre-emphasis' and 'post-emphasis' factors, but the described operations are magnitude compression and expansion of a spectrogram rather than temporal pre-emphasis; please clarify the terminology.
[Section 3.2, Eq. (2)] The notation M′ = M1:L − M0:L−1 is not fully defined: please specify the exact frame indices and how the mel input history is aligned with the text and pitch sequences.
[Section 4.4 and Figure 3] The discussion refers to DM and Mask images and to spectrograms from model4, model5, and ground truth, but the provided text does not make the figure contents legible; please ensure the figure is high resolution and that the relevant regions are clearly indicated.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular reasoning found; the model4-vs-model5 comparison is confounded but not circular.

full rationale

The paper's central claims are empirical and evaluated on held-out test data against ablated models; none of the proposed losses or evaluation metrics is equivalent by construction to the claimed outcome. The adversarial loss in Eq. (5)-(6) does update both MS and SR as a joint generator, so the model5/model4 comparison does not isolate the super-resolution network; however, this is an experimental attribution confound, not a circular reduction. The quantitative f0 check compares extracted pitch to the input condition on test data and merely verifies conditioning was learned, which is not a fitted parameter being renamed as a prediction. The explicit modeling assumption p(M)≈p(M_hat) in Section 3.3.2 is stated as an assumption rather than hidden as a definition. The only citation with author overlap ([5]) is used as background on LSTM-based SVS and is not load-bearing. Therefore no circularity score is warranted.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central empirical results rest on the private dataset, the network architecture inherited from DCTTS, and the stated modeling assumptions. No new physical entities are introduced.

free parameters (2)

pre-emphasis factor delta = 0.6
Hand-chosen normalization exponent applied to linear-spectrogram magnitudes before mel-filtering; affects the input/output representation across all models.
post-emphasis factor zeta = 1.3
Hand-chosen exponent applied to generated spectrograms before waveform synthesis; influences final audio quality.

assumptions (3)

domain assumption The mel-spectrogram generated by the mel-synthesis network approximately follows the true mel-spectrogram distribution, enabling joint training of the two modules.
Stated in Section 3.3.2; if false, the adversarial conditioner receives mismatched inputs.
domain assumption A Korean syllable can be represented by three phonemes (onset, nucleus, coda) and the timing can be approximated by assigning onset and coda to the first and last frames.
Described in Section 3.1; the network is expected to compensate for coarse timing via a wide receptive field.
standard math The Griffin-Lim algorithm reconstructs an acceptable waveform from a magnitude spectrogram.
Used to convert the final linear-spectrogram to audio; it is a known approximate phase recovery method.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Adversarially Trained End-to-end Korean Singing Voice Synthesis System." pith.science (2026). https://pith.science/paper/OTVBOIQ5

@misc{pith2026190801919,
  author       = {Pith},
  title        = {Pith review of: Adversarially Trained End-to-end Korean Singing Voice Synthesis System},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OTVBOIQ5}},
  note         = {Machine review of arXiv:1908.01919}
}

read the original abstract

In this paper, we propose an end-to-end Korean singing voice synthesis system from lyrics and a symbolic melody using the following three novel approaches: 1) phonetic enhancement masking, 2) local conditioning of text and pitch to the super-resolution network, and 3) conditional adversarial training. The proposed system consists of two main modules; a mel-synthesis network that generates a mel-spectrogram from the given input information, and a super-resolution network that upsamples the generated mel-spectrogram into a linear-spectrogram. In the mel-synthesis network, phonetic enhancement masking is applied to generate implicit formant masks solely from the input text, which enables a more accurate phonetic control of singing voice. In addition, we show that two other proposed methods -- local conditioning of text and pitch, and conditional adversarial training -- are crucial for a realistic generation of the human singing voice in the super-resolution process. Finally, both quantitative and qualitative evaluations are conducted, confirming the validity of all proposed methods.

Figures

Figures reproduced from arXiv: 1908.01919 by the authors.

**Figure 1.** Proposed system overview (left), detailed structure of each sub-module (right). X|F| denotes F(X). pitch information as conditional input, the super-resolution network upsamples the generated mel-spectrogram M to a linearspectrogram S. Finally, the discriminator takes the upsampled result with generated mel-spectrogram to train the network in an adversarial manner. During the test phase, a sequence of mel-spectrog… view at source ↗

**Figure 2.** Input representation method overview 3.1. Input representation Our training data includes recorded singing voice along with the corresponding text and midi. A single midi note represents pitch information with onset and offset. For the single midi note, one syllable and its corresponding vocal audio section are manually aligned [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗

**Figure 3.** Generated spectrograms. showed a significant increase in score. From this we can confirm that training the network in an adversarial manner improves the quality of the generated audio. Finally, for naturalness, there was a significant improvement when all methods were applied. 4.4. Analysis on generated spectrogram In this section we analyze the features generated by the melsynthesis and super-resolution networks.… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

33 extracted references · 21 canonical work pages

[5]

We showed that using text information to model the phonetic enhancement mask actually worked, and produced more accurate pronunciation

Conclusions In this paper, we proposed the end-to-end Korean singing vo- cie synthesis system. We showed that using text information to model the phonetic enhancement mask actually worked, and produced more accurate pronunciation. Also, we successfully applied the conditional adversarial training method to the super- resolution stage, which resulted in a ...

work page
[1]

For example, three SVS sys- tems based on DNN, LSTM, and Wavenet architecture were proposed, respectively [4, 5, 6]

Introduction With the recent development of deep learning, a learning- based singing voice synthesis (SVS) system, which synthesizes sounds as natural as the concatenative method [1, 2, 3], but can expand more ﬂexibly, is proposed. For example, three SVS sys- tems based on DNN, LSTM, and Wavenet architecture were proposed, respectively [4, 5, 6]. These sy...

work page
[2]

Recently, the end-to-end TTS system, which is trained as an autoregressive manner, such as Tacotron[8], Deep voice[9], is showing better performance than the conventional method

Related work The SVS system is similar to the TTS system in terms of syn- thesizing natural human speech. Recently, the end-to-end TTS system, which is trained as an autoregressive manner, such as Tacotron[8], Deep voice[9], is showing better performance than the conventional method. In addition, various follow-up stud- ies are being conducted that have f...

work page
[3]

Adversarially Trained End-to-end Korean Singing Voice Synthesis System

Proposed Network As illustrated in Figure 1, our proposed model consists of two main modules, a mel-synthesis network and a super-resolution network. The mel-synthesis network is trained to produce a mel-spectrogram M1:L from previous mel input M0:L−1, time-aligned text T1:L, and pitch inputs P1:L. With text and arXiv:1908.01919v1 [cs.SD] 6 Aug 2019 Figur...

work page Pith review arXiv 1908
[4]

Dataset Since there is no publicly available Korean singing voice dataset, we created the dataset as follows

Experiments 4.1. Dataset Since there is no publicly available Korean singing voice dataset, we created the dataset as follows. First, we prepared accompaniment and singing voice MIDI ﬁles of 60 Korean pop songs. Next, a professional female vocalist was told to sing to the accompaniment. Then, the singing voice MIDI ﬁles were manually realigned so that the...

work page
[6]

In the case of sound quality, methods 1 and 2 did not sig- niﬁcantly affect the improvement, but the applying method 3 Figure 3: Generated spectrograms

In other words, all of the proposed methods helped to cre- ate more accurate pronunciation singing voices, and the perfor- mance was improved to the greatest extent with all three meth- ods. In the case of sound quality, methods 1 and 2 did not sig- niﬁcantly affect the improvement, but the applying method 3 Figure 3: Generated spectrograms. showed a sign...

work page
[7]

First, in order to enter pitch information, we added pitch encoders with the same structure as text encoders

we modiﬁed it to ﬁt the SVS system. First, in order to enter pitch information, we added pitch encoders with the same structure as text encoders. In addition, the local conditioning method proposed by [17] was used to conduct a conditioning of the encoded pitch on the mel decoder. Second, we assumed that among the various elements form- ing a singing voic...

work page
[8]

Acknowledgements This work has partly supported by National Research Founda- tion of Korea (NRF) funded by the Korea government (NRF- 2017R1E1A1A01076284), and partly by Institute for Infor- mation & Communications Technology Planning & Evalua- tion(IITP) grant funded by the Korea government (No.2019-0- 01367)

work page 2019

Show all 33 references

[9]

Concatenation-based midi-to-singing voice syn- thesis,

M. Macon, L. Jensen-Link, E. B. George, J. Oliverio, and M. Clements, “Concatenation-based midi-to-singing voice syn- thesis,” in Audio Engineering Society Convention 103 . Audio Engineering Society, 1997

1997
[10]

Sample- based singing voice synthesizer using spectral models and source- ﬁlter decomposition,

J. Bonada, A. Loscos, O. Mayor, and H. Kenmochi, “Sample- based singing voice synthesizer using spectral models and source- ﬁlter decomposition,” in Third International Workshop on Mod- els and Analysis of Vocal Emissions for Biomedical Applications, 2003

2003
[11]

V ocaloid-commercial singing syn- thesizer based on sample concatenation,

H. Kenmochi and H. Ohshita, “V ocaloid-commercial singing syn- thesizer based on sample concatenation,” in Eighth Annual Con- ference of the International Speech Communication Association , 2007

2007
[12]

Singing voice synthesis based on deep neural net- works

M. Nishimura, K. Hashimoto, K. Oura, Y . Nankaku, and K. Tokuda, “Singing voice synthesis based on deep neural net- works.” inInterspeech, 2016, pp. 2478–2482

2016
[13]

Korean singing voice synthesis system based on an lstm recurrent neural network,

J. Kim, H. Choi, J. Park, S. Kim, J. Kim, and M. Hahn, “Korean singing voice synthesis system based on an lstm recurrent neural network,” in INTERSPEECH 2018. International Speech Com- munication Association, 2018

2018
[14]

A neural parametric singing synthe- sizer modeling timbre and expression from natural songs,

M. Blaauw and J. Bonada, “A neural parametric singing synthe- sizer modeling timbre and expression from natural songs,” Ap- plied Sciences, vol. 7, no. 12, p. 1313, 2017

2017
[15]

Efﬁciently train- able text-to-speech system based on deep convolutional networks with guided attention,

H. Tachibana, K. Uenoyama, and S. Aihara, “Efﬁciently train- able text-to-speech system based on deep convolutional networks with guided attention,” in2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2018, pp. 4784–4788

2018
[16]

Tacotron: Towards end-to-end speech synthesis,

Y . Wang, R. Skerry-Ryan, D. Stanton, Y . Wu, R. J. Weiss, N. Jaitly, Z. Yang, Y . Xiao, Z. Chen, S. Bengio et al. , “Tacotron: Towards end-to-end speech synthesis,”arXiv preprint arXiv:1703.10135, 2017

2017 arXiv
[17]

Deep voice 2: Multi-speaker neural text- to-speech,

A. Gibiansky, S. Arik, G. Diamos, J. Miller, K. Peng, W. Ping, J. Raiman, and Y . Zhou, “Deep voice 2: Multi-speaker neural text- to-speech,” inAdvances in neural information processing systems, 2017, pp. 2962–2970

2017
[18]

Towards end-to-end prosody transfer for expressive speech synthesis with tacotron,

R. Skerry-Ryan, E. Battenberg, Y . Xiao, Y . Wang, D. Stanton, J. Shor, R. J. Weiss, R. Clark, and R. A. Saurous, “Towards end-to-end prosody transfer for expressive speech synthesis with tacotron,”arXiv preprint arXiv:1803.09047, 2018

2018 arXiv
[19]

Style tokens: Unsupervised style modeling, control and transfer in end-to-end speech synthesis,

Y . Wang, D. Stanton, Y . Zhang, R. Skerry-Ryan, E. Battenberg, J. Shor, Y . Xiao, F. Ren, Y . Jia, and R. A. Saurous, “Style tokens: Unsupervised style modeling, control and transfer in end-to-end speech synthesis,”arXiv preprint arXiv:1803.09017, 2018

2018 arXiv
[20]

Semi-supervised training for improving data efﬁciency in end-to- end speech synthesis,

Y .-A. Chung, Y . Wang, W.-N. Hsu, Y . Zhang, and R. Skerry-Ryan, “Semi-supervised training for improving data efﬁciency in end-to- end speech synthesis,”arXiv preprint arXiv:1808.10128, 2018

2018 arXiv
[21]

Speech band- width extension using generative adversarial networks,

S. Li, S. Villette, P. Ramadas, and D. J. Sinder, “Speech band- width extension using generative adversarial networks,” in 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2018, pp. 5029–5033

2018
[22]

cgans with projection discriminator,

T. Miyato and M. Koyama, “cgans with projection discriminator,” arXiv preprint arXiv:1802.05637, 2018

2018 arXiv
[23]

Large scale gan train- ing for high ﬁdelity natural image synthesis,

A. Brock, J. Donahue, and K. Simonyan, “Large scale gan train- ing for high ﬁdelity natural image synthesis,” arXiv preprint arXiv:1809.11096, 2018

2018 arXiv
[24]

Signal estimation from modiﬁed short- time fourier transform,

D. Grifﬁn and J. Lim, “Signal estimation from modiﬁed short- time fourier transform,”IEEE Transactions on Acoustics, Speech, and Signal Processing, vol. 32, no. 2, pp. 236–243, 1984

1984
[25]

Wavenet: A generative model for raw audio,

A. v. d. Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. Senior, and K. Kavukcuoglu, “Wavenet: A generative model for raw audio,” arXiv preprint arXiv:1609.03499, 2016

2016 arXiv
[26]

Dropout: a simple way to prevent neural net- works from overﬁtting,

N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: a simple way to prevent neural net- works from overﬁtting,” The Journal of Machine Learning Re- search, vol. 15, no. 1, pp. 1929–1958, 2014

1929
[27]

Improved training of wasserstein gans,

I. Gulrajani, F. Ahmed, M. Arjovsky, V . Dumoulin, and A. C. Courville, “Improved training of wasserstein gans,” in Advances in Neural Information Processing Systems, 2017, pp. 5767–5777

2017
[28]

Spectral normalization for generative adversarial networks,

T. Miyato, T. Kataoka, M. Koyama, and Y . Yoshida, “Spectral normalization for generative adversarial networks,”arXiv preprint arXiv:1802.05957, 2018

2018 arXiv
[29]

Which train- ing methods for gans do actually converge?

L. Mescheder, A. Geiger, and S. Nowozin, “Which train- ing methods for gans do actually converge?” arXiv preprint arXiv:1801.04406, 2018

2018 arXiv
[30]

Generative adver- sarial nets,

I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde- Farley, S. Ozair, A. Courville, and Y . Bengio, “Generative adver- sarial nets,” in Advances in neural information processing sys- tems, 2014, pp. 2672–2680

2014
[31]

Adam: A method for stochastic opti- mization,

D. P. Kingma and J. Ba, “Adam: A method for stochastic opti- mization,”arXiv preprint arXiv:1412.6980, 2014

2014 arXiv
[32]

Understanding the difﬁculty of train- ing deep feedforward neural networks,

X. Glorot and Y . Bengio, “Understanding the difﬁculty of train- ing deep feedforward neural networks,” in Proceedings of the thirteenth international conference on artiﬁcial intelligence and statistics, 2010, pp. 249–256

2010
[33]

World: a vocoder-based high-quality speech synthesis system for real-time applications,

M. Morise, F. Yokomori, and K. Ozawa, “World: a vocoder-based high-quality speech synthesis system for real-time applications,” IEICE TRANSACTIONS on Information and Systems , vol. 99, no. 7, pp. 1877–1884, 2016

2016

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [5]

We showed that using text information to model the phonetic enhancement mask actually worked, and produced more accurate pronunciation

Conclusions In this paper, we proposed the end-to-end Korean singing vo- cie synthesis system. We showed that using text information to model the phonetic enhancement mask actually worked, and produced more accurate pronunciation. Also, we successfully applied the conditional adversarial training method to the super- resolution stage, which resulted in a ...

work page

[2] [1]

For example, three SVS sys- tems based on DNN, LSTM, and Wavenet architecture were proposed, respectively [4, 5, 6]

Introduction With the recent development of deep learning, a learning- based singing voice synthesis (SVS) system, which synthesizes sounds as natural as the concatenative method [1, 2, 3], but can expand more ﬂexibly, is proposed. For example, three SVS sys- tems based on DNN, LSTM, and Wavenet architecture were proposed, respectively [4, 5, 6]. These sy...

work page

[3] [2]

Recently, the end-to-end TTS system, which is trained as an autoregressive manner, such as Tacotron[8], Deep voice[9], is showing better performance than the conventional method

Related work The SVS system is similar to the TTS system in terms of syn- thesizing natural human speech. Recently, the end-to-end TTS system, which is trained as an autoregressive manner, such as Tacotron[8], Deep voice[9], is showing better performance than the conventional method. In addition, various follow-up stud- ies are being conducted that have f...

work page

[4] [3]

Adversarially Trained End-to-end Korean Singing Voice Synthesis System

Proposed Network As illustrated in Figure 1, our proposed model consists of two main modules, a mel-synthesis network and a super-resolution network. The mel-synthesis network is trained to produce a mel-spectrogram M1:L from previous mel input M0:L−1, time-aligned text T1:L, and pitch inputs P1:L. With text and arXiv:1908.01919v1 [cs.SD] 6 Aug 2019 Figur...

work page Pith review arXiv 1908

[5] [4]

Dataset Since there is no publicly available Korean singing voice dataset, we created the dataset as follows

Experiments 4.1. Dataset Since there is no publicly available Korean singing voice dataset, we created the dataset as follows. First, we prepared accompaniment and singing voice MIDI ﬁles of 60 Korean pop songs. Next, a professional female vocalist was told to sing to the accompaniment. Then, the singing voice MIDI ﬁles were manually realigned so that the...

work page

[6] [6]

In the case of sound quality, methods 1 and 2 did not sig- niﬁcantly affect the improvement, but the applying method 3 Figure 3: Generated spectrograms

In other words, all of the proposed methods helped to cre- ate more accurate pronunciation singing voices, and the perfor- mance was improved to the greatest extent with all three meth- ods. In the case of sound quality, methods 1 and 2 did not sig- niﬁcantly affect the improvement, but the applying method 3 Figure 3: Generated spectrograms. showed a sign...

work page

[7] [7]

First, in order to enter pitch information, we added pitch encoders with the same structure as text encoders

we modiﬁed it to ﬁt the SVS system. First, in order to enter pitch information, we added pitch encoders with the same structure as text encoders. In addition, the local conditioning method proposed by [17] was used to conduct a conditioning of the encoded pitch on the mel decoder. Second, we assumed that among the various elements form- ing a singing voic...

work page

[8] [8]

Acknowledgements This work has partly supported by National Research Founda- tion of Korea (NRF) funded by the Korea government (NRF- 2017R1E1A1A01076284), and partly by Institute for Infor- mation & Communications Technology Planning & Evalua- tion(IITP) grant funded by the Korea government (No.2019-0- 01367)

work page 2019