REVIEW 4 major objections 6 minor 36 references
Conditional LSTM-GAN for Melody Generation from Lyrics
T0 review · 4 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read Conditional LSTM-GAN can generate plausible, tuneful melodies aligned syllable-by-syllable with input lyrics.
desk verdict A plausible lyrics-to-melody model with a genuinely useful aligned dataset, but the evaluation needs more rigorous baselines and validation of the alignment parser before the claims can be taken at face value. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the conditional LSTM-GAN, an adversarial pair in which both the generator and discriminator are LSTM networks that receive the same sequence of syllable embeddings as context. The generator's LSTM cells are unrolled once per syllable, concatenating each syllable vector with random noise and with the previous cell's output, so the model must learn a sequential alignment between text positions and note positions; the discriminator tries to tell real syllable-note pairs from generated ones, pushing the generator's output distribution toward the distribution of the training melodies. A final tuning step quantizes the continuous triplets to discrete MIDI, duration, and rest values and corrects out-of-tune notes to the nearest standard scale.
What would settle it
Compute the MMD or BLEU between melodies generated from the same noise but different lyrics; if the resulting distributions are statistically indistinguishable, the claimed lyric conditioning has no effect on the output.
Extended reading notes
Core claim
The central claim is that a deep LSTM-based generator and an LSTM-based discriminator, both conditioned on syllable embeddings of the lyrics, can jointly infer a melody and the alignment between each syllable and its note. Instead of sampling notes independently, the generator takes a 20-dimensional embedding that concatenates word-level and syllable-level skip-gram vectors plus a 30-dimensional noise vector, and emits a triplet of music attributes per syllable: MIDI number, note duration, and rest duration. A quantizer then rounds these continuous outputs to the nearest discrete musical values and maps out-of-scale notes into the closest matching standard scale. The paper reports that this setup produces melodies whose note-range, repetition, and transition statistics are closer to the ground-truth testing set than those of the random and MLE baselines, and that ablating the lyrics conditioning narrows the generated MIDI-number distribution away from the human distribution.
Load-bearing premise
The model depends on the assumption that the automatically extracted one-syllable-to-one-note alignments in the MIDI files are accurate and that discarding all multi-note syllables does not systematically distort the learned lyrics-to-melody mapping or the evaluation.
Editorial extensions
If this is right
- A single adversarial model can jointly produce pitch, rhythm, and syllable-to-note alignment from text, rather than treating alignment as a separate preprocessing step.
- Lyrics conditioning changes the generated pitch distribution measurably, so the model is not merely reproducing the marginal statistics of the training set.
- The newly built 12,197-song paired dataset gives later work a common training and evaluation ground for English lyrics-to-melody generation.
- The MLE baseline's lower BLEU and higher repetition metrics suggest that adversarial training adds something beyond next-note likelihood for this task.
- Because the model outputs quantized triplets, the same framework could be extended to other discrete symbolic music representations.
Reading between the lines
- The one-syllable-to-one-note constraint, which discards multi-note syllables, likely limits the model to syllabic singing; extending the alignment to one-syllable-to-multiple-notes would be a direct test of whether the learned mapping generalizes to melismatic vocal lines.
- The reported scale-consistency of around 49 percent suggests the tuning step is not resolving key ambiguity; combining the generator with an explicit key or scale condition could be a cheap way to raise musical plausibility.
- Because the discriminator is conditioned on the same lyrics, the model implicitly learns a lyric-to-melody association; one could test this by holding lyrics fixed and measuring how much melody variance remains, which would quantify how strongly text actually constrains the output.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a conditional LSTM-GAN for generating melodies from lyrics. It introduces a lyrics-melody dataset built from 12,197 MIDI files with syllable-note alignment extracted from lyrics and MIDI metadata, and it restricts attention to one-syllable-to-one-note pairs. The generator and discriminator are LSTM networks conditioned on syllable- and word-level embeddings, and a tuning scheme quantizes the continuous outputs and snaps out-of-tune notes to a nearby standard scale. The authors evaluate against a random baseline and a single-layer LSTM MLE baseline using attribute statistics, BLEU scores, scale consistency, a permutation-style test for lyric conditioning on note/rest durations, and a small listening study. They conclude that the model generates more plausible and tuneful melodies than the baselines while jointly learning syllable-note alignment.
Significance. If the central claim holds, the paper would make a useful contribution to lyrics-conditioned melody generation: the dataset is a substantial resource, the architecture is a reasonable application of conditional GANs to symbolic music, and the open-source release supports reproducibility. The permutation-style duration experiment and the subjective evaluation go beyond what many generative-music papers provide. However, the significance is currently limited by two load-bearing gaps: the alignment extraction, on which every training and evaluation step depends, is not validated or even fully specified; and the comparison set does not include published neural lyrics-to-melody systems, so the claimed advantage over the current state of the art is not established. The paper's own limitation note in Section III-C acknowledges the one-to-one alignment restriction, but the main claims are stated without that restriction, and no parser-error statistics or manual validation are reported.
major comments (4)
- [Section III-C and Section V-2] The key premise that the lyrics-melody alignment is accurately extracted is not supported. Section V-2 never specifies how 'a note has a corresponding English syllable' is determined, while Section III-C states that only one-syllable-to-one-note pairs are used and multi-note syllables are deferred to future work. No counts of discarded pairs, no parser-error statistics, and no manual validation are reported. Because the same parsed alignments are used as training targets, as references for MMD model selection, and as ground truth for BLEU and attribute metrics, any systematic parser error is baked into both the learned model and the evaluation. Please add a precise alignment rule, validation statistics, and at least a manually checked subset, or restrict the central claim to the validated one-to-one case.
- [Section VI.E, Tables V and VI] All reported quantitative results are single point estimates without error bars, confidence intervals, or repeated training runs. The BLEU-3 difference between the proposed model (0.460) and the MLE baseline (0.331) could be within run-to-run variability, and Table V shows the proposed model below ground truth on 3-MIDI repetitions and song length despite the text saying it outperforms the baseline 'in every aspect.' The evaluation also omits published neural baselines such as the lyrics-to-melody model of Bao et al. [9], so the central claim is not established against current state-of-the-art methods. Please report variance estimates or statistical tests over multiple runs and add comparisons to at least one existing neural lyrics-to-melody system.
- [Section VI.A and Section VI.D] The tuning scheme remaps out-of-tune notes to the inferred scale before validation and testing, so scale-consistency and some pitch metrics reflect the post-processing rule as much as the learned model. The Random baseline is also tuned, which makes the comparison fair in that narrow sense, but the contribution of tuning to the reported metrics should be quantified by ablating it, for example by reporting results with tuning disabled at test time.
- [Section VI.F, Figures 11 and 12] The permutation-style experiment establishes that the model uses lyric conditioning for note and rest durations, but no analogous test is provided for MIDI pitch, although Figures 11 and 12 are used to claim that lyric conditioning shapes pitch distributions. The two-lyric illustration in Figure 12 is anecdotal. Please provide a statistical test for the pitch attribute analogous to the duration experiment in Section VI.F.2.
minor comments (6)
- [Section IV.C] The section heading contains a typo: 'Condtional' should be 'Conditional.'
- [Introduction] The phrase 'national language processing' should be 'natural language processing.'
- [Section VI.E] The description of Table V should be clarified: the proposed model does not outperform the baselines on every reported attribute, since 3-MIDI repetitions and song length are further from the ground-truth values than the MLE baseline for some rows.
- [Section VI.E] The scale-consistency metric is not defined precisely; please state how 'most likely scale' is inferred and how 'mean accuracy' is computed, especially for notes that belong to multiple standard scales.
- [Section V.2] The notation for the rest calculation is ambiguous: the formula uses note-on_k and note-off_{k-1} but the prior note index is not defined.
- [Section VI.G] The subjective evaluation uses only 7 subjects and reports averaged scores without any significance test; please add at least basic variability information or acknowledge this limitation explicitly.
Circularity Check
No significant circularity: the central derivation is self-contained, with model selection and evaluation on held-out data.
full rationale
The paper's core claim is that a conditional LSTM-GAN can generate melodies from lyrics. The generation model is trained on paired lyrics-melody data; model selection uses MMD on a held-out validation split, and all reported quantitative metrics (MIDI span, repetitions, BLEU, scale consistency, transition distributions) plus the subjective listening test are computed on the held-out testing split. The comparison baselines (Random and MLE) are trained and evaluated under the same protocol, so the reported advantage is not forced by construction. The dataset and alignment extraction are external inputs, not outputs of the model; the one-syllable-to-one-note simplification is an acknowledged limitation (Section III-C) and a data-quality concern, not a circular step. The reference [5] by one of the authors is used only as general related work and is not load-bearing. The scale-tuning post-processing is a deterministic correction applied during validation and testing; it may affect what the metrics measure, but it is not a fitted parameter and does not make the predicted melody equal to a training input by construction. No equation in the paper defines a predicted quantity in terms of the quantity it is supposed to predict, and no load-bearing argument reduces to a self-citation. Therefore no specific circular step can be exhibited.
Assumptions & free parameters
free parameters (6)
- sequence_length =
20
- syllable_and_word_embedding_dimension =
10 each (20 total)
- noise_vector_dimension =
30
- hidden_units_per_lstm_layer =
400
- note_duration_and_rest_quantization_sets =
durations {0.25,0.5,0.75,1,1.5,2,3,4,6,8,16,32}; rests {0,1,2,4,8,16,32}
- scale_inference_and_tuning_rule =
not specified
assumptions (6)
- standard math LSTM cell equations and GAN minimax objective are standard and assumed correct.
- domain assumption One-syllable-to-one-note alignment is a sufficient representation of lyrics-melody relationship.
- domain assumption MIDI files and their embedded lyrics are correctly parsed and aligned from the LMD-full and reddit datasets.
- domain assumption Skip-gram embeddings capture semantic and syntactic information relevant to melody.
- ad hoc to paper Quantizing and remapping out-of-tune notes to an inferred scale is an acceptable post-processing step.
- ad hoc to paper Truncating songs to 20-note sequences preserves the training signal.
Cite this review
Pith. "Pith review of Conditional LSTM-GAN for Melody Generation from Lyrics." pith.science (2026). https://pith.science/paper/BQMBVXNA
@misc{pith2026190805551,
author = {Pith},
title = {Pith review of: Conditional LSTM-GAN for Melody Generation from Lyrics},
year = {2026},
howpublished = {\url{https://pith.science/paper/BQMBVXNA}},
note = {Machine review of arXiv:1908.05551}
}
read the original abstract
Melody generation from lyrics has been a challenging research issue in the field of artificial intelligence and music, which enables to learn and discover latent relationship between interesting lyrics and accompanying melody. Unfortunately, the limited availability of paired lyrics-melody dataset with alignment information has hindered the research progress. To address this problem, we create a large dataset consisting of 12,197 MIDI songs each with paired lyrics and melody alignment through leveraging different music sources where alignment relationship between syllables and music attributes is extracted. Most importantly, we propose a novel deep generative model, conditional Long Short-Term Memory - Generative Adversarial Network (LSTM-GAN) for melody generation from lyrics, which contains a deep LSTM generator and a deep LSTM discriminator both conditioned on lyrics. In particular, lyrics-conditioned melody and alignment relationship between syllables of given lyrics and notes of predicted melody are generated simultaneously. Experimental results have proved the effectiveness of our proposed lyrics-to-melody generative model, where plausible and tuneful sequences can be inferred from lyrics.
Figures
Figures from the paper (9 more)
Reference graph
Works this paper leans on
-
[1]
A preliminary framework for description, analysis and comparison of creative systems,
G. A. Wiggins, “A preliminary framework for description, analysis and comparison of creative systems,” Journal of Knowledge Based Systems , vol. 19, no. 7, pp. 449–458, 2006
work page 2006
-
[2]
Musical composition with a high-speed digital computer,
L. A. Hiller and L. M. Isaacson, “Musical composition with a high-speed digital computer,”Journal of the Audio Engineering Society, vol. 6, no. 3, pp. 154–160, 1958
work page 1958
-
[3]
Statistical learning of harmonic movement,
D. Ponsford, G. Wiggins, and C. Mellish, “Statistical learning of harmonic movement,” Journal of New Music Research , vol. 28, no. 2, pp. 150–177, 1999
work page 1999
-
[4]
Music Generation by Deep Learning - Challenges and Directions
J. Briot and F. Pachet, “Music generation by deep learning - challenges and directions,” CoRR, vol. abs/1712.04371, 2017. [Online]. Available: http://arxiv.org/abs/1712.04371
work page Pith review arXiv 2017
-
[5]
Deep cross-modal correlation learning for audio and lyrics in music retrieval,
Y . Yu, S. Tang, F. Raposo, and L. Chen, “Deep cross-modal correlation learning for audio and lyrics in music retrieval,” ACM Trans. Multimedia Comput. Commun. Appl. , vol. 15, no. 1
-
[6]
https://en.wikipedia.org/wiki/Melody
-
[7]
Smug: Scientific music generator,
M. Scirea, G. A. B. Barros, N. Shaker, and J. Togelius, “Smug: Scientific music generator,” in Sixth International Conference on Computational Creativity, 2015, pp. 204–211
work page 2015
-
[8]
Algorithmic Songwriting with ALYSIA
M. Ackerman and D. Loker, “Algorithmic songwriting with ALYSIA,” CoRR, vol. abs/1612.01058, 2016. [Online]. Available: http://arxiv.org/ abs/1612.01058
work page Pith review arXiv 2016
Show all 36 references
-
[10]
Generative Adversarial Networks,
I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde- 10 Fig. 11: Left: estimated distribution of generated MIDI numbers conditioned on lyrics. Right: estimated distribution of ground truth MIDI numbers. Fig. 12: Estimated distributions of MIDI numbers generated by tw...
2014
-
[11]
AI methods in algorithmic compo- sition: A comprehensive survey,
J. D. F. Rodriguez and F. J. Vico, “AI methods in algorithmic compo- sition: A comprehensive survey,” CoRR, vol. abs/1402.0585, 2014
2014 arXiv
-
[12]
Constraint programming systems for modeling music theories and composition,
T. Anders and E. R. Miranda, “Constraint programming systems for modeling music theories and composition,” ACM Comput. Surv
-
[13]
Inmamusys: Intelligent multiagent music system,
M. Delgado, W. Fajardo, and M. Molina-Solana, “Inmamusys: Intelligent multiagent music system,” Expert Syst. Appl
-
[14]
Music generation from statistical models,
D. Conklin, “Music generation from statistical models,” in SAISB Symposium on Artificial Intelligence and Creativity in the Arts and Sciences, 2003, pp. 30–35
2003
-
[15]
Realtime generation of harmonic progres- sions using controlled markov selection,
A. Eigenfeldt and P. Pasquier, “Realtime generation of harmonic progres- sions using controlled markov selection,” in International Conference on Fig. 14: Boxplots of the distributions of drs, drn and drns (for the rest duration attribute). d = 1 .336 is highlighted in red in ...
2010
-
[16]
Cope, Computer Models of Musical Creativity
D. Cope, Computer Models of Musical Creativity . The MIT Press, 2005
2005
-
[17]
A hierarchical re- current neural network for symbolic melody generation,
J. Wu, C. Hu, Y . Wang, X. Hu, and J. Zhu, “A hierarchical re- current neural network for symbolic melody generation,” CoRR, vol. abs/1712.05274, 2017
2017 arXiv
-
[18]
Generating polyphonic music using tied parallel net- works,
D. D. Johnson, “Generating polyphonic music using tied parallel net- works,” in International Conference on Evolutionary and Biologically Inspired Music and Art , 2017, pp. 128–143
2017
-
[19]
C-RNN-GAN: continuous recurrent neural networks with adversarial training,
O. Mogren, “C-RNN-GAN: continuous recurrent neural networks with adversarial training,” CoRR, vol. abs/1611.09904, 2016
2016 arXiv
-
[20]
Automatic song composition from the lyrics exploiting prosody of the japanese language,
S. Fukayama, K. Nakatsuma, S. Sako, T. Nishimoto, and S. Sagayama, “Automatic song composition from the lyrics exploiting prosody of the japanese language,” in International Conference of Sound and Music Computing, 2010, pp. 299–302
2010
-
[21]
Automatic generation of melodic accompaniments for lyrics,
K. Monteith, T. R. Martinez, and D. Ventura, “Automatic generation of melodic accompaniments for lyrics,” in Proceedings of the Third International Conference on Computational Creativity, 2012. , 2012, pp. 87–94
2012
-
[22]
http://www.musiccrashcourses.com/lessons/pitch.html/
-
[23]
Available: https://en.wikipedia.org/wiki/Duration (music)/
[Online]. Available: https://en.wikipedia.org/wiki/Duration (music)/
-
[24]
Available: https://en.wikipedia.org/wiki/Rest (music)/
[Online]. Available: https://en.wikipedia.org/wiki/Rest (music)/
-
[25]
https://en.wikipedia.org/wiki/Syllable
-
[26]
Long short-term memory,
S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural Computation, vol. 9, no. 8, pp. 1735–1780, 1997. [Online]. Available: https://doi.org/10.1162/neco.1997.9.8.1735
1997 doi
-
[27]
Conditional generative adversarial nets,
M. Mirza and S. Osindero, “Conditional generative adversarial nets,” CoRR, vol. abs/1411.1784, 2014. [Online]. Available: http: //arxiv.org/abs/1411.1784
2014 arXiv
-
[28]
https://colinraffel.com/projects/lmd/
-
[29]
https://www.reddit.com/r/datasets/
-
[30]
A hilbert space embedding for distributions,
A. Smola, A. Gretton, L. Song, and B. Sch ¨olkopf, “A hilbert space embedding for distributions,” inAlgorithmic Learning Theory, M. Hutter, R. A. Servedio, and E. Takimoto, Eds. Springer Berlin Heidelberg, 2007. 11 Fig. 15: Subjective evaluation results
2007
-
[31]
A Test of Relative Similarity For Model Selection in Generative Models,
W. Bounliphone, E. Belilovsky, M. B. Blaschko, I. Antonoglou, and A. Gretton, “A Test of Relative Similarity For Model Selection in Generative Models,” arXiv e-prints, 2015
2015
-
[32]
S. J. Reddi, A. Ramdas, B. Poczos, A. Singh, and L. Wasserman, 2014
2014
-
[33]
iComposer: An automatic song- writing system for Chinese popular music,
H.-P. Lee, J.-S. Fang, and W.-Y . Ma, “iComposer: An automatic song- writing system for Chinese popular music,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)
2019
-
[34]
Seqgan: Sequence generative adversarial nets with policy gradient,
L. Yu, W. Zhang, J. Wang, and Y . Yu, “Seqgan: Sequence generative adversarial nets with policy gradient,” 2016
2016
-
[35]
https://synthesizerv.com/en/
-
[36]
Categorical reparameterization with gumbel-softmax,
E. Jang, S. Gu, and B. Poole, “Categorical reparameterization with gumbel-softmax,” 2016
2016
-
[2018]
Available: http://arxiv.org/abs/1809.04318
[Online]. Available: http://arxiv.org/abs/1809.04318
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.