Pith. sign in

REVIEW 4 major objections 4 minor 27 references

Improving Automatic Jazz Melody Generation by Transfer Learning Techniques

T0 review · 4 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Multitask learning beats fine-tuning for jazz melody generation

desk verdict A competent first look at transfer learning for jazz melody generation, with a useful new dataset, but the objective evidence for multitask learning is thin and the authors' own listening test cuts against it. read the letter →

arxiv 1908.09484 v1 pith:HFYFITRB submitted 2019-08-26 cs.SD cs.LGeess.AS

Hsiao-Tzu Hung , Chung-Yang Wang , Yi-Hsuan Yang , Hsin-Min Wang This is my paper · ORCID
classification cs.SDcs.LGeess.AS
keywords transferlearningjazzmelodygenerationmultitaskfine-tuningrecurrentvariationalautoencoderoverlappingareagenreclassifiersymbolicmusic
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether a large, genre-unspecified collection of MIDI melodies can help a model generate Jazz melodies when Jazz-specific data is scarce. It finds that yes, transfer learning helps, but the way it is done matters: multitask learning, which trains the model on source and target data together while conditioning on a genre label, outperforms the more common fine-tuning approach on the objective overlapping-area metric. The authors also report that six source-to-target data ratios behave differently for the two methods, with fine-tuning peaking at a ratio of three and multitask learning at one. A subjective listening test complicates the story: listeners who know Jazz preferred the fine-tuned melodies even though the objective metric favored multitask learning.

What carries the argument

The central object is a recurrent variational autoencoder (RVAE) for four-bar monophonic melody phrases represented as 4×16×48 pianorolls. The load-bearing mechanism is Method 2's addition of a one-hot genre label $y$ to the latent vector $z$ and a separately pre-trained genre classifier whose sigmoid output feeds a genre-prediction loss $L_{\text{genre}}$, giving the objective $L = L_{\text{recon}} + L_{\text{lat}} + L_{\text{genre}}$. This lets the model be trained on source and target data at once, using the classifier to steer generation toward Jazz. The evaluation machinery is the overlapping-area (OA) metric, which compares kernel-density estimates of feature distributions of generated and training melodies.

What would settle it

Generate melodies by copying phrases at random from the CY+R training set and compute their overlapping-area scores against that same training set; if verbatim copied phrases score higher than Method 2's generated melodies on the same features, the metric is rewarding distributional mimicry, and the claimed transfer-learning improvement could be memorization rather than better Jazz writing.

Watch

Extended reading notes

Core claim

Using a recurrent variational autoencoder that encodes four-bar melody phrases as pianorolls, the paper compares two transfer-learning strategies for Jazz melody generation. The first pre-trains the model on the TheoryTab genre-unspecified source set and fine-tunes on the small CY+R Jazz set. The second trains on both sets jointly, concatenating a one-hot genre label to the latent vector and adding a genre-classifier loss to the variational objective. Measured by overlapping area between generated and training melodies across nine pitch- and rhythm-related features, the multitask method achieves the highest average scores at R=1, and it beats the source-only and target-only baselines and fine-tuning on most features. The authors conclude that multitask learning is slightly better overall for the objective metric, while acknowledging that expert listeners in their subjective test preferred the fine-tuning method.

Load-bearing premise

The central claim rests on assuming that the overlapping-area metric, which rewards similarity between generated melodies and the Jazz training set, actually measures the quality of Jazz melody generation, and that random Gaussian sampling rules out memorization.

Editorial extensions

If this is right

  • Under-resourced musical genres can be tackled with transfer learning from a large genre-unspecified MIDI corpus, not just by collecting more genre-specific data.
  • Multitask learning with a genre label and classifier is a stronger transfer strategy than fine-tuning for the objective OA metric, and it works best when the source set is not much larger than the target set (R=1).
  • Fine-tuning helps the model learn what a melody is, but it does not beat directly training on the small Jazz set alone, so the source corpus adds little under this strategy.
  • The best method still underperforms or ties on pitch class histogram and pitch class transition features, meaning transfer gains are not uniform across musical attributes.
  • Because the subjective test and objective metric disagree, conclusions about which generated melodies are 'better' depend heavily on the evaluation metric chosen.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The OA metric measures distributional similarity to the training set, so the reported gains for Method 2 may partly reflect better mimicry of the target distribution, not necessarily higher musical quality; a held-out listener study with original compositions would separate these.
  • The genre classifier trained in Method 2 could be reused as a steerable control: by setting the genre label at generation time, the same model could produce both Jazz and non-Jazz melodies, which the paper does not exploit.
  • The expert-preference result suggests a testable extension: replace the generic OA features with jazz-specific vocabulary detectors (e.g., chromatic enclosures), which the paper itself gestures at, and the ranking of methods may flip.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper studies transfer learning for generating Jazz melodies with a recurrent variational autoencoder (RVAE). It uses a large genre-unspecified source dataset (TheoryTab, TT) and a small Jazz-only target dataset (CY+R), comparing two transfer strategies: fine-tuning a source-pretrained model on the target data (Method 1) and multitask training on both datasets with an auxiliary genre classifier (Method 2), across six source-to-target data ratios R. The authors evaluate generated melodies with the overlapping-area (OA) metric from Yang and Lerch [17] and with a subjective listening test, concluding that Method 2 outperforms Method 1 and both baselines on OA, while the subjective test shows that more experienced listeners prefer Method 1.

Significance. If the central claim holds, the paper would provide a practical recommendation for transfer learning in under-resourced music generation, a relatively underexplored application compared to classification/regression tasks. The paper is also one of the few to explicitly study the effect of the source-to-target data ratio in generative transfer. The authors include a detailed dataset construction effort and implement both fine-tuning and multitask variants, which are reasonable baselines for the community. The main contribution, however, is weakened by an evaluation protocol whose validity is not established: the OA metric is computed against the training set, the differences between methods are small and without significance testing, and the subjective test points in the opposite direction for the most expert listeners.

major comments (4)
  1. [V-C and Tables III-V] The OA metric is computed between the generated melodies and the training melodies, so a model that memorizes or closely mimics training phrases will score highly. The statement in Section V-C that 'Since the melodies are generated under random sampling conditions of a Gaussian distribution, there is no overfitting problem' is not justified: the decoder can map latent vectors sampled from N(0,1) into outputs very close to training points, and random sampling alone does not prevent that. Because the central claim (Method 2 outperforms) is built solely on OA numbers against the training set, the paper needs to rule out memorization, for example by reporting OA against a held-out test split, or by measuring the distance between generated and nearest training phrases. As written, the reported improvements could reflect distributional mimicry rather than better jazz writing.
  2. [Tables III-V] The numerical evidence for the central claim is thin and lacks statistical support. Method 2's average OA at R=1 is 0.7536 versus 0.7490 for Baseline 2, a difference of 0.0046, and the per-feature differences are also small. No error bars, multiple seeds, or significance tests are reported. In addition, the best R for each method is selected post hoc (R=3 for Method 1, R=1 for Method 2) from the same tables used for the comparison, without any correction for multiple comparisons or an independent validation set. The claim that 'Method 2 outperforms Method 1' should be supported by a proper statistical comparison, ideally with confidence intervals and a pre-specified selection rule, or at least by reporting all configurations rather than only the best ones.
  3. [VI-C, Table VI] The subjective test contradicts the objective ranking: Types II and III (music lovers and professional composers) prefer Method 1 over Method 2, and Method 2 only receives the highest score from Type I (subjects who seldom listen to soft jazz). The paper's explanatory paragraph is speculative (e.g., the chromatic-enclosure example is based on a single round), and no significance testing is reported for the listening scores. This discrepancy directly undermines the paper's implicit assumption that higher OA means higher musical quality. The authors should either reconcile the two evaluations or explicitly limit the objective claim to distributional similarity, not quality.
  4. [IV-C] The genre classifier used inside the multitask objective is trained on the same source (TT) and target (CY+R) data used to train the generative model. The Lgenre term can then be satisfied by reproducing label-specific training statistics, and the classifier's own accuracy or generalization is never reported. This makes it difficult to attribute Method 2's higher OA to better Jazz-specific generation rather than to the model learning to output melodies that the classifier (trained on the same data) recognizes as Jazz. An evaluation of the classifier on held-out data, or an ablation without the classifier, would strengthen the claim.
minor comments (4)
  1. [General] There are several typos and wording issues, e.g., 'datatsets' in Section III, 'discrminative' in Section II-B, and 'unconditioned' used in the introduction should be 'unconditional'. The paper would benefit from proofreading.
  2. [III] The description of the CY+R dataset says '1,446 phrases for training, and 162 phrases for testing', but Table II reports 1,608 total phrases; the numbers are consistent (1,446+162=1,608), but the split is not stated to be random or stratified. Please clarify how the split was performed.
  3. [V-B] The feature metrics section does not define exactly how the overlapping-area PDFs are computed from the histograms, particularly the kernel density estimation bandwidth. Since the OA numbers are central to the paper, a precise description (or a reference to a specific implementation) is needed for reproducibility.
  4. [IV-A] The notation for the variational lower bound in Eq. (1) is slightly confusing: Lrecon(x) is written without the expectation subscript in the equation but defined below with Eqφ(z|x). Please make the notation consistent.

Circularity Check

1 steps flagged · score 5.0 of 10

Central 'improvement' claim rests on an OA metric computed against the same training melodies used to fit the model, making the evaluation partly self-referential.

  1. fitted input called prediction [Section V-C (OA definition) and Tables III/IV captions]
    "OVERLAPPING AREA (OA) BETWEEN THE TRAINING MELODIES AND THE MELODIES GENERATED BY METHOD 1. ... Since the melodies are generated under random sampling conditions of a Gaussian distribution, there is no overfitting problem."

    The VAE is trained on the CY+R training melodies, and the OA metric in Tables III-V compares generated melodies with those same training melodies. Thus the reported 'improvement' is a measure of similarity to the fitting distribution, not an out-of-sample or externally grounded quality score. A model that reproduced training phrases would achieve OA close to 1. Random Gaussian sampling does not break the equivalence: the VAE's prior is regularized to match the aggregate posterior of training encodings, so N(0,1) samples can decode near training points.

full rationale

The paper's quantitative comparison (Tables III-V) is built entirely on the overlapping-area metric, which is computed between generated samples and the training set of CY+R (Section V-C, Tables III/IV captions). Because the same training set is used to fit the VAE (Section III: 1,446 training phrases) and to define the reference distribution for OA, the metric measures how well the generator mimics its own training distribution. The VAE reconstruction loss L_recon already optimizes fidelity to those training phrases, while the extra L_genre in Method 2 optimizes a classifier trained on the same source/target data; hence the objective numbers do not constitute an external test of jazz quality. The paper's own 'no overfitting' claim (Section V-C) is asserted without argument, and the subjective test (Table VI) actually shows expert listeners prefer Method 1, contradicting the OA ranking. However, the OA comparison between methods is not forced by any equation: better mimicry is not logically identical to better jazz, so this is partial circularity rather than a fully self-definitional derivation. Self-citations (e.g., [26] for the RVAE architecture) are present but not load-bearing; the central issue is the in-sample evaluation loop. Overall score 5.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The empirical claims rest on hand-chosen hyperparameters, a post hoc selection of the best data ratio, and an evaluation metric that measures similarity to the training distribution. No new physical or theoretical entities are postulated.

free parameters (4)
  • Learning rate schedule = pre-train 1e-3; fine-tune 1e-5, 1e-7, 1e-9
    Hand-selected for training; different schedules would change the comparison.
  • Best source-to-target ratio R = R=3 for Method 1, R=1 for Method 2
    Chosen post hoc from Tables III and IV to present the best comparison in Table V; this inflates the apparent performance of each method.
  • Architecture hyperparameters (latent dimension, layer sizes) = unspecified
    The RVAE capacity is not stated; the central result depends on these choices and they are not documented.
  • KDE bandwidth in OA computation = unspecified
    The OA values depend on the kernel density estimation smoothing, which is not reported.
assumptions (3)
  • domain assumption Source and target tasks share low-level features that transfer across genres
    Invoked in Section II-A as the premise of transfer learning; if generic pop phrasing does not transfer to jazz, the method cannot help.
  • ad hoc to paper Overlapping Area against the training set is a valid measure of generated melody quality
    Section V-C defines OA and claims no overfitting because sampling from a Gaussian is used; this assumption underpins the objective evaluation.
  • domain assumption The genre classifier can learn a meaningful Jazz/non-Jazz distinction from the two datasets
    The TT labels are acknowledged as unreliable (Section III), yet the multitask objective uses y=[1,0] for all TT and y=[0,1] for CY+R, assuming the source set is cleanly non-Jazz.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Improving Automatic Jazz Melody Generation by Transfer Learning Techniques." pith.science (2026). https://pith.science/paper/HFYFITRB

@misc{pith2026190809484,
  author       = {Pith},
  title        = {Pith review of: Improving Automatic Jazz Melody Generation by Transfer Learning Techniques},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HFYFITRB}},
  note         = {Machine review of arXiv:1908.09484}
}
read the original abstract

In this paper, we tackle the problem of transfer learning for Jazz automatic generation. Jazz is one of representative types of music, but the lack of Jazz data in the MIDI format hinders the construction of a generative model for Jazz. Transfer learning is an approach aiming to solve the problem of data insufficiency, so as to transfer the common feature from one domain to another. In view of its success in other machine learning problems, we investigate whether, and how much, it can help improve automatic music generation for under-resourced musical genres. Specifically, we use a recurrent variational autoencoder as the generative model, and use a genre-unspecified dataset as the source dataset and a Jazz-only dataset as the target dataset. Two transfer learning methods are evaluated using six levels of source-to-target data ratios. The first method is to train the model on the source dataset, and then fine-tune the resulting model parameters on the target dataset. The second method is to train the model on both the source and target datasets at the same time, but add genre labels to the latent vectors and use a genre classifier to improve Jazz generation. The evaluation results show that the second method seems to perform better overall, but it cannot take full advantage of the genre-unspecified dataset.

Figures

Figures reproduced from arXiv: 1908.09484 by the authors.

Figure 1
Figure 1. Model architectures and training flows of the methods evaluated in this paper. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Pitch histograms of (a) the source (TT) dataset (upper) and the target (CY+R) dataset (lower), (b) the melodies generated by Baseline 1 trained on TT (upper)and Baseline 2 trained on CY+R (lower), and (c) the melodies generated by Method 1 with R = 3 (upper) and Method 2 with R = 1 (lower). C C# D Eb E F F# G Ab A Bb B Major        Minor        0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 C C# D Eb E F F# G A… view at source ↗
Figure 3
Figure 3. (a) Upper: Pitch class histograms of the target (CY+R) and source (TT) datasets. Lower: the scales of C-Major and C-Minor; (b) Pitch class histograms of the target dataset and the melodies generated by Baseline 2 and Method 2 (R = 1). model trained on the source dataset by fine-tuning it with a small target dataset, but the improvement is not much. Third, surprisingly, Method 1 is worse than Baseline 2, indicating t… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Score of melody generated by (a) method 1, (b)method 2. TABLE VI THE RESULTS OF THE SUBJECTIVE TEST. CY+R Baseline 1 (source) Baseline 2 (target) Method 1 (R=3) Method 2 (R=1) Type I 3.5905 2.6286 2.7524 2.8095 2.8857 Type II 3.7744 2.3179 2.3282 2.6564 2.3538 Type III…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

27 extracted references · 23 canonical work pages

  1. [17]

    On the evaluation of generative models in music,

    L.-C. Yang and A. Lerch, “On the evaluation of generative models in music,” Neural Computing and Applications , pp. 1–12, 2018

    work page 2018

  2. [1]

    MidiNet: A convolutional generative adversarial network for symbolic-domain music generation,

    L.-C. Yang, S.-Y . Chou, and Y .-H. Yang, “MidiNet: A convolutional generative adversarial network for symbolic-domain music generation,” Proc. Int. Soc. Music Information Retrieval Conf. , 2017

    work page 2017

  3. [2]

    Project Magenta: Generating long-term structure in songs and stories,

    E. Waite, D. Eck, A. Roberts, and D. Abolafia, “Project Magenta: Generating long-term structure in songs and stories,” [Online] https: //magenta.tensorflow.org/2016/07/15/lookback-rnn-attention-rnn/, 2016

    work page 2016

  4. [3]

    JazzGAN: Improvising with generative adver- sarial networks,

    N. Trieu and R. Keller, “JazzGAN: Improvising with generative adver- sarial networks,” Proc. Int. Workshop on Musical Metacreation , 2018

    work page 2018

  5. [4]

    Generative adversarial nets,

    I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y . Bengio, “Generative adversarial nets,” Proc. Advances in Neural Information Processing Systems , pp. 2672– 2680, 2014

    work page 2014

  6. [5]

    Auto-encoding variational bayes

    D. P. Kingma and M. Welling, “Auto-encoding variational bayes”, arXiv preprint arXiv:1312.6114, 2013

    arXiv 2013

  7. [6]

    MuseGAN: Symbolic-domain music generation and accompaniment with multi-track sequential generative adversarial networks,

    H.-W. Dong, W.-Y . Hsiao, L.-C. Yang, and Y .-H. Yang, “MuseGAN: Symbolic-domain music generation and accompaniment with multi-track sequential generative adversarial networks,” Proc. AAAI Conf. Artificial Intelligence, 2018

    work page 2018

  8. [7]

    A hierarchical latent vector model for learning long-term structure in music,

    A. Roberts, J. Engel, C. Raffel, C.Hawthorne and D. Eck, “A hierarchical latent vector model for learning long-term structure in music,” Proc. Int. Conf. Machine Learning , 2018,

    work page 2018

Show all 27 references
  1. [8]

    AI methods in algorithmic composition: A comprehensive survey,

    J. D. Fern ´andez and F. Vico, “AI methods in algorithmic composition: A comprehensive survey,” J. Artificial Intelligence Research , vol. 48, no. 1, pp. 513–582, 2013

    2013

  2. [9]

    Song from PI: A musically plausible network for pop music generation,

    H. Chu, R. Urtasun, and S. Fidler, “Song from PI: A musically plausible network for pop music generation,” Proc. Int. Conf. Learning Represen- tations, Workshop Track, 2017

    2017

  3. [10]

    Music transcription modelling and composition using deep learning,

    B. L. Sturm, J. Felipe Santos, O. Ben-Tal, and I. Korshunova, “Music transcription modelling and composition using deep learning, ” arXiv preprint arXiv:1604.08723, 2016

    2016 arXiv

  4. [11]

    Hadjeres, F

    G. Hadjeres, F. Pachet and F. Nielsen, “DeepBach: A steerable model for Bach chorales generation, Proc. Int. Conf. Machine Learning , 2017

    2017

  5. [12]

    A Survey on Transfer Learning,

    S. J. Pan and Q. Yang, “A Survey on Transfer Learning,” IEEE Trans. Knowledge and Data Engineering , vol. 22, no. 10, pp. 1345–1359, 2010

    2010

  6. [13]

    Learning and trans- ferring mid-level image representations using convolutional neural net- works,

    M. Oquab, L. Bottou, I. Laptev, and J. Sivic, “Learning and trans- ferring mid-level image representations using convolutional neural net- works,” Proc. IEEE Conf. Computer Vision and Pattern Recognition , pp. 17171724, 2014

    2014

  7. [14]

    What makes ImageNet good for transfer learning?,

    M.-Y . Huh, P. Agrawal and A. A. Efros, “What makes ImageNet good for transfer learning?,” arXiv preprint arXiv:1608.08614, 2016

    2016 arXiv

  8. [15]

    BERT: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2018

    2018 arXiv

  9. [16]

    ASR for under-resourced languages from probabilistic transcription,

    M. A. Hasegawa-Johnson et al. , “ASR for under-resourced languages from probabilistic transcription,” IEEE/ACM Transactions on Audio, Speech, and Language Processing , vol. 25, no. 1, pp. 50-63, 2017

    2017

  10. [18]

    B.W.: Density estimation for statistics and data analysis, vol. 26. CRC press,1986

    1986

  11. [19]

    Transfer learning for music classification and regression tasks,

    K. Choi, G. Fazekas, M. Sandler, and K. Cho, “Transfer learning for music classification and regression tasks,” Proc. Int. Society of Music Information Retrieval Conf. , 2017

    2017

  12. [20]

    Transfer learning,

    H.-Y . Lee, “Transfer learning,” National Taiwan University, class lecture, [Online] http://speech.ee.ntu.edu.tw/ ∼tlkagk/courses/ML 2017/ Lecture/transfer.pdf, 2017

    2017

  13. [21]

    Transfer learning in MIR: sharing learned latent representations for music audio classification and similarity,

    M. E. P. Davies, K. Yoshii, and M. Goto, “Transfer learning in MIR: sharing learned latent representations for music audio classification and similarity,”Proc. Int. Society of Music Information Retrieval Conf. , 2013

    2013

  14. [22]

    Representation learning of music using artist labels,

    J. Park, J. Lee, J. Park, J. W. Ha, J. Nam, “Representation learning of music using artist labels,” Proc. Int. Society of Music Information Retrieval Conf., 2018

    2018

  15. [23]

    V ocal melody extraction with semantic segmenta- tion and audio-symbolic domain transfer learning,

    W.-T. Lu and L. Su, “V ocal melody extraction with semantic segmenta- tion and audio-symbolic domain transfer learning,” Proc. Int. Society of Music Information Retrieval Conf. , 2018

    2018

  16. [24]

    Learning domain-adaptive latent representations of music signals using variational autoencoders,

    Y .-J. Luo and L. Su, “Learning domain-adaptive latent representations of music signals using variational autoencoders,” Proc. Int. Society of Music Information Retrieval Conf. , 2018

    2018

  17. [25]

    Pypianoroll: Open source Python package for handling multitrack pianoroll,

    H.-W. Dong, W.-Y . Hsiao, and Yi-Hsuan Yang, “Pypianoroll: Open source Python package for handling multitrack pianoroll,” Proc. Int. Society of Music Information Retrieval Conf. , Late-breaking and demo paper, 2018

    2018

  18. [26]

    Lead sheet generation and arrangement via a hybrid generative model,

    H.-M. Liu, M.-H. Wu, and Y .-H. Yang, “Lead sheet generation and arrangement via a hybrid generative model,” Proc. Int. Society of Music Information Retrieval Conf. , Late-breaking and demo paper, 2018

    2018

  19. [27]

    Realtime chord recognition of musical sound: A system using common Lisp,

    T. Fujishima, “Realtime chord recognition of musical sound: A system using common Lisp,” Proc. Int. Computer Music Conf. , pp. 464–467, 1999

    1999

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.