REVIEW 4 major objections 7 minor 46 references
Synthesising Handwritten Music with GANs: A Comprehensive Evaluation of CycleWGAN, ProGAN, and DCGAN
T0 review · 4 major / 7 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read This paper claims that CycleWGAN, a CycleGAN variant with Wasserstein loss, outperforms DCGAN and ProGAN at generating handwritten music crops, scoring FID 41.87, IS 2.29, and KID 0.05.
desk verdict Useful benchmark of GAN variants for handwritten music, but the headline 'CycleWGAN significantly outperforms' is confounded by a conditional-vs-unconditional task mismatch and a 200-image evaluation set. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the CycleWGAN architecture: two generators and two discriminators translating between printed and handwritten music domains, trained with Wasserstein adversarial losses plus a cycle-consistency loss that forces round trips to reconstruct the input. Supporting machinery includes instance normalisation, chosen because batch normalisation is incompatible with Wasserstein training, nine ResNet blocks in the generator, DCGAN as a 64x64 baseline, and ProGAN with progressive growth from 4x4 to 128x128 and WGAN-GP. The comparison is carried by standard generative metrics, FID, IS, and KID, supplemented by qualitative object detection and PCA.
What would settle it
Compute FID, IS, and KID on at least a few thousand generated images with bootstrap confidence intervals; the CycleWGAN claim collapses if its FID advantage over ProGAN falls inside the noise band. A second falsifier is a downstream OMR test: if a recogniser trained with CycleWGAN crops does not match or beat training on ProGAN crops or real data, the practical claim for improving OMR is unsupported.
Extended reading notes
Core claim
CycleWGAN replaces the standard adversarial losses in CycleGAN with the Wasserstein distance, uses instance normalisation in both networks, and employs a ResNet-based generator with nine residual blocks while keeping an L2 cycle-consistency loss with lambda equal to 10. The paper's finding is that these changes improve training stability and style transfer enough that CycleWGAN substantially outperforms DCGAN and ProGAN on FID, IS, and KID, and that PCA places its outputs closest to real handwritten crops. The paper presents this as evidence that CycleWGAN is a promising data-augmentation tool for OMR, while noting that symbol-level errors, such as imperfect note heads and accidentals, mean the images are not yet ready for direct use in training OMR systems.
Load-bearing premise
The ranking rests on the assumption that FID, IS, and KID computed on 200 generated crops are stable enough to separate the models, despite the paper's own note that FID typically needs larger samples and the absence of confidence intervals or significance tests.
Editorial extensions
If this is right
- CycleWGAN provides a stronger baseline than the earlier CycleGAN for handwriting style transfer in music, so future image-to-image work can start from the Wasserstein variant.
- The generated crops, after correcting note-head and accidental errors, could augment OMR training sets and reduce the annotation bottleneck.
- ProGAN's greater symbol completeness and CycleWGAN's superior style closeness are complementary, suggesting that combining or conditioning these approaches is a plausible next step.
- The reported metric margins need validation on larger evaluation sets before being used as a design decision for OMR pipelines, as the paper itself notes FID typically requires larger samples.
- If CycleWGAN is right, data-hungry OMR systems can benefit from synthetic handwritten crops rather than relying only on scarce archives.
Reading between the lines
- Editorial inference: the 200-image evaluation set makes FID the least reliable of the three metrics; KID's unbiased estimator is more believable at this sample size, and its large gap (0.05 vs 0.17) is the strongest quantitative evidence for CycleWGAN.
- Editorial inference: because the models are trained and evaluated on square crops, the ranking may not survive the harder task of generating full pages; a full-page or stream-based evaluation could invert the ordering.
- Editorial inference: a direct downstream test, training an OMR recogniser on real plus synthetic crops and measuring symbol error rate on held-out manuscripts, would settle practical value faster than FID, IS, or KID.
- Editorial inference: incorporating WGAN-GP into CycleWGAN, as the paper suggests, is a concrete next experiment; the ProGAN results show gradient penalty helps stability, but it may change CycleWGAN's style-transfer behaviour.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper applies three GAN architectures—DCGAN, ProGAN, and CycleWGAN (a CycleGAN variant using Wasserstein losses and instance normalization)—to synthesize handwritten music crops from the CVC-MUSCIMA and DoReMi datasets. It reports quantitative metrics (FID, IS, KID) and qualitative analyses including object detection and PCA, and claims that CycleWGAN significantly outperforms the other two models and offers a promising route to improving OMR training data. The paper also provides hyperparameter details and a discussion of limitations.
Significance. If the comparison were task-equivalent and statistically grounded, this would be a useful empirical baseline for GAN-based handwritten music synthesis. The CycleWGAN adaptation is a reasonable engineering contribution, and the qualitative evaluation with a pre-trained object detector is a valuable addition. However, the central claim is not supported as stated: the comparison confounds conditioning with generation ability, the quantitative metrics lack uncertainty quantification, and the paper's own qualitative results contradict the headline conclusion. The work is nevertheless a reasonable starting point for a more careful evaluation.
major comments (4)
- [Sec. III-B1/2/3; Sec. V-B, Table I] The comparison is not task-equivalent. CycleWGAN is a conditional image-to-image translator that takes printed music crops as input, while DCGAN and ProGAN are unconditional generators that map latent noise z to images. The lower FID of CycleWGAN (41.87 vs 49.54 vs 210.18) may therefore reflect the strong conditioning signal rather than superior generative ability. To support the abstract's claim of 'significantly outperforms', the evaluation must either condition all models on the same printed input or evaluate CycleWGAN in an unconditional setting; at a minimum, the claims should be re-framed as 'best on the conditional translation task'.
- [Sec. V-B, Table I; Sec. V-D] The quantitative ranking relies on point estimates from a 200-image evaluation set with no confidence intervals or significance tests. The Inception Scores overlap within one standard deviation (CycleWGAN 2.29 ± 0.19, DCGAN 2.17 ± 0.14, ProGAN 2.01 ± 0.20), so the word 'significantly' is not supported. The paper also does not specify how the 200 evaluation images were chosen, whether the same real-image reference set was used for all models, or how many generated images were evaluated. Please provide confidence intervals (e.g., bootstrap) or a significance test, and describe the evaluation set construction; otherwise the reported margins may be evaluation noise.
- [Sec. V-A; Sec. VI] The qualitative results contradict the central claim. The paper states that CycleWGAN outputs contain gaps, pitch shifts, and missing noteheads and are 'unsuitable for direct use in training OMR systems', while ProGAN produces 'more complete and balanced images'. This internal contradiction needs to be resolved: either the quantitative advantage of CycleWGAN is not diagnostically meaningful for the stated application, or the conclusion that CycleWGAN is 'a promising solution for improving OMR systems' should be substantially weakened.
- [Sec. V-B, Eq. (5)] The Inception Score is computed with an ImageNet-pretrained Inception network, but handwritten music crops are far outside that domain, making the conditional label distribution p(y|x) largely arbitrary and the IS values difficult to interpret. The paper itself concedes that the direct applicability of these metrics 'is less established', yet IS is used as headline evidence. Either justify the use of IS for this domain (e.g., with a domain-adapted classifier) or remove it from the main claims and rely on FID/KID plus the qualitative evaluation.
minor comments (7)
- [Sec. V-B] The text contains the literal placeholder '[cite key papers]' in the paragraph motivating metric selection; this must be completed before submission.
- [Sec. V-A] The statement that DCGAN suffers from mode collapse '(see Figure 6(c))' is incorrect because Figure 6(c) is the printed input image, not a DCGAN output; please fix the cross-reference and check all figure references.
- [Sec. V-C, Fig. 9] The text describes 'Figure 9(b)' as comparing real handwritten, real printed, and CycleWGAN-generated images, but in the figure caption panel (b) is the ProGAN panel; correct either the text or the caption.
- [Throughout] Capitalization of model names is inconsistent in the text, Table I, and figures (DCGAN, PROGAN, ProGAN, CycleWGAN); please standardize.
- [References] Reference [26], cited as 'Improved CycleGAN for image-to-image translation, 2021', lacks author names and venue; provide full bibliographic details.
- [Sec. IV-C] The phrase 'Experimented with custom beta values' should read 'We experimented with custom beta values'.
- [Sec. III-A] The text refers to 'Figures 1a and 1b' but the sub-caption labels in Figure 1 are not defined in the caption; adding explicit '(a)' and '(b)' labels would improve clarity.
Circularity Check
No significant circularity: the paper is an empirical GAN comparison with measured metrics, and its self-citations are not load-bearing.
full rationale
The paper's central claim is an empirical ranking of CycleWGAN, ProGAN, and DCGAN based on measured IS, FID, and KID values reported in Section V-B and Table I. These numbers are outputs of an evaluation procedure, not quantities forced by the model definitions or loss equations in Section III-B. The CycleWGAN loss functions (Wasserstein adversarial losses plus cycle consistency) do not algebraically determine the FID/IS/KID values, and no fitted parameter is renamed as a prediction. The self-citations, most notably Ref. [7] for an earlier CycleGAN baseline, are used for comparison and for motivating inpainting-style refinements, but the CycleWGAN result does not reduce to a claim imported from that reference. The paper even flags a relevant external-validity limitation in Section V-D: the evaluation uses only 200 images, and FID typically requires larger sample sizes; this weakens the strength of the ranking but is not a circularity. The possible task mismatch between the conditional image-to-image model (CycleWGAN) and unconditional generators (DCGAN, ProGAN) is a substantive fairness-of-comparison concern, but it does not make the reported measurements equivalent to the model's inputs by construction. Overall, the derivation chain here is empirical and self-contained, so there is no significant circularity.
Assumptions & free parameters
free parameters (6)
- Cycle consistency weight lambda_cycle =
10
- Gradient penalty coefficient lambda_GP (ProGAN) =
10
- Learning rates =
DCGAN 2e-4, ProGAN 1e-3, CycleWGAN 1e-5
- ProGAN latent dimension z =
256
- CycleWGAN residual blocks =
9
- Evaluation set size =
200 images
assumptions (4)
- domain assumption CVC-MUSCIMA and DoReMi crops form a valid paired/unpaired training set for printed-to-handwritten music translation.
- domain assumption Inception-network features used by FID/IS/KID are meaningful for handwritten music images.
- domain assumption The 200-image evaluation set gives stable estimates of FID/IS/KID.
- domain assumption Cycle-consistency loss with L2 norm preserves musical content during translation.
invented entities (1)
-
CycleWGAN architecture variant
Cite this review
Pith. "Pith review of Synthesising Handwritten Music with GANs: A Comprehensive Evaluation of CycleWGAN, ProGAN, and DCGAN." pith.science (2026). https://pith.science/paper/5HLET42V
@misc{pith2026241116405,
author = {Pith},
title = {Pith review of: Synthesising Handwritten Music with GANs: A Comprehensive Evaluation of CycleWGAN, ProGAN, and DCGAN},
year = {2026},
howpublished = {\url{https://pith.science/paper/5HLET42V}},
note = {Machine review of arXiv:2411.16405}
}
read the original abstract
The generation of handwritten music sheets is a crucial step toward enhancing Optical Music Recognition (OMR) systems, which rely on large and diverse datasets for optimal performance. However, handwritten music sheets, often found in archives, present challenges for digitisation due to their fragility, varied handwriting styles, and image quality. This paper addresses the data scarcity problem by applying Generative Adversarial Networks (GANs) to synthesise realistic handwritten music sheets. We provide a comprehensive evaluation of three GAN models - DCGAN, ProGAN, and CycleWGAN - comparing their ability to generate diverse and high-quality handwritten music images. The proposed CycleWGAN model, which enhances style transfer and training stability, significantly outperforms DCGAN and ProGAN in both qualitative and quantitative evaluations. CycleWGAN achieves superior performance, with an FID score of 41.87, an IS of 2.29, and a KID of 0.05, making it a promising solution for improving OMR systems.
Figures
Figures from the paper (5 more)
Reference graph
Works this paper leans on
-
[1]
J. Haji ˇc and P. Pecina, ”The MUSCIMA++ Dataset for Handwritten Optical Music Recognition,” in Proc. 14th IAPR Int. Conf. Document Analysis and Recognition (ICDAR) , Kyoto, Japan, 2017, pp. 39–46, doi: 10.1109/ICDAR.2017.16
- [2]
-
[3]
J. Y . Zhu, T. Park, P. Isola, and A. A. Efros, ”Unpaired image-to-image translation using cycle-consistent adversarial networks,” in Proc. IEEE Int. Conf. Computer Vision (ICCV) , 2017, pp. 2223–2232
work page 2017
- [4]
-
[5]
E. Shatri and G. Fazekas, ”Optical music recognition: State of the art and major challenges,” arXiv preprint, arXiv:2006.07885, 2020
arXiv 2006
-
[6]
E. Shatri and G. Fazekas, ”DoReMi: First glance at a universal OMR dataset,” arXiv preprint, arXiv:2107.07786, Jul. 2021
arXiv 2021
- [7]
-
[8]
J. Calvo-Zaragoza, J. H. Jr., and A. Pacha, ”Understanding optical music recognition,” ACM Comput. Surveys (CSUR) , vol. 53, no. 4, pp. 1–35, 2020
work page 2020
Show all 46 references
-
[9]
Pacha, K
A. Pacha, K. Y . Choi, B. Cou ¨asnon, Y . Ricquebourg, R. Zanibbi, and H. Eidenberger, ”Handwritten music object detection: Open issues and baseline results,” in Proc. 13th IAPR Int. Workshop on Document Analysis Systems (DAS) , 2018, pp. 163–168
2018
-
[10]
S. Ren, K. He, R. Girshick, and J. Sun, ”Faster R-CNN: Towards real-time object detection with region proposal networks,” Advances in Neural Information Processing Systems (NeurIPS) , vol. 28, 2015
2015
-
[11]
Z. Wang, Q. She, and T. E. Ward, ”Generative Adversarial Networks in Computer Vision: A Survey and Taxonomy,” ACM Comput. Surveys , vol. 54, no. 2, Article 37, pp. 1–38, Mar. 2022, doi: 10.1145/3439723
2022 doi
-
[12]
J. Dai, Y . Li, K. He, and J. Sun, ”R-FCN: Object detection via region- based fully convolutional networks,” in Advances in Neural Information Processing Systems (NeurIPS) , vol. 29, 2016
2016
-
[13]
W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C. Y . Fu, and A. C. Berg, ”SSD: Single shot multibox detector,” in Proc. Eur . Conf. Computer Vision (ECCV) , Amsterdam, The Netherlands, 2016, pp. 21– 37, Springer International Publishing
2016
-
[14]
Mirza and S
M. Mirza and S. Osindero, ”Conditional generative adversarial nets,” arXiv preprint, arXiv:1411.1784, 2014
2014 arXiv
-
[15]
Cardoso, S
R. Cardoso, S. Vallecorsa, and E. Nemni, ”Conditional progressive generative adversarial network for satellite image generation,” arXiv preprint, arXiv:2211.15303, Nov. 2022
2022 arXiv
-
[16]
L. Sun, J. Chen, Y . Xu, M. Gong, K. Yu, and K. Batmanghelich, ”Hierarchical amortized GAN for 3D high-resolution medical image synthesis,” IEEE J. Biomed. Health Informatics , vol. 26, no. 8, pp. 3966– 3975, May 2022
2022
-
[17]
F. F. Farooqui, M. Hassan, M. S. Younis, and M. K. Siddhu, ”Offline hand-written Urdu word spotting using random data generation,” IEEE Access, vol. 8, pp. 131119–131136, Jul. 2020
2020
-
[18]
R ´ıos-Vila, J
A. R ´ıos-Vila, J. M. I ˜nesta, and J. Calvo-Zaragoza, ”On the use of transformers for end-to-end optical music recognition,” in Proc. Iberian Conf. Pattern Recognition and Image Analysis , Apr. 2022, pp. 470–481, Cham: Springer International Publishing
2022
-
[19]
Huang, X
Z. Huang, X. Jia, and Y . Guo, ”State-of-the-art model for music object recognition with deep learning,” Applied Sciences , vol. 9, no. 13, p. 2645, Jun. 2019
2019
-
[20]
Bar ´o, P
A. Bar ´o, P. Riba, and A. Forn´es, ”Towards the recognition of compound music notes in handwritten music scores,” in Proc. 15th Int. Conf. Frontiers in Handwriting Recognition (ICFHR) , Oct. 2016, pp. 465– 470
2016
-
[21]
F. A. Croitoru, V . Hondru, R. T. Ionescu, and M. Shah, ”Diffusion models in vision: A survey,” IEEE Trans. Pattern Anal. Mach. Intell. , vol. 45, no. 9, pp. 10850–10869, Mar. 2023
2023
-
[22]
Gulrajani, F
I. Gulrajani, F. Ahmed, M. Arjovsky, V . Dumoulin, and A. C. Courville, ”Improved training of Wasserstein GANs,” in Advances in Neural Information Processing Systems (NeurIPS) , vol. 30, 2017
2017
-
[23]
Ulyanov, A
D. Ulyanov, A. Vedaldi, and V . Lempitsky, ”Instance normaliza- tion: The missing ingredient for fast stylization,” arXiv preprint , arXiv:1607.08022, Jul. 2016
2016 arXiv
-
[24]
Arjovsky, S
M. Arjovsky, S. Chintala, and L. Bottou, ”Wasserstein generative ad- versarial networks,” in Proc. Int. Conf. Machine Learning (ICML) , Jul. 2017, pp. 214–223
2017
-
[25]
Gemp and B
I. Gemp and B. McWilliams, ”The unreasonable effectiveness of Adam on cycles,” in Proc. NeurIPS Workshop Bridging Game Theory and Deep Learning, 2019
2019
-
[26]
W. Hu, M. Li, and X. Ju, ”Improved CycleGAN for image-to-image translation,” 2021
2021
-
[27]
Parmar, R
G. Parmar, R. Zhang, and J. Y . Zhu, ”On aliased resizing and surprising subtleties in GAN evaluation,” in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition (CVPR) , 2022, pp. 11410–11420
2022
-
[28]
Salimans, I
T. Salimans, I. Goodfellow, W. Zaremba, V . Cheung, A. Radford, and X. Chen, ”Improved techniques for training GANs,” in Advances in Neural Information Processing Systems (NeurIPS) , vol. 29, 2016
2016
-
[29]
Szegedy, V
C. Szegedy, V . Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, ”Rethinking the inception architecture for computer vision,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR) , 2016, pp. 2818– 2826
2016
-
[30]
Borji, ”Pros and cons of GAN evaluation measures: New develop- ments,” Computer Vision and Image Understanding , vol
A. Borji, ”Pros and cons of GAN evaluation measures: New develop- ments,” Computer Vision and Image Understanding , vol. 215, Jan. 2022, Art. no. 103329
2022
-
[31]
Heusel, H
M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter, ”GANs trained by a two-time-scale update rule converge to a local Nash equilibrium,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 30, 2017
2017
-
[32]
K. He, X. Zhang, S. Ren, and J. Sun, ”Deep residual learning for image recognition,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR) , 2016, pp. 770–778
2016
-
[33]
Bi ´nkowski, D
M. Bi ´nkowski, D. J. Sutherland, M. Arbel, and A. Gretton, ”Demysti- fying MMD GANs,” arXiv preprint, arXiv:1801.01401, Jan. 2018
2018 arXiv
-
[34]
Havelka, J
J. Havelka, J. Mayer, and P. Pecina, ”Symbol Generation via Autoen- coders for Handwritten Music Synthesis,” in Proc. 5th Int. Workshop on Reading Music Systems , Nov. 2023, p. 20
2023
-
[35]
Mayer and P
J. Mayer and P. Pecina, ”Synthesizing training data for handwritten music recognition,” in Proc. 16th Int. Conf. Document Analysis and Recognition (ICDAR), Lausanne, Switzerland, Sep. 2021, pp. 626–641, Springer International Publishing
2021
-
[36]
Radford, ”Unsupervised representation learning with deep convolutional generative adversarial networks,” arXiv preprint , arXiv:1511.06434, 2015
A. Radford, ”Unsupervised representation learning with deep convolutional generative adversarial networks,” arXiv preprint , arXiv:1511.06434, 2015
2015 arXiv
-
[37]
Goodfellow, J
I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y . Bengio, ”Generative adversarial nets,” in Advances in Neural Information Processing Systems (NeurIPS) , vol. 27, 2014
2014
-
[38]
R ´ıos-Vila, J
A. R ´ıos-Vila, J. Calvo-Zaragoza, and T. Paquet, ”Sheet Music Trans- former: End-To-End Optical Music Recognition Beyond Monophonic Transcription,” arXiv preprint, arXiv:2402.07596, Feb. 2024
2024 arXiv
-
[39]
van Der Wel and K
E. van Der Wel and K. Ullrich, ”Optical music recognition with convolu- tional sequence-to-sequence models,” arXiv preprint, arXiv:1707.04877, Jul. 2017
2017 arXiv
-
[40]
Forn ´es, A
A. Forn ´es, A. Dutta, A. Gordo, and J. Llad ´os, ”CVC-MUSCIMA: A ground truth of handwritten music score images for writer identification and staff removal,” Int. J. Document Anal. Recognit. (IJDAR) , vol. 15, pp. 243–251, Sep. 2012
2012
-
[41]
Tirupati, E
N. Tirupati, E. Shatri, and G. Fazekas, ”Crafting Handwritten Notations: Towards Sheet Music Generation,” in To appear in Proc. 6th Int. Workshop on Reading Music Systems , 2024
2024
-
[42]
Bar ´o, C
A. Bar ´o, C. Badal, and A. Forn ´es, ”Handwritten historical music recog- nition by sequence-to-sequence with attention mechanism,” in Proc. 17th Int. Conf. Frontiers in Handwriting Recognition (ICFHR), Sep. 2020, pp. 205–210
2020
-
[43]
I. B. Mustapha, S. Hasan, H. Nabus, and S. M. Shamsuddin, ”Condi- tional deep convolutional generative adversarial networks for isolated handwritten Arabic character generation,” Arabian J. Sci. Eng. , vol. 47, no. 2, pp. 1309–1320, Feb. 2022
2022
-
[44]
Calvo-Zaragoza and D
J. Calvo-Zaragoza and D. Rizo, ”End-to-end neural optical music recognition of monophonic scores,” Applied Sciences , vol. 8, no. 4, p. 606, Apr. 2018
2018
-
[45]
Gretton, K
A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Sch ¨olkopf, and A. Smola, ”A kernel two-sample test,” J. Mach. Learn. Res. , vol. 13, no. 1, pp. 723–773, Mar. 2012
2012
-
[46]
Shatri and G
E. Shatri and G. Fazekas, ”Knowledge Discovery in Optical Music Recognition: Enhancing Information Retrieval with Instance Segmenta- tion,” in Proc. Int. Conf. Knowledge Discovery and Information Retrieval (KDIR), 2024
2024
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.