{"id":"f7bd92b5-1f3a-4656-ad7e-689938e1b8bd","arxiv_id":"2411.16405","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"CycleWGAN, a CycleGAN variant with Wasserstein loss, beats DCGAN and ProGAN at generating handwritten music images, with FID 41.87, IS 2.29, and KID 0.05.","lead":"The authors tested three GAN models, including a modified CycleGAN called CycleWGAN, for generating fake handwritten music sheets and found CycleWGAN scored best on standard image-quality metrics. The work aims to ease the data shortage in optical music recognition by synthesizing training images, though the generated sheets still have visible errors.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Quantitative ranking is confounded: CycleWGAN is a conditional image-to-image translator (printed-to-handwritten), while DCGAN/ProGAN are unconditional generators; lower FID may reflect easier conditioning, not better generation.","rationale":"I read the paper as claiming that CycleWGAN is the best of the three architectures for synthesizing handwritten music crops. For that claim to be supported, the quantitative evaluation must compare equivalent generation tasks. It does not. CycleWGAN is an image-to-image translator conditioned on printed scores; DCGAN and ProGAN are unconditional samplers from noise. The FID/IS/KID scores in Table I compare output distributions, but CycleWGAN can use the printed input to place staff lines and noteheads, while the baselines must invent structure from scratch. This is not a subtle statistical issue; it is a confound in experimental design. The reader's concern about evaluating on only 200 images is valid and compounds the problem, but even an ideal FID estimator would not fix the task mismatch. I agree with the reader that the paper has addressable weaknesses, but I would add a new condition: the ranking must be re-established under matched conditioning. The qualitative object-detection results and PCA give some independent support for CycleWGAN's style-transfer ability, and the authors are candid about limitations, so I would not reject outright; however, the central claim of 'significantly outperforms' is not currently supported by the reported numbers. A scoped conclusion ('best for style transfer, not unconditional synthesis') or a matched-conditioning experiment would resolve the issue. Therefore I keep the verdict CONDITIONAL, with the condition now explicitly including a fair comparison protocol.","tokens_in":12542,"tokens_out":7451,"duration_ms":77152,"concrete_test":"Train conditional variants of DCGAN and ProGAN that take the same printed-score crop as input (e.g., U-Net generator with the printed image concatenated to the noise/latent input), following the same training schedule as the originals. Generate outputs from all three models on identical held-out printed crops and recompute FID, IS, and KID against the same real handwritten reference set used in Table I, with bootstrap confidence intervals. If either baseline matches or beats CycleWGAN's FID of 41.87, or the gap collapses within noise, the headline ranking is an artifact of conditioning rather than model quality; if CycleWGAN still leads, the claim is supported.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The comparison supporting the central claim is not task-equivalent. In Sec. III-B3, CycleWGAN is a CycleGAN variant with two generators (Gp, Gh) translating between printed and handwritten domains using Wasserstein and cycle-consistency losses; its outputs are functions of printed-score crops. In contrast, DCGAN (Sec. III-B1) and ProGAN (Sec. III-B2) are unconditional generators that map latent noise z to images. Sec. V-B then compares FID/IS/KID as if the three were competing generative models. This comparison is not task-equivalent: CycleWGAN receives structured printed music as input, so its output distribution is constrained to resemble notation, making lower FID (41.87 vs 49.54 vs 210.18) largely expected. The margin is therefore evidence of a conditioning advantage, not of superior generative ability as claimed in the abstract. The paper's own conclusion reinforces this: CycleWGAN outputs contain gaps and pitch shifts and are 'unsuitable for direct use in training OMR systems,' while ProGAN produces 'more complete and balanced images.' A fair test must either make all models conditional on the same printed input or evaluate CycleWGAN unconditionally.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper applies three GAN architectures—DCGAN, ProGAN, and CycleWGAN (a CycleGAN variant using Wasserstein losses and instance normalization)—to synthesize handwritten music crops from the CVC-MUSCIMA and DoReMi datasets. It reports quantitative metrics (FID, IS, KID) and qualitative analyses including object detection and PCA, and claims that CycleWGAN significantly outperforms the other two models and offers a promising route to improving OMR training data. The paper also provides hyperparameter details and a discussion of limitations.","tokens_in":12787,"tokens_out":4140,"duration_ms":37539,"significance":"If the comparison were task-equivalent and statistically grounded, this would be a useful empirical baseline for GAN-based handwritten music synthesis. The CycleWGAN adaptation is a reasonable engineering contribution, and the qualitative evaluation with a pre-trained object detector is a valuable addition. However, the central claim is not supported as stated: the comparison confounds conditioning with generation ability, the quantitative metrics lack uncertainty quantification, and the paper's own qualitative results contradict the headline conclusion. The work is nevertheless a reasonable starting point for a more careful evaluation.","major_comments":[{"comment":"The comparison is not task-equivalent. CycleWGAN is a conditional image-to-image translator that takes printed music crops as input, while DCGAN and ProGAN are unconditional generators that map latent noise z to images. The lower FID of CycleWGAN (41.87 vs 49.54 vs 210.18) may therefore reflect the strong conditioning signal rather than superior generative ability. To support the abstract's claim of 'significantly outperforms', the evaluation must either condition all models on the same printed input or evaluate CycleWGAN in an unconditional setting; at a minimum, the claims should be re-framed as 'best on the conditional translation task'.","section":"Sec. III-B1/2/3; Sec. V-B, Table I"},{"comment":"The quantitative ranking relies on point estimates from a 200-image evaluation set with no confidence intervals or significance tests. The Inception Scores overlap within one standard deviation (CycleWGAN 2.29 ± 0.19, DCGAN 2.17 ± 0.14, ProGAN 2.01 ± 0.20), so the word 'significantly' is not supported. The paper also does not specify how the 200 evaluation images were chosen, whether the same real-image reference set was used for all models, or how many generated images were evaluated. Please provide confidence intervals (e.g., bootstrap) or a significance test, and describe the evaluation set construction; otherwise the reported margins may be evaluation noise.","section":"Sec. V-B, Table I; Sec. V-D"},{"comment":"The qualitative results contradict the central claim. The paper states that CycleWGAN outputs contain gaps, pitch shifts, and missing noteheads and are 'unsuitable for direct use in training OMR systems', while ProGAN produces 'more complete and balanced images'. This internal contradiction needs to be resolved: either the quantitative advantage of CycleWGAN is not diagnostically meaningful for the stated application, or the conclusion that CycleWGAN is 'a promising solution for improving OMR systems' should be substantially weakened.","section":"Sec. V-A; Sec. VI"},{"comment":"The Inception Score is computed with an ImageNet-pretrained Inception network, but handwritten music crops are far outside that domain, making the conditional label distribution p(y|x) largely arbitrary and the IS values difficult to interpret. The paper itself concedes that the direct applicability of these metrics 'is less established', yet IS is used as headline evidence. Either justify the use of IS for this domain (e.g., with a domain-adapted classifier) or remove it from the main claims and rely on FID/KID plus the qualitative evaluation.","section":"Sec. V-B, Eq. (5)"}],"minor_comments":[{"comment":"The text contains the literal placeholder '[cite key papers]' in the paragraph motivating metric selection; this must be completed before submission.","section":"Sec. V-B"},{"comment":"The statement that DCGAN suffers from mode collapse '(see Figure 6(c))' is incorrect because Figure 6(c) is the printed input image, not a DCGAN output; please fix the cross-reference and check all figure references.","section":"Sec. V-A"},{"comment":"The text describes 'Figure 9(b)' as comparing real handwritten, real printed, and CycleWGAN-generated images, but in the figure caption panel (b) is the ProGAN panel; correct either the text or the caption.","section":"Sec. V-C, Fig. 9"},{"comment":"Capitalization of model names is inconsistent in the text, Table I, and figures (DCGAN, PROGAN, ProGAN, CycleWGAN); please standardize.","section":"Throughout"},{"comment":"Reference [26], cited as 'Improved CycleGAN for image-to-image translation, 2021', lacks author names and venue; provide full bibliographic details.","section":"References"},{"comment":"The phrase 'Experimented with custom beta values' should read 'We experimented with custom beta values'.","section":"Sec. IV-C"},{"comment":"The text refers to 'Figures 1a and 1b' but the sub-caption labels in Figure 1 are not defined in the caption; adding explicit '(a)' and '(b)' labels would improve clarity.","section":"Sec. III-A"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: this is a usable empirical comparison of three GAN approaches for generating handwritten music crops, and CycleWGAN (CycleGAN with Wasserstein loss) is a reasonable thing to try. But the central claim that CycleWGAN 'significantly outperforms' the others does not hold up, because the comparison is not apples-to-apples. CycleWGAN is an image-to-image translator conditioned on a printed score; DCGAN and ProGAN are unconditional generators that start from noise. A model that is told what to draw is naturally going to get a lower FID than models that must invent notation. The paper's own conclusion concedes CycleWGAN outputs contain gaps, pitch shifts, and are unsuitable for direct OMR training, while ProGAN produces more complete and balanced images. So the quantitative ranking is real in the table but much weaker as evidence of superior generative quality.\n\nWhat is genuinely useful: the paper documents a concrete CycleWGAN implementation for handwritten music, reports FID/IS/KID numbers on that task, and supplements the metrics with object-detection checks and PCA plots. That is more than many GAN papers do. The limitations section is honest about the small evaluation set and about the remaining artifacts in CycleWGAN outputs. The method section is clear enough to reproduce.\n\nSoft spots, in order of severity. First, the conditional-vs-unconditional confound above; the headline result is not task-equivalent. Second, no confidence intervals or significance tests on FID/IS/KID; the margin between ProGAN (49.54) and CycleWGAN (41.87) could easily be noise on 200 images. Third, no code or generated-sample release, which makes verification harder. Fourth, the text contains an unresolved '[cite key papers]' placeholder in Section V-B, and a few typos like missing parenthesis around 'IS of 2.29'. None of these kill the paper as a baseline reference, but together they kill the abstract's strong claim as written.\n\nThis paper is for researchers working on OMR data augmentation and GAN evaluation for structured documents. I would send it to peer review with a request for major revision: either make the comparison task-equivalent (e.g., conditional DCGAN/ProGAN on the same printed input) or reframe the claim to 'CycleWGAN, a conditional translator, achieves better perceptual metrics than unconditional generators on this dataset.' I would cite it cautiously as an architecture and hyperparameter reference, not as evidence that CycleWGAN beats ProGAN at unconditional generation.","headline":"Useful benchmark of GAN variants for handwritten music, but the headline 'CycleWGAN significantly outperforms' is confounded by a conditional-vs-unconditional task mismatch and a 200-image evaluation set.","tokens_in":13331,"tokens_out":2852,"would_cite":true,"duration_ms":24551,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that CycleWGAN, a CycleGAN variant with Wasserstein loss, outperforms DCGAN and ProGAN at generating handwritten music crops, scoring FID 41.87, IS 2.29, and KID 0.05.","keywords":["handwritten music generation","optical music recognition","CycleWGAN","generative adversarial networks","image-to-image translation","Wasserstein GAN","FID and KID evaluation","data augmentation"],"falsifier":"Compute FID, IS, and KID on at least a few thousand generated images with bootstrap confidence intervals; the CycleWGAN claim collapses if its FID advantage over ProGAN falls inside the noise band. A second falsifier is a downstream OMR test: if a recogniser trained with CycleWGAN crops does not match or beat training on ProGAN crops or real data, the practical claim for improving OMR is unsupported.","tokens_in":49,"feed_emoji":"🎼","tokens_out":4995,"duration_ms":104231,"temperature":0.7,"pith_summary":"The paper addresses the scarcity of annotated handwritten music for Optical Music Recognition by testing whether GANs can generate convincing handwritten score crops. It compares three architectures and argues that CycleWGAN, a CycleGAN modified with Wasserstein loss, instance normalisation, and residual blocks, produces the most realistic and diverse images of the three. On 200 evaluation images, CycleWGAN reaches FID 41.87, IS 2.29, and KID 0.05, beating ProGAN (FID 49.54) and DCGAN (FID 210.18). A sympathetic reader would take the central claim as: for crop-level handwritten music synthesis, the Wasserstein-stabilised cycle-consistency architecture is the strongest of the tested options and a better baseline than the earlier CycleGAN. The paper also concedes remaining flaws, such as gaps in note-head transfer, that keep the generated images from direct OMR training.","feed_headline":"CycleWGAN tops DCGAN and ProGAN on handwritten music","feed_subtitle":"It reaches FID 41.87 versus 49.54 and 210.18, a step toward synthetic training data for music recognition.","key_machinery":"The load-bearing object is the CycleWGAN architecture: two generators and two discriminators translating between printed and handwritten music domains, trained with Wasserstein adversarial losses plus a cycle-consistency loss that forces round trips to reconstruct the input. Supporting machinery includes instance normalisation, chosen because batch normalisation is incompatible with Wasserstein training, nine ResNet blocks in the generator, DCGAN as a 64x64 baseline, and ProGAN with progressive growth from 4x4 to 128x128 and WGAN-GP. The comparison is carried by standard generative metrics, FID, IS, and KID, supplemented by qualitative object detection and PCA.","core_discovery":"CycleWGAN replaces the standard adversarial losses in CycleGAN with the Wasserstein distance, uses instance normalisation in both networks, and employs a ResNet-based generator with nine residual blocks while keeping an L2 cycle-consistency loss with lambda equal to 10. The paper's finding is that these changes improve training stability and style transfer enough that CycleWGAN substantially outperforms DCGAN and ProGAN on FID, IS, and KID, and that PCA places its outputs closest to real handwritten crops. The paper presents this as evidence that CycleWGAN is a promising data-augmentation tool for OMR, while noting that symbol-level errors, such as imperfect note heads and accidentals, mean the images are not yet ready for direct use in training OMR systems.","pith_inferences":["Editorial inference: the 200-image evaluation set makes FID the least reliable of the three metrics; KID's unbiased estimator is more believable at this sample size, and its large gap (0.05 vs 0.17) is the strongest quantitative evidence for CycleWGAN.","Editorial inference: because the models are trained and evaluated on square crops, the ranking may not survive the harder task of generating full pages; a full-page or stream-based evaluation could invert the ordering.","Editorial inference: a direct downstream test, training an OMR recogniser on real plus synthetic crops and measuring symbol error rate on held-out manuscripts, would settle practical value faster than FID, IS, or KID.","Editorial inference: incorporating WGAN-GP into CycleWGAN, as the paper suggests, is a concrete next experiment; the ProGAN results show gradient penalty helps stability, but it may change CycleWGAN's style-transfer behaviour."],"forward_implications":["CycleWGAN provides a stronger baseline than the earlier CycleGAN for handwriting style transfer in music, so future image-to-image work can start from the Wasserstein variant.","The generated crops, after correcting note-head and accidental errors, could augment OMR training sets and reduce the annotation bottleneck.","ProGAN's greater symbol completeness and CycleWGAN's superior style closeness are complementary, suggesting that combining or conditioning these approaches is a plausible next step.","The reported metric margins need validation on larger evaluation sets before being used as a design decision for OMR pipelines, as the paper itself notes FID typically requires larger samples.","If CycleWGAN is right, data-hungry OMR systems can benefit from synthetic handwritten crops rather than relying only on scarce archives."],"supporting_citations":[{"why":"Provides the improved CycleGAN design that CycleWGAN modifies.","marker":"[26]"},{"why":"Supplies the Wasserstein distance that replaces standard adversarial losses.","marker":"[24]"},{"why":"Defines the original CycleGAN and the cycle-consistency objective used here.","marker":"[3]"},{"why":"Earlier CycleGAN applied to handwritten music; the paper's baseline to beat.","marker":"[7]"},{"why":"Defines the DCGAN architecture and hyperparameter recommendations used as the baseline.","marker":"[36]"},{"why":"Defines ProGAN's progressive training used as the comparison baseline.","marker":"[4]"},{"why":"WGAN-GP stabilisation used in ProGAN and proposed for future CycleWGAN work.","marker":"[22]"},{"why":"Analysis of GAN evaluation measures that grounds the paper's warning about FID sample size.","marker":"[30]"},{"why":"Instance normalisation used in CycleWGAN generators and discriminators for style transfer.","marker":"[23]"},{"why":"ResNet architecture providing the residual blocks in CycleWGAN's generators.","marker":"[32]"}],"fun_headline_variants":["CycleWGAN beats DCGAN and ProGAN on handwritten music","Wasserstein GAN improves synthetic handwritten music realism","CycleWGAN wins for synthetic handwritten music generation","Best GAN for fake handwritten music: CycleWGAN","CycleWGAN yields realistic synthetic music sheets for OMR"],"cache_read_input_tokens":15488,"weakest_assumption_plain":"The ranking rests on the assumption that FID, IS, and KID computed on 200 generated crops are stable enough to separate the models, despite the paper's own note that FID typically needs larger samples and the absence of confidence intervals or significance tests.","fun_headline_variants_meta":{"raw":{"variants":["CycleWGAN beats DCGAN and ProGAN on handwritten music","Wasserstein GAN improves synthetic handwritten music realism","CycleWGAN wins for synthetic handwritten music generation","Best GAN for fake handwritten music: CycleWGAN","CycleWGAN yields realistic synthetic music sheets for OMR"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000646,"raw_usage":{"total_tokens":2939,"prompt_tokens":885,"completion_tokens":2054,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":501,"completion_tokens_details":{"reasoning_tokens":1989}},"tokens_in":501,"tokens_out":2054,"duration_ms":14784,"temperature":1.0,"reasoning_tokens":1989,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T13:08:24.403567+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute FID, IS, and KID on at least a few thousand generated images with bootstrap confidence intervals; the CycleWGAN claim collapses if its FID advantage over ProGAN falls inside the noise band. A second falsifier is a downstream OMR test: if a recogniser trained with CycleWGAN crops does not match or beat training on ProGAN crops or real data, the practical claim for improving OMR is unsupported.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the improved CycleGAN design that CycleWGAN modifies."},{"cited_title":"Arjovsky, S","cited_arxiv_id":null,"evidence_quote":"Supplies the Wasserstein distance that replaces standard adversarial losses."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the original CycleGAN and the cycle-consistency objective used here."},{"cited_title":"Hande, E","cited_arxiv_id":null,"evidence_quote":"Earlier CycleGAN applied to handwritten music; the paper's baseline to beat."},{"cited_title":"Karras, S","cited_arxiv_id":null,"evidence_quote":"Defines ProGAN's progressive training used as the comparison baseline."},{"cited_title":"Gulrajani, F","cited_arxiv_id":null,"evidence_quote":"WGAN-GP stabilisation used in ProGAN and proposed for future CycleWGAN work."},{"cited_title":"Borji, ”Pros and cons of GAN evaluation measures: New develop- ments,” Computer Vision and Image Understanding , vol","cited_arxiv_id":null,"evidence_quote":"Analysis of GAN evaluation measures that grounds the paper's warning about FID sample size."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"ResNet architecture providing the residual blocks in CycleWGAN's generators."}],"review_version":1}