{"id":"64802f6b-5264-4146-bb3d-d04f9e6bedf0","arxiv_id":"2412.09887","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A controllable Transformer generator produces full-song melodies from lyrics, conditioned on syllable tones, part-of-speech tags, semantic sentence embeddings, and user-specified musical attributes.","lead":"This paper presents a neural system that turns Chinese pop lyrics into full-song melodies while letting the user steer musical attributes such as pitch level, note density, and structure. It claims that feeding the generator detailed per-sentence musical statistics and lyric features improves quality and controllability over earlier lyric-to-melody systems.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline comparison is confounded by target-derived statistical controls: Table 1's CSL-L2M is conditioned on sentence-level musical statistics extracted from the same ground-truth test melody used to compute PD/DD/MD, so the reported superiority may largely reflect information leakage rather…","rationale":"I read the paper in good faith: the proposed REMI-Aligned representation, the in-attention conditioning mechanism, and the multi-granularity control design are technically coherent, and the controllability analysis in Figure 5 is a reasonable way to test whether generated melodies track user-specified attribute classes. The central weakness is the evaluation protocol for the objective head-to-head comparison. The model's conditioning inputs include statistical attributes that, if taken from the test ground truth, trivially improve distribution-similarity metrics like PD, DD, and MD. The paper does not specify that test-time controls are chosen independently of the target melody, and the ablation table strongly suggests target-derived controls are the default. This does not invalidate the method, because user-specified controls are a legitimate part of the task, but it does invalidate the specific claim that the reported objective scores reflect higher generation quality rather than the model being handed a summary of the answer. The reader's weakest assumption identifies exactly this issue, and my independent reading agrees. A revised evaluation with fixed or independently sampled controls, plus error bars and a fair baseline protocol, would let the central claim be judged. Since the concern is addressable and does not refute the architecture, the appropriate verdict remains CONDITIONAL, so I recommend no change to the reader's verdict.","tokens_in":12202,"tokens_out":3672,"duration_ms":42527,"concrete_test":"Re-run the objective evaluation of Table 1 and Table 2 with the statistical controls fixed to the training-set median values (or randomly sampled from the training-set distribution) instead of being extracted from the test ground-truth melody, using the released code and keeping lyrics and all other settings identical for CSL-L2M, TeleMelody, and SongComposer. If CSL-L2M's PD/DD/MD advantage over TeleMelody/SongComposer collapses (e.g., MD gap below 0.3 or PD values overlapping within error bars), the headline superiority is an artifact of target-derived conditioning. As a secondary check, compute the same objective metrics on the 'w/o musical Ctls' row from Table 2 and verify whether it still significantly outperforms both baselines; if not, the claimed advantage depends on leaked controls.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing assumption is that Eq. (5)'s sentence-level statistical musical controls (cPM, cPV, cPR, cDMM, cAA, cCM, cDM, cDV, cDR, cMCD, cND, cAlign) are truly user-specified values independent of the evaluation target when Table 1 and Table 2 are computed. The paper never states this; the natural reading of the experimental section is that for each test song the controls are extracted from the ground-truth sentence-level melody, because the objective metrics compare generated output against that same ground truth. Under that protocol, the model receives a 12-dimensional summary of the target at every syllable, including pitch mean/variance/range, duration mean/variance/range, note density, and the syllable-to-note alignment ratio. PD/DD/MD then measure how well the model reproduces statistics it was explicitly given, not how well it generates from lyrics alone. This would explain the large gap in Table 1 (86.35/93.50/1.27 vs. 33-40/45-50/2.93-3.12) and the monotone degradation in Table 2 as controls are removed: removing target-derived controls lowers the metrics almost by construction. The 'w/ learned Ctls' row (MD 0.25) further demonstrates how much target-derived conditioning alone can inflate similarity; although the main model excludes learned features, the statistical controls already leak the same information at coarser granularity. Consequently, the claim that CSL-L2M outperforms state-of-the-art models is not established unless the comparison is repeated with controls that are not derived from the test target, or unless the baselines are given the same target-derived controls.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CSL-L2M, a controllable full-song lyric-to-melody generation system based on a Transformer decoder with in-attention conditioning. The method introduces a REMI-Aligned music representation that encodes syllable- and sentence-level lyric-melody alignments, and it feeds a multi-granularity set of lyric controls (tone, part-of-speech, sentence-level semantic embeddings) and musical controls (human-labeled tags, 12 sentence-level statistical musical attributes, and learned VQ-VAE features) into the decoder. The authors present a new dataset of 10,170 Chinese pop songs with alignment annotations and report objective and subjective evaluations comparing CSL-L2M against TeleMelody and SongComposer, along with ablations and a controllability study.","tokens_in":12564,"tokens_out":5989,"duration_ms":66180,"significance":"If the reported results are credible, the paper makes several worthwhile contributions: a full-song lyric-to-melody formulation rather than sentence-level concatenation, explicit support for one-to-many syllable-note alignment through a new representation, a broad set of fine-grained control mechanisms, and a newly collected aligned dataset. The in-attention conditioning architecture and the use of a pre-trained VQ-VAE for high-fidelity musical features are interesting ideas, and the public availability of code and demos is a strength. However, the central claim of outperforming state-of-the-art systems is not supported by the reported experiments because the evaluation protocol appears to condition the model on statistical summaries of the ground-truth test melody, which can trivially inflate the objective similarity metrics. The controllability study using Spearman correlations is a good idea and partially demonstrates attribute control, but it does not resolve the circularity in the main comparison.","major_comments":[{"comment":"The authors should clarify whether the baseline models were trained or fine-tuned on the same 10,170-song dataset and the same train/validation/test split. The paper states only that 'we first compare our CSL-L2M with two state-of-the-art models, i.e. TeleMelody and SongComposer' without describing whether these models were retrained on the new dataset, whether they use the same full-song generation setting, or whether the same test songs are used. If the baselines were used off-the-shelf without adapting to the new data, differences in training data domain and scale could explain much of the gap in Table 1. Please document the baseline setup in detail or retrain the baselines on the same data and report the results.","section":"§Experimental Settings, Table 1"},{"comment":"The subjective evaluation is not sufficiently controlled or reported. The paper mentions 10 test songs and 70 participants but provides no inter-annotator agreement, no statistical significance tests, and no description of whether the raters were blinded to the system or to the conditioning information. Since the CSL-L2M outputs evaluated in Table 1 may have been conditioned on ground-truth musical attributes, the subjective ratings could reflect similarity to the original song rather than independent musical quality. A controlled listening test with matched generative conditions and appropriate statistical analysis is needed to support the subjective quality claim.","section":"§Subjective Metrics, Table 1"},{"comment":"The controllability study is informative but incomplete. The paper reports Spearman correlations between user-specified attribute classes and attribute values computed from generated melodies, which is a reasonable approach. However, the analysis is only performed for CSL-L2M and does not compare against the controllability of the baseline methods. To support the claim of 'better controllability' over state-of-the-art models, the same Spearman-based controllability evaluation should be applied to the baselines. Additionally, the paper does not specify how many generated samples are used for the correlations or whether the results are averaged over repeated generations.","section":"§Controllability Study, Figure 5"}],"minor_comments":[{"comment":"There are several typographical errors, including 'V AE' instead of 'VAE' and 'vise versa' instead of 'vice versa'.","section":"Throughout"},{"comment":"The definitions of several statistical attributes (DMM, AA, CM) are not given in the paper; the reader is referred to a website. For an archival publication, these definitions should be stated in the text or in an appendix.","section":"§Statistical Musical Attributes"},{"comment":"Figure 4 shows a 'w/o all Ctls' setting, but Table 2 does not include that row. Please add the corresponding results to Table 2 or explain the omission.","section":"§Main Results, Table 2"},{"comment":"The description of the correlation matrix in Figure 5 would benefit from a precise explanation of how 'unrelated attribute' correlations are computed and whether they control for the other specified attributes.","section":"§Controllability Study, Figure 5"},{"comment":"The new dataset is a valuable contribution, but it is not released along with the code. The paper should state whether the dataset will be made publicly available for reproducibility.","section":"§Dataset"}],"recommendation":"major_revision","confidential_remarks":"The core issue is the evaluation protocol: the reported objective improvements are very likely inflated by conditioning on ground-truth-derived statistical controls. This is a load-bearing problem for the 'outperforms state-of-the-art' claim. The paper has promising architectural ideas and a new dataset, but the experiments must be redesigned so that the controls are independent of the test target, and the baselines must be trained/evaluated under comparable conditions. I would not support acceptance until this is addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: the main result as stated isn't supported. The paper compares CSL-L2M against TeleMelody and SongComposer, but CSL-L2M is fed 12 sentence-level statistical attributes extracted from the ground-truth test melody (Eq. 5, Table 2), and then evaluated by how well its output matches that same ground truth. The baselines don't get that information. So the huge gap in PD/DD/MD (86.35/93.50/1.27 vs 33-40/45-50/2.93-3.12) is mostly a leakage artifact. The stress-test note is right about this.\n\nThat said, the paper is not a waste. The new things are real: REMI-Aligned explicitly encodes syllable- and sentence-level alignment, supports one-to-many syllable-note relations, and the representation is a reasonable extension of REMI. The authors also built a 10,170-song Chinese pop dataset with key/emotion/structure tags, which is a genuine resource. The controllability study using Spearman correlations between specified attribute classes and extracted attributes from generated melodies is a good idea and, if the protocol is cleaned up, a solid way to measure control strength.\n\nThe soft spots are all in the evaluation. The main comparison is confounded as described. Even the lyric-only ablation (49.20/59.13/2.26) beats the baselines, so the model may have real advantages, but we can't tell from these numbers because the baselines are under-specified: no info on whether they were retrained on this dataset, what decoding was used, or whether the metrics were computed on full songs for all models. Subjective eval: 10 songs, 70 participants, no agreement stats or significance tests. The 'learned controls' row (MD 0.25) shows the control pathway can essentially copy the target when given learned features from it, which is fine for style transfer but doesn't belong in a 'lyric-to-melody' quality comparison.\n\nOne more conceptual point: calling the VQ-VAE features 'musical controls' is generous. They're extracted from an existing melody, so they're a style-transfer reference, not a user-specified attribute. That's acknowledged in the case study, but the framing in the abstract overstates it.\n\nWho's this for? People working on lyric-to-melody and controllable music generation, especially for tonal languages. The representation and dataset are worth engaging with. The generation results need to be re-evaluated with a fair protocol: either use fixed user-specified controls that don't come from the test target, or give identical controls to baselines, and report error bars.\n\nRecommendation: send to peer review. It deserves a serious referee, but make clear that a revision needs to fix the evaluation before publication. This is fixable.","headline":"REMI-Aligned and the new dataset are solid contributions, but the headline comparison is confounded by target-derived controls and the paper needs a major evaluation overhaul before its claims can be trusted.","tokens_in":13097,"tokens_out":3707,"would_cite":true,"duration_ms":40058,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A conditional Transformer generates full-song melodies that align each lyric syllable to one or more notes and follow user-specified musical attributes.","keywords":["lyric-to-melody generation","controllable music generation","in-attention conditioning","REMI-Aligned","Transformer decoder","syllable-note alignment","Chinese pop songs","VQ-VAE musical features"],"falsifier":"Run the ablation with the sentence-level statistical controls computed from held-out reference songs or from user-specified values instead of from the ground-truth test melody; if pitch and duration similarity and melody distance collapse toward the no-musical-control baseline, the reported advantage depends on target leakage rather than on learning the lyric-melody mapping.","tokens_in":11985,"feed_emoji":"🎵","tokens_out":8040,"duration_ms":74023,"temperature":0.7,"pith_summary":"This paper proposes CSL-L2M, a lyric-to-melody system that aims to generate complete songs rather than short fragments. The central claim is that a Transformer decoder conditioned through in-attention on multi-granularity lyric and musical controls can produce full-song melodies in which every syllable is aligned to one or more notes and user-chosen musical attributes are respected. The paper introduces a new token representation, REMI-Aligned, that makes syllable- and sentence-level lyric-melody alignment explicit, and a dataset of over ten thousand annotated Chinese pop songs. If the claim is right, lyric-based songwriting tools could move from stitching sentence-level melodies into coherent whole songs with controllable key, emotion, structure, and rhythm.","feed_headline":"Lyrics and user tags now steer full-song melody generation","feed_subtitle":"A conditional Transformer aligns every syllable to one or more notes and obeys key, emotion, and rhythm controls.","key_machinery":"Two components carry the argument. REMI-Aligned extends the REMI music representation by adding ALIGN tokens at syllable and sentence boundaries, converting the strict but weak lyric-melody correspondence into discrete, learnable tokens that support both one-to-one and one-to-many alignments. In-attention conditioning is the mechanism that makes the controls effective over long sequences: each segment-level control vector is projected into the decoder's hidden-state space and added to the input of every self-attention layer, so the model is continually reminded of the current sentence's lyric and musical conditions. The per-syllable control vector concatenates all lyric and musical embeddings (Eq. 5), which is what lets the same architecture handle both tight alignment and user-directed attribute control.","core_discovery":"The discovery the paper is trying to establish is that the long-standing weaknesses of lyric-to-melody generation - weak controllability, low quality, and poor structure - can be addressed simultaneously at the full-song level by a single conditional autoregressive Transformer. On the lyric side, the model consumes syllable-level tone embeddings, word-level part-of-speech embeddings, and sentence-level semantic embeddings; on the musical side, it consumes human-labeled tags (key, emotion, structure), sentence-level statistical attributes (pitch, duration, note density, alignment ratio), and learned VQ-VAE features. These are concatenated per syllable and injected into the decoder through in-attention, so the conditions are present at every layer and every time step. The paper reports that on its collected dataset of 10,170 Chinese pop songs, CSL-L2M outperforms prior state-of-the-art models on objective similarity and subjective listening scores, and that ablating control groups progressively degrades performance.","pith_inferences":["One implication the paper leaves implicit is that a user-facing deployment must source the sentence-level statistical controls without access to the target melody; if users cannot supply them, the controls would need to be predicted or estimated.","Another open question is whether the alignment-token representation transfers to other tonal languages, where tone embeddings might similarly guide pitch, or to non-tonal languages, where the tone channel would be dropped.","Because the dataset is limited to Chinese pop songs in 4/4 time, the method's generality to other genres, meters, and languages is untested; adapting REMI-Aligned and the control stack to those settings would be a natural next experiment.","Running the same Spearman rank-correlation controllability analysis on prior baselines would show whether the fine-grained control this paper claims is a new capability or simply a more favorable evaluation protocol."],"forward_implications":["Users could specify key, emotion, and song structure at the whole-song level and fine-grained sentence-level statistics such as pitch mean or note density, and receive melodies that track those specifications.","Syllables that naturally span multiple notes are no longer forced into a one-to-one mapping, so melismatic vocal writing can be generated directly rather than avoided.","Full-song structure, including repetition patterns and verse-chorus contrast, can emerge from a single decoding pass instead of concatenated sentence-level fragments.","The progressive ablation results indicate that each control group - duration, pitch, rhythm, alignment, and global tags - contributes independent information to melody quality, so controllability and quality are not in tension."],"supporting_citations":[{"why":"Supplies the in-attention conditioning technique that lets the decoder receive time-varying controls at every layer.","marker":"Wu and Yang 2023"},{"why":"Supplies the REMI token representation that REMI-Aligned extends with explicit alignment tokens.","marker":"Huang and Yang 2020"},{"why":"Supplies the idea of learned high-fidelity musical features extracted from a pre-trained VQ-VAE for controllable generation.","marker":"von Rütte et al. 2023"},{"why":"TeleMelody is a state-of-the-art baseline whose template-based two-stage approach CSL-L2M compares against.","marker":"Ju et al. 2021"},{"why":"SongComposer is a large-language-model baseline that CSL-L2M compares against on full-song generation.","marker":"Ding et al. 2024"},{"why":"SongMASS provides the objective similarity metrics (pitch and duration distribution similarity, melody distance) and the alignment-constraint pre-training idea.","marker":"Sheng et al. 2021"}],"fun_headline_variants":["Lyrics and user tags now shape full-song melodies","Syllable-level control yields better song melodies","Full-song melodies from lyrics with fine-grained controls","CSL-L2M: precise lyric-to-melody with musical tags","Conditional Transformer aligns lyrics and tags for songs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that conditioning the decoder on sentence-level statistical attributes extracted from the ground-truth test melody is a fair measure of generation quality, rather than a shortcut that lets the model reproduce the target's summary statistics and inflate similarity scores.","fun_headline_variants_meta":{"raw":{"variants":["Lyrics and user tags now shape full-song melodies","Syllable-level control yields better song melodies","Full-song melodies from lyrics with fine-grained controls","CSL-L2M: precise lyric-to-melody with musical tags","Conditional Transformer aligns lyrics and tags for songs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000514,"raw_usage":{"total_tokens":2551,"prompt_tokens":1054,"completion_tokens":1497,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":670,"completion_tokens_details":{"reasoning_tokens":1419}},"tokens_in":670,"tokens_out":1497,"duration_ms":13447,"temperature":1.0,"reasoning_tokens":1419,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T16:36:04.576384+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the ablation with the sentence-level statistical controls computed from held-out reference songs or from user-specified values instead of from the ground-truth test melody; if pitch and duration similarity and melody distance collapse toward the no-musical-control baseline, the reported advantage depends on target leakage rather than on learning the lyric-melody mapping.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"SongMASS provides the objective similarity metrics (pitch and duration distribution similarity, melody distance) and the alignment-constraint pre-training idea."}],"review_version":1}