{"id":"128bfff0-19db-4668-8010-1e3f529e4bf5","arxiv_id":"2411.13100","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A GPT-2 fine-tuned on Genius lyrics with structured song-form and syllable tokens generates or infills full-song lyrics that match requested syllable counts at four levels of granularity.","lead":"The paper trains GPT-2 on Genius lyrics with special song-form and syllable-count tokens, enabling generation or infilling of full songs that match requested syllable counts at word, phrase, line, and paragraph levels. The best configuration reports very low syllable-count error, far below OpenAI's ChatGPT baselines.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 2a shows paragraph-level SCErr of 68.8% for the best model, undermining the abstract's claim of multi-level syllable control at the paragraph level.","rationale":"I identify a different load-bearing concern from the reader's weakest assumption. The reader focused on the reliability of the Syllables library [27], which is a legitimate external-validity issue but secondary: if the library is biased, the model could still be seen as controlling the library's counts. The more direct problem is internal to the paper's own results. Table 2a shows that paragraph-level SCErr is 68.8% for the best model, while the paper highlights the 'Full' SCErr of 5.025%. Given the definition of SCErr as the frequency of incorrect syllable counts, the paragraph-level figure is hard to reconcile with the abstract's claim that the framework 'enables multi-level syllable control at the word, phrase, line, and paragraph levels.' The authors never discuss this discrepancy, and it is the single most load-bearing weakness because it concerns the central claim of the paper, not just the evaluation toolchain. The verdict remains CONDITIONAL (hence UNCHANGED): the framework does demonstrate strong control at the line, phrase, and word levels, and the paragraph-level SCD (0.027) indicates a small average deviation, so the work may be salvageable if the authors either improve paragraph-level control or temper the claim to 'approximate control at the paragraph level' and explain the metric gap. A REJECT would be too harsh given the strong fine-grained results, but the paper should not be accepted without addressing this inconsistency.","tokens_in":9527,"tokens_out":14306,"duration_ms":126491,"concrete_test":"Recompute SCD and SCErr separately for each structural level on a random subset of 200 generated songs from the 10K evaluation set, using the definitions in Section 4.4. Specifically, count the fraction of generated paragraphs whose total syllable count exactly equals the paragraph-level condition. If this fraction is approximately 31% (i.e., an error rate near 68.8%) while the full-song total condition is satisfied about 95% of the time, then the headline 'Full SCErr' is masking a paragraph-level failure. Also examine the distribution of per-paragraph syllable-count errors: if they are not centered at zero (e.g., consistently over- or under-counting), this would indicate a systematic bias in how paragraph-level conditions are encoded or in how the model sums line-level outputs. The linked generation samples make this check feasible without model retraining.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's own Table 2a reports that the best generation model (Back-S) has a paragraph-level SCErr of 68.777%, meaning that in more than two-thirds of generated paragraphs the total syllable count differs from the specified condition, while the headline 'Full' SCErr is only 5.025%. The 'Full' metric appears to aggregate over the entire song, allowing positive and negative per-paragraph errors to cancel, so it does not demonstrate paragraph-level control. The abstract claims the framework 'enables multi-level syllable control at the word, phrase, line, and paragraph levels,' but the paragraph-level exact-match rate is only about 31%, which is close to ChatGPT's overall error rate (79.8%) reported in Table 1. The paper never acknowledges or explains this large discrepancy; it highlights only the Full numbers. Since SCErr is defined as how frequently the model generates text with incorrect syllable counts, a 68.8% paragraph-level error directly contradicts the central claim of precise, multi-level syllable control.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a GPT-2-based framework for full-song lyrics generation and infilling, conditioned on semantic embeddings, song-form tokens, and multi-granularity syllable count tokens at the word, phrase, line, and paragraph levels. The authors train on a processed Genius Song Lyrics dataset, synthesize hierarchical generation plans via subtree masking, and evaluate with SCD, SCErr, PPL, BERT-S, and NLD. The best generation model, Back-S, is reported to achieve Full SCD 0.003 and SCErr 5.025%, well below ChatGPT-3.5/4 baselines, and the best infilling model, ILM-S, reports Full SCD 0.006 and SCErr 5.479% with BERT-S 0.856. Ablation studies show that specialized masking tokens and song-form tokens improve performance.","tokens_in":9679,"tokens_out":7360,"duration_ms":71205,"significance":"If the reported numbers are taken at face value, the paper makes a useful contribution: it is one of the first attempts to demonstrate syllable-count control at word, phrase, and line granularity in full-song generation, and the infilling formulation and song-form-consistency evaluation are novel. The controlled comparisons against ChatGPT-3.5/4, the use of Wilcoxon signed-rank tests across multiple model variants, and the ablation study are strengths. However, the significance is conditional on unresolved issues: SCErr is never formally defined, the abstract's paragraph-level control claim is not supported by the paragraph-level numbers in Table 2a, and the entire evaluation relies on a single syllable-counting library without independent validation. These issues must be addressed before the claimed multi-level syllable control can be accepted.","major_comments":[{"comment":"SCErr is described only verbally as 'measures how frequently the model generates text with incorrect syllable counts', but no formula is given. The unit of analysis is ambiguous: is it the fraction of evaluated segments (word, phrase, line, paragraph, or full song) with any mismatch, or a pooled count of mismatched syllables? Because of this ambiguity, the relationship between Full SCErr 5.025% and Paragraph SCErr 68.777% for Back-S in Table 2a is uninterpretable. This is load-bearing because SCErr is the headline metric in Table 1 and the basis for the multi-level control claims. Please provide the exact definition, the aggregation rule across levels, and error distributions (e.g., percentages of segments off by 1, 2, or more syllables).","section":"Section 4.4, Eq. (1), Tables 1-3"},{"comment":"Back-S, presented as the best generation model, has paragraph-level SCErr 68.777%, meaning that under the natural reading of the verbal definition, roughly two-thirds of generated paragraphs do not exactly match the specified syllable counts, while the Full SCErr is only 5.025%. The Full aggregate can hide consistent per-paragraph errors through cancellation of positive and negative differences. The paper does not acknowledge or discuss this discrepancy; Section 5.2 reports overall syllable accuracy without mentioning the paragraph-level failure rate. Since the Abstract claims the framework 'enables multi-level syllable control at the word, phrase, line, and paragraph levels', this claim is not supported as stated. Please report exact-match rates and error statistics at each level, discuss the limitation, or revise the claim.","section":"Table 2a, Abstract, Section 5.2"},{"comment":"All syllable counts in both the training conditions and the evaluation metrics are computed with the same Syllables library [27], with no validation against a reference dictionary or human annotation. If this estimator is systematically wrong for the vocabulary of song lyrics (e.g., contractions, informal spellings, or word-stress ambiguity), then both the conditioning plans and the reported SCD/SCErr values are biased in the same direction, making the absolute accuracy numbers unreliable as evidence of true rhythmic fit. Please validate the estimator against an independent standard such as CMU Pronouncing Dictionary or manually annotated samples, report its per-word accuracy, and discuss how residual errors would affect the reported SCD/SCErr values.","section":"Sections 4.1 and 4.4"},{"comment":"The SCD formula in Eq. (1) divides by \\hat{s}_i, which is undefined if the generated unit has zero syllables; such cases may be rare but are not impossible in lyrics with interjections or incomplete tokens. Please either clip syllable counts to a minimum of 1 or specify how zero-count generated segments are handled in the metric.","section":"Section 4.2, Eq. (1)"}],"minor_comments":[{"comment":"The pre-order traversal selection probability p is introduced in Section 4.2 without a value and is set only later in Section 4.3; please state the value where the preprocessing is defined and report sensitivity to p.","section":"Section 4.2, 4.3"},{"comment":"The table headers mix metric abbreviations and granularity levels (Full, Para., Line, Phrase, Word) in a way that is difficult to parse; please use clearly labeled sub-tables or separate column groups for SCD, SCErr, PPL, and BERT-S.","section":"Tables 2a and 2b"},{"comment":"The caption says 'Each metric is calculated between two distinct paragraphs' but does not explain what the rows and columns represent or whether diagonal values are within-form comparisons; please label the axes and clarify the construction of the matrices.","section":"Figure 3"},{"comment":"The phrase 'using both scratch and pre-trained model' should be 'trained both from scratch and from pre-trained weights', and the initialization of newly introduced special-token embeddings in the pre-trained case should be described.","section":"Section 4.3"},{"comment":"The preliminary ChatGPT evaluation reports success rates for producing the correct number of lines, but the interaction between line-count success and syllable-count accuracy is not quantified; please clarify whether the 688 successful samples are filtered before computing SCD/SCErr.","section":"Section 5.1"}],"recommendation":"major_revision","confidential_remarks":"The paper's central methodology is plausible and the controlled experiments are a strength, but the unresolved definition of SCErr and the stark gap between Full and Paragraph-level SCErr currently support an overclaim in the abstract. The authors should be asked to provide the exact metric formula, report per-level error distributions, validate the syllable estimator, and either temper or substantiate the paragraph-level claim. This is fixable within the scope of a major revision, so I do not recommend rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read the paper. The headline claim of multi-level syllable control at the paragraph level is not supported by the paper's own Table 2a: the best generation model Back-S has a paragraph-level SCErr of 68.8%, meaning only about 31% of paragraphs hit their specified total syllable count. The paper reports Full SCErr of 5.0% and mostly ignores the paragraph-level number, so the abstract overstates the result. That said, the line, phrase, and word-level control is genuinely strong (SCErr 2.9%, 0.8%, 0.3%), and the integrated framework—full-song generation with multi-granularity conditioning, song-form tokens, and an infilling variant—is a real contribution beyond the cited prior work.\n\nWhat it does well: the system is clearly described, the experiments are controlled with internal ablations (Front/Back/Both, scratch vs pre-trained, infilling variants), and the comparison to ChatGPT is revealing: LLMs are terrible at respecting syllable constraints. The BERT-S upper bound (0.799) shows they are pushing against a plausible ceiling. The infilling model (ILM-S) also improves semantic coherence with full context.\n\nSoft spots:\n- The paragraph-level SCErr is never discussed. The paper must either fix the paragraph control or stop claiming it. This is the main issue.\n- SCErr is never formally defined; the reader has to infer from the vague description.\n- All syllable counts come from the same Syllables library used to build training conditions and evaluation metrics. A systematic bias would inflate all numbers, and there is no human validation.\n- No comparison with prior specialized syllable-controlled lyric generators (e.g., SongMASS or Chinese poetry/lyrics systems). Only ChatGPT and internal ablations.\n- No code or data release; the sample URL is a tinyurl that may be ephemeral.\n\nNone of these are fatal to the core engineering claim that a GPT-2-style model can be trained to respect fine-grained syllable constraints at word/line level. But the paragraph-level gap is substantial, and the paper's reporting is selective at best.\n\nVerdict: worth a serious referee, but the authors need to address the per-level reporting and ideally add human evaluation or at least a qualitative analysis of paragraph errors. As it stands, a referee would likely request major revision.","headline":"The paper's multi-level syllable control is genuinely strong at word/line/phrase level, but the paragraph-level claim is contradicted by its own Table 2a, so the abstract overreaches and the paper needs a forthright revision.","tokens_in":10250,"tokens_out":3240,"would_cite":false,"duration_ms":31136,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A language-model decoder conditioned on special syllable and song-form tokens can generate complete lyrics with controllable syllable counts at word, phrase, line, and paragraph levels.","keywords":["lyrics generation","syllable count control","song form","text-to-lyrics","lyrics infilling","conditional text generation","special tokens","semantic embedding"],"falsifier":"Compare the automated syllable-counting library used in the paper with hand or dictionary syllable counts on a random sample of the test vocabulary; if disagreement is common, the model's near-zero syllable distance may be an artifact of the estimator rather than genuine rhythmic control.","tokens_in":9298,"feed_emoji":"🎵","tokens_out":7625,"duration_ms":73202,"temperature":0.7,"pith_summary":"The paper sets out to show that the rhythmic core of lyrics—syllable counts at every level from a single word to a whole verse—can be controlled by a single language-model decoder, as long as the requested structure is handed to it as explicit tokens. It builds a full-song generation framework that takes an arbitrary text prompt, a song form (verse, chorus, bridge, and similar), and syllable-count targets for words, phrases, lines, and paragraphs, and produces complete lyrics under those constraints. This matters because fluent general-purpose language models count syllables poorly, and most lyrics systems either work line by line or require melody-audio paired data that is scarce. The paper reports syllable count distance of 0.003 and syllable error rate of 5.025% in its best generation setting, far below the large-language-model baselines it compares against, with comparable semantic coherence.","feed_headline":"Syllable-exact lyrics model beats ChatGPT's count accuracy","feed_subtitle":"It hits required counts at word, phrase, line, and paragraph levels while respecting verse and chorus structure.","key_machinery":"The machinery is a structured token vocabulary: song-form tokens (<VERSE>, <CHORUS>, etc.), syllable-count tokens (<SYL:s>) that state a budget, and generation/infilling directive tokens (<GEN *>, <INF *>) with matching end tokens at each granularity. These tokens turn a lyrical plan into a sequence the decoder is trained to follow; the model is not left to infer syllable counts from sub-word tokenization. Training plans are synthesized by parsing lyrics into hierarchical trees (paragraphs, lines, phrases, word leaves) and traversing them in pre-order, and the semantic embedding of the input text is injected into the initial token embedding. During infilling, masked subtrees are marked with special tokens so the model can repair or replace a word, phrase, line, or paragraph using both past and future context.","core_discovery":"The central claim is that precise multi-level syllable control and song-form awareness can both be engineered into a lyrics decoder through a token interface rather than through explicit musical audio or rule-based post-processing. The model is trained on real lyrics converted into hierarchical generation plans; each plan tells it which song-form section to write, the exact syllable budget of the section, and the granularity of each segment. Semantic embeddings of the input text stand in for paired text-lyrics data, so any text prompt can steer content. In evaluation, the best model (back-conditioned, trained from scratch) achieves full-level syllable count distance 0.003 and syllable count error rate 5.025%, versus 0.194 and 79.828% for a large general-purpose language model, and an infilling variant preserves semantic similarity while matching syllable constraints.","pith_inferences":["A direct extension would be to test the same token-conditioning idea on other constrained generation tasks, such as metered poetry, subtitles with character limits, or spoken lines with exact pause counts.","The reported syllable accuracy has not been checked against human perception; an independent human-annotation study of syllable counts on generated lyrics would show whether the near-zero error rates reflect true rhythmic fit or the quirks of the automated estimator used to build and score the data.","Because training plans and evaluation metrics are derived from the same corpus and the same syllable counter, the model may be partly memorizing the estimator's behavior; training on plans from a second syllable counter or an annotated dataset would separate genuine control from estimator alignment."],"forward_implications":["A songwriter can specify a song form and exact syllable counts per paragraph, line, phrase, or word and receive complete lyrics that meet those counts, without needing melody-audio paired training data.","The infilling mode can revise an existing lyric locally—replacing a word, phrase, line, or paragraph—while keeping the rest of the song and the syllable budget intact.","Training from scratch with the special-token structure is what drives syllable accuracy; pretrained weights help language quality but hurt count control, so the two objectives trade off in this design.","Song-form conditioning makes generated songs internally consistent: the paper finds that paragraphs of the same form (verse versus verse) are semantically and structurally closer to each other than to paragraphs of other forms."],"supporting_citations":[{"why":"Supplies the song-form and syllable-count token format used for conditioning the decoder.","marker":"[17]"},{"why":"Provides the idea of distinct end tokens for each generation granularity, used to delimit words, phrases, lines, and paragraphs.","marker":"[19]"},{"why":"Supplies the sentence-embedding model that encodes the input text as the semantic condition.","marker":"[20]"},{"why":"Provides the masked-infilling training strategy and subtree-selection method used to synthesize generation plans.","marker":"[24]"},{"why":"Supplies the large lyrics corpus with song-form annotations used for training and evaluation.","marker":"[25]"},{"why":"Supplies the syllable-count estimator used to build syllable conditions and to compute the syllable metrics.","marker":"[27]"},{"why":"Supplies the summarization model used to create input-text and lyric pairs for evaluation.","marker":"[28]"},{"why":"Defines the syllable count distance metric used to measure deviation from expected syllable counts.","marker":"[31]"},{"why":"Defines the BERTScore metric used to measure semantic coherence between input text and generated lyrics.","marker":"[32]"},{"why":"Defines the normalized edit distance metric used to measure song-form consistency between paragraphs.","marker":"[33]"}],"fun_headline_variants":["Model hits exact syllable counts at 4 levels, beats ChatGPT","Multi-level syllable control in full-song lyrics generation","Song-form aware lyrics model nails syllable counts precisely","Beat ChatGPT in syllable accuracy with song-aware generator","New lyrics generator masters word-to-paragraph syllable counts"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported syllable accuracy is only as trustworthy as the off-the-shelf syllable-counting library used to build the training plans and to score the outputs, and the paper does not independently verify those counts against a dictionary or human annotators.","fun_headline_variants_meta":{"raw":{"variants":["Model hits exact syllable counts at 4 levels, beats ChatGPT","Multi-level syllable control in full-song lyrics generation","Song-form aware lyrics model nails syllable counts precisely","Beat ChatGPT in syllable accuracy with song-aware generator","New lyrics generator masters word-to-paragraph syllable counts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000479,"raw_usage":{"total_tokens":2301,"prompt_tokens":802,"completion_tokens":1499,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":418,"completion_tokens_details":{"reasoning_tokens":1424}},"tokens_in":418,"tokens_out":1499,"duration_ms":10237,"temperature":1.0,"reasoning_tokens":1424,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T16:49:42.420694+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare the automated syllable-counting library used in the paper with hand or dictionary syllable counts on a random sample of the test vocabulary; if disagreement is common, the model's near-zero syllable distance may be an artifact of the estimator rather than genuine rhythmic control.","supporting_citations":[{"cited_title":"K-pop lyric transla- tion: Dataset, analysis, and neural-modelling,","cited_arxiv_id":null,"evidence_quote":"Supplies the large lyrics corpus with song-form annotations used for training and evaluation."},{"cited_title":"Melody-conditioned lyrics generation with seqgans,","cited_arxiv_id":null,"evidence_quote":"Supplies the song-form and syllable-count token format used for conditioning the decoder."},{"cited_title":"Un- supervised melody-to-lyrics generation,","cited_arxiv_id":null,"evidence_quote":"Provides the idea of distinct end tokens for each generation granularity, used to delimit words, phrases, lines, and paragraphs."},{"cited_title":"Seqgan: sequence genera- tive adversarial nets with policy gradient,","cited_arxiv_id":null,"evidence_quote":"Supplies the sentence-embedding model that encodes the input text as the semantic condition."},{"cited_title":"Generating chinese classical poems with rnn encoder-decoder,","cited_arxiv_id":null,"evidence_quote":"Provides the masked-infilling training strategy and subtree-selection method used to synthesize generation plans."},{"cited_title":"Semantics of the unwritten: The effect of end of paragraph and sequence tokens on text generation with GPT2,","cited_arxiv_id":null,"evidence_quote":"Supplies the syllable-count estimator used to build syllable conditions and to compute the syllable metrics."},{"cited_title":"Sentence-bert: Sen- tence embeddings using siamese bert-networks,","cited_arxiv_id":null,"evidence_quote":"Supplies the summarization model used to create input-text and lyric pairs for evaluation."},{"cited_title":"An image is worth 16x16 words: Transformers for image recognition at scale,","cited_arxiv_id":null,"evidence_quote":"Defines the syllable count distance metric used to measure deviation from expected syllable counts."},{"cited_title":"Enabling language models to fill in the blanks,","cited_arxiv_id":null,"evidence_quote":"Defines the BERTScore metric used to measure semantic coherence between input text and generated lyrics."},{"cited_title":"Genius song lyrics dataset,","cited_arxiv_id":null,"evidence_quote":"Defines the normalized edit distance metric used to measure song-form consistency between paragraphs."}],"review_version":1}