{"id":"c53c45c1-05e1-48c4-9673-232fb419ad37","arxiv_id":"2508.05011","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"PER-based preference optimization (DPO, PPO, GRPO) reduces lyric-to-song hallucination in an audio language model, with the largest gains from DPO plus reject sampling.","lead":"This paper applies three reinforcement-learning preference optimization methods to reduce lyric-to-song alignment errors in generated music, measuring errors by phoneme error rate. It reports fewer misaligned songs without hurting musical quality, though the headline numbers rest partly on the same reward model used for training.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"GRPO's subjective LYC (2.18) is significantly worse than Origin (2.61) despite a higher reward score (0.834 vs 0.771), showing the reward model is not a reliable hallucination oracle; Table 1's gains may be proxy artifacts.","rationale":"The reader's weakest assumption was that the PER-based reward model is an unbiased oracle. I agree that this is the central risk, but I found a sharper internal piece of evidence: GRPO's reward score is much higher than Origin yet its subjective LYC is significantly worse, and PPO's MEL is significantly lower. This shows the reward model is not merely unvalidated—it disagrees with human perception in the very dimension it is meant to optimize. Table 1's headline numbers are reward-model predictions, not measured PER, so the abstract's '7.4% PER reduction' and the conclusion's '16.50% reduction in hallucinated songs' are likely circular if the reward model is biased. The reader's CONDITIONAL verdict is appropriate: the method may still work for RS+DPO, whose LYC improvement and stable musicality suggest a real effect, but the framework-wide claim needs independent PER measurement and proper significance testing. I do not see grounds for REJECT because the direction of the best result is credible and the fix is concrete. My concern partially overlaps with the reader's weakest_assumption but is more specific and grounded in the reported subjective results.","tokens_in":15861,"tokens_out":4183,"duration_ms":47478,"concrete_test":"Re-run the full independent PER pipeline (Demucs vocal separation, Whisper-large-v2 + Zipformer ASR, G2P) on the 90 validation songs per method, without using the learned reward model, and report mean PER with bootstrap CIs and paired tests. If GRPO's measured PER is not significantly lower than Origin (or if it worsens), Table 1's reward-score gains are an artifact of reward-model bias. Also report LYC/HAM significance for GRPO/PPO vs Origin; if the current 20-song CIs already show significant degradation, the 'no decrease in musicality' claim requires revision.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that PER-based preference optimization suppresses hallucination without hurting musicality rests on the reward model R=1-PER being a faithful oracle. Table 1 reports predicted reward scores (Sec 4.4), not measured PER, yet the abstract/caption treat the 0.845 vs 0.771 delta as a '7.4% PER reduction'. Independent subjective LYC in Table 3 contradicts the reward-model ranking: GRPO has reward 0.834 (higher than Origin 0.771) but LYC 2.180±0.053, significantly below Origin 2.605±0.098; PPO also has lower MEL (3.330±0.067 vs 3.455±0.073). Thus the on-policy methods' apparent Table 1 gains are not corroborated by the human ratings, and the claim that 'preference optimization successfully aligns generated songs... without a decrease in musicality' (Sec 4.6) fails for GRPO/PPO. Since early stopping was also performed on the same proxy (Sec 4.3), the headline numbers may largely reflect overfitting to a biased reward model rather than genuine hallucination reduction. RS+DPO's LYC improvement is suggestive, but the framework-wide claim is not.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a reinforcement learning (RL) preference optimization framework to reduce hallucination in lyric-to-song generation, formulated as a reduction of phoneme error rate (PER). Using the LeVo audio language model as the policy, the authors construct a large preference dataset by generating songs from lyric/audio prompts, transcribing them with ASR, converting to phonemes via G2P, and pairing outputs according to PER-based rules. They then evaluate three preference optimization strategies: off-policy DPO, and on-policy PPO and GRPO, optionally preceded by reject sampling (RS). A reward model is trained to predict R = 1 − PER for the on-policy methods. Results in Table 1 show higher reward scores and shifted reward distributions after optimization; Table 3 reports subjective MOS ratings; the conclusion claims that DPO with RS reduces hallucinated samples by 16.50% and increases hallucination-free samples by 38.47% while preserving musicality and audio quality.","tokens_in":16110,"tokens_out":2672,"duration_ms":31245,"significance":"If the central claim is correct, the paper makes a useful contribution by demonstrating that a PER-derived preference signal can improve lyric following in a modern song-generation LM, and it provides a systematic comparison of DPO/PPO/GRPO in this domain. The authors construct and release a sizable preference dataset (25,459 paired samples) and provide demo materials, which are valuable assets. The method's transferability to other song-level alignment tasks is plausible. However, the primary evidence for hallucination reduction is a reward score computed by the same PER-based reward model used in training, so the headline gains are not independently verified. The subjective evaluation partially supports RS+DPO but contradicts the reward-model ranking for GRPO. The framework-wide claim that preference optimization aligns songs without decreasing musicality is therefore not established by the present evidence.","major_comments":[{"comment":"Table 1 does not report measured PER; it reports average reward scores predicted by the reward model trained in Section 4.2, with the caption defining PER = 1 − r. The abstract's '7.4% PER reduction' and the conclusion's percentages are therefore reward-score deltas, not verified PER reductions. Because the same reward model is used for DPO pairing, PPO/GRPO rewards, early stopping (Section 4.3), and the Table 1 evaluation, Table 1 is circular evidence for hallucination mitigation. Appendix D.4 explicitly conditions the conclusion on 'ideal, unbiased reward modeling conditions,' which is exactly the assumption that needs independent validation. The authors should report actual PER/alignment metrics computed from ASR+G2P on a held-out set, ideally with error bars and significance tests, and should adjust the abstract and conclusion to distinguish reward-score gains from measured alignment","section":"Section 4.4, Table 1, Appendix D.4"},{"comment":"The subjective results contradict the reward-model ranking and undermine the framework-wide claim. GRPO has a higher reward score (0.834) than Origin (0.771) in Table 1, but its LYC is 2.180 ± 0.053, significantly worse than Origin's 2.605 ± 0.098; its OVL, MEL, and HAM are also numerically lower. Thus the PER-based reward model is not a reliable oracle for hallucination in GRPO's case, and the claim in Section 4.6 that 'preference optimization successfully aligns generated songs with provided lyrics without a decrease in musicality and audio quality' is false for GRPO and unsupported for PPO. The authors should either restrict the claim to RS+DPO (where LYC improves and other metrics are comparable) or explain the discrepancy and provide a more faithful reward signal.","section":"Table 5 (Appendix D.1) vs. Table 1; Section 4.6"},{"comment":"Early stopping is performed on the same reward score used for training (Figure 5(b) shows a non-monotonic reward trajectory). This selection-on-the-evaluation-proxy further inflates the apparent effectiveness of PPO/GRPO: the reported checkpoint is chosen precisely because it maximizes the proxy score. Independent validation on a held-out human-evaluation set or on a separately computed PER from the ASR pipeline is needed before conclusions about hallucination reduction can be drawn. At minimum, the paper should report the performance of the final (not early-stopped) checkpoints and justify the stopping rule without recourse to the evaluation metric.","section":"Section 4.3 and Figure 5(b)"}],"minor_comments":[{"comment":"The abstract states 'achieving a significant 7.4% PER reduction' but the corresponding entry in Table 1 is a reward-score increase, not a measured PER reduction. The wording should be corrected throughout.","section":"Abstract/Conclusion"},{"comment":"The hyperparameter table lists 'β1 = 0.9, β1 = 0.95'; the second should presumably be β2.","section":"Appendix C.3"},{"comment":"The text says 'There are three genres' but then enumerates ten; this is likely a typo.","section":"Appendix B.1"},{"comment":"The notation 'Lact = − 1/4 i∈I X i=1 Ai' is confusing. Clarify the summation range and the normalization.","section":"Equation (9)"},{"comment":"The columns 'r < 0.7', '0.7–0.8', and 'r > 0.8' are reward-model thresholds, not measured PER thresholds. This should be stated explicitly and not conflated with PER.","section":"Table 1"},{"comment":"The rule 'the difference in the number of incorrect phonemes between the two songs exceeds 40' is arbitrary and depends on song length; consider reporting the distribution of PER and the resulting pair statistics.","section":"Section 3.3"}],"recommendation":"major_revision","confidential_remarks":"The core methodology is plausible and the dataset is a useful contribution, but the paper's headline evidence is largely self-referential. A revision that reports held-out measured PER, adds error bars/significance tests, and substantially narrows the claims to the methods actually supported by human evaluation (notably RS+DPO) could make this publishable. The comparison with prior RL-for-music work (MusicRL, seed-music, LeVo) is somewhat thin; the novelty beyond LeVo's multi-preference alignment should be clarified."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: there's one good result here—RS+DPO improves lyric-following without hurting musicality—and the rest of the paper overclaims on a circular evaluation. The subjective LYC score for RS+DPO (3.16 vs 2.61 for the baseline, with non-overlapping 95% CIs) is credible and worth taking seriously. That's the nugget.\n\nWhat's new is the PER-based preference dataset and the three-step pairing rules, plus the long-sequence adjustments (lambda=1 for GAE, truncated GRPO). The comparison of DPO, PPO, and GRPO on the same base model is useful, though the algorithms themselves are off-the-shelf.\n\nThe soft spots are substantial. The abstract's '7.4% PER reduction' is not a measured PER; it's the difference between reward-model scores (0.845 vs 0.771). Table 1 reports reward-model predictions, not PER from an ASR/G2P pipeline, and the caption 'PER = 1 - r' papers over that. Since the same reward model is used for training and evaluation, the objective table is circular. No error bars or significance tests on those reward scores, and early stopping is done on the same proxy.\n\nThe subjective results make the circularity concrete. GRPO gets a higher reward score than baseline (0.834 vs 0.771) but a significantly worse LYC (2.18 vs 2.61). PPO's melody score (MEL 3.33) is below baseline (3.46). So the claim that preference optimization aligns songs 'without a decrease in musicality' fails for the on-policy methods. The data support DPO and RS+DPO only. The limitations section mentions reward sparsity but doesn't own this contradiction.\n\nThe authors should re-run an independent PER measurement on final outputs, report the subjective results as primary, and reframe the paper around DPO+RS. No code or data is released, which limits reproducibility.\n\nBottom line: a legitimate engineering contribution with one reliable result and a flawed evaluation wrapper. It deserves peer review because the problem is real and the DPO/RS result is practical, but the revision needs to be major.","headline":"Useful engineering result for RS+DPO lyric alignment, but the evaluation is circular and the framework-wide claim collapses once you look at the subjective scores.","tokens_in":16726,"tokens_out":3843,"would_cite":true,"duration_ms":39991,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Reinforcement preference optimization, driven by a phoneme-error-rate reward, measurably suppresses lyric-to-song hallucination while preserving musicality.","keywords":["hallucination mitigation","lyric-to-song generation","preference optimization","phoneme error rate","direct preference optimization","proximal policy optimization","group relative policy optimization","audio language model"],"falsifier":"Score the same 90-prompt, 360-song validation set with human-annotated hallucination spans instead of the trained reward model, and check whether RS+DPO's drop in low-reward songs (17.30% to 4.80%) reproduces as a drop in human-labeled hallucination. A second check: swap a component of the PER pipeline (e.g., replace Whisper-large-v2 or the grapheme-to-phoneme converter) and see whether the 0.7/0.8 hallucination thresholds stay calibrated.","tokens_in":15668,"feed_emoji":"🎵","tokens_out":18740,"duration_ms":171489,"temperature":0.7,"pith_summary":"This paper claims that the hallucination problem in AI lyric-to-song generation — generated vocals singing words that are not in the provided lyrics — can be attacked at training time instead of patched at inference. The authors measure hallucination as phoneme error rate (PER) between the sung words and the intended lyrics, build a large preference dataset of better/worse song pairs from that signal, and post-train an audio language model with three reinforcement-learning preference-optimization strategies: DPO, PPO, and GRPO. They report that all three suppress hallucinated outputs on validation prompts, with reject-sampled DPO the strongest: a 16.50% reduction in hallucinated songs and a 38.47% increase in hallucination-free samples, plus a 9.60% reward-score gain, while subjective ratings of quality, melody, and harmony stay flat. If right, this matters because it turns lyric alignment in generative audio from an uncontrolled side-effect of autoregressive decoding into a learnable objective, and it offers a recipe — score with an automated error metric, pair, optimize — that could be reapplied to style adherence or other song-generation attributes.","feed_headline":"RL training cuts hallucinated AI songs by 16.5%","feed_subtitle":"Phoneme-error rewards teach music AI to follow its lyrics, and musicality holds steady.","key_machinery":"The load-bearing object is phoneme error rate (PER) — the edit distance between the phoneme sequence of the sung vocal and that of the intended lyrics, obtained by vocal separation (Demucs), double ASR (Whisper-large-v2 and Zipformer), and grapheme-to-phoneme conversion. PER does triple duty: it labels the chosen–rejected pairs of the preference dataset through a three-rule pairing strategy; it sets the reward model's regression target ($R = 1 - \\mathrm{PER}$, L1 loss on a head attached to the same audio LM); and it is the scoring metric behind the headline gains. Around it sit three optimization losses: DPO's Bradley–Terry contrast of policy against reference log-probabilities; PPO's token-","core_discovery":"The paper's central claim is that lyric-following accuracy in audio language models can be optimized directly: treat hallucination as excess phoneme error rate (PER) and post-train the model with preference optimization. The authors build a preference dataset of 86,746 songs yielding 25,459 chosen–rejected pairs, train a reward model with score $R=1-\\mathrm{PER}$, and tune an audio LM with three strategies: DPO, PPO, and GRPO. On validation prompts every strategy lowers the share of hallucinated outputs; reject-sampled DPO is strongest, raising the reward score 9.60%, cutting hallucinated songs 16.50%, and lifting hallucination-free samples from 43.26% to 81.73%. Subjective quality, melody,","pith_inferences":["My inference: because the reward model behind Table 1 is itself trained on the same PER pipeline, the headline gains mix genuine alignment with optimization against that specific proxy; the subjective LYC ratings — which rise for some methods but fall for GRPO — are the only independent evidence, so re-scoring validation songs from human transcription would separate the two.","My reading of the paper's own data: its diagnosis of sparse, sequence-level rewards predicts the pattern in Table 3 where GRPO's reward score rises while its subjective lyric-following score falls (2.180 vs. 2.605 baseline); the token-level GRPO variant trialed in Appendix D.3 is the direct test of that explanation.","My inference: PER as a training signal should transfer to any language with a grapheme-to-phoneme converter, but tonal languages, where ASR confuses homophones, may understate hallucination; a cross-lingual replication would map the metric's limits.","My inference: the pair-and-optimize loop generalizes to any generation attribute with a computable error score — style adherence, instrumentation, tempo stability — so the framework's practical ceiling is set by the quality of the scoring function, not by the RL machinery."],"forward_implications":["Hallucination in lyric-to-song generation is addressable at training time: post-trained policies produce fewer outputs that sing words absent from the lyrics, with overall-quality, melody, and harmony ratings unmoved.","Automated PER scoring lets preference data be assembled at scale — 25,459 chosen–rejected pairs from roughly 1,700 hours of generated songs — without human hallucination annotation, removing the main bottleneck for RL post-training in audio.","The best reported configuration is reject sampling followed by DPO; on-policy PPO and GRPO also cut hallucination but require early stopping because the sparse sequence-level reward makes training non-monotonic.","The reward score doubles as a deployable output filter: songs scoring above 0.8 count as deterministically hallucination-free, below 0.7 as hallucinated.","The framework's two adaptable parts — preference-data construction and reward-model design — transfer to other song-generation objectives such as style adherence and musicality enhancement, per the paper's discussion."],"supporting_citations":[{"why":"The audio language model (LeVo) used as the policy to be optimized; every baseline and post-trained result is produced by this model.","marker":"Lei et al. 2025"},{"why":"Supplies the DPO loss (Eq. 1) contrasting chosen against rejected songs; DPO and RS+DPO are the strongest reported configurations.","marker":"Rafailov et al. 2023"},{"why":"Supplies the PPO machinery — token-level reward shaping, GAE, and clipping — used for the on-policy experiments.","marker":"Yao et al. 2023a"},{"why":"Supplies GRPO and the K3 KL estimator (Eq. 10) that keep the policy near the reference model.","marker":"Shao et al. 2024"},{"why":"Demucs separates the vocal track, the first stage of the PER pipeline that labels the preference data.","marker":"Défossez et al. 2019"},{"why":"Whisper-large-v2 performs the primary ASR whose recognized text feeds the phoneme error computation.","marker":"Radford et al. 2023"},{"why":"Zipformer double-checks the ASR output before grapheme-to-phoneme conversion, reducing recognition noise in PER.","marker":"Yao et al. 2023b"},{"why":"Audiobox-Aesthetics supplies the objective content scores (CE, CU, PC, PQ) supporting the claim that musicality is preserved.","marker":"Tjandra et al. 2025"},{"why":"MuQ-Mulan measures prompt–song similarity, supporting the claim that audio-prompt conditioning is unchanged.","marker":"Zhu et al. 2025"}],"fun_headline_variants":["RL preference tuning cuts AI song hallucinations by 16.5%","Phoneme-error rewards slash lyric mismatch in AI songs","RL framework trims hallucinated AI tracks by 16.5%","Preference optimization: 16.5% fewer AI songs go off-lyrics","DPO beats PPO and GRPO for lyric-aligned AI singing"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The load-bearing premise is that the PER-based reward model is an unbiased oracle for hallucination — the paper trains this model on ASR-derived error and then measures its own success with it — so if that proxy is miscalibrated, the headline gains could partly reflect optimization of the proxy rather than genuine lyric alignment, and the subjective LYC ratings are the only independent check.","fun_headline_variants_meta":{"raw":{"variants":["RL preference tuning cuts AI song hallucinations by 16.5%","Phoneme-error rewards slash lyric mismatch in AI songs","RL framework trims hallucinated AI tracks by 16.5%","Preference optimization: 16.5% fewer AI songs go off-lyrics","DPO beats PPO and GRPO for lyric-aligned AI singing"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000781,"raw_usage":{"total_tokens":3329,"prompt_tokens":831,"completion_tokens":2498,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":575,"completion_tokens_details":{"reasoning_tokens":2418}},"tokens_in":575,"tokens_out":2498,"duration_ms":18365,"temperature":1.0,"reasoning_tokens":2418,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T23:37:05.745518+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Score the same 90-prompt, 360-song validation set with human-annotated hallucination spans instead of the trained reward model, and check whether RS+DPO's drop in low-reward songs (17.30% to 4.80%) reproduces as a drop in human-labeled hallucination. A second check: swap a component of the PER pipeline (e.g., replace Whisper-large-v2 or the grapheme-to-phoneme converter) and see whether the 0.7/0.8 hallucination thresholds stay calibrated.","supporting_citations":[],"review_version":1}