Pith. sign in

REVIEW 3 major objections 6 minor 44 references

Towards Hallucination-Free Music: A Reinforcement Learning Preference Optimization Framework for Reliable Song Generation

T0 review · 3 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Reinforcement preference optimization, driven by a phoneme-error-rate reward, measurably suppresses lyric-to-song hallucination while preserving musicality.

desk verdict Useful engineering result for RS+DPO lyric alignment, but the evaluation is circular and the framework-wide claim collapses once you look at the subjective scores. read the letter →

arxiv 2508.05011 v1 pith:BVCAATDF submitted 2025-08-07 cs.SD cs.AIeess.AS

Huaicheng Zhang , Wei Tan , Guangzheng Li , Yixuan Zhang , Hangting Chen , Shun Lei , Chenyu Yang , Zhiyong Wu
show 3 more authors
Shuai Wang Qijun Huang Dong Yu
This is my paper
classification cs.SDcs.AIeess.AS
keywords hallucinationmitigationlyric-to-songgenerationpreferenceoptimizationphonemeerrorratedirectproximalpolicygrouprelativeaudiolanguagemodel
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that the hallucination problem in AI lyric-to-song generation — generated vocals singing words that are not in the provided lyrics — can be attacked at training time instead of patched at inference. The authors measure hallucination as phoneme error rate (PER) between the sung words and the intended lyrics, build a large preference dataset of better/worse song pairs from that signal, and post-train an audio language model with three reinforcement-learning preference-optimization strategies: DPO, PPO, and GRPO. They report that all three suppress hallucinated outputs on validation prompts, with reject-sampled DPO the strongest: a 16.50% reduction in hallucinated songs and a 38.47% increase in hallucination-free samples, plus a 9.60% reward-score gain, while subjective ratings of quality, melody, and harmony stay flat. If right, this matters because it turns lyric alignment in generative audio from an uncontrolled side-effect of autoregressive decoding into a learnable objective, and it offers a recipe — score with an automated error metric, pair, optimize — that could be reapplied to style adherence or other song-generation attributes.

What carries the argument

The load-bearing object is phoneme error rate (PER) — the edit distance between the phoneme sequence of the sung vocal and that of the intended lyrics, obtained by vocal separation (Demucs), double ASR (Whisper-large-v2 and Zipformer), and grapheme-to-phoneme conversion. PER does triple duty: it labels the chosen–rejected pairs of the preference dataset through a three-rule pairing strategy; it sets the reward model's regression target ($R = 1 - \mathrm{PER}$, L1 loss on a head attached to the same audio LM); and it is the scoring metric behind the headline gains. Around it sit three optimization losses: DPO's Bradley–Terry contrast of policy against reference log-probabilities; PPO's token-

What would settle it

Score the same 90-prompt, 360-song validation set with human-annotated hallucination spans instead of the trained reward model, and check whether RS+DPO's drop in low-reward songs (17.30% to 4.80%) reproduces as a drop in human-labeled hallucination. A second check: swap a component of the PER pipeline (e.g., replace Whisper-large-v2 or the grapheme-to-phoneme converter) and see whether the 0.7/0.8 hallucination thresholds stay calibrated.

Watch

Extended reading notes

Core claim

The paper's central claim is that lyric-following accuracy in audio language models can be optimized directly: treat hallucination as excess phoneme error rate (PER) and post-train the model with preference optimization. The authors build a preference dataset of 86,746 songs yielding 25,459 chosen–rejected pairs, train a reward model with score $R=1-\mathrm{PER}$, and tune an audio LM with three strategies: DPO, PPO, and GRPO. On validation prompts every strategy lowers the share of hallucinated outputs; reject-sampled DPO is strongest, raising the reward score 9.60%, cutting hallucinated songs 16.50%, and lifting hallucination-free samples from 43.26% to 81.73%. Subjective quality, melody,

Load-bearing premise

The load-bearing premise is that the PER-based reward model is an unbiased oracle for hallucination — the paper trains this model on ASR-derived error and then measures its own success with it — so if that proxy is miscalibrated, the headline gains could partly reflect optimization of the proxy rather than genuine lyric alignment, and the subjective LYC ratings are the only independent check.

Editorial extensions

If this is right

  • Hallucination in lyric-to-song generation is addressable at training time: post-trained policies produce fewer outputs that sing words absent from the lyrics, with overall-quality, melody, and harmony ratings unmoved.
  • Automated PER scoring lets preference data be assembled at scale — 25,459 chosen–rejected pairs from roughly 1,700 hours of generated songs — without human hallucination annotation, removing the main bottleneck for RL post-training in audio.
  • The best reported configuration is reject sampling followed by DPO; on-policy PPO and GRPO also cut hallucination but require early stopping because the sparse sequence-level reward makes training non-monotonic.
  • The reward score doubles as a deployable output filter: songs scoring above 0.8 count as deterministically hallucination-free, below 0.7 as hallucinated.
  • The framework's two adaptable parts — preference-data construction and reward-model design — transfer to other song-generation objectives such as style adherence and musicality enhancement, per the paper's discussion.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • My inference: because the reward model behind Table 1 is itself trained on the same PER pipeline, the headline gains mix genuine alignment with optimization against that specific proxy; the subjective LYC ratings — which rise for some methods but fall for GRPO — are the only independent evidence, so re-scoring validation songs from human transcription would separate the two.
  • My reading of the paper's own data: its diagnosis of sparse, sequence-level rewards predicts the pattern in Table 3 where GRPO's reward score rises while its subjective lyric-following score falls (2.180 vs. 2.605 baseline); the token-level GRPO variant trialed in Appendix D.3 is the direct test of that explanation.
  • My inference: PER as a training signal should transfer to any language with a grapheme-to-phoneme converter, but tonal languages, where ASR confuses homophones, may understate hallucination; a cross-lingual replication would map the metric's limits.
  • My inference: the pair-and-optimize loop generalizes to any generation attribute with a computable error score — style adherence, instrumentation, tempo stability — so the framework's practical ceiling is set by the quality of the scoring function, not by the RL machinery.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes a reinforcement learning (RL) preference optimization framework to reduce hallucination in lyric-to-song generation, formulated as a reduction of phoneme error rate (PER). Using the LeVo audio language model as the policy, the authors construct a large preference dataset by generating songs from lyric/audio prompts, transcribing them with ASR, converting to phonemes via G2P, and pairing outputs according to PER-based rules. They then evaluate three preference optimization strategies: off-policy DPO, and on-policy PPO and GRPO, optionally preceded by reject sampling (RS). A reward model is trained to predict R = 1 − PER for the on-policy methods. Results in Table 1 show higher reward scores and shifted reward distributions after optimization; Table 3 reports subjective MOS ratings; the conclusion claims that DPO with RS reduces hallucinated samples by 16.50% and increases hallucination-free samples by 38.47% while preserving musicality and audio quality.

Significance. If the central claim is correct, the paper makes a useful contribution by demonstrating that a PER-derived preference signal can improve lyric following in a modern song-generation LM, and it provides a systematic comparison of DPO/PPO/GRPO in this domain. The authors construct and release a sizable preference dataset (25,459 paired samples) and provide demo materials, which are valuable assets. The method's transferability to other song-level alignment tasks is plausible. However, the primary evidence for hallucination reduction is a reward score computed by the same PER-based reward model used in training, so the headline gains are not independently verified. The subjective evaluation partially supports RS+DPO but contradicts the reward-model ranking for GRPO. The framework-wide claim that preference optimization aligns songs without decreasing musicality is therefore not established by the present evidence.

major comments (3)
  1. [Section 4.4, Table 1, Appendix D.4] Table 1 does not report measured PER; it reports average reward scores predicted by the reward model trained in Section 4.2, with the caption defining PER = 1 − r. The abstract's '7.4% PER reduction' and the conclusion's percentages are therefore reward-score deltas, not verified PER reductions. Because the same reward model is used for DPO pairing, PPO/GRPO rewards, early stopping (Section 4.3), and the Table 1 evaluation, Table 1 is circular evidence for hallucination mitigation. Appendix D.4 explicitly conditions the conclusion on 'ideal, unbiased reward modeling conditions,' which is exactly the assumption that needs independent validation. The authors should report actual PER/alignment metrics computed from ASR+G2P on a held-out set, ideally with error bars and significance tests, and should adjust the abstract and conclusion to distinguish reward-score gains from measured alignment
  2. [Table 5 (Appendix D.1) vs. Table 1; Section 4.6] The subjective results contradict the reward-model ranking and undermine the framework-wide claim. GRPO has a higher reward score (0.834) than Origin (0.771) in Table 1, but its LYC is 2.180 ± 0.053, significantly worse than Origin's 2.605 ± 0.098; its OVL, MEL, and HAM are also numerically lower. Thus the PER-based reward model is not a reliable oracle for hallucination in GRPO's case, and the claim in Section 4.6 that 'preference optimization successfully aligns generated songs with provided lyrics without a decrease in musicality and audio quality' is false for GRPO and unsupported for PPO. The authors should either restrict the claim to RS+DPO (where LYC improves and other metrics are comparable) or explain the discrepancy and provide a more faithful reward signal.
  3. [Section 4.3 and Figure 5(b)] Early stopping is performed on the same reward score used for training (Figure 5(b) shows a non-monotonic reward trajectory). This selection-on-the-evaluation-proxy further inflates the apparent effectiveness of PPO/GRPO: the reported checkpoint is chosen precisely because it maximizes the proxy score. Independent validation on a held-out human-evaluation set or on a separately computed PER from the ASR pipeline is needed before conclusions about hallucination reduction can be drawn. At minimum, the paper should report the performance of the final (not early-stopped) checkpoints and justify the stopping rule without recourse to the evaluation metric.
minor comments (6)
  1. [Abstract/Conclusion] The abstract states 'achieving a significant 7.4% PER reduction' but the corresponding entry in Table 1 is a reward-score increase, not a measured PER reduction. The wording should be corrected throughout.
  2. [Appendix C.3] The hyperparameter table lists 'β1 = 0.9, β1 = 0.95'; the second should presumably be β2.
  3. [Appendix B.1] The text says 'There are three genres' but then enumerates ten; this is likely a typo.
  4. [Equation (9)] The notation 'Lact = − 1/4 i∈I X i=1 Ai' is confusing. Clarify the summation range and the normalization.
  5. [Table 1] The columns 'r < 0.7', '0.7–0.8', and 'r > 0.8' are reward-model thresholds, not measured PER thresholds. This should be stated explicitly and not conflated with PER.
  6. [Section 3.3] The rule 'the difference in the number of incorrect phonemes between the two songs exceeds 40' is arbitrary and depends on song length; consider reporting the distribution of PER and the resulting pair statistics.

Circularity Check

2 steps flagged · score 6.0 of 10

Headline PER reductions are computed from the same reward model used as the training objective, not from independently measured PER; early stopping on that reward further selects the reported gains.

  1. fitted input called prediction [Sec. 3.4 (Eq. 2), Sec. 4.2, Sec. 4.4 (Table 1)]
    "In the hallucination mitigation task, we directly leverage the PER and define the reward score as Eq. 2, where R denotes the reward score. In other words, the reward model’s regressive target becomes PER. ... The average reward scores ˆr of songs generated from validation prompts are presented. ... PER= 1−r."

    The reward model is trained to predict R = 1 − PER, PPO/GRPO maximize that R, and Table 1's 'PER reduction' is computed from the same reward model's predicted scores, not from independently measured PER on validation outputs. The headline gains (7.4%, 4.9%, 4.7%) are therefore changes in the training objective itself. Appendix D.4 concedes the conclusion holds only 'under ideal, unbiased reward modeling conditions', which is exactly the assumption needed to break the loop but is not established.

  2. fitted input called prediction [Sec. 4.3 (early stopping) and Sec. 4.4 (Table 1)]
    "The non-monotonic reward trajectory suggests limitations of sparse reward design. ... Therefore, an early stopping strategy was adopted to record the better-performing policy before model collapse occurs."

    Early stopping selects the checkpoint with the highest reward score on the validation set, and Table 1 then reports reward scores for validation prompts. Selecting the model on the same metric that is later reported as the result makes the improvement partly an artifact of checkpoint selection: the chosen policy is, by construction, one that maximizes the reported score. This does not invalidate the independent subjective LYC results, but it means the Table 1 gains are not independent evidence of hallucination reduction.

full rationale

The paper's central quantitative claim of PER reduction is measured with the same PER-based reward model used as the training objective. On-policy methods (PPO, GRPO) optimize R=1-PER, DPO pairs are constructed from PER, and Table 1 reports the reward model's predicted scores as if they were measured PER. This creates a circularity burden for the headline numbers. Early stopping on that same reward further selects the reported gains. The subjective LYC ratings provide an independent anchor, and RS+DPO's LYC improvement is suggestive, but GRPO's LYC is significantly worse than Origin (2.18 vs 2.61) despite a higher reward score, showing the reward model is not a reliable oracle. Overall, the framework-wide claim of effective hallucination suppression is partially circular and not fully corroborated.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical or mathematical entities. Its free parameters are the RL hyperparameters and the thresholds used to convert PER into preference pairs and reward-score categories. The most consequential assumptions are that PER is a faithful hallucination measure and that the reward model trained on it can serve as an unbiased evaluation oracle.

free parameters (6)
  • DPO temperature beta = 0.3
    Controls KL deviation from reference policy in Eq. 1; hand-set.
  • PPO KL weight alpha = 0.0005
    Weight of KL penalty in Eq. 3; selected to avoid collapse, with ablation in Appendix D.2.
  • PPO GAE discount gamma and lambda = gamma=1.0, lambda=1.0
    Forced to Monte Carlo returns due to 3000-token sequences; lambda=1 is argued, gamma=1 is set.
  • GRPO KL weight beta = 0.0 (disabled)
    Eq. 9 includes KL term but Table 4 sets beta=0, so the claimed KL regularization is not active in the main GRPO runs.
  • PER pairing thresholds = error difference >40; insertion >5; omission >10 within 15-phoneme span
    Hand-defined cutoffs in the 3-step strategy, calibrated on 360 human-annotated songs but not independently justified.
  • Reward classification thresholds = 0.7 and 0.8
    Empirical cutoffs used to label hallucinated and hallucination-free outputs in Table 1; no distributional uncertainty reported.
assumptions (4)
  • domain assumption PER computed via Demucs/Whisper/Zipformer and G2P is a valid proxy for hallucination.
    Section 3.1 and C.1 assert PER captures intelligibility and hallucination, but no human-correlation measurement is reported.
  • domain assumption The reward model trained to predict R=1-PER is an unbiased oracle for final evaluation.
    Section 4.2 only shows L1 loss convergence; Appendix D.4 says conclusions hold 'under ideal, unbiased reward modeling conditions'.
  • domain assumption PER-based chosen/rejected pairs express human preferences for hallucination.
    Section 3.3 derives pairing rules from 360 annotated songs; no inter-annotator agreement or validation of the pair labels is given.
  • standard math DPO implicit reward formulation (Bradley-Terry) is valid for song tokens.
    Equation 1 inherits DPO's assumptions from Rafailov et al. (2023); the paper does not re-derive or justify them for coarse audio tokens.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards Hallucination-Free Music: A Reinforcement Learning Preference Optimization Framework for Reliable Song Generation." pith.science (2026). https://pith.science/paper/BVCAATDF

@misc{pith2026250805011,
  author       = {Pith},
  title        = {Pith review of: Towards Hallucination-Free Music: A Reinforcement Learning Preference Optimization Framework for Reliable Song Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BVCAATDF}},
  note         = {Machine review of arXiv:2508.05011}
}
read the original abstract

Recent advances in audio-based generative language models have accelerated AI-driven lyric-to-song generation. However, these models frequently suffer from content hallucination, producing outputs misaligned with the input lyrics and undermining musical coherence. Current supervised fine-tuning (SFT) approaches, limited by passive label-fitting, exhibit constrained self-improvement and poor hallucination mitigation. To address this core challenge, we propose a novel reinforcement learning (RL) framework leveraging preference optimization for hallucination control. Our key contributions include: (1) Developing a robust hallucination preference dataset constructed via phoneme error rate (PER) computation and rule-based filtering to capture alignment with human expectations; (2) Implementing and evaluating three distinct preference optimization strategies within the RL framework: Direct Preference Optimization (DPO), Proximal Policy Optimization (PPO), and Group Relative Policy Optimization (GRPO). DPO operates off-policy to enhance positive token likelihood, achieving a significant 7.4% PER reduction. PPO and GRPO employ an on-policy approach, training a PER-based reward model to iteratively optimize sequences via reward maximization and KL-regularization, yielding PER reductions of 4.9% and 4.7%, respectively. Comprehensive objective and subjective evaluations confirm that our methods effectively suppress hallucinations while preserving musical quality. Crucially, this work presents a systematic, RL-based solution to hallucination control in lyric-to-song generation. The framework's transferability also unlocks potential for music style adherence and musicality enhancement, opening new avenues for future generative song research.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

44 extracted references · 8 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; et al

    Achiam, J.; Adler, S.; Agarwal, S.; Ahmad, L.; Akkaya, I.; Aleman, F. L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  4. [4]

    I.; Borsos, Z.; Engel, J.; Verzetti, M.; Caillon, A.; Huang, Q.; Jansen, A.; Roberts, A.; Tagliasacchi, M.; et al

    Agostinelli, A.; Denk, T. I.; Borsos, Z.; Engel, J.; Verzetti, M.; Caillon, A.; Huang, Q.; Jansen, A.; Roberts, A.; Tagliasacchi, M.; et al. 2023. Musiclm: Generating music from text. arXiv preprint arXiv:2301.11325

    arXiv 2023

  5. [5]

    Anastassiou, P.; Chen, J.; Chen, J.; Chen, Y.; Chen, Z.; Chen, Z.; Cong, J.; Deng, L.; Ding, C.; Gao, L.; et al. 2024. Seed-tts: A family of high-quality versatile speech generation models. arXiv preprint arXiv:2406.02430

    arXiv 2024

  6. [6]

    Bai, Y.; Chen, H.; Chen, J.; Chen, Z.; Deng, Y.; Dong, X.; Hantrakul, L.; Hao, W.; Huang, Q.; Huang, Z.; et al. 2024. Seed-music: A unified framework for high quality and controlled music generation. arXiv preprint arXiv:2409.09214

    arXiv 2024

  7. [7]

    Borsos, Z.; Marinier, R.; Vincent, D.; Kharitonov, E.; Pietquin, O.; Sharifi, M.; Roblek, D.; Teboul, O.; Grangier, D.; Tagliasacchi, M.; et al. 2023. Audiolm: a language modeling approach to audio generation. IEEE/ACM transactions on audio, speech, and language processing, 31: 2523--2533

    work page 2023

  8. [8]

    S.; and Zhang, C

    Chen, C.; Hu, Y.; Wu, W.; Wang, H.; Chng, E. S.; and Zhang, C. 2024. Enhancing zero-shot text-to-speech synthesis with human feedback. arXiv preprint arXiv:2406.00654

    arXiv 2024

Show all 44 references
  1. [9]

    Chen, Y.; Ge, Y.; Wang, R.; Ge, Y.; Cheng, J.; Shan, Y.; and Liu, X. 2025. GRPO-CARE: Consistency-Aware Reinforcement Learning for Multimodal Reasoning. arXiv preprint arXiv:2506.16141

    2025 arXiv

  2. [10]

    Cideron, G.; Girgin, S.; Verzetti, M.; Vincent, D.; Kastelic, M.; Borsos, Z.; Mcwilliams, B.; Ungureanu, V.; Bachem, O.; Pietquin, O.; et al. 2024. MusicRL: Aligning Music Generation to Human Preferences. In International Conference on Machine Learning, 8968--8984. PMLR

    2024

  3. [11]

    D \'e fossez, A.; Usunier, N.; Bottou, L.; and Bach, F. 2019. Demucs: Deep extractor for music sources with extra unlabeled data remixed. arXiv preprint arXiv:1909.01174

    2019 arXiv

  4. [12]

    Du, Z.; Wang, Y.; Chen, Q.; Shi, X.; Lv, X.; Zhao, T.; Gao, Z.; Yang, Y.; Gao, C.; Wang, H.; et al. 2024. Cosyvoice 2: Scalable streaming speech synthesis with large language models. arXiv preprint arXiv:2412.10117

    2024 arXiv

  5. [13]

    Gong, J.; Zhao, S.; Wang, S.; Xu, S.; and Guo, J. 2025. Ace-step: A step towards music generation foundation model. arXiv preprint arXiv:2506.00045

    2025 arXiv

  6. [14]

    Huang, L.; Yu, W.; Ma, W.; Zhong, W.; Feng, Z.; Wang, H.; Chen, Q.; Peng, W.; Feng, X.; Qin, B.; et al. 2025. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems, 43(2): 1--55

    2025

  7. [15]

    Ju, Z.; Wang, Y.; Shen, K.; Tan, X.; Xin, D.; Yang, D.; Liu, E.; Leng, Y.; Song, K.; Tang, S.; et al. 2024. NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models. In International Conference on Machine Learning, 22605--22623. PMLR

    2024

  8. [16]

    Lei, S.; Xu, Y.; Lin, Z.; Zhang, H.; Tan, W.; Chen, H.; Yu, J.; Zhang, Y.; Yang, C.; Zhu, H.; et al. 2025. LeVo: High-Quality Song Generation with Multi-Preference Alignment. arXiv preprint arXiv:2506.07520

    2025

  9. [17]

    Lin, Z.; Lin, M.; Xie, Y.; and Ji, R. 2025. Cppo: Accelerating the training of group relative policy optimization-based reasoning models. arXiv preprint arXiv:2503.22342

    2025

  10. [18]

    Liu, M.; Diao, S.; Lu, X.; Hu, J.; Dong, X.; Choi, Y.; Kautz, J.; and Dong, Y. 2025 a . Prorl: Prolonged reinforcement learning expands reasoning boundaries in large language models. arXiv preprint arXiv:2505.24864

    2025 arXiv

  11. [19]

    Liu, Z.; Ding, S.; Zhang, Z.; Dong, X.; Zhang, P.; Zang, Y.; Cao, Y.; Lin, D.; and Wang, J. 2025 b . Songgen: A single stage auto-regressive transformer for text-to-song generation. arXiv preprint arXiv:2502.13128

    2025 arXiv

  12. [20]

    Ning, Z.; Chen, H.; Jiang, Y.; Hao, C.; Ma, G.; Wang, S.; Yao, J.; and Xie, L. 2025. DiffRhythm: Blazingly fast and embarrassingly simple end-to-end full-length song generation with latent diffusion. arXiv preprint arXiv:2503.01183

    2025 arXiv

  13. [21]

    Ouali, Y.; Bulat, A.; Martinez, B.; and Tzimiropoulos, G. 2024. Clip-dpo: Vision-language models as a source of preference for fixing hallucinations in lvlms. In European Conference on Computer Vision, 395--413. Springer

    2024

  14. [22]

    W.; Xu, T.; Brockman, G.; McLeavey, C.; and Sutskever, I

    Radford, A.; Kim, J. W.; Xu, T.; Brockman, G.; McLeavey, C.; and Sutskever, I. 2023. Robust speech recognition via large-scale weak supervision. In International conference on machine learning, 28492--28518. PMLR

    2023

  15. [23]

    D.; Ermon, S.; and Finn, C

    Rafailov, R.; Sharma, A.; Mitchell, E.; Manning, C. D.; Ermon, S.; and Finn, C. 2023. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36: 53728--53741

    2023

  16. [24]

    \"O .; and Pfister, T

    Sarkar, P.; Ebrahimi, S.; Etemad, A.; Beirami, A.; Ar k, S. \"O .; and Pfister, T. 2024. Mitigating Object Hallucination in MLLMs via Data-augmented Phrase-level Alignment. arXiv preprint arXiv:2405.18654

    2024 arXiv

  17. [25]

    Schulman, J.; Wolski, F.; Dhariwal, P.; Radford, A.; and Klimov, O. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347

    2017 arXiv

  18. [26]

    Shao, Z.; Wang, P.; Zhu, Q.; Xu, R.; Song, J.; Bi, X.; Zhang, H.; Zhang, M.; Li, Y.; Wu, Y.; et al. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300

    2024 arXiv

  19. [27]

    F.; Ben-Tal, O.; and Korshunova, I

    Sturm, B.; Santos, J. F.; Ben-Tal, O.; and Korshunova, I. 2016. Music Transcription Modelling and Composition Using Deep Learning. In 1st Conference on Computer Simulation of Musical Creativity

    2016

  20. [28]

    Sun, X.; Chen, Y.; Huang, Y.; Xie, R.; Zhu, J.; Zhang, K.; Li, S.; Yang, Z.; Han, J.; Shu, X.; et al. 2024. Hunyuan-large: An open-source moe model with 52 billion activated parameters by tencent. arXiv preprint arXiv:2411.02265

    2024 arXiv

  21. [29]

    Tjandra, A.; Wu, Y.-C.; Guo, B.; Hoffman, J.; Ellis, B.; Vyas, A.; Shi, B.; Chen, S.; Le, M.; Zacharov, N.; et al. 2025. Meta audiobox aesthetics: Unified automatic quality assessment for speech, music, and sound. arXiv preprint arXiv:2502.05139

    2025 arXiv

  22. [30]

    Touvron, H.; Martin, L.; Stone, K.; Albert, P.; Almahairi, A.; Babaei, Y.; Bashlykov, N.; Batra, S.; Bhargava, P.; Bhosale, S.; et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  23. [31]

    Wang, C.; Chen, S.; Wu, Y.; Zhang, Z.; Zhou, L.; Liu, S.; Chen, Z.; Liu, Y.; Wang, H.; Li, J.; et al. 2023. Neural codec language models are zero-shot text to speech synthesizers. arXiv preprint arXiv:2301.02111

    2023 arXiv

  24. [32]

    Wang, S.; Yu, L.; Gao, C.; Zheng, C.; Liu, S.; Lu, R.; Dang, K.; Chen, X.; Yang, J.; Zhang, Z.; et al. 2025. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for llm reasoning. arXiv preprint arXiv:2506.01939

    2025 arXiv

  25. [33]

    Wijmans, E.; Kadian, A.; Morcos, A.; Lee, S.; Essa, I.; Parikh, D.; Savva, M.; and Batra, D. 2019. Dd-ppo: Learning near-perfect pointgoal navigators from 2.5 billion frames. arXiv preprint arXiv:1911.00357

    2019 arXiv

  26. [34]

    A.; and Engel, J

    Wu, Y.; Manilow, E.; Deng, Y.; Swavely, R.; Kastner, K.; Cooijmans, T.; Courville, A.; Huang, C.-Z. A.; and Engel, J. 2022. MIDI-DDSP: Detailed Control of Musical Performance via Hierarchical Modeling. In International Conference on Learning Representations

    2022

  27. [35]

    Y.; Ruwase, O.; Rajbhandari, S.; Wu, X.; Awan, A

    Yao, Z.; Aminabadi, R. Y.; Ruwase, O.; Rajbhandari, S.; Wu, X.; Awan, A. A.; Rasley, J.; Zhang, M.; Li, C.; Holmes, C.; et al. 2023 a . Deepspeed-chat: Easy, fast and affordable rlhf training of chatgpt-like models at all scales. arXiv preprint arXiv:2308.01320

    2023 arXiv

  28. [36]

    Yao, Z.; Guo, L.; Yang, X.; Kang, W.; Kuang, F.; Yang, Y.; Jin, Z.; Lin, L.; and Povey, D. 2023 b . Zipformer: A faster and better encoder for automatic speech recognition. arXiv preprint arXiv:2310.11230

    2023 arXiv

  29. [37]

    Yu, Q.; Zhang, Z.; Zhu, R.; Yuan, Y.; Zuo, X.; Yue, Y.; Fan, T.; Liu, G.; Liu, L.; Liu, X.; et al. 2025. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476

    2025 arXiv

  30. [38]

    Yuan, R.; Lin, H.; Guo, S.; Zhang, G.; Pan, J.; Zang, Y.; Liu, H.; Liang, Y.; Ma, W.; Du, X.; et al. 2025 a . YuE: Scaling Open Foundation Models for Long-Form Music Generation. arXiv preprint arXiv:2503.08638

    2025

  31. [39]

    Yuan, Y.; Yu, Q.; Zuo, X.; Zhu, R.; Xu, W.; Chen, J.; Wang, C.; Fan, T.; Du, Z.; Wei, X.; et al. 2025 b . VAPO: Efficient and reliable reinforcement learning for advanced reasoning tasks. arXiv preprint arXiv:2504.05118

    2025 arXiv

  32. [40]

    Zeghidour, N.; Luebs, A.; Omran, A.; Skoglund, J.; and Tagliasacchi, M. 2021. Soundstream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30: 495--507

    2021

  33. [41]

    Zhang, D.; Li, Z.; Li, S.; Zhang, X.; Wang, P.; Zhou, Y.; and Qiu, X. 2024. Speechalign: Aligning speech generation to human preferences. Advances in Neural Information Processing Systems, 37: 50343--50360

    2024

  34. [42]

    Zhang, J.; and Zuo, C. 2025. Grpo-lead: A difficulty-aware reinforcement learning approach for concise mathematical reasoning in language models. arXiv preprint arXiv:2504.09696

    2025

  35. [43]

    Zhu, H.; Zhou, Y.; Chen, H.; Yu, J.; Ma, Z.; Gu, R.; Luo, Y.; Tan, W.; and Chen, X. 2025. Muq: Self-supervised music representation learning with mel residual vector quantization. arXiv preprint arXiv:2501.01108

    2025 arXiv

  36. [44]

    M.; Stiennon, N.; Wu, J.; Brown, T

    Ziegler, D. M.; Stiennon, N.; Wu, J.; Brown, T. B.; Radford, A.; Amodei, D.; Christiano, P.; and Irving, G. 2019. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593

    2019 arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.