Pith. sign in

REVIEW 4 major objections 8 minor 2 cited by

Differentiable Reward Optimization for LLM based TTS system

T0 review · 4 major / 8 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read DiffRO makes TTS reward training differentiable by scoring predicted codec tokens directly, bypassing synthesized audio and preference pairs.

desk verdict DiffRO is a real methodological step—token-level differentiable rewards for codec TTS—but its SOTA and zero-shot quality-control claims outrun the evidence, mainly because the paper itself shows token-level optimization need not transfer to the final audio. read the letter →

arxiv 2507.05911 v1 pith:TNGIYICM submitted 2025-07-08 cs.SD cs.AIeess.AS

Changfeng Gao , Zhihao Du , Shiliang Zhang This is my paper
classification cs.SDcs.AIeess.AS
keywords text-to-speechneuralcodeclanguagemodelRLHFdifferentiablerewardoptimizationGumbel-Softmaxmulti-taskworderrorratezero-shotemotioncontrol
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that a text-to-speech system built on a neural codec language model can be trained with rewards computed directly on the predicted codec tokens, before any audio waveform is made. Instead of decoding candidates and judging the sound, DiffRO uses a Gumbel-Softmax sample in place of the argmax token choice, so the reward's gradient flows straight back into the language model. The authors report that this yields state-of-the-art word-error-rate results on the seed-tts-eval benchmark, and that a multi-task reward model adds zero-shot control over emotion and other acoustic attributes. The practical stake is large: if token-level rewards are enough, reinforcement learning for TTS no longer needs expensive vocoder synthesis of every candidate or hand-constructed preference pairs.

What carries the argument

DiffRO rests on three pieces. Token2Reward prediction: a speech-understanding model is trained directly on codec-token sequences, so rewards can be scored without reconstructing the waveform. Gumbel-Softmax relaxation: the language model's nondifferentiable argmax token selection is replaced by a differentiable sample, which converts the reward loss into an ordinary backpropagation target and lets DiffRO update the LM directly instead of running PPO or DPO. The multi-task reward (MTR) model: one shared token encoder with separate heads for ASR, SER, SQA, and age/gender prediction, whose per-task log posteriors can be summed into one reward. This design is what lets one training loop both improve pronunciation and follow instructions such as an emotion label.

What would settle it

Take the MOS-control setup from the paper and extend it: train DiffRO with MOS targets of 2 and 3 on a larger, diverse set of texts, then measure the final synthesized audio with both a reference-free MOS predictor and human raters. If the codec-token MOS tracks the target closely while the audio MOS stays flat at baseline across the full set, the assumption that optimizing token-level rewards improves the rendered speech is falsified. For the ASR reward, an equivalent test is to collect synthesized samples that score well under the token-level ASR reward but are misheard by human listeners; if a DiffRO-ASR model systematically produces such samples beyond chance, the improvement is an artifact of the reward model rather than genuine pronunciation.

Watch

Extended reading notes

Core claim

The paper's central claim is that pronunciation accuracy and style controllability can be improved by maximizing the log-probability a token-level reward model assigns to the desired label, with sampling made differentiable. The ASR reward is the log posterior of the correct text given the predicted token sequence; the multi-task reward adds heads for speech emotion recognition, speech quality assessment, and age/gender prediction. On the seed-tts-eval benchmark, DiffRO-ASR lowers WER across Chinese, English, Japanese, and Korean relative to both the baseline and a DPO baseline, including state-of-the-art results on Chinese and the hard subset. The emotion-control experiment reports large gains on held-out samples for happy, sad, and angry speech even though the reinforcement stage never sees emotion-labeled data, and the system spontaneously learns non-verbal cues such as laughter and breath.

Load-bearing premise

The load-bearing premise is that a reward computed on predicted codec tokens transfers to the quality of the final synthesized audio; if the flow-matching and vocoder stages can override the language model's token choices, as the paper's own MOS-control experiment shows, the reward may move token scores without moving what listeners hear.

Editorial extensions

If this is right

  • A codec-LM TTS system optimized by DiffRO-ASR beats its DPO counterpart on every language reported in the seed-tts-eval benchmark, and also recovers much of the multilingual accuracy lost during speaker fine-tuning.
  • The RL stage becomes cheaper: because rewards are read from tokens, the flow-matching and vocoder stages never need to run for training candidates.
  • Emotion controllability is achieved zero-shot: the multi-task reward supplies the emotion signal without any emotion-labeled reinforcement data, and the model learns non-verbal expressions like laughter and breath as a side effect.
  • The same DiffRO loop can be redirected to other attributes by swapping reward heads; the authors demonstrate quality attributes with the speech-quality-assessment head, though they report that age and gender cannot be decisively controlled because speaker information enters at the flow-matching stage.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's own MOS table draws the boundary the authors leave implicit: token-level MOS can be pulled toward the target while the final audio MOS stays at baseline, because the vocoder and flow model are trained on clean audio and denoise the LM's attempts. The same gap likely appears for any attribute the post-LM modules can override.
  • A token-level ASR reward could in principle be exploited by token patterns that satisfy the reward model but do not decode into the intended text; the cross-ASR evaluation is partial evidence against this for pronunciation, and an adversarial search for such patterns would settle it.
  • DiffRO is modular, so a natural next step is to apply the same differentiable-reward trick to the flow-matching module rather than only the LM; the authors name this as future work, and it is where speaker and age/gender control would likely become effective.
  • The method's ceiling is set by how well token-level classifiers predict human judgments of the waveform, so the reward model's calibration, not the optimizer, is the real bottleneck.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 8 minor

Summary. The paper proposes DiffRO, a method for optimizing a neural codec language model TTS system by computing rewards directly from predicted codec tokens rather than from synthesized audio, and by using a Gumbel-Softmax relaxation to make the reward differentiable so the LM can be optimized by backpropagation without PPO or DPO. A multi-task reward (MTR) model incorporating ASR, speech emotion recognition, speech quality assessment, and age/gender prediction is introduced. Experiments on a CosyVoice2.0-based system report improved ASR WER on the seed-tts-eval benchmark, emotion classification improvements, and an attempt at quality/attribute control, with the authors claiming state-of-the-art WER and zero-shot attribute control.

Significance. If the central claims hold, DiffRO is a practical contribution: it removes the need to decode audio during RL training, offers a lightweight alternative to DPO for TTS alignment, and demonstrates that a single multi-task token-level reward model can address multiple objectives. The evaluation design is partially commendable: pronunciation is evaluated with different ASR models (Paramformer/Whisper) and emotion with a different classifier (emo2vec-plus-large) than the reward model, which reduces direct circularity. The paper also honestly reports a failure mode in the MOS control experiment (Table 4), which is useful for the community. However, the load-bearing transfer assumption, the unsupported SOTA claim, and the absence of statistical rigor prevent the paper from being accepted in its current form.

major comments (4)
  1. [§4.2, Table 2] The claim of 'state-of-the-art WER results on the seed-tts-eval benchmark' is not supported by the experiments. Table 2 compares only CosyVoice2.0 and its SFT/DPO/DiffRO variants; no external systems with published seed-tts-eval WER numbers are included. To substantiate the SOTA claim, the authors should report published WER results from Seed-TTS and other recent TTS systems on the same benchmark, or explicitly weaken the claim to an improvement over the internal CosyVoice2.0 baseline.
  2. [§3.1, Eq. (8) and §4.2, Table 2] The paper does not establish that maximizing the Gumbel-Softmax soft-token reward transfers to the final synthesized audio. The token2reward model is trained on hard codec tokens from real audio (§4.1.2), while Eq. (8) is computed over continuous soft token mixtures, and no temperature schedule, straight-through estimator, or analysis of soft-versus-hard reward agreement is provided. Table 4 directly shows this transfer can fail: the codec-level MOS tracks the target while the audio-level MOS remains near baseline. For the ASR claim, the authors should provide a direct alignment check, e.g., by measuring whether the soft ASR reward per utterance correlates with the audio WER of the same utterance, or by comparing DiffRO against a hard-token reward variant.
  3. [§4.2, Table 2] No error bars, confidence intervals, or significance tests are reported for any WER result. Several improvements are small in absolute terms (e.g., zh WER 1.50 to 1.09 after DiffRO-ASR), and the ja/ko numbers are based on an unspecified number of evaluation utterances. The authors should report the number of test utterances and either bootstrap confidence intervals or repeated-seed training runs to show the improvements are not noise.
  4. [§4.3.1, Table 3] The emotion control results are reported as point accuracies on 100 samples per emotion, with no statistical significance testing or confidence intervals. Since the zero-shot emotion control claim is one of the two main contributions, the authors should provide more evidence that the improvements over CosyVoice2.0 are reliable, for example by reporting per-language breakdowns and bootstrap intervals, or by providing listening examples.
minor comments (8)
  1. [Abstract] The abstract contains grammatical errors, e.g., 'DiffRO directly compute the rewards' should be 'DiffRO directly computes the rewards', and 'reward function' should be 'reward functions' or 'the reward function'.
  2. [§2.1, Eq. (1)] Eq. (1) writes U_t = argmax P(...), which is inconsistent with the sampling-based generation described in Section 2.2. The equation should use a sampling distribution or explicitly describe the decoding strategy.
  3. [§3.1, Eq. (6)] Eq. (6) still uses an argmax over tokens, but the text says Gumbel-Softmax replaces the argmax. Please clarify the exact forward pass: is the soft mixture fed directly to PASR, or is a hard sample drawn for each token? This affects the interpretation of Eq. (8).
  4. [§3.1] The phrase 'post-probability' should be 'posterior probability'.
  5. [Table 1] The table caption contains a typo: 'Performace' should be 'Performance', and 'inhome' in §4.1.2 should be 'in-house'.
  6. [Table 4] The caption says 'MOS score compassion for the TTS system'; this should be 'comparison'.
  7. [§4.1.2] The description of the reward model is incomplete: it should state which codec (e.g., CosyVoice2.0's tokenizer), how many codebooks are used, how the token sequence is converted to embeddings, and the exact architecture modifications to SenseVoice. This is needed for reproducibility.
  8. [Table 1] The abbreviation 'SQA-BVCC-LCC' is not defined in the text; the authors should expand it when first used.

Circularity Check

0 steps flagged · score 2.0 of 10

No load-bearing circularity: the central WER claim rests on independent ASR evaluation, with only a minor non-load-bearing self-citation and one secondary metric-alignment concern in the emotion-control evaluation.

full rationale

The paper's central derivation is not circular. The token2reward ASR model (Eqs. 7-8) is a fixed model trained on an external ASR corpus, and the pronunciation evaluation in Section 4.2 uses different recognizers (Paramformer-zh, whisper-large-v3) on decoded audio, so the reported WER improvement is an independent empirical outcome rather than an identity with the training objective. The Gumbel-Softmax relaxation introduces a soft-token/hard-token mismatch that the paper's own Table 4 exposes for MOS and age/gender control; the authors honestly report that audio-level MOS fails to follow the codec-level target, which is a transfer limitation, not a circular definition. The self-citations to CosyVoice 2, SenseVoice, and Paraformer provide baselines, components, and evaluation tools, but none is invoked as a uniqueness theorem or as a reason the result must hold. One secondary concern: the MTR emotion head is trained on emo2vec-plus-large pseudo-labels and Table 3 evaluates emotion with the same emo2vec-plus-large classifier, so the 'zero-shot emotion control' result is partly a self-consistent metric rather than an independent verification; this is an evaluation-design weakness, not a load-bearing circularity in the central derivation. The 'SOTA' WER claim is unsupported by external comparisons in Table 2, but that is an evidentiary overclaim, not circularity.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claim rests on three main assumptions: the Gumbel-Softmax relaxation works as intended, codec tokens retain enough information for the reward tasks, and token-level optimization transfers to audio. The first is standard; the second is partially validated by Table 1; the third is partially contradicted by the MOS and age/gender control results. Two unstated hyperparameters (reward weight and Gumbel temperature) also affect the result.

free parameters (2)
  • Reward combination weight between ASR and SER rewards in DiffRO-MTR emotion control = not reported
    Equation (9) sums multiple task rewards, but the weight used to combine the ASR reward with the SER reward in the emotion control experiments is not reported.
  • Gumbel-Softmax temperature = not reported
    The temperature controls the bias-variance tradeoff of the differentiable sampling operation and is not stated anywhere in the paper.
assumptions (3)
  • standard math The Gumbel-Softmax relaxation provides a differentiable approximation to sampling from the categorical token distribution.
    Invoked in Section 3.1 and Figure 1 to enable backpropagation through the token sampling step.
  • domain assumption Codec tokens contain sufficient information for ASR, SER, SQA, age and gender prediction.
    The whole DiffRO method depends on predicting rewards from token input; Table 1 provides supporting evidence, though token-based ASR is slightly worse than audio-based ASR.
  • domain assumption Optimizing token-level rewards transfers to improvements in the final synthesized audio.
    The WER improvements support this for pronunciation, but the MOS experiment in Table 4 shows it fails for quality, and Section 4.3.2 concedes it is limited for age and gender.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Differentiable Reward Optimization for LLM based TTS system." pith.science (2026). https://pith.science/paper/TNGIYICM

@misc{pith2026250705911,
  author       = {Pith},
  title        = {Pith review of: Differentiable Reward Optimization for LLM based TTS system},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TNGIYICM}},
  note         = {Machine review of arXiv:2507.05911}
}
read the original abstract

This paper proposes a novel Differentiable Reward Optimization (DiffRO) method aimed at enhancing the performance of neural codec language models based text-to-speech (TTS) systems. In contrast to conventional reinforcement learning from human feedback (RLHF) approaches applied to TTS, DiffRO directly compute the rewards based on neural codec tokens, rather than relying on synthesized audio. Furthermore, we employ the Gumbel-Softmax technique to render the reward function differentiable, thereby streamlining the RLHF training process. Additionally, we introduce a multi-task reward (MTR) model which can provide feedback from different perspectives and find that it can augment the system's capability to follow instructions effectively.Experimental results indicate that DiffRO significantly improves the pronunciation accuracy of the TTS system, achieving state-of-the-art (SOTA) WER results on the seed-tts-eval benchmark. Moreover, with the integration of the MTR model, we demonstrate the ability to control emotional and quality attributes in a zero-shot manner.

Figures

Figures reproduced from arXiv: 2507.05911 by the authors.

Figure 1
Figure 1. Comparison between the DPO an the proposed DiffRO. DiffRO can directly predict the reward score from the codec token sequence rather than the audio, and it can directly optimize the LM with back-propagation rather than the RL training loop like DPO. encodes the speech into a token sequence U1:T . Subsequently, the LM is trained using a next-token prediction task: Ut = argmax1:QPπ(Ut = q|U1:t−1, Y1:N ) (1) where Q is… view at source ↗
Figure 2
Figure 2. The Structure of the proposed MTR model. ASR, SER, SQA and AED are used for the downstream task in this figure and can be adjusted according to requirements. train a codec-based speech understanding model using a multi￾task training approach, enabling it to perform tasks such as SER, SQA, AED and other audio understanding functionalities. Subsequently, we employ this model as a MTR model to guide the TTS system in g… view at source ↗
Figure 3
Figure 3. Laughter (top) and breath (down) generated in differ￾ent emotion with same text. Model MOS Codec Audio Recodec Baseline 3.82 3.84 3.88 MOSt = 2 2.20 3.77 3.47 MOSt = 3 3.14 3.81 3.66 MOSt = 4 3.96 3.86 3.92 [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Evaluating and Rewarding LALMs for Expressive Role-Play TTS via Mean Continuation Log-Probability

    cs.SD 2026-01 conditional novelty 6.0 of 10

    A new metric, MCLP, uses a frozen audio-LLM's continuation likelihood to grade speaking-style consistency and doubles as a reward that improves role-play TTS on a new 1,435-hour drama dataset.

  2. End-to-End Training for Discrete Token LLM based TTS System

    cs.SD 2026-06 unverdicted novelty 5.0 of 10

    An end-to-end optimization framework jointly trains the speech tokenizer, LLM, FM model, and reward model for discrete-token TTS, reporting new SOTA WER of 0.78% and 1.56% on Seed-TTS-Eval with 0.6B LLM and 0.5B FM.

Reference graph

Works this paper leans on

34 extracted references · 15 canonical work pages · cited by 2 Pith papers

  1. [1]

    Introduction Inspired by the success of text-based large language mod- els (LLMs), neural codec token language modeling (LM) has emerged as a leading approach for text-to-speech (TTS) genera- tion. This methodology employs a pre-trained speech tokenizer to encode speech into a discrete sequence of tokens, which are subsequently modeled using a decoder-onl...

    work page

  2. [2]

    Related work 2.1. Neural Codec LLM based TTS systems Most neural codec LLM based TTS systems comprise four pri- mary building blocks: a pre-trained speech tokenizer, a neural codec token LM, a FM model, and an acoustic vocoder. Given a paired input of speech X1:T and text Y1:N , the speech tokenizer arXiv:2507.05911v1 [cs.SD] 8 Jul 2025 Figure 1: Comparis...

    arXiv 2025

  3. [3]

    Figure 1 shows the difference between the DiffRO and the existing RL method like DPO

    Differentiable Reward Optimization This section will introduce the DiffRO, which can further sim- plify the RL training process and can provide feedback on different aspects. Figure 1 shows the difference between the DiffRO and the existing RL method like DPO. 3.1. Token2Reward Prediction Unlike previous work, DiffRO directly predicts the reward from the ...

    work page

  4. [4]

    Experimental Setup

    Experiments 4.1. Experimental Setup. 4.1.1. DataSet and Baseline TTS system We use the Cosyvoice2.0-0.5B [13] model as the baseline sys- tem and fine-tune it with our in-home target speaker data as the SFT baseline. The SFT data contains 4000 audio samples from 5 speakers (4 female and 1 male) and all of them speak in Man- darin. For the RL data, we colle...

    work page

  5. [5]

    Compared to other reinforcement learning methods, DiffRO is capable of directly predicting reward scores from speech tokens rather than from synthesized audio

    Conclusions In this paper, we propose a novel DiffRO based reinforcement learning strategy for neural codec LM based text-to-speech systems. Compared to other reinforcement learning methods, DiffRO is capable of directly predicting reward scores from speech tokens rather than from synthesized audio. Moreover, it allows for the direct optimization of LM pa...

    work page

  6. [6]

    Denoising diffusion probabilistic models,

    J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” in Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, Eds., vol. 33. Curran Associates, Inc., 2020, pp. 6840–6851

    work page 2020

  7. [7]

    Neural codec language models are zero-shot text to speech synthesizers,

    S. Chen, C. Wang, Y . Wu, Z. Zhang, L. Zhou, S. Liu, Z. Chen, Y . Liu, H. Wang, J. Li, L. He, S. Zhao, and F. Wei, “Neural codec language models are zero-shot text to speech synthesizers,” IEEE Transactions on Audio, Speech and Language Processing, vol. 33, pp. 705–718, 2025

    work page 2025

  8. [8]

    Speak foreign languages with your own voice: Cross-lingual neural codec language modeling,

    Z. Zhang, L. Zhou, C. Wang, S. Chen, Y . Wu, S. Liu, Z. Chen, Y . Liu, H. Wang, J. Li, L. He, S. Zhao, and F. Wei, “Speak foreign languages with your own voice: Cross-lingual neural codec language modeling,” 2023. [Online]. Available: https://arxiv.org/abs/2303.03926

    arXiv 2023

Show all 34 references
  1. [9]

    Fine-tuning language models from human preferences,

    D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irving, “Fine-tuning language models from human preferences,” 2020. [Online]. Available: https://arxiv.org/abs/1909.08593

    2020 arXiv

  2. [10]

    Baton: Align- ing text-to-audio model with human preference feedback,

    H. Liao, H. Han, K. Yang, T. Du, R. Yang, Z. Xu, Q. Xu, J. Liu, J. Lu, and X. Li, “Baton: Align- ing text-to-audio model with human preference feedback,” ArXiv, vol. abs/2402.00744, 2024. [Online]. Available: https://api.semanticscholar.org/CorpusID:267365170

    2024 arXiv

  3. [11]

    Enhancing Zero-shot Text-to-Speech Synthesis with Human Feedback,

    C. Chen, Y . Hu, W. Wu, H. Wang, E. S. Chng, and C. Zhang, “Enhancing Zero-shot Text-to-Speech Synthesis with Human Feedback,” Jun. 2024, arXiv:2406.00654. [Online]. Available: http://arxiv.org/abs/2406.00654

    2024 arXiv

  4. [12]

    Robust zero- shot text-to-speech synthesis with reverse inference optimization,

    Y . Hu, C. Chen, S. Wang, E. S. Chng, and C. Zhang, “Robust zero- shot text-to-speech synthesis with reverse inference optimization,”

  5. [13]

    Cosyvoice 2: Scalable streaming speech synthesis with large language models,

    Z. Du, Y . Wang, Q. Chen, X. Shi, X. Lv, T. Zhao, Z. Gao, Y . Yang, C. Gao, H. Wang, F. Yu, H. Liu, Z. Sheng, Y . Gu, C. Deng, W. Wang, S. Zhang, Z. Yan, and J. Zhou, “Cosyvoice 2: Scalable streaming speech synthesis with large language models,”

  6. [14]

    Seed- TTS: A Family of High-Quality Versatile Speech Generation Models,

    P. Anastassiou, J. Chen, J. Chen, Y . Chen, Z. Chen, Z. Chen, J. Cong, L. Deng, C. Ding, L. Gao, M. Gong, P. Huang, Q. Huang, Z. Huang, Y . Huo, D. Jia, C. Li, F. Li, H. Li, J. Li, X. Li, X. Li, L. Liu, S. Liu, S. Liu, X. Liu, Y . Liu, Z. Liu, L. Lu, J. Pan, X. Wang, Y . Wang,...

    2024 arXiv

  7. [15]

    Emo-DPO: Controllable Emotional Speech Synthesis through Direct Preference Optimization,

    X. Gao, C. Zhang, Y . Chen, H. Zhang, and N. F. Chen, “Emo-DPO: Controllable Emotional Speech Synthesis through Direct Preference Optimization,” Sep. 2024, arXiv:2409.10157 [eess]. [Online]. Available: http://arxiv.org/abs/2409.10157

    2024 arXiv

  8. [16]

    Proximal Policy Optimization Algorithms,

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal Policy Optimization Algorithms,” Aug. 2017, arXiv:1707.06347 [cs]. [Online]. Available: http://arxiv.org/abs/1707.06347

    2017 arXiv

  9. [17]

    Direct Preference Optimization: Your Language Model is Secretly a Reward Model,

    R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, and C. Finn, “Direct Preference Optimization: Your Language Model is Secretly a Reward Model,” May 2023. [Online]. Available: https://arxiv.org/abs/2305.18290v3

    2023 arXiv

  10. [18]

    Asq: An ultra-low bit rate asr-oriented speech quantization method,

    L. Ye, C. Gao, G. Cheng, L. Luo, and Q. Zhao, “Asq: An ultra-low bit rate asr-oriented speech quantization method,” IEEE Signal Processing Letters, vol. 31, pp. 221–225, 2024

    2024

  11. [19]

    Air-bench: Automated heterogeneous information retrieval benchmark,

    J. Chen, N. Wang, C. Li, B. Wang, S. Xiao, H. Xiao, H. Liao, D. Lian, and Z. Liu, “Air-bench: Automated heterogeneous information retrieval benchmark,” 2024. [Online]. Available: https://arxiv.org/abs/2412.13102

    2024 arXiv

  12. [20]

    Available: https://arxiv.org/abs/2412.10117

    [Online]. Available: https://arxiv.org/abs/2412.10117

    arXiv

  13. [21]

    Torchaudio-squim: Reference-less speech quality and intelligibility measures in torchaudio,

    A. Kumar, K. Tan, Z. Ni, P. Manocha, X. Zhang, E. Henderson, and B. Xu, “Torchaudio-squim: Reference-less speech quality and intelligibility measures in torchaudio,” ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1–5, ...

    2023

  14. [22]

    Panns: Large-scale pretrained audio neural networks for audio pattern recognition,

    Q. Kong, Y . Cao, T. Iqbal, Y . Wang, W. Wang, and M. D. Plumbley, “Panns: Large-scale pretrained audio neural networks for audio pattern recognition,” IEEE/ACM Transactions on Audio, Speech, and Language Process- ing, vol. 28, pp. 2880–2894, 2019. [Online]. Available: https:/...

    2019

  15. [23]

    Common voice: A massively-multilingual speech corpus,

    R. Ardila, M. Branson, K. Davis, M. Henretty, M. Kohler, J. Meyer, R. Morais, L. Saunders, F. M. Tyers, and G. Weber, “Common voice: A massively-multilingual speech corpus,” in Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), 2020, pp. 4211–4215

    2020

  16. [24]

    The voicemos challenge 2022,

    W. C. Huang, E. Cooper, Y . Tsao, H.-M. Wang, T. Toda, and J. Ya- magishi, “The voicemos challenge 2022,” in Interspeech 2022, 2022, pp. 4536–4540

    2022

  17. [25]

    Iemocap: Interactive emotional dyadic motion capture database,

    C. Busso, M. Bulut, C.-C. Lee, A. Kazemzadeh, E. Mower, S. Kim, J. N. Chang, S. Lee, and S. S. Narayanan, “Iemocap: Interactive emotional dyadic motion capture database,”Language resources and evaluation, vol. 42, pp. 335–359, 2008

    2008

  18. [26]

    Robust speech recognition via large-scale weak supervision,

    A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” 2022. [Online]. Available: https://arxiv.org/abs/2212.04356

    2022 arXiv

  19. [27]

    Funaudiollm: V oice understanding and generation foundation models for natural interaction between humans and llms,

    K. An, Q. Chen, C. Deng, Z. Du, C. Gao, Z. Gao, Y . Gu, T. He, H. Hu, K. Hu, S. Ji, Y . Li, Z. Li, H. Lu, H. Luo, X. Lv, B. Ma, Z. Ma, C. Ni, C. Song, J. Shi, X. Shi, H. Wang, W. Wang, Y . Wang, Z. Xiao, Z. Yan, Y . Yang, B. Zhang, Q. Zhang, S. Zhang, N. Zhao, and S. Zheng, “F...

    2024 arXiv

  20. [28]

    Dnsmos p.835: A non- intrusive perceptual objective speech quality metric to evaluate noise suppressors,

    C. K. Reddy, V . Gopal, and R. Cutler, “Dnsmos p.835: A non- intrusive perceptual objective speech quality metric to evaluate noise suppressors,” in ICASSP 2022 IEEE International Con- ference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2022

    2022

  21. [29]

    Emobox: Multilingual multi-corpus speech emotion recognition toolkit and benchmark,

    Z. Ma, M. Chen, H. Zhang, Z. Zheng, W. Chen, X. Li, J. Ye, X. Chen, and T. Hain, “Emobox: Multilingual multi-corpus speech emotion recognition toolkit and benchmark,” 2024. [Online]. Available: https://arxiv.org/abs/2406.07162

    2024 arXiv

  22. [30]

    Qwen-audio: Advancing universal audio understand- ing via unified large-scale audio-language models,

    Y . Chu, J. Xu, X. Zhou, Q. Yang, S. Zhang, Z. Yan, C. Zhou, and J. Zhou, “Qwen-audio: Advancing universal audio understand- ing via unified large-scale audio-language models,”arXiv preprint arXiv:2311.07919, 2023

    2023 arXiv

  23. [31]

    Minmo: A multimodal large language model for seamless voice interaction,

    Q. Chen, Y . Chen, Y . Chen, M. Chen, Y . Chen, C. Deng, Z. Du, R. Gao, C. Gao, Z. Gao, Y . Li, X. Lv, J. Liu, H. Luo, B. Ma, C. Ni, X. Shi, J. Tang, H. Wang, H. Wang, W. Wang, Y . Wang, Y . Xu, F. Yu, Z. Yan, Y . Yang, B. Yang, X. Yang, G. Yang, T. Zhao, Q. Zhang, S. Zhang, N...

    2025 arXiv

  24. [32]

    Paraformer: Fast and accurate parallel transformer for non-autoregressive end-to- end speech recognition,

    Z. Gao, S. Zhang, I. McLoughlin, and Z. Yan, “Paraformer: Fast and accurate parallel transformer for non-autoregressive end-to- end speech recognition,” in Proc. Interspeech 2022 , 2022, pp. 2063–2067

    2022

  25. [34]

    F5-tts: A fairytaler that fakes fluent and faithful speech with flow matching,

    Y . Chen, Z. Niu, Z. Ma, K. Deng, C. Wang, J. Zhao, K. Yu, and X. Chen, “F5-tts: A fairytaler that fakes fluent and faithful speech with flow matching,” arXiv preprint arXiv:2410.06885, 2024

    2024 arXiv

  26. [2024]

    Available: https://arxiv.org/abs/2407.02243

    [Online]. Available: https://arxiv.org/abs/2407.02243

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.