Pith. sign in

REVIEW 4 major objections 6 minor 48 references

Audio Turing Test: Benchmarking the Human-likeness of Large Language Model-based Text-to-Speech Systems in Chinese

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A Turing-style listening test shows that even the strongest LLM-based Chinese TTS system, Seed-TTS, scores only about 0.4 on human-likeness, well below real human speech.

desk verdict ATT is a genuinely useful Chinese TTS evaluation benchmark; trust the human-study results, but treat the Auto-ATT 'strong alignment' claim as unverified until it is shown on held-out voices. read the letter →

arxiv 2505.11200 v1 pith:HJSVNFU7 submitted 2025-05-16 cs.SD cs.AIcs.CLcs.HCcs.LGeess.AS

Xihuai Wang , Ziyi Zhao , Siyu Ren , Shao Zhang , Song Li , Xiaoyu Li , Ziwen Wang , Lin Qiu
show 4 more authors
Guanglu Wan Xuezhi Cao Xunliang Cai Weinan Zhang
This is my paper · ORCID
classification cs.SDcs.AIcs.CLcs.HCcs.LGeess.AS
keywords text-to-speechevaluationTuringtesthuman-likenessscoreChinesespeechsynthesismeanopinionautomaticQwen2-Audiocode-switching
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes the Audio Turing Test (ATT), a Chinese-language benchmark that replaces the five-point Mean Opinion Score with a simple question: does this voice sound human? It contributes a multi-dimensional corpus (ATT-Corpus) covering code-switching, polyphonic characters, paralinguistic emotion, classical poetry, and numerals, plus a protocol in which listeners label clips Human, Unclear, or Machine and must pass hidden trap items. The central result is that ATT cleanly separates five state-of-the-art LLM-based TTS systems, and that the best system, Seed-TTS, reaches a Human-likeness Score of only about 0.4, far below what the paper expects of real human speech. The paper also trains Auto-ATT, an automatic judge built on Qwen2-Audio-Instruct, and reports that it agrees with human rankings and detects flawed synthetic clips far better than existing MOS predictors. If right, ATT gives the field a more sensitive, interpretable yardstick for the remaining gap between synthetic and human voices.

What carries the argument

The load-bearing object is the Human-likeness Score (HLS), defined as the average over clips of $s_i = \mathbf{1}(\text{Label}=\text{Human}) + 0.5\,\mathbf{1}(\text{Label}=\text{Unclear})$, with Machine scored 0. The measurement apparatus is ATT-Corpus, a semi-automatically built corpus spanning five Chinese linguistic difficulty dimensions, combined with a protocol that inserts one flawed synthetic clip and two genuine human recordings into every ten clips and discards batches that miss them. Auto-ATT carries the automatic-evaluation half: Qwen2-Audio-Instruct is adapted by LoRA, and its output logits for the tokens Human, Unclear, and Machine are softmaxed and converted into a weighted score, trained with a combination of Bradley-Terry and mean-squared-error losses.

What would settle it

Two checks would settle the central claims. First, compute the HLS of the genuine human recordings already used as trap items: if their score is not clearly above Seed-TTS's 0.4, the 'considerably lower than real human speech' claim fails. Second, retrain Auto-ATT on three voices per model and evaluate only on the held-out fourth voice; if the Kendall-tau agreement with human rankings drops to chance, Auto-ATT is not a general judge of new voices.

Watch

Extended reading notes

Core claim

The paper's claim is that human-likeness in TTS can be measured directly by asking listeners whether synthesized speech is human, and that this simpler ternary judgment yields sharper distinctions than MOS. Under ATT, a clip earns 1 point for a Human label and 0.5 for Unclear, and a system's Human-likeness Score is the average across clips. Evaluated on 20 voice styles from five model families, Seed-TTS ranks first at roughly 0.4, MiniMax-Speech follows near 0.39, Step-Audio and CosyVoice sit around 0.22-0.29, and GPT-4o lags at 0.13; each model retains its rank in a held-out black-box split. The paper argues this ordering, and the finding that no system comes close to a score of 1, shows ATT exposes a large gap that MOS-style scores hide. In addition, the fine-tuned Auto-ATT judge reproduces the human voice ranking (Kendall distances around 0.27-0.34 across dimensions) and scores trap items with F1 of 0.92, while UTMOSv2 scores 0.14 and DNSMOS Pro 0.00 at the same threshold.

Load-bearing premise

The claim that Auto-ATT reliably predicts human judgments assumes that the voices it is scored on are not the voices it was trained on; the paper trains Auto-ATT on three voices per model family but reports agreement on audio that includes all four voices, so the agreement numbers could partly reflect in-sample familiarity.

Editorial extensions

If this is right

  • TTS developers can use ATT-Corpus and the HLS protocol to compare systems on specific weaknesses: Seed-TTS is strongest at code-switching and numerals but falls behind MiniMax-Speech on classical Chinese prose.
  • Auto-ATT provides a fast proxy that ranks voices in nearly the same order as human listeners, so model iteration no longer has to wait for crowdsourced listening tests.
  • Because the black-box and white-box splits give the same model ordering, published white-box results can be read as a fair preview of blind evaluation outcomes.
  • The qualitative justifications collected by the protocol locate the common failure modes, such as abrupt prosody, missing micro-pauses, flattened emotion, and artifacts like foreign accent and hiss, giving system designers concrete targets.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implicit consequence is that published MOS claims that modern TTS is nearly indistinguishable from human speech may be systematically over-optimistic; the paper's 0.4 ceiling suggests the metric rather than the technology deserves scrutiny.
  • A testable extension would be to report HLS for genuine human recordings under the same protocol as a calibration anchor; the paper uses human clips as trap items but does not publish their HLS, leaving the size of that gap to be quantified.
  • The Auto-ATT results would generalize more convincingly if the Kendall-tau comparison were rerun on voices never seen in training; the paper reserves one voice per family for testing but evaluates on the same audio as the human study, so part of the agreement may reflect familiar voices.
  • If the ATT protocol transfers to other languages, the same ternary-judgment design with trap items could become a common currency for cross-lingual human-likeness benchmarks, with Auto-ATT's zero-shot cross-lingual transfer as a natural next test.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes the Audio Turing Test (ATT), a Chinese TTS human-likeness evaluation framework consisting of a five-dimension corpus (ATT-Corpus), a Turing-test-style human listening protocol with trap items and free-text justifications, and a Human-likeness Score (HLS) computed from ternary Human/Unclear/Machine labels. The authors collect judgments from 857 native Chinese listeners for 20 voices across five TTS systems, report model-level HLS rankings, and claim that even the best system (Seed-TTS) reaches only about 0.4 HLS, far below real human speech. They also fine-tune Qwen2-Audio-Instruct on human labels to create Auto-ATT, a model-as-a-judge automatic evaluator, and evaluate it on trap items and on agreement with human rankings. The main human benchmark protocol is carefully designed, but the Auto-ATT generalization claim and some central quantitative comparisons are not fully supported by the reported experiments.

Significance. If the claims hold, ATT would be a valuable contribution to TTS evaluation: it addresses genuine limitations of MOS by using a simple forced-choice human-likeness judgment, includes trap items for attention screening, applies expert consistency checks, and provides a reproducible white-box corpus. The multi-dimensional design and the finding that leading LLM-based TTS systems still fall substantially short of human speech on a human-likeness criterion are important and potentially impactful results. The machine-checkable strengths include the detailed protocol, the GLMM convergence diagnostics, the qualitative attribution coding, and the public release of the corpus and tools. However, the strongest auxiliary claim, that Auto-ATT is a fast and reliable automatic judge with strong alignment to human evaluations, is weakened by the absence of held-out-voice results and by the moderate reported agreement; the headline comparison with human speech also lacks a measured human-reference HLS. These issues are fixable with additional analyses rather than being fundamental to the ATT human benchmark itself.

major comments (4)
  1. [Section 3.5 and Section 4.2.2, Table 4] The claim that Auto-ATT shows strong alignment with human evaluations and is reliable for new voices is not supported by the reported experiment. Section 3.5 states that within each of four model families, one voice was reserved for testing and the remaining three voices were used for training, but Section 4.2.2 evaluates Auto-ATT on the same audio data as the human evaluation, which includes all four voices per family. At least 12 of the 20 voices ranked in Table 4 are therefore training voices, and GPT-4o voices are not separated in the reported agreement. The Kendall tau of 0.3316 for the All dimension is moderate, not strong, and the held-out-voice agreement is not reported. Please report the ranking agreement computed only on the reserved held-out voices, with confidence intervals, and restrict the generalization claim to what that analysis supports.
  2. [Section 4.1.2] The headline result that Seed-TTS reaches only about 0.4 HLS and is 'considerably lower than that of real human speech' lacks a measured human reference. Since HLS is defined from ternary labels, human recordings are not guaranteed to receive a score of 1.0; listeners may label genuine human clips as Unclear or Machine. The protocol includes human recordings as trap items, so the authors have the data to compute an HLS for real human speech. Please report the mean HLS and uncertainty for the human reference clips used in the evaluation, and use this measured value rather than the nominal value of 1.0 when claiming a large remaining gap between synthetic and human speech.
  3. [Table 4 caption and rows] The interpretation of Table 4 is internally inconsistent. The caption says 'Lower tau is better,' but Kendall's tau is a rank correlation coefficient for which higher values indicate better agreement, and the p-value column is consistent with testing a correlation. If the authors instead intend a Kendall distance or discordant-pair proportion, that quantity and its reference should be defined explicitly. In addition, the 'All' row reports exactly the same value as the 'Polyphonic Characters' row (0.3316, p=0.0398); an aggregate over all dimensions would not be expected to coincide exactly with one subset, so this appears to be a reporting error that should be corrected.
  4. [Section 4.2.1, Figure 3] The comparison with UTMOSv2 and DNSMOS Pro on trap items is not calibrated fairly. The figure normalizes all predictions to a 0-1 scale and applies the same 0.5 decision threshold to models that were not designed to output ternary human-likeness probabilities; MOS predictors are not trained to separate human speech from deliberately flawed synthetic speech on this scale, so an F1 score at an arbitrary threshold does not establish that Auto-ATT is intrinsically superior. Please report threshold-free measures such as AUROC or area under the precision-recall curve, and show the raw score distributions for all three models on the trap items.
minor comments (6)
  1. [Section 3.5 and Table 4] The description of Auto-ATT training data is inconsistent with the in-distribution/out-of-distribution split in Table 4. Section 3.5 says training focuses on 'three capability subsets' but lists four phenomena and does not clearly state whether Special Characters and Numerals was included; Table 4 labels Special Characters and Numerals as an in-distribution dimension. Please reconcile the number of subsets and the dimension names.
  2. [Section 3.3 and Section 3 text] There are several typographical errors: 'shownshown' in Section 3.3, 'ofof' in the paragraph introducing Section 3, and 'choral quality' in the Related Works section, which should likely be 'vocal quality.'
  3. [Section 4.2.1] The name of the DNSMOS baseline is inconsistent: the text refers to 'DMSMOS Pro' in one place and 'DNSMOSPro' in another, while the figure uses 'DNSMOS Pro.' Please use a single consistent name matching the cited reference.
  4. [Figure 2 and Table 6] Figure 2 shows point estimates without error bars, and Table 6 gives voice-level HLS values to four decimal places without uncertainty intervals. Since the text draws conclusions about differences between voice styles within a model (for example Seed-TTS 'Skye' at 0.47 versus lower-ranked voices), please provide confidence intervals or posterior intervals for these per-dimension and per-voice estimates.
  5. [Table 6] The voice style names are inconsistent between Table 2 and Table 6: Table 2 lists 'Sky' for Seed-TTS while Table 6 uses 'skye,' and MiniMax names such as 'siyuan' and 'xinyue' in Table 6 do not match the platform-style names in Table 2. Please harmonize the naming and clarify whether these are the same voice styles.
  6. [References] The text refers to 'Kendall's distance [1],' but the cited reference is titled 'The Kendall rank correlation coefficient.' Please use the correct terminology or cite a genuine distance measure, and ensure the statistic reported in Table 4 matches the definition used.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: ATT's human-likeness metric is a direct empirical aggregate, and Auto-ATT is not shown to be validated on its own training data.

full rationale

The central ATT human benchmark is self-contained. HLS is defined directly from independently collected ternary human labels (Section 3.4), and the between-model differences in Figure 2 are empirical observations from 857 listeners, not outputs of any fitted model. Auto-ATT's prediction formula (Section 3.5, Eq. 1) uses the same 1/0.5/0 weights as HLS, but that is the intended regression target, not a circular reduction: the model is trained to predict HLS and then compared with human judgments. The paper states that Auto-ATT was trained on 'additional private evaluation data' using 'internal TTS systems' (Section 3.5), while Section 4.2.2 evaluates on 'the same audio data as the human evaluation' from the public benchmark (Section 4.1). Nothing in the text equates the internal training voices with the five public model families in Table 2, so the reported Kendall agreement is not shown to be an in-sample fit. A reporting caveat is that Section 4.2.2 does not break out agreement on held-out voices only, which would strengthen the generalization claim, but this is a reporting limitation rather than a circular step. The trap-item comparison (F1=0.92 vs. UTMOSv2's 0.14 and DNSMOS's 0.00) provides an additional external check. No self-citation chains, imported uniqueness theorems, or definitional equivalences that force the paper's conclusions were found.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central HLS metric depends on hand-chosen label weights and trap-item thresholds. The Auto-ATT evaluation depends on the assumption that the token logits correspond to meaningful human-likeness judgments, and on the five corpus dimensions being representative. No new physical entities are introduced.

free parameters (4)
  • HLS label weights = Human: 1, Unclear: 0.5, Machine: 0
    The 0.5 weight for 'Unclear' is an arbitrary design choice that directly defines the central metric; changing it changes all model rankings.
  • Trap-item validation rule = 1 failed synthetic + 2 human recordings per 10 clips; require correct on failed synthetic and at least 1 human
    The specific trap ratio and pass threshold are chosen by hand and affect which participant responses are included in every reported score.
  • Auto-ATT loss weights = 0.4 BT, 0.6 MSE
    The combination and weights of Bradley-Terry and MSE losses are chosen without sensitivity analysis, yet they determine the trained Auto-ATT's predictions.
  • LoRA hyperparameters = rank 32, alpha 32, dropout 0.05
    These hyperparameters are stated but no ablation is provided; they affect Auto-ATT performance.
assumptions (4)
  • domain assumption Native Chinese crowdsourced listeners' human/machine judgments are the ground truth for human-likeness.
    The entire ATT framework treats these judgments as the target metric, without validating against any external criterion of human-likeness.
  • domain assumption Trap items (one flawed synthetic and two human recordings per 10 clips) are unambiguously identifiable by attentive listeners.
    The protocol assumes the flawed synthetic clip is clearly defective and the human recordings are clearly human; failure on these is used to discard responses.
  • ad hoc to paper Qwen2-Audio-Instruct's logits for the tokens 'Human', 'Unclear', and 'Machine' at the final prompt position carry meaningful signal for HLS regression.
    The Auto-ATT design extracts probabilities from these three tokens and treats them as judgment probabilities; this is an architectural choice specific to this paper.
  • domain assumption The five selected corpus dimensions are representative of the capabilities that matter for Chinese TTS human-likeness.
    No evidence is given that these five dimensions cover the distribution of real-world Chinese TTS usage.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Audio Turing Test: Benchmarking the Human-likeness of Large Language Model-based Text-to-Speech Systems in Chinese." pith.science (2026). https://pith.science/paper/HJSVNFU7

@misc{pith2026250511200,
  author       = {Pith},
  title        = {Pith review of: Audio Turing Test: Benchmarking the Human-likeness of Large Language Model-based Text-to-Speech Systems in Chinese},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HJSVNFU7}},
  note         = {Machine review of arXiv:2505.11200}
}
read the original abstract

Recent advances in large language models (LLMs) have significantly improved text-to-speech (TTS) systems, enhancing control over speech style, naturalness, and emotional expression, which brings TTS Systems closer to human-level performance. Although the Mean Opinion Score (MOS) remains the standard for TTS System evaluation, it suffers from subjectivity, environmental inconsistencies, and limited interpretability. Existing evaluation datasets also lack a multi-dimensional design, often neglecting factors such as speaking styles, context diversity, and trap utterances, which is particularly evident in Chinese TTS evaluation. To address these challenges, we introduce the Audio Turing Test (ATT), a multi-dimensional Chinese corpus dataset ATT-Corpus paired with a simple, Turing-Test-inspired evaluation protocol. Instead of relying on complex MOS scales or direct model comparisons, ATT asks evaluators to judge whether a voice sounds human. This simplification reduces rating bias and improves evaluation robustness. To further support rapid model development, we also finetune Qwen2-Audio-Instruct with human judgment data as Auto-ATT for automatic evaluation. Experimental results show that ATT effectively differentiates models across specific capability dimensions using its multi-dimensional design. Auto-ATT also demonstrates strong alignment with human evaluations, confirming its value as a fast and reliable assessment tool. The white-box ATT-Corpus and Auto-ATT can be found in ATT Hugging Face Collection (https://huggingface.co/collections/meituan/audio-turing-test-682446320368164faeaf38a4).

Figures

Figures reproduced from arXiv: 2505.11200 by the authors.

Figure 1
Figure 1. Audio Turing Test Evaluation Framework: (1) Corpus Generation: a semi-automatic corpus generation pipeline for generating the challenge TTS synthesis corpus for ATT evaluation; (2) Human Evaluation: a human-evaluation protocol that enables precise, comparable assessments and lowers evaluation costs through a simple yet effective Turing-test-style design, (3) Automatic Evaluation: Auto-ATT, an automatic tool to predi… view at source ↗
Figure 2
Figure 2. The Key Benchmark Results of ATT Human Evaluation. 4.1.2 Benchmarking via Human Evaluation Effectiveness of ATT. As shown in [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. The prediction results of Trap Item through DMSMOS Pro, UTMOSv2, and Auto￾ATT. For a human speech clip, the ideal outcome is a true positive: the red dot should fall within the red zone; for a flawed synthetic speech clip, the ideal outcome is a true negative: the gray dot should fall within the gray zone. 4.2.1 Comparison with Other Auto Evaluation in Trap Item To evaluate model reliability, we conduct experiments … view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: The Screen of One Audio Clip in ATT Evaluation. [PITH_FULL_IMAGE:figures/full_fig_p016_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

48 extracted references · 32 canonical work pages

  1. [1]

    The kendall rank correlation coefficient

    Hervé Abdi. The kendall rank correlation coefficient. Encyclopedia of measurement and statistics, 2:508–510, 2007

    work page 2007

  2. [2]

    Seed-tts: A family of high-quality versatile speech generation models

    Philip Anastassiou, Jiawei Chen, Jitong Chen, Yuanzhe Chen, Zhuo Chen, Ziyi Chen, Jian Cong, Lelai Deng, Chuang Ding, Lu Gao, et al. Seed-tts: A family of high-quality versatile speech generation models. arXiv preprint arXiv:2406.02430, 2024

    arXiv 2024

  3. [3]

    The t05 system for the voicemos challenge 2024: Transfer learning from deep image classifier to naturalness mos prediction of high-quality synthetic speech

    Kaito Baba, Wataru Nakata, Yuki Saito, and Hiroshi Saruwatari. The t05 system for the voicemos challenge 2024: Transfer learning from deep image classifier to naturalness mos prediction of high-quality synthetic speech. In Proc. IEEE Spoken Language Technology Workshop (SLT),

    work page 2024

  4. [4]

    Generalized linear mixed models: a practical guide for ecology and evolution

    Benjamin M Bolker, Mollie E Brooks, Connie J Clark, Shane W Geange, John R Poulsen, M Henry H Stevens, and Jada-Simone S White. Generalized linear mixed models: a practical guide for ecology and evolution. Trends in ecology & evolution, 24(3):127–135, 2009

    work page 2009

  5. [5]

    Why we should report the details in subjective evaluation of tts more rigorously

    Cheng-Han Chiang, Wei-Ping Huang, and Hung yi Lee. Why we should report the details in subjective evaluation of tts more rigorously. In Interspeech 2023, pages 5551–5555, 2023. doi: 10.21437/Interspeech.2023-416

    work page doi:10.21437/interspeech.2023-416 2023

  6. [6]

    Qwen2-audio technical report

    Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuan- jun Lv, Jinzheng He, Junyang Lin, et al. Qwen2-audio technical report. arXiv preprint arXiv:2407.10759, 2024

    arXiv 2024

  7. [7]

    Fredrik Cumlin, Xinyu Liang, Victor Ungureanu, Chandan K. A. Reddy, Christian Schüldt, and Saikat Chatterjee. Dnsmos pro: A reduced-size dnn for probabilistic mos of speech. In Interspeech 2024, pages 4818–4822, 2024. doi: 10.21437/Interspeech.2024-478

    doi:10.21437/interspeech.2024-478 2024

  8. [8]

    Disambiguation of chinese polyphones in an end-to-end framework with semantic features extracted by pre-trained bert

    Dongyang Dai, Zhiyong Wu, Shiyin Kang, Xixin Wu, Jia Jia, Dan Su, Dong Yu, and Helen Meng. Disambiguation of chinese polyphones in an end-to-end framework with semantic features extracted by pre-trained bert. arXiv preprint arXiv:2501.01102, 2025

    arXiv 2025

Show all 48 references
  1. [9]

    Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens

    Zhihao Du, Qian Chen, Shiliang Zhang, Kai Hu, Heng Lu, Yexin Yang, Hangrui Hu, Siqi Zheng, Yue Gu, Ziyang Ma, et al. Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens. arXiv preprint arXiv:2407.05407, 2024

    2024 arXiv

  2. [10]

    Assessing the impact of contextual framing on subjective tts quality

    Jens Edlund, Christina Tånnander, Sébastien Le Maguer, and Petra Wagner. Assessing the impact of contextual framing on subjective tts quality. In25th Interspeech Conferece 2024, Kos Island, Greece, Sep 1 2024-Sep 5 2024, pages 1205–1209. International Speech Communication Asso...

    2024

  3. [11]

    The turing test: the first 50 years

    Robert M French. The turing test: the first 50 years. Trends in cognitive sciences, 4(3):115–122, 2000

    2000

  4. [12]

    Analysis of speaker similarity in the statistical speech synthesis systems using a hybrid approach

    Ekrem Guner, Amir Mohammadi, and Cenk Demiroglu. Analysis of speaker similarity in the statistical speech synthesis systems using a hybrid approach. In 2012 Proceedings of the 20th European Signal Processing Conference (EUSIPCO), pages 2055–2059. IEEE, 2012

    2012

  5. [13]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  6. [14]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1 (2):3, 2022. 10

    2022

  7. [15]

    Step-audio: Unified understanding and generation in intelligent speech interaction

    Ailin Huang, Boyong Wu, Bruce Wang, Chao Yan, Chen Hu, Chengli Feng, Fei Tian, Feiyu Shen, Jingbei Li, Mingrui Chen, et al. Step-audio: Unified understanding and generation in intelligent speech interaction. arXiv preprint arXiv:2502.11946, 2025

    2025 arXiv

  8. [16]

    Mm algorithms for generalized bradley-terry models

    David R Hunter. Mm algorithms for generalized bradley-terry models. The annals of statistics, 32(1):384–406, 2004

    2004

  9. [17]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  10. [18]

    Method for the Subjective Assessment of Intermediate Quality Level of Audio Systems, 2015

    International Telecommunication Union. Method for the Subjective Assessment of Intermediate Quality Level of Audio Systems, 2015. MUltiple Stimulus with Hidden Reference and Anchor (MUSHRA)

    2015

  11. [19]

    Subjective evaluation of speech quality with a crowd- sourcing approach, 2018

    International Telecommunication Union. Subjective evaluation of speech quality with a crowd- sourcing approach, 2018

    2018

  12. [20]

    Compact neural tts voices for accessibility

    Kunal Jain, Eoin Murphy, Deepanshu Gupta, Jonathan Dyke, Saumya Shah, Vasilieios Tsiaras, Petko Petkov, and Alistair Conkie. Compact neural tts voices for accessibility. arXiv preprint arXiv:2501.17332, 2025

    2025 arXiv

  13. [21]

    Stuck in the mos pit: A critical analysis of mos test methodology in tts evaluation

    Ambika Kirkland, Shivam Mehta, Harm Lameris, Gustav Eje Henter, Eva Székely, and Joakim Gustafson. Stuck in the mos pit: A critical analysis of mos test methodology in tts evaluation. In 12th Speech Synthesis Workshop (SSW) 2023, 2023

    2023

  14. [22]

    Issues in chinese prosody: conceptual foundations of a linguistically-motivated text-to-speech system for mandarin

    Richard S Lavin. Issues in chinese prosody: conceptual foundations of a linguistically-motivated text-to-speech system for mandarin. In Language, Information, and Computation: Proceedings of The 16th Pacific Asia Conference: January 31-February 2 2002, Jeju, Korea, pages 259–2...

    2002

  15. [23]

    The limits of the mean opinion score for speech synthesis evaluation

    Sébastien Le Maguer, Simon King, and Naomi Harte. The limits of the mean opinion score for speech synthesis evaluation. Computer Speech & Language, 84:101577, 2024. ISSN 0885-

    2024

  16. [24]

    Styletts-zs: Efficient high- quality zero-shot text-to-speech synthesis with distilled time-varying style diffusion

    Yinghao Aaron Li, Xilin Jiang, Cong Han, and Nima Mesgarani. Styletts-zs: Efficient high- quality zero-shot text-to-speech synthesis with distilled time-varying style diffusion. arXiv preprint arXiv:2409.10058, 2024

    2024 arXiv

  17. [25]

    Hyper-realistic, multi-emotion generative speech model speech-01

    MiniMax. Hyper-realistic, multi-emotion generative speech model speech-01. https://www. minimax.io/news/speech-01, 2025

    2025

  18. [26]

    Speech quality assessment in crowdsourcing: Comparison category rating method

    Babak Naderi, Sebastian Möller, and Ross Cutler. Speech quality assessment in crowdsourcing: Comparison category rating method. arXiv preprint arXiv:2104.04371, 2021

    2021 arXiv

  19. [27]

    The blizzard challenge

    Olivier Perrotin, Brooke Stephenson, Silvain Gerber, and Gérard Bailly. The blizzard challenge

  20. [28]

    Dnsmos p

    Chandan KA Reddy, Vishak Gopal, and Ross Cutler. Dnsmos p. 835: A non-intrusive perceptual objective speech quality metric to evaluate noise suppressors. In ICASSP 2022-2022 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 886–890. IEEE, 2022

    2022

  21. [29]

    Utmos: Utokyo-sarulab system for voicemos challenge 2022

    Takaaki Saeki, Detai Xin, Wataru Nakata, Tomoki Koriyama, Shinnosuke Takamichi, and Hiroshi Saruwatari. Utmos: Utokyo-sarulab system for voicemos challenge 2022. arXiv preprint arXiv:2204.02152, 2022

    2022 arXiv

  22. [30]

    Mean opinion score (mos) revisited: methods and applications, limitations and alternatives

    Robert C Streijl, Stefan Winkler, and David S Hands. Mean opinion score (mos) revisited: methods and applications, limitations and alternatives. Multimedia Systems, 22(2):213–227, 2016. 11

    2016

  23. [31]

    Rethinking mushra: Addressing modern challenges in text-to-speech evaluation

    Praveen Srinivasa Varadhan, Amogh Gulati, Ashwin Sankar, Srija Anand, Anirudh Gupta, Anirudh Mukherjee, Shiva Kumar Marepally, Ankur Bhatia, Saloni Jaju, Suvrat Bhooshan, et al. Rethinking mushra: Addressing modern challenges in text-to-speech evaluation. arXiv preprint arXiv:...

    2024 arXiv

  24. [32]

    Neural codec language models are zero-shot text to speech synthesizers

    Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al. Neural codec language models are zero-shot text to speech synthesizers. arXiv preprint arXiv:2301.02111, 2023

    2023 arXiv

  25. [33]

    Contextual interactive evaluation of tts models in dialogue systems

    Siyang Wang, Eva Székely, and Joakim Gustafson. Contextual interactive evaluation of tts models in dialogue systems. In 25th Interspeech Conferece 2024, Kos Island, Greece, Sep 1 2024-Sep 5 2024, pages 2965–2969. International Speech Communication Association, 2024

    2024

  26. [34]

    Spark-tts: An efficient llm-based text-to-speech model with single-stream decoupled speech tokens

    Xinsheng Wang, Mingqi Jiang, Ziyang Ma, Ziyu Zhang, Songxiang Liu, Linqin Li, Zheng Liang, Qixi Zheng, Rui Wang, Xiaoqin Feng, et al. Spark-tts: An efficient llm-based text-to-speech model with single-stream decoupled speech tokens. arXiv preprint arXiv:2503.01710, 2025

    2025 arXiv

  27. [35]

    Bilingual and code- switching tts enhanced with denoising diffusion model and gan

    Huai-Zhe Yang, Chia-Ping Chen, Shan-Yun He, and Cheng-Ruei Li. Bilingual and code- switching tts enhanced with denoising diffusion model and gan. In Proc. Interspeech 2024, pages 4938–4942, 2024

    2024

  28. [36]

    Talk2care: An llm-based voice assistant for communication between healthcare providers and older adults

    Ziqi Yang, Xuhai Xu, Bingsheng Yao, Ethan Rogers, Shao Zhang, Stephen Intille, Nawar Shara, Guodong Gordon Gao, and Dakuo Wang. Talk2care: An llm-based voice assistant for communication between healthcare providers and older adults. Proc. ACM Interact. Mob. Wearable Ubiquitous...

    2024 doi

  29. [37]

    Dialog modeling in audiobook synthesis

    Cheng-Chieh Yeh, Amirreza Shirani, Weicheng Zhang, Tuomo Raitio, Ramya Rasipuram, Ladan Golipour, and David Winarsky. Dialog modeling in audiobook synthesis. InICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 13341–13345...

    2024

  30. [40]

    orders of magnitude larger

    for an additional colloquial rewrite, using the prompt shown below: 将给出的文本改写为更加口语化，有沟通感的文本，并添加一定的背景及前后连贯信息， 你可以从以下的6个示例中获得灵感，但不允许照搬照抄，或者仿照句式，不允许用同样重 复的开头 示例1：原始：开始用 Notion这个app之后，发现它真的太强大了，不仅可以用来记笔记， 还能用来管理项目和计划，非常实用，简直是提高效率的利器呢。更改为：我跟你讲，我 最近在用Notion这个app，我的天我真的发现它真的很强，不仅可以用来记...

  31. [41]

    Pure machine voice

    " Pure machine voice ."

  32. [42]

    The i m i t a t i o n of human speech is too forced

    " The i m i t a t i o n of human speech is too forced ."

  33. [43]

    O b v i o u s l y a machine tone - doesn ’ t sound like a real person

    " O b v i o u s l y a machine tone - doesn ’ t sound like a real person ."

  34. [44]

    Sounds like a late - night radio host

    " Sounds like a late - night radio host ." Ex am pl es of good reasons ( A cc ur at e an al ys is that cites c on cre te details in the clip )

  35. [45]

    Many thins

    The phrase " Many thins " should end with a falling intonation , but here it rises - it sounds u n n a t u r a l

  36. [46]

    The clip is machine - g e n e r a t e d : each word pops out i n d i v i d u a l l y with poor flow

  37. [47]

    go away

    The phrase " go away " lacks the angry / i m p a t i e n t tone that should be present

  38. [48]

    angry ,

    After the word " angry ," the breath has a n o t i c e a b l e e l e c t r o n i c / robotic ar ti fa ct . And the user interface for the task are shown in Figure 4 with explanation in English. 15 Figure 4: The Screen of One Audio Clip in ATT Evaluation. B.3 Qualitative Analys...

  39. [2023]

    doi: 10.21437/Blizzard.2023-1

    In 18th Blizzard Challenge Workshop, pages 1–27, 2023. doi: 10.21437/Blizzard.2023-1

    2023 doi

  40. [2308]

    URL https://www.sciencedirect

    doi: https://doi.org/10.1016/j.csl.2023.101577. URL https://www.sciencedirect. com/science/article/pii/S0885230823000967

    2023

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.