{"id":"8ee80fb1-f2c8-4f42-9a00-73168fe1c4b2","arxiv_id":"2608.10606","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"ASR roundtrip evaluation masked 46 of 110 human-confirmed reading errors in Chinese news TTS, showing it is not reliable standalone ground truth.","lead":"A human-audited study of Chinese news text-to-speech found that ASR roundtrip checks can hide wrong readings of scores, models, and units. The authors release a benchmark and argue that human listening, not ASR transcripts, is needed to catch these errors.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 46-case masked count is not robustly established: it rests on one annotator's convention judgments with κ=0.634 on 30 cases and a pool selected using suspected-masking signals; the existence claim survives, the rate is not.","rationale":"The reader's weakest assumption is exactly the right one: the key counts assume the primary annotator's convention judgments are reliable and representative, and validation is limited to a 30-case IAA subset. My reading of §3.2–§3.3 and Table 1 agrees. The concern is load-bearing because the strongest claim is stated numerically (46 vs 9 vs 55), not merely as 'there exists at least one masked case.' If the primary annotator's notion of a wrong reading is too prescriptive, or if the 30-case agreement does not generalize, the specific counts lose their evidentiary weight. The pool-construction signal—including ASR transcripts that already appear surface-correct—compounds this by making the denominator a targeted yield rather than an unbiased sample. I do not think this threatens the central existence claim: Table 2 gives concrete examples, the span-isolation result and the Qwen3-ASR same-decoder control support the mechanism independently of any single ASR route, and the paper is unusually transparent in §5 that the audit is targeted and not a prevalence estimate. The paper's own disclosure that Structured is an oracle-style upper bound and that CosyVoice reuses the same pool also shows appropriate scope discipline. For these reasons I would keep the reader's CONDITIONAL verdict rather than reject or accept outright. The condition should be an independent, blind re-labeling of the full 110-case audit and a listener-perception check on the confirmed-masked examples; without that, the 46-case figure should be cited only as an annotator- and pool-dependent yield.","tokens_in":6303,"tokens_out":7420,"duration_ms":137731,"concrete_test":"Re-run the complete 110-row audit with two independent annotators who are blind to the paper's labels and ASR transcripts. Use the released audio and source text, require an audio-first forced-choice judgment between the expected reading and known negative readings, and only then reveal ASR transcripts to assign the confirmed-masked/exposed/no-error category. Compare the resulting confirmed-masked count with 46; if the count shifts by more than 10 cases, or if the two new annotators disagree with each other on a comparable 30-case subset at exact κ < 0.7, the headline rates are annotation-dependent and only the existence claim should be reported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.3 labels a row 'confirmed masked' only after a human listener judges the Raw TTS reading as wrong and at least one ASR route returns the expected/surface-correct form. The 46/110 count therefore inherits two dependencies. First, the pool of 110 was mined using 'suspected masking signals' that include ASR transcripts that already look surface-correct (Table 1, §3.3); the denominator is complete but not random, so 46/110 is a yield, not a rate. Second, the ground-truth 'wrong' judgment is the primary annotator's application of predefined expected readings (e.g., 13-11 must be 十三比十一, not 十三至十一). The 30-case blind subset has exact-label κ=0.634 and binary κ=0.800; on the remaining 80 rows there is no independent label. These are not fatal to the existence claim—even one verified wrong-audio/surface-ASR case would establish that the failure mode can occur—but they mean the headline counts should not be treated as a stable quantity, and the 'perceived by listeners' phrasing goes beyond what a correctness-against-gold annotation demonstrates.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper documents a failure mode of ASR-roundtrip evaluation for Chinese news TTS: short written spans whose correct reading depends on context or domain conventions (sports scores, aircraft models, units, membership names) can be read incorrectly by a TTS system while ASR transcribes the audio as the intended or surface-correct text. The authors construct a 200-case frozen benchmark from production news scripts and synthetic cases, define three risk labels (CDRD-entity, CDRD-polyphone, CDRD-adjacent), run Raw and oracle-style Structured text conditions with MiMo and CosyVoice TTS, and perform human span audits. A targeted 110-case audit reports 46 confirmed masked cases, 9 exposed TTS errors, and 55 no-error cases for MiMo; context isolation re-exposes 18/46; a Raw-only CosyVoice audit finds 51 masked cases; and cross-ASR comparisons show strong evaluator dependence. The central claim is that ASR-roundtrip can miss listener-perceived reading errors and should not be treated as standalone ground truth for Chinese news reading-risk evaluation.","tokens_in":6499,"tokens_out":4918,"duration_ms":55120,"significance":"If the existence claim holds, this is a useful and falsifiable caution for the TTS evaluation community. The paper has several strengths: the targeted audit is reported with a complete denominator rather than positive-only examples; the main failure cases are documented with human listening judgments and concrete transcripts; the authors include cross-TTS and cross-ASR controls; and they release prompts, settings, transcripts, labels, audio, summaries, and scoring code on GitHub and Zenodo. The paper also explicitly distinguishes targeted-audit yields from production prevalence and labels the Structured condition as an oracle-style diagnostic rather than a deployable frontend. These design choices make the central mechanism credible even though some headline counts need more careful qualification.","major_comments":[{"comment":"The headline count of 46 confirmed masked cases in Table 1 rests on the primary annotator's audio judgments for the full 110-case pool, but the blind relabel subset consists of only 30 rows and achieves 23/30 exact full-label agreement (κ=0.634) and 27/30 binary masked-versus-other agreement (κ=0.800). For the exact four-category partition that defines the 46/9/55 counts, this is moderate agreement, and the remaining 80 rows have no independent labels. The existence claim survives—one verified case would establish the mechanism—but the magnitude '46 confirmed masked' should be presented as protocol-dependent, ideally with per-annotator counts or a sensitivity analysis, rather than as an unqualified confirmation.","section":"§3.3, Table 1"},{"comment":"The 110-case pool is selected using suspected masking signals that include ASR transcripts that already appear surface-correct, so the counts in Table 1 are yields conditional on a non-random screening procedure, not rates or prevalence estimates. The paper does state this in §4.1 and §5, but the abstract's 'complete denominator' framing and Table 1's 'total targeted pool' column can easily be misread as incidence. I recommend removing 'complete denominator' from the abstract or explicitly writing 'among 110 pre-selected high-risk candidates' whenever the 46/110 count is invoked.","section":"§3.3, §4.1, Abstract"},{"comment":"The claim that context isolation re-exposes 18/46 masked errors is not a clean context manipulation for MiMo: the original full-sentence condition uses the case-specific audit route, while the clips use the main strict ASR route, so the ASR protocol changes together with the context. The same-decoder Qwen control in §4.5 provides supporting evidence for the isolation mechanism, but Table 3 should be relabeled as a combined context-plus-route comparison, or a same-route isolation comparison should be reported as the primary evidence.","section":"§4.3, Table 3"}],"minor_comments":[{"comment":"The text contains a typo in 'official score' that should read 'official score'.","section":"§3.2"},{"comment":"The phrase 'insufficient' should be corrected to 'insufficient'.","section":"Abstract and §1"},{"comment":"The description of the blind relabel subset would be clearer if it stated whether the blind annotators saw the same predefined expected readings and negative readings as the primary annotator, since this affects how the reported κ values should be interpreted.","section":"§3.3"},{"comment":"The table would benefit from a note that the 0/46 and 2/51 counts are occurrence-aware recovery counts on previously confirmed masked files, not full-corpus error rates, to prevent readers from interpreting them as ASR quality rankings.","section":"§4.5, Table 4"}],"recommendation":"major_revision","confidential_remarks":"The central existence claim is credible and the authors are appropriately cautious in the Discussion, but the abstract and Table 1 present the 46-case count and the 110-case denominator in a way that is stronger than the inter-annotator reliability supports. The manuscript will be acceptable once the quantitative claims are recalibrated and the isolation comparison is presented without the ASR-route confound. I do not see grounds for rejection, and I do not see fit or citation concerns."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this one. It makes a narrow claim and mostly supports it: ASR-roundtrip can miss reading errors a human listener would catch in Chinese news TTS, in spans where the correct reading depends on context or convention (scores, aircraft models, units, membership names). The evidence that the failure mode exists is solid—46 confirmed masked cases out of a 110-case targeted audit with human-audited wrong audio and surface-correct ASR transcript, plus 9 exposed errors and 55 no-error cases, so the denominator is complete and the labeling is not positive-only. Cross-TTS (CosyVoice) and cross-ASR (Whisper, Paraformer, Qwen3-ASR) controls add real weight. They also release the benchmark, transcripts, audio, and scoring code, so the result is checkable. The CDRD taxonomy and the frozen 200-case benchmark are new and useful for the speech evaluation subfield.\n\nI agree with the stress-test note, with one clarification: the flaw is not in the existence claim, it is in the headline count. The 46/110 is a yield from a targeted pool, not a prevalence or error rate. The pool was mined using suspected-masking signals, including ASR transcripts that already look surface-correct, so 46 is an upper-bound-ish demonstration of mechanism, not a measured frequency. The ground truth is one primary annotator applying predefined expected readings (e.g., 13-11 must be 十三比十一). Validation on 30 cases gives exact-label κ=0.634 and binary κ=0.800; that is real but modest, and 80 of the 110 rows have no independent label. So the specific counts should be cited cautiously, and the paper mostly says this itself—it repeatedly labels the numbers as targeted audit yields and explicitly says the experiments establish mechanism, not production incidence. Good.\n\nWhere I push back on the stress-test note a little: calling it 'not robustly established' might be too strong for the existence claim. Even one verified wrong-audio/surface-ASR case establishes the failure mode; 46 such cases with a complete denominator and a documented protocol is more than enough. I would frame it as 'the existence claim is established; the rate is not.'\n\nThe bigger concern is the span-isolation diagnostic crossing ASR routes (full-context uses the original audit route, clips use strict ASR). The paper discloses this. It weakens the isolation result as a clean mechanism argument, but the Qwen in-house control partially addresses it.\n\nBottom line: this is a solid, honest paper for the speech evaluation community. It deserves peer review. I would accept it with a request to tone down rate language and add a second annotator on the full 110 or at least a larger random subset. A serious referee should engage.","headline":"Useful, well-scoped demonstration that ASR-roundtrip can mask human-perceived reading errors in Chinese news TTS; treat the 46/110 as a yield, not a rate.","tokens_in":7010,"tokens_out":2167,"would_cite":true,"duration_ms":22659,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ASR roundtrip checks can certify TTS audio as correct even when human listeners hear a wrong reading.","keywords":["text-to-speech evaluation","ASR roundtrip","Chinese news TTS","reading errors","context-dependent reading decisions","text normalization","speech evaluation","polyphone disambiguation"],"falsifier":"Run the same audio-first span audit on a random sample of Chinese news TTS clips that is not pre-filtered for high-risk forms and compare the masked-error rate to the transcript-implied error rate; if ASR roundtrip never produces a surface-correct transcript for a human-confirmed wrong reading in that sample, the paper's central claim would be refuted. Alternatively, if an ASR decoder with all normalization and language-model recovery disabled still recovers the surface form in the same 46 cases, the effect would be an artifact of the protocol rather than of ASR priors.","tokens_in":6065,"feed_emoji":"🔊","tokens_out":6814,"duration_ms":65219,"temperature":0.7,"pith_summary":"The paper targets a blind spot in a cheap, widely used text-to-speech check: synthesize speech, run it through speech recognition, and compare the transcript to the source text. It argues that for Chinese news spans whose correct reading depends on context or domain conventions, such as sports scores, aircraft models, technical units, and membership names, this ASR-roundtrip can return the intended written form even when the spoken audio is wrong. A human audio-first audit of 110 high-risk cases confirms 46 such masked errors for one TTS system and 51 for a second, reported with the full denominator of exposed-error and no-error cases. The paper concludes that roundtrip transcription remains useful for screening but should not stand alone as ground truth for reading-risk evaluation.","feed_headline":"ASR roundtrip hides 46 wrong readings a listener would catch","feed_subtitle":"When Chinese news TTS misreads scores and units, ASR still writes the intended text; audio-first audits expose it.","key_machinery":"The load-bearing object is the Context-Dependent Reading Decision (CDRD) span: a written string whose correct spoken form depends on context beyond the local characters, such as whether a hyphen means a score, a range, a model number, or something else. The mechanism that masks errors is ASR surface recovery, in which the recognizer's language-model and text-normalization priors map fluent-but-wrong audio back to the conventional written form. The paper also uses a span-isolation diagnostic that cuts a target span out of its sentence, showing that full-sentence context is what pushes many transcripts back to surface-correct.","core_discovery":"The central discovery is that ASR-based roundtrip evaluation produces false negatives precisely for fluent reading errors that listeners do notice. When a TTS frontend misreads 13-11 as a range instead of a snooker score, or 伊尔-76 as a negative number instead of an aircraft model, the ASR decoder's contextual and normalization priors can reconstruct the intended or surface-correct text, hiding the error. Evidence comes from a 110-case targeted audit in which the primary annotator heard wrong Raw TTS in 55 of the MiMo cases (46 masked plus 9 exposed), with the remaining 55 clean; a CosyVoice run on the same pool found 51 masked cases and 27 exposed errors. Isolating the span from sentence context re-exposed 18 of the 46 MiMo masked cases, and across the 97 confirmed-masked files Qwen3-ASR recovered surface-correct text in 40 while Paraformer did so in only 2, showing that masking is ASR- and protocol-dependent.","pith_inferences":["Beyond the paper's claims, a random-sample audit would be needed to estimate a production masked-error rate; the 46 of 110 figure comes from a deliberately high-risk pool and should not be quoted as prevalence.","We would expect the same surface-recovery route to appear in any modern Chinese news TTS pipeline that feeds short numeric or mixed-script spans without explicit spoken-form rewriting, although the paper only documents two TTS systems.","The large gap between Qwen3-ASR and Paraformer recovery rates suggests that newer decoder-based ASR systems may be more prone to normalizing away reading errors, an implication the paper does not explicitly draw.","A natural testable extension is to apply the span-isolation diagnostic to convention-heavy spans in other languages, such as sports scores and unit strings in English or Japanese, to see whether this masking is a general speech-evaluation phenomenon."],"forward_implications":["For Chinese news TTS, transcript-only scores should be reported alongside human span-level listening whenever CDRD or CDRD-adjacent material is at stake.","Masking concentrates in scores, aircraft models, and unit strings, so evaluations built on news of those kinds need audio-first checks.","A span-isolation pass can serve as a cheap diagnostic that re-exposes many masked reading errors, recovering 18 of 46 in this audit.","Roundtrip comparison remains appropriate for screening and ablation but should be treated as an upper-bound sanity check, not standalone ground truth.","The choice of ASR system changes the measured masking rate, so cross-ASR controls are needed before drawing conclusions from roundtrip scores."],"supporting_citations":[{"why":"Supplies the prior evidence that ASR-based TTS evaluation is task- and protocol-sensitive, motivating the failure-mode study.","marker":"[1]"},{"why":"Establishes that ASR metrics do not always track human utility, the background for distinguishing transcript accuracy from perceived correctness.","marker":"[2]"},{"why":"Provides the normalization-of-non-standard-words taxonomy that grounds the CDRD span categories.","marker":"[5]"},{"why":"Supplies the TTS text-normalization system background that defines convention-dependent readings such as units and scores.","marker":"[6]"},{"why":"Documents the MiMo audio-understanding API used as the primary ASR and fallback route in the audits.","marker":"[7]"},{"why":"Whisper-small is the non-MiMo ASR control used in the span-isolation diagnostic.","marker":"[8]"},{"why":"CosyVoice is the second Raw-only TTS system used to show that masking recurs across different TTS engines.","marker":"[9]"},{"why":"Paraformer is the open-source ASR baseline that preserves wrong or noncanonical forms in most cases, quantifying evaluator dependence.","marker":"[10]"},{"why":"Qwen3-ASR is the comparison ASR that recovers surface-correct text in 40 of 97 confirmed-masked files, showing decoder dependence.","marker":"[11]"}],"fun_headline_variants":["ASR roundtrip hides 46 Chinese news TTS misreadings","46 wrong TTS readings slip past ASR but not listeners","Audio-first audit exposes 46 errors ASR roundtrip misses","ASR roundtrip: 46 masked mispronunciations in Chinese TTS","When ASR agrees but listeners hear errors: 46 cases"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The audit's category counts rest on one primary listener's judgment that the expected readings define correctness, for example that 13-11 must be read as a snooker score with 比, and that judgment was checked on only a 30-case subset with 0.634 agreement, so if those conventions are disputed the 46 of 110 numbers shift.","fun_headline_variants_meta":{"raw":{"variants":["ASR roundtrip hides 46 Chinese news TTS misreadings","46 wrong TTS readings slip past ASR but not listeners","Audio-first audit exposes 46 errors ASR roundtrip misses","ASR roundtrip: 46 masked mispronunciations in Chinese TTS","When ASR agrees but listeners hear errors: 46 cases"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000664,"raw_usage":{"total_tokens":3054,"prompt_tokens":987,"completion_tokens":2067,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":603,"completion_tokens_details":{"reasoning_tokens":1972}},"tokens_in":603,"tokens_out":2067,"duration_ms":15470,"temperature":1.0,"reasoning_tokens":1972,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T21:01:19.856102+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same audio-first span audit on a random sample of Chinese news TTS clips that is not pre-filtered for high-risk forms and compare the masked-error rate to the transcript-implied error rate; if ASR roundtrip never produces a surface-correct transcript for a human-confirmed wrong reading in that sample, the paper's central claim would be refuted. Alternatively, if an ASR decoder with all normalization and language-model recovery disabled still recovers the surface form in the same 46 cases, the effect would be an artifact of the protocol rather than of ASR priors.","supporting_citations":[{"cited_title":"It scales cheaply and often tracks intelligibility, but its reliability depends on the task and protocol [1, 2]","cited_arxiv_id":null,"evidence_quote":"Supplies the prior evidence that ASR-based TTS evaluation is task- and protocol-sensitive, motivating the failure-mode study."},{"cited_title":"Index Terms— text-to-speech evaluation, ASR-roundtrip, Chinese news TTS, reading errors, speech evaluation","cited_arxiv_id":null,"evidence_quote":"Establishes that ASR metrics do not always track human utility, the background for distinguishing transcript accuracy from perceived correctness."},{"cited_title":"ASR false negatives in targeted audits Table 1 reports the targeted audit, with a complete denomina- tor, that supports the ASR false-negative claim","cited_arxiv_id":null,"evidence_quote":"Provides the normalization-of-non-standard-words taxonomy that grounds the CDRD span categories."},{"cited_title":"This does not imply that ASR always masks such errors, that 46/110 is a natural production rate, or that Structured is a deployable frontend","cited_arxiv_id":null,"evidence_quote":"Supplies the TTS text-normalization system background that defines convention-dependent readings such as units and scores."},{"cited_title":"Human annotators performed span-level listening audits of synthetic audio and authorized the use of their anonymized judgments for this study","cited_arxiv_id":null,"evidence_quote":"Documents the MiMo audio-understanding API used as the primary ASR and fallback route in the audits."},{"cited_title":"Confidence intervals for ASR-based TTS evaluation,","cited_arxiv_id":null,"evidence_quote":"Whisper-small is the non-MiMo ASR control used in the span-isolation diagnostic."},{"cited_title":"Automatic human utility evaluation of ASR systems: Does WER really predict performance?","cited_arxiv_id":null,"evidence_quote":"CosyVoice is the second Raw-only TTS system used to show that masking recurs across different TTS engines."},{"cited_title":"An end-to-end Chinese text normalization model based on rule-guided flat-lattice Transformer,","cited_arxiv_id":null,"evidence_quote":"Paraformer is the open-source ASR baseline that preserves wrong or noncanonical forms in most cases, quantifying evaluator dependence."},{"cited_title":"Label embedding for Chinese grapheme-to-phoneme conver- sion,","cited_arxiv_id":null,"evidence_quote":"Qwen3-ASR is the comparison ASR that recovers surface-correct text in 40 of 97 confirmed-masked files, showing decoder dependence."}],"review_version":1}