{"id":"9ca125e0-92a5-4196-8d4b-a990b90f0f61","arxiv_id":"2509.05399","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Graph Temporal Classification trained on multiple G2P pronunciations per word reduces phoneme error rates versus CTC with one pronunciation: English 32.9 to 28.8, Dutch 23.9 to 23.0.","lead":"Using a training loss that accepts several possible pronunciations of each word, the authors improved phoneme recognition over the standard single-pronunciation baseline on both English and Dutch. The gains are moderate, about 4% absolute on English and 1% on Dutch, but consistent across the two languages.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing significance test for the headline English result; reported p-values are contradictory and omit the CTC-vs-GTC contrast.","rationale":"The central claim is an empirical improvement in PER. The paper's own statistical reporting omits the decisive tests and contains a clear error (calling p=0.04 non-significant), so the evidence as written does not establish the claim. This is more load-bearing than the reader's oracle-LER mismatch because even a perfect oracle LER would not rescue a non-significant PER difference; conversely, a significant PER difference would make the oracle-LER issue a matter of mechanistic interpretation rather than existence of the effect. The proposed paired test on the actual test-set speakers directly settles whether the observed gains are reliable. I agree with the reader's overall conditional verdict, so no verdict change is needed.","tokens_in":9485,"tokens_out":9243,"duration_ms":95391,"concrete_test":"Compute per-speaker PER on TIMIT test (168 speakers) and CGN test (102 speakers) for the 1-best (CTC) system and each GTC system. Run paired Wilcoxon signed-rank tests for 1-best vs 2-best and 1-best vs 3-best in each language, and also for 1-best vs the post-hoc-selected best GTC. Report all p-values with a multiple-comparison correction. If the English 1-vs-2 p-value exceeds 0.05, the headline result is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Table II (Section VI-B), the central claim rests on PER reductions from 32.9 to 28.8 (TIMIT, 1-best vs 2-best) and 23.9 to 23.0 (CGN, 1-best vs 3-best). Yet no Wilcoxon signed-rank test is reported for either the English 1-vs-2 contrast or the Dutch 1-vs-3 contrast. The only English p-values are between GTC variants: p=0.04 (2-best vs 3-best) and p=0.11 (2-best vs all); the text calls p=0.04 'not statistically significant', which is internally inconsistent. For Dutch, 1-vs-2 is significant (p=1e-5) but 1-vs-3, the best outcome, is not directly tested. Moreover, the number of pronunciations was selected post hoc per language based on test-set performance (2 for English, 3 for Dutch), inflating the chance of finding a favorable configuration. If the untested English contrast is not significant on a per-speaker paired test, the abstract's 'consistently improves' claim is not supported for English. This is a direct, testable challenge to the main conclusion, independent of the oracle-LER analysis.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes using Graph Temporal Classification (GTC) instead of CTC for training phoneme recognizers when G2P systems provide multiple candidate pronunciations per word. A WFSA-based label graph accepts any concatenation of candidate pronunciations, so all variants are treated as valid supervision. Experiments on US English (Common Voice training, TIMIT test) and Dutch (CGN) compare a 1-best CTC baseline with GTC trained on up to 2, 3, or all pronunciations. The reported PERs are 32.9% -> 28.8% (TIMIT) and 23.9% -> 23.0% (CGN), leading the authors to conclude that incorporating multiple pronunciations consistently improves phoneme error rates.","tokens_in":9819,"tokens_out":4257,"duration_ms":45280,"significance":"If the result holds, the contribution is useful and practical: it adapts an existing sequence-level training objective (GTC) from ASR lattice/hypothesis use to APR with G2P candidate sets, with a clean WFSA formulation and a two-language evaluation. The method is well motivated by oracle label-error rates showing candidate-set gains. The main caveats are statistical: the headline English improvement is not significance-tested against the CTC baseline, the reported p-value interpretation is internally inconsistent, and the oracle-LER upper bounds are computed on different splits from the test PERs. These issues affect the strength of the central empirical claim rather than the validity of the method, so the paper warrants major revision. The authors provide a detailed algorithm description and implementation choices, which is a strength for reproducibility.","major_comments":[{"comment":"The central claim 'consistently improves' over CTC rests on 32.9 vs 28.8 (TIMIT) and 23.9 vs 23.0 (CGN). For English, no Wilcoxon signed-rank test is reported for the 1-best vs 2-best contrast; the only reported English p-values are between GTC variants, and the text says these differences are 'not statistically significant' while listing p=0.04, which is significant at the conventional 0.05 level. For Dutch, the best 3-best system is not directly tested against 1-best (the p=1e-5 compares 1-best vs 2-best). Please report per-speaker paired tests for each 1-best vs k-best contrast, state the alpha/multiple-comparison policy, and correct the p-value interpretation.","section":"§VI-B, Table II"},{"comment":"The oracle LER upper bounds are measured on TIMIT train and CGN-phoneme dev, whereas the PER gains in Table II are measured on TIMIT test and CGN-phoneme test. The claim that GTC's benefit is bounded by candidate-set quality is only as strong as the transferability of these LERs to the test splits, which is assumed rather than demonstrated. Compute oracle LER on the same test sets used in Table II (or provide explicit evidence that train/dev LERs are representative of test words).","section":"§VI-A, Table I"},{"comment":"The number of pronunciations used in the headline result (2 for English, 3 for Dutch) appears to be selected after inspecting test-set PERs, and the conclusion highlights the best configuration. Because multiple configurations are evaluated and only the best is featured, the reported gain is a selected maximum; no multiple-testing correction or validation-based selection is described. At minimum, the authors should state whether the choice was made on a separate validation set or treat the results as exploratory.","section":"§VI-B"}],"minor_comments":[{"comment":"The definition of β^{-1}(G) as 'the set of sequences π such that β(π)=ŷ_k, and ŷ_k ∈ G' should be written as a union over all label sequences ŷ in G: β^{-1}(G) = ∪_{ŷ∈G} β^{-1}(ŷ). The current wording is imprecise.","section":"§III-B, Eq. (4)"},{"comment":"The caption contains a dangling construction: 'The arc with label −1 pointing to the final state of a design requirement for all k2 WFSAs.' This should be rewritten for clarity.","section":"Figure 1"},{"comment":"The model-selection criterion is described as 'the minimum loss value on the validation set,' but it is unclear whether this is the CTC/GTC loss on the pseudo-labels or a phoneme-level error metric. Please specify.","section":"§V-C"},{"comment":"The sentence 'the differences between the 2-best, 3-best, and all systems are not statistically significant (p=0.04 between 2-best and 3-best, p=0.11 between 2-best and all)' is internally inconsistent with the usual interpretation of p=0.04. Clarify the significance level and correction method.","section":"§VI-B"}],"recommendation":"major_revision","confidential_remarks":"I found no evidence of circularity in the training setup; GTC uses only G2P-derived candidates, not test labels. The main risk is statistical: the untested English baseline contrast and the oracle-LER split mismatch could weaken the headline claim if not addressed. The paper is within the scope of the journal, and the WFSA implementation is a solid contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The conditional verdict is about right. GTC itself is prior work, so the novelty here is the application to APR with G2P ambiguity, not the algorithm. That application is genuinely useful: it is a clean solution to a real annotation problem, and the paper demonstrates it on two languages. The consistent pattern across all multi-pronunciation settings (2/3/all) beating 1-best CTC in both English and Dutch means the headline claim does not stand or fall on the choice of n. The oracle LER analysis in Table I supports the mechanism.\n\nThe soft spots are all in the statistics and presentation. The biggest one: the English CTC-vs-GTC contrast, the headline 32.9 to 28.8, is never significance-tested. The only p-values are among GTC variants, and the text says p=0.04 is \"not statistically significant,\" which is internally inconsistent unless some other alpha is in play. Since the reader has to rely on this one number, a per-speaker Wilcoxon on 1-best vs 2-best should be trivial to report, and it should have been included. For Dutch, 1-best vs 2-best is tested and significant, so that language's claim is fine; the lack of a test for 3-best is less important because 2-best already beats CTC.\n\nSecond, n was selected post hoc based on test-set performance. As noted, this does not change the main conclusion because every GTC variant beats CTC, but it does inflate the odds of reporting a favorable configuration, so it should be acknowledged or chosen on dev.\n\nThird, oracle LER is computed on TIMIT train and CGN dev, not on the test sets used in the headline PER table. If candidate-set quality differs on test, the upper-bound argument weakens. Also, calling oracle LER an \"upper bound on performance\" is conceptually sloppy; it is a lower bound on achievable PER under the candidate-set assumption, and not a hard bound if the model can emit phonemes outside the candidates.\n\nNo code or data release, which is a minor reproducibility mark against an otherwise clearly described method.\n\nBottom line: this is a legitimate, competent extension, not a breakthrough. With the missing significance test, a fix to the p-value wording, and a slightly more careful treatment of the oracle bound, it is a solid paper for the APR community. I would send it to review, not desk reject.","headline":"Useful application of existing GTC idea to APR with G2P pronunciation ambiguity; the empirical pattern is consistent, but the English headline gain needs a proper significance test and the upper-bound framing needs cleaning up.","tokens_in":10214,"tokens_out":5254,"would_cite":false,"duration_ms":54752,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Training a phoneme recognizer with several G2P pronunciations per word, via a graph variant of CTC, cuts phoneme error rate compared with a single-pronunciation CTC baseline.","keywords":["phoneme recognition","graph temporal classification","connectionist temporal classification","grapheme-to-phoneme","pronunciation variants","label error rate","weakly supervised speech recognition","automatic speech recognition"],"falsifier":"Compute, on the exact TIMIT test and CGN test word lists, the lowest label error rate achievable by cherry-picking among the top candidate pronunciations. If that coverage gap between the best single candidate and the best of the top three is near zero on those test words, the reported PER gains have no source and should not be expected to replicate.","tokens_in":9438,"feed_emoji":"🎤","tokens_out":8533,"duration_ms":91849,"temperature":0.7,"pith_summary":"Automatic phoneme recognizers are usually trained from text converted to phonemes by grapheme-to-phoneme tools, which often give several candidate pronunciations per word. Standard CTC training has to pick one candidate, throwing away the ambiguity; the paper adapts Graph Temporal Classification to instead treat every concatenation of the candidate pronunciations as a valid target. On English and Dutch data this lowers phoneme error rate relative to a 1-best CTC baseline: from 32.9% to 28.8% on the English TIMIT test and from 23.9% to 23.0% on the Dutch CGN test. The reason the approach can work is shown by oracle label error rates: the best single G2P pronunciation is frequently not the closest to what was actually said, and allowing two or three candidates closes much of that gap. If true, this means noisy text-derived supervision can be used more effectively without manual phoneme labels.","feed_headline":"Training on several pronunciations per word cuts phoneme errors","feed_subtitle":"Training with pronunciation-variant labels instead of a single best guess cuts phoneme error rate in English and Dutch.","key_machinery":"The central object is the label graph: a weighted finite-state acceptor in which the candidate pronunciations of each word are placed in parallel and the word-level graphs are joined in sequence, so any path from a start to an end node spells an acceptable phoneme string for the utterance. The GTC loss is the negative log of the total probability of all CTC alignments that collapse to some path in this graph, computed with the forward-backward algorithm. CTC is the special case where the graph contains a single path.","core_discovery":"The paper's claim is that pronunciation ambiguity in G2P output is a usable training signal rather than noise to be averaged away. By compiling multiple pronunciations per word into a label graph and training with GTC, the model is free to weight whichever pronunciation path best explains the acoustics. The authors report lower phoneme error rates than the CTC baseline in both languages, with the best English model using up to two pronunciations per word (28.8% vs 32.9%) and the best Dutch model using up to three (23.0% vs 23.9%). The oracle label error rates in Table I support the mechanism: 1-best G2P output has 11.8% LER on English training words versus 8.5% with the best of three candida","pith_inferences":["The headline numbers depend on candidate pronunciation coverage on the exact test words; a direct check of oracle coverage on TIMIT test and CGN test would confirm whether the reported gains transfer or are an artifact of dev-set selection.","The same graph-loss idea generalizes to any sequence task with multiple valid reference transcriptions, such as accented or code-mixed speech, or tasks with inter-annotator disagreement.","If candidate coverage is the real driver, then languages or dictionaries with richer pronunciation variation should show larger GTC gains; this is testable by comparing G2P systems with different variant counts.","Adding decoding-side pronunciation variation could compound the training-side gain, but the paper leaves that untested."],"forward_implications":["APR can be trained from utterance-text pairs with G2P uncertainty, avoiding costly manual phoneme annotations.","Because GTC reduces to CTC when each word has one pronunciation, one training pipeline covers both unambiguous and ambiguous words.","The benefit appears already at 2 or 3 candidate pronunciations, so full G2P enumeration is not needed.","The gap between single-best and oracle multi-best label error rates quantifies how much headroom remains for better pronunciation modeling.","The method does not model coarticulation across word boundaries; extending the label graph with insertion, substitution, or deletion arcs is the paper's stated next step."],"supporting_citations":[{"why":"Defines the CTC loss that the paper treats as the single-pronunciation baseline.","marker":"[8]"},{"why":"Introduces Graph Temporal Classification and supplies the graph-loss definition and forward-backward computation that the paper adapts to phoneme recognition.","marker":"[9]"},{"why":"Supplies the English grapheme-to-phoneme model outputs with multiple candidate pronunciations per word.","marker":"[29]"},{"why":"Supplies the Dutch pronunciation dictionary and its rule-based alternative pronunciations.","marker":"[7]"},{"why":"Supplies the English speech training data used to train the models.","marker":"[26]"},{"why":"Supplies the phonemically annotated English test set whose PER is reported.","marker":"[27]"},{"why":"Supplies the phonemically annotated Dutch test and validation sets whose PER is reported.","marker":"[28]"}],"fun_headline_variants":["Pronunciation graphs cut phoneme errors","Training on G2P graphs beats CTC for phonemes","Multiple pronunciations per word lower error","Graph CTC harnesses pronunciation ambiguity","GTC exploits G2P variation for phoneme models"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The candidate pronunciation lists for the words in an utterance must, for enough words, actually contain the pronunciation the speaker used; the paper demonstrates this on development data, not on the exact test words used for the headline error rates.","fun_headline_variants_meta":{"raw":{"variants":["Pronunciation graphs cut phoneme errors","Training on G2P graphs beats CTC for phonemes","Multiple pronunciations per word lower error","Graph CTC harnesses pronunciation ambiguity","GTC exploits G2P variation for phoneme models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000157,"raw_usage":{"total_tokens":1029,"prompt_tokens":686,"completion_tokens":343,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":430,"completion_tokens_details":{"reasoning_tokens":277}},"tokens_in":430,"tokens_out":343,"duration_ms":4309,"temperature":1.0,"reasoning_tokens":277,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T05:31:19.725594+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute, on the exact TIMIT test and CGN test word lists, the lowest label error rate achievable by cherry-picking among the top candidate pronunciations. If that coverage gap between the best single candidate and the best of the top three is near zero on those test words, the reported PER gains have no source and should not be expected to replicate.","supporting_citations":[{"cited_title":"Connection- ist temporal classification: labelling unsegmented sequence data with recurrent neural networks,","cited_arxiv_id":null,"evidence_quote":"Defines the CTC loss that the paper treats as the single-pronunciation baseline."},{"cited_title":"Semi-supervised speech recognition via graph-based temporal classification,","cited_arxiv_id":null,"evidence_quote":"Introduces Graph Temporal Classification and supplies the graph-loss definition and forward-backward computation that the paper adapts to phoneme recognition."},{"cited_title":"Mertens and F","cited_arxiv_id":null,"evidence_quote":"Supplies the Dutch pronunciation dictionary and its rule-based alternative pronunciations."},{"cited_title":"Common V oice: A Massively-Multilingual Speech Corpus,","cited_arxiv_id":null,"evidence_quote":"Supplies the English speech training data used to train the models."},{"cited_title":"DARPA TIMIT Acoustic-Phonetic Continuous Speech Corpus CD-ROM,","cited_arxiv_id":null,"evidence_quote":"Supplies the phonemically annotated English test set whose PER is reported."},{"cited_title":"CGN, an annotated corpus of spoken Dutch,","cited_arxiv_id":null,"evidence_quote":"Supplies the phonemically annotated Dutch test and validation sets whose PER is reported."}],"review_version":1}