{"id":"7aa85bb9-b803-47c5-9c0a-0b34c60dc352","arxiv_id":"2505.03697","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"Severity-aware mixing of normal and cleft lip and palate speech during ASR training improved word error rates and a proposed fairness score on two small datasets, but the evaluation has reliability issues.","lead":"The paper tests whether adding cleft lip and palate (CLP) speech, grouped by severity, to normal speech training data improves automatic speech recognition accuracy and fairness. It reports word error rate gains on Kannada and English child-speech datasets, but the evaluation has inconsistent headline numbers and an unclear speaker split.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Severity-aware mixing gains are confounded by speaker overlap and unbounded training-set size; no speaker-disjoint or size-matched control is reported.","rationale":"The reader's weakest assumption correctly identifies speaker overlap as a serious threat, and I agree that the utterance-level 80/20 split is insufficient for a generalization claim about ASR fairness for CLP speech. However, I see an equally or more load-bearing problem that the reader did not explicitly foreground: the augmentation comparisons are not size-matched, so simply adding more CLP utterances—rather than using severity information—could explain the improvements. The CLP-only condition already improves fairness, and no control isolates severity-aware selection from data quantity. This means the central claim is currently unverified in both directions: it could survive a clean speaker-disjoint reanalysis, or it could collapse to a trivial 'more training data helps' result. The internal numerical inconsistencies (Table 5 vs. Table 7, and the abstract discrepancy) compound the problem by making it hard to know which numbers to trust. I therefore recommend UNVERDICTED: not because the direction is implausible, but because the current evidence does not yet adjudicate between severity-aware mixing and confounds of speaker identity and dataset size. A concrete reanalysis with speaker-disjoint splits and size-matched controls would settle the question and could move the paper to acceptance if the effect survives.","tokens_in":18901,"tokens_out":5240,"duration_ms":50512,"concrete_test":"Recompute the main comparisons in Tables 7 and 8 using speaker-disjoint splits (e.g., leave-one-speaker-out or stratified speaker groups) for GMM-HMM and Whisper on both datasets, and add two size-matched control training sets: (a) the same total number of utterances randomly sampled from all CLP speakers regardless of severity, and (b) the same total number of additional normal utterances. If the severity-mixed WER/FS advantage over these controls disappears, or if speaker-disjoint WER degrades sharply relative to the utterance-level split, the reported fairness gains are not evidence for severity-aware generalization. Report means and 95% confidence intervals over at least 5 random speaker splits.","verdict_should_be":"UNVERDICTED","load_bearing_attack":"The central claim—that severity-aware mixing of CLP and normal speech improves ASR fairness—requires ruling out two simpler explanations that the current experiments do not address. First, Section 3 states that 'Both datasets are randomly divided into training and testing partitions with 80:20 ratio.' This is an utterance-level split, not a speaker-level split. AIISH has only 60 speakers and 19 unique utterances, and NMCPC has 65 speakers and 76 unique utterances, so the same speaker almost certainly contributes utterances to both training and evaluation. CLP speech carries speaker-specific hypernasality and articulation patterns, so improved WER on the evaluation set may reflect memorization of seen speakers rather than generalization to unseen CLP speakers. Second, the augmentation conditions in Tables 7 and 8 are not size-matched: 'Mild+Normal', 'Mild+Moderate+Normal', and 'Mild+Moderate+Severe+Normal' add monotonically more CLP utterances, and there is no control adding an equivalent amount of normal-only data or an equivalent amount of severity-blind CLP data. The 'CLP' training condition alone also improves fairness substantially—e.g., AIISH GMM-HMM FS rises from -31.57 with Normal training to -26.67 with CLP training—so the reported gains are not specifically attributable to severity-aware selection. Internal reporting inconsistencies, such as Table 5 listing the GMM-HMM AIISH CLP-trained CLP WER as 36.44 while Table 7 lists 11.00, and the differing headline WERs between the metadata abstract and the full-text abstract, further weaken confidence. A speaker-disjoint, size-matched reanalysis is required before the central claim can be accepted.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses automatic speech recognition (ASR) fairness for cleft lip and palate (CLP) speech. It introduces a fairness score (FS) that balances average word error rate and error disparity between normal and CLP groups, evaluates the fairness of a public Google ASR API, and proposes severity-aware mixing of CLP and normal speech during ASR training. Experiments are conducted with GMM-HMM, Whisper, and XLS-R on two datasets: AIISH (Kannada) and NMCPC (English). The authors report that severity-aware augmentation improves FS and reduces WER; for example, the full-text abstract reports WER decreasing from 22.64% to 18.76% (GMM-HMM, AIISH) and from 28.45% to 18.89% (Whisper, NMCPC).","tokens_in":19233,"tokens_out":3877,"duration_ms":33623,"significance":"If the reported effect is robust, the paper would be a useful empirical contribution to ASR fairness for disordered speech: it provides a simple fairness metric, a concrete data-mixing recipe, and evaluations across a public API, a traditional HMM system, and two foundation models in two languages. The paper also documents a striking performance gap in a public ASR service for CLP speech. However, the central claim depends on unaddressed confounds and internal inconsistencies. The utterance-level split, lack of size-matched controls, and post hoc selection of the best augmentation condition make the current evidence suggestive rather than demonstrative; the reported headline numbers are also internally inconsistent.","major_comments":[{"comment":"The 80/20 split is utterance-level, not speaker-disjoint. With only 60 speakers in AIISH and 65 in NMCPC, the same speaker very likely contributes utterances to both training and evaluation. CLP speech carries speaker-specific hypernasality and articulation patterns, so the reported WER gains may reflect speaker-specific memorization rather than generalization to unseen CLP speakers. A speaker-disjoint split, and ideally speaker-independent evaluation, is needed to support the claim that severity-aware mixing improves fairness for new CLP speakers.","section":"Section 3 (Database setup)"},{"comment":"The augmentation conditions are not matched in training-set size or total CLP content. 'Mild+Normal', 'Mild+Moderate+Normal', and 'Mild+Moderate+Severe+Normal' add monotonically more CLP utterances, and there is no control adding an equivalent amount of normal-only data or an equivalent amount of severity-blind CLP data. Moreover, the 'CLP' training condition alone improves FS substantially (AIISH GMM-HMM FS rises from -31.57 with Normal training to -26.67 with CLP training), so the reported gains are not specifically attributable to severity-aware selection.","section":"Tables 7 and 8"},{"comment":"The headline numbers are internally inconsistent. The metadata abstract states WER decreases from 37.58% to 25.47% (GMM-HMM, AIISH) and from 35.74% to 21.72% (Whisper, NMCPC), while the full-text abstract states 22.64% to 18.76% and 28.45% to 18.89%, respectively. In addition, Table 5 lists the GMM-HMM AIISH CLP-trained CLP WER as 36.44, whereas Table 7 lists the same quantity as 11.00; the Whisper values also differ (55.66 vs. 44.77). These discrepancies undermine the reliability of the reported improvements and must be reconciled.","section":"Abstract and Tables 5/7"},{"comment":"The evaluation lacks error bars, confidence intervals, and significance tests. All WER and FS comparisons are point estimates from a single random split, and the best augmentation recipe is chosen post hoc per dataset (Mild+Moderate+Severe+Normal for AIISH, Mild+Moderate+Normal for NMCPC). Without multiple seeds, bootstrap confidence intervals, or a pre-specified selection rule, the paper cannot rule out chance variation or selection effects.","section":"Section 6.2, Tables 7-9"}],"minor_comments":[{"comment":"The phrase 'crisis cross' should be 'criss-cross'.","section":"Section 6.2.1"},{"comment":"The header 'datsets' should be 'datasets'.","section":"Table 3"},{"comment":"Reference [49] contains an encoding artifact ('children²s'); references [55] and [56] are identical and one should be removed.","section":"Reference list"},{"comment":"The Whisper AIISH CLP-trained row reports '7' for the normal test column without a decimal place; clarify whether this is 7.00 or another value.","section":"Table 7"},{"comment":"The FS definition states 'α,β> = 0' but the experiments later use α+β=1; clarify the constraint on the weights.","section":"Section 4.2"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the scope of an ASR or health-equity venue, but the authors should be asked to address the speaker-overlap confound, add size-matched controls, reconcile the inconsistent reported numbers, and provide uncertainty quantification before the manuscript can be considered further. The claim of being the first study on ASR fairness for CLP speech should also be checked carefully against [40] and related enhancement work, which may partially overlap."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a read if you work on atypical speech ASR. The paper is the first to explicitly study fairness of ASR for cleft lip and palate speech, and the central idea—severity-aware mixing of CLP and normal speech during training—is simple and plausible. They evaluate three model families on two small real clinical corpora, Kannada and English, and report consistent improvements in a fairness score that captures both average WER and between-group disparity. That is a real contribution, and the DTW-based motivation for thinking mild/moderate speech is closer to normal is reasonable, though it's only illustrative. Credit where due: the problem is meaningful, the framing is new, and they are transparent about the datasets and splits, which makes the missing controls easy to spot.\n\nThe soft spots are serious. First, the random 80/20 split is at the utterance level, not the speaker level. With 60–65 speakers and short utterances, the same speakers almost certainly appear in both train and test, so the WER gains could come from speaker memorization rather than generalization to unseen CLP speakers. Second, the augmentation conditions are not size-matched. Adding mild, then mild+moderate, then mild+moderate+severe monotonically increases the amount of CLP training data, and there is no control that adds an equivalent amount of normal-only or severity-blind CLP data. The 'CLP' training condition alone substantially improves fairness too, so the specific claim about severity awareness is not isolated. Third, the numbers are internally inconsistent: the full-text abstract reports different headline WERs than the metadata abstract, and Table 5 lists the GMM-HMM AIISH CLP-trained CLP WER as 36.44 while Table 7 lists 11.00 for the same condition. There are also no error bars or significance tests, and the best augmentation recipe appears to be chosen post hoc per dataset. The fairness score itself is a simple weighted sum; that's fine, but the alpha/beta sweep is arbitrary and doesn't change the core results.\n\nThose confounds mean the central claim—severity-aware mixing improves fairness—is plausibly true but not established by this evidence. A speaker-disjoint, size-matched reanalysis would be needed before I'd trust the effect sizes. The literature review is solid and self-citation is not excessive.\n\nFor whom: speech researchers interested in disordered speech, accessibility, and fairness metrics. It deserves a serious referee, but the recommendation should be major revision with the controls I mentioned. I'd want to see the reanalysis before citing it myself.","headline":"A first fairness evaluation for CLP speech with a sensible augmentation idea, but the evidence is confounded by speaker overlap and uncontrolled training set size.","tokens_in":19780,"tokens_out":1037,"would_cite":false,"duration_ms":11941,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Severity-aware mixing of cleft lip and palate (CLP) speech with normal speech during ASR training improves accuracy and fairness, cutting pooled word error rate from 22.64% to 18.76% on Kannada and from 28.45% to 18.89% on English child…","keywords":["automatic speech recognition","cleft lip and palate speech","ASR fairness","severity-aware data mixing","hypernasality","Whisper","XLS-R","fairness score"],"falsifier":"Re-run the best severity-mixed training condition with a speaker-disjoint split, where all recordings of each speaker are assigned wholly to either training or testing, on both corpora, and compare the pooled word error rate and fairness score against the paper's utterance-level split. If the improvement disappears or falls below a meaningful margin, the reported fairness gain is speaker memorization rather than generalization to unseen CLP speakers.","tokens_in":18739,"feed_emoji":"🗣️","tokens_out":8920,"duration_ms":78950,"temperature":0.7,"pith_summary":"The paper asks whether automatic speech recognition (ASR) is fair to people with cleft lip and palate, whose speech is hypernasal and breathy because of structural differences in the mouth and nose. It first shows that a commercial speech-to-text API errs far more on CLP speech than on typical speech. It then argues that mild and moderate CLP speech still preserves enough spectral structure to be useful in training, and proposes a simple intervention: mix severity-graded CLP utterances with normal speech when training ASR models. On Kannada and English child speech, this lowers word error rate and improves a two-group fairness score across GMM-HMM, Whisper, and XLS-R models. The headline numbers are pooled word error rate falling from 22.64% to 18.76% for GMM-HMM on the Kannada corpus and from 28.45% to 18.89% for Whisper on the English corpus.","feed_headline":"Mixing cleft-palate and normal speech cuts ASR errors by up to 34%","feed_subtitle":"On Kannada and English child speech, adding graded CLP samples to training improves fairness and lowers word error rate.","key_machinery":"The load-bearing object is the severity ordering of CLP speech, measured by dynamic time warping distance between voiced spectrogram frames of normal and CLP utterances, together with the two-group fairness score. The dynamic time warping distance supplies a graded distortion axis from mild to moderate to severe, motivating which utterances to add during training. The fairness score, defined as $FS = -\\alpha \\cdot \\text{Average Error Rate} - \\beta \\cdot \\text{Error Disparity}$, collapses the normal-versus-CLP word error rates into a single number, with values closer to zero meaning fairer. The mixing strategy trains a model on normal speech plus a severity subset of CLP speech and evaluates on the full test set, allowing the choice of severity composition to be tuned for a given model and language.","core_discovery":"The central claim is that ASR fairness for cleft lip and palate speech can be improved by severity-aware data mixing: rather than training only on normal speech or only on CLP speech, the authors train on normal speech augmented with CLP utterances in order of increasing severity. They introduce a fairness score that trades off the average error rate against the error disparity between normal and CLP groups, with values closer to zero indicating a fairer system. Spectrogram, formant-contour, and dynamic time warping diagnostics show that spectral distortion grows progressively from mild to moderate to severe CLP speech, which justifies mixing by severity groups. In their experiments, adding mild, moderate, and severe CLP utterances to the normal training data improves pooled word error rate and moves the fairness score closer to zero on both corpora; the best configuration for GMM-HMM on the Kannada corpus includes severe speech, while for Whisper on the English corpus the best configuration stops at moderate. The reported fairness-score improvements are 17.89% on the Kannada corpus and 47.50% on the English corpus.","pith_inferences":["If the effect is driven by spectral overlap rather than speaker identity, the same severity-graded mixing protocol may transfer to other graded speech disorders, such as dysarthria or stuttering, where distortion increases with severity.","Because the reported experiments split recordings randomly rather than by speaker, a speaker-disjoint evaluation is the direct test of whether the gain is acoustic generalization or speaker memorization; that check is not reported.","A natural extension is to condition the model on the severity label itself, for instance through an auxiliary loss or an embedding, which could combine the benefit of mixing with explicit severity awareness.","The two-group fairness score could be extended to multi-severity fairness by summing pairwise disparities or using the maximum group error, rather than collapsing all CLP speakers into one group."],"forward_implications":["On both Kannada child speech and English child speech, replacing a normal-only training set with a severity-mixed set reduces the gap between normal and CLP word error rates.","The best mixing recipe is not universal: for GMM-HMM on the Kannada corpus it includes severe CLP speech, while for Whisper on the English corpus adding severe utterances hurts, so the recipe must be chosen per model and language.","The fairness score supports deployment choices: setting $\\alpha=0.9$, $\\beta=0.1$ prioritizes overall accuracy, while $\\alpha=0.1$, $\\beta=0.9$ prioritizes reducing disparity between normal and CLP groups.","Training with CLP data barely degrades performance on normal speech, so the fairness improvement is not bought by sacrificing typical speech recognition.","Severity-aware mixing improves fairness for a conventional GMM-HMM model, a self-supervised foundation model, and a supervised foundation model, suggesting the mechanism is not tied to one architecture."],"supporting_citations":[{"why":"Supplies the English child-speech corpus with mild, moderate, and severe CLP labels used for the Whisper word error rate improvement from 28.45% to 18.89%.","marker":"[45]"},{"why":"Supplies the Kannada child-speech corpus used for the main GMM-HMM severity-mixing experiments and the improvement from 22.64% to 18.76%.","marker":"[48]"},{"why":"Supplies the self-supervised XLS-R model whose performance is compared across mixing conditions.","marker":"[18]"},{"why":"Supplies the Whisper model behind the English-corpus word error rate drop and the best fairness-score gain.","marker":"[19]"},{"why":"Provide the fairness-metric ideas on which the paper's fairness score is based.","marker":"[50, 51]"},{"why":"Provides the dynamic time warping distance used to quantify spectral distortion across CLP severity levels.","marker":"[52]"},{"why":"Supplies prior evidence that data augmentation helps CLP speech recognition, which this paper extends to fairness.","marker":"[38]"}],"fun_headline_variants":["Severity-aware mixing of CLP speech improves ASR fairness by 47.5%","Mixing cleft-palate speech by severity cuts ASR WER by up to 39%","ASR fairness for cleft-palate speech boosted via severity-graded data mixing","Graded mixing of cleft and normal speech reduces ASR errors and disparity","Severity-based training data mixing: fairer ASR for cleft-palate speech"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation uses a random 80/20 split of individual recordings, so the same speaker can appear in both the training and testing sets; if that overlap drives the results, the reported gains may come from remembering a speaker's voice rather than from learning to recognize cleft-palate speech in general.","fun_headline_variants_meta":{"raw":{"variants":["Severity-aware mixing of CLP speech improves ASR fairness by 47.5%","Mixing cleft-palate speech by severity cuts ASR WER by up to 39%","ASR fairness for cleft-palate speech boosted via severity-graded data mixing","Graded mixing of cleft and normal speech reduces ASR errors and disparity","Severity-based training data mixing: fairer ASR for cleft-palate speech"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001111,"raw_usage":{"total_tokens":4682,"prompt_tokens":1049,"completion_tokens":3633,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":665,"completion_tokens_details":{"reasoning_tokens":3525}},"tokens_in":665,"tokens_out":3633,"duration_ms":27840,"temperature":1.0,"reasoning_tokens":3525,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:44:23.382798+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the best severity-mixed training condition with a speaker-disjoint split, where all recordings of each speaker are assigned wholly to either training or testing, on both corpora, and compare the pooled word error rate and fairness score against the paper's utterance-level split. If the improvement disappears or falls below a meaningful margin, the reported fairness gain is speaker memorization rather than generalization to unseen CLP speakers.","supporting_citations":[{"cited_title":"Nikitha, S","cited_arxiv_id":null,"evidence_quote":"Supplies the Kannada child-speech corpus used for the main GMM-HMM severity-mixing experiments and the improvement from 22.64% to 18.76%."},{"cited_title":"Sakoe, Dynamic programming algorithm optimization for spoken word recognition, IEEE Transactions on Acoustics, Speech, and Signal Processing 26 (1978) 159–165","cited_arxiv_id":null,"evidence_quote":"Provides the dynamic time warping distance used to quantify spectral distortion across CLP severity levels."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies prior evidence that data augmentation helps CLP speech recognition, which this paper extends to fairness."}],"review_version":1}