{"id":"35f6750d-47bc-4b88-8788-0f5e255a8883","arxiv_id":"2412.03784","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"ASR transcriptions and word-boundary timestamps as classification features raise balanced accuracy for dysarthria severity to 83.72 percent on a Korean dataset, beating waveform and deep-model baselines.","lead":"The authors extract features from a speech-recognition system's transcriptions, rather than only from sound waveforms, to classify how severe a person's dysarthria is. In tests on Korean patients reading a standard paragraph, the new features raised balanced accuracy to 83.72 percent and improved detection of the most severe cases.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unknown speaker split—same-patient utterances may leak across train/test, and the balanced-accuracy (83.72%) result rests on 18 severity-2 test samples with no significance or error bars.","rationale":"The reader identified the same load-bearing issue: the 8:1:1 split is stratified only by severity and does not state that speakers are disjoint, so the reported 83.72% balanced accuracy may reflect speaker leakage rather than severity-relevant generalization. My independent check of the manuscript confirms that no speaker-level exclusion or patient-ID-based splitting is described anywhere in Sections 4.1-4.4, and the test set for the decisive severity-2 class contains only 18 utterances. Because the central claim is explicitly empirical (\"balanced accuracy of 83.72%\"), and because the paper provides no confidence intervals, no repeated-seed variance, and no per-speaker evaluation, the result is currently unfalsifiable from the text alone. The paper has real strengths: the feature design is clearly grounded in clinical categories (Duffy's subsystems), the ablation and integration experiments (Tables 6-8) partially control for feature-count effects, and the code is promised public, which makes the requested check feasible. I do not see an internal inconsistency in the feature construction itself, only an insufficiently controlled evaluation. Hence I agree with the reader's conditional verdict and would keep it conditional pending a speaker-disjoint retest. The recommended action is not to reject the contribution but to require the speaker-disjoint experiment and a significance statement before the headline number can be accepted.","tokens_in":10206,"tokens_out":1595,"duration_ms":15048,"concrete_test":"Ask the authors to release a speaker-disjoint 8:1:1 split (or a speaker-ID-restricted retest: hold out all utterances from a set of patients, then rerun both SR-features and the Waveform baseline through the same AutoGluon pipeline). If the severity-2 test cell must come from unseen speakers, recompute balanced accuracy and report the number of distinct speakers in each test cell, plus bootstrap 95% CIs. The central claim survives only if balanced accuracy remains above the baselines with disjoint speakers and the precision on severity 2 does not collapse.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that SR-features give 83.72% balanced accuracy, mainly by perfect recall (18/18) on severity 2. The paper's Section 4.1 says only \"the ratio of 8:1:1, considering the ratio of each severity level,\" with no statement that speakers/patients are disjoint between train and test. The dataset contains paragraph readings, so it is natural for one patient to contribute multiple utterances (the cited AI-Hub Korean dysarthric corpus is organized by speaker and has multiple records per speaker). If a speaker appears in both training and test partitions, then speaker-specific attributes—including how that speaker reads the fixed Autumn paragraph—are shared, and the classifier can memorize or latch onto speaker-level patterns rather than generalizable severity cues. The severity-2 class has only 18 test utterances; even a few speakers in that cell could dominate recall. No standard deviation, confidence interval, or significance test is reported, and the binary formula in Equations 1-2 does not formally match the three-class balanced accuracy (macro-average recall). Because the reported improvement over the waveform baseline (83.72 vs 68.96) and over DNNs (61.28-62.47) is the entire evidence for the contribution, an unknown speaker split makes the headline empirical result unverifiable from the manuscript alone. The limitation statement also notes an unresolved class disparity (accuracy for severity 1 degrades), which does not rescue the central claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an automatic dysarthria severity classification method based on features extracted from a fine-tuned Whisper ASR model (DysarthricWhisper). The features fall into two families: Pronunciation Correctness (ASR error metrics, BERTScore, disfluency measures) and Structural Prosody (pause location/duration, articulation duration, rhythm). These are fed to an AutoGluon tabular classifier and evaluated on a Korean dysarthric speech dataset from AI-Hub, with a reported balanced accuracy of 83.72% for the proposed SR-features, compared with 68.96% for waveform-based features and 53.69–62.47% for the tested DNN baselines. The paper includes an ablation of the two feature families and an integration experiment to argue that the gain is not simply due to a larger feature count. Code is made publicly available.","tokens_in":10395,"tokens_out":9446,"duration_ms":101400,"significance":"If the evaluation is valid, the contribution is practically useful: the feature taxonomy maps naturally onto clinically used perceptual categories, the pipeline retains feature-level explainability, and the reported balanced accuracy is competitive with DNNs while improving recall for the severe class. The paper's strengths are its clearly specified feature definitions (Table 2), the inclusion of ablation and integration experiments that rule out feature-count as the sole explanation, and the stated intention to release code. The central claim is not circular: severity labels are not used to train the ASR feature extractor, and the pause-token method is external prior work. However, the empirical headline rests on a very small severe-class test set and an underspecified data-split procedure, so the key quantitative claim is not yet verifiable from the manuscript.","major_comments":[{"comment":"The evaluation does not establish that the train and test sets are speaker-disjoint. Section 4.1 states only that the corpus is split 'in the ratio of 8:1:1, considering the ratio of each severity level,' but does not state whether the unit of random assignment is an utterance or a speaker. The test-set sizes in Table 5 (54 severity-0, 300 severity-1, 18 severity-2; 372 total) are also inconsistent with a strict 10% holdout of the 2,567 utterances in Table 1, so the actual split procedure is unclear. If speakers contribute multiple utterances and the same speaker appears in both training and test, the classifier and the fine-tuned Whisper model can exploit speaker-specific voice characteristics, and the reported 83.72% balanced accuracy would not reflect generalization to unseen patients. Please report the number of speakers, the assignment unit, and results under a speaker-exclusive split, or demonstrate that each speaker contributes exactly one utterance.","section":"Section 4.1, Table 5"},{"comment":"The balanced-accuracy definition is given only for a binary classifier, but the task has three severity classes. The reported 83.72% appears to match macro-average recall across the three classes, but this is not stated, and Equations (1)–(2) do not define a multiclass balanced accuracy. Please correct the metric definition, state explicitly how the multiclass balanced accuracy was computed, and report per-class recall and precision alongside the aggregate number.","section":"Equations (1)–(2), Section 4.2"},{"comment":"The main improvement is driven by perfect recall on the 18 severity-2 test utterances. The 95% confidence interval for 18/18 successes is approximately [81.5%, 100%], and the balanced-accuracy gap between SR-features (83.72%) and the waveform baseline (68.96%) is reported without any variance estimate or significance test. Please provide bootstrap confidence intervals or a permutation test, ideally clustered by speaker, to show that the difference is not attributable to the small size of the severe class.","section":"Section 4.2, Table 5"},{"comment":"The reference duration sequence used for WS DTW and Abnormal Speed is described as 'the average duration sequence of 360 samples read by healthy individuals,' but the paper does not specify the provenance of these 360 samples. If they are drawn from the severity-0 utterances of the same corpus and if those speakers appear in the test split, the reference statistics are partly constructed from test data, which can inflate the reported performance for severity 0 and potentially for other classes. Please state where the reference samples come from, and if they are from the corpus, ensure they are disjoint from the test set or re-estimate the reference on training data only.","section":"Section 3.2"}],"minor_comments":[{"comment":"There are typos in the labels: 'Integerated-without-FS', 'Integarted-before-FS', and 'Integrated-wihtout-FS' should be 'Integrated-without-FS' and 'Integrated-before-FS'.","section":"Section 4.4, Table 8"},{"comment":"The entry '00.00%' in the last row of the severity-2 column should be '0.00%'.","section":"Table 5(a)"},{"comment":"The filler-word tokens used in Filler Words Similarity ([2], [Wm], [W], [kW]) are not defined; please describe the token set and how fillers were identified in the transcription.","section":"Table 2, Section 3.1"},{"comment":"The Top-30% short/long word-segment threshold is a free parameter, and no sensitivity analysis is reported; a short experiment varying this threshold would clarify how dependent the results are on this choice.","section":"Section 3.2, Table 2"},{"comment":"The training details for the DNN baselines are incomplete (e.g., number of epochs, learning-rate schedule, whether Whisper/wav2vec2.0 were fine-tuned or used as frozen feature extractors); adding these details would improve reproducibility. The Discussion also acknowledges the severity-1 accuracy drop, but because severity 1 is the majority class, the paper should discuss whether the balanced-accuracy improvement corresponds to a clinically preferable operating point.","section":"Section 4.1"}],"recommendation":"major_revision","confidential_remarks":"The unresolved speaker-leakage risk is the main threat to the headline result, and the test-size inconsistency in Table 5 compounds it. I would ask the authors for a speaker-independent split (or explicit evidence that speakers are unique) and for confidence intervals before considering acceptance. The paper is otherwise a reasonable empirical contribution, and the feature design is well motivated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The new thing here is real: using ASR transcriptions and word-boundary timestamps as the feature source for dysarthria severity classification, rather than waveform, glottal, prosodic, or DNN embeddings. The feature design is clinically motivated, careful, and clearly described. The ablation showing that structural prosody alone nearly matches the full set, the integration experiment that rules out feature count as the driver, and the public code all earn credit. The writing is straightforward and the limitations section is honest about severity-1 degradation.\n\nThe soft spot is the one the stress-test flags, and it is load-bearing. The split is described only as 8:1:1 stratified by severity, with no statement that speakers are disjoint between train and test. The dataset is paragraph readings with multiple utterances per speaker, so same-speaker leakage is plausible. If present, the 83.72% balanced accuracy and the perfect 18/18 recall on severity 2 become unreliable. The severe class has only 18 test samples, and there are no confidence intervals or significance tests. The binary balanced-accuracy formula in Equations 1–2 also does not match the three-class macro-average recall actually reported. That said, the concern is fixable: a speaker-independent split and per-class error bars would settle it. The central idea could still stand even if the reported number shrinks.\n\nWho is this for? Anyone working on clinically oriented dysarthria assessment, especially in Korean, and researchers interested in explainable features derived from ASR outputs. The paper is worth a serious referee, but it needs revision before the empirical claim can be accepted. My recommendation: send it to peer review, and require speaker-independent evaluation and proper statistical reporting in the revision.","headline":"A useful, well-designed feature idea for dysarthria severity classification, but the headline 83.72% balanced accuracy is unverifiable from the paper as written because the train/test split may not be speaker-disjoint.","tokens_in":11006,"tokens_out":721,"would_cite":false,"duration_ms":9155,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that speech-recognition-based features, built from transcriptions and word boundaries, classify dysarthria severity with higher balanced accuracy than waveform-based features or tested deep models on a Korean post-stroke…","keywords":["dysarthria severity classification","speech recognition features","Whisper fine-tuning","explainable machine learning","pronunciation correctness","structural prosody","Korean dysarthric speech","balanced accuracy"],"falsifier":"Run the same feature extraction and AutoML pipeline on a speaker-disjoint split of the same dataset, so that no patient contributes utterances to both training and test. If balanced accuracy drops to the waveform baseline or below, the method's advantage is an artifact of speaker leakage; if it stays near 83.72%, the ASR-feature claim survives.","tokens_in":9927,"feed_emoji":"🗣️","tokens_out":7240,"duration_ms":63558,"temperature":0.7,"pith_summary":"This paper argues that the output of a speech recognizer—transcription text plus word and pause boundaries—can serve as a stronger feature source for automatic dysarthria severity classification than the raw waveform alone. The authors fine-tune Whisper on Korean dysarthric speech, extract two feature families (pronunciation correctness and structural prosody) from the resulting transcription, and feed them to an off-the-shelf AutoML classifier. On a post-stroke Korean dataset graded by clinicians, this reaches 83.72% balanced accuracy, compared with 68.96% for waveform-based features and no better than 62.47% for the deep models tested. Most of the gain is recall for severity 2, the most severe class that the DNN baselines miss entirely. A sympathetic reader would take the paper's claim to be that ASR-based features combine explainability with performance, and that this feature source deserves attention in clinical speech assessment.","feed_headline":"ASR transcripts outscore waveforms in dysarthria severity grading","feed_subtitle":"Features pulled from Whisper transcriptions reach 83.72% balanced accuracy and catch the most severe cases.","key_machinery":"The machinery is a two-stage pipeline whose carrier is a fine-tuned speech recognizer. Whisper is fine-tuned on dysarthric Korean speech with a special pause token added, and the resulting model outputs both a transcription and word segment boundaries obtained by aligning the audio to the inferred transcription. From these outputs the paper computes 'SR-features' in two families: Pronunciation Correctness, using syntactic edit metrics, semantic BERT score, and disfluency measures; and Structural Prosody, using pause-sequence edit distances, pause duration statistics, word-segment duration statistics, and rhythm measures such as speed change rate and syllables per second. These features carry the argument because they encode clinical description criteria that pure waveform features cannot express, such as whether a pause occurs at a linguistically expected location.","core_discovery":"The paper's central discovery is that the mismatch between what a speaker intended to read and what a dysarthria-adapted recognizer transcribes is itself a diagnostic signal, and so is the timing of the word segments the recognizer produces. Using Whisper fine-tuned on dysarthric speech, the authors obtain transcriptions and word-level timestamps, then define two feature families: Pronunciation Correctness, covering ASR-style edit metrics, BERT score, repetition and filler-word similarity; and Structural Prosody, covering pause locations and durations, articulation durations, and rhythm. In the main experiment these features give balanced accuracy of 83.72% (accuracy 72.85%) with an AutoML classifier, versus 68.96% balanced accuracy for waveform features and 62.47% for the best DNN baseline. The confusion matrix shows the SR-feature model finds 18 of 18 severity-2 utterances, while Whisper+Linear finds 0 of 18 and waveform features find 8 of 18. The paper also reports that the method sacrifices some accuracy on the majority class (severity 1), which it attributes to remaining ASR error.","pith_inferences":["The reported 8:1:1 split is at the utterance level, not the speaker level; if the same patient contributes utterances to both training and test, part of the balanced-accuracy gain could be speaker-specific rather than severity-specific. A speaker-disjoint split is the natural test of this.","The feature families are language-agnostic in principle, but they depend on having a dysarthria-adapted recognizer for the target language; in languages without such a model, the method's advantage may shrink.","The same transcription-alignment pipeline could be extended to predict continuous severity scores or to flag which words a patient mispronounced as direct therapy feedback, neither of which the paper evaluates.","Because the authors note severity-1 accuracy falls, the method currently trades majority-class accuracy for minority-class recall; clinical use may require a decision threshold tuned to the cost of missing severe cases."],"forward_implications":["If the central claim is right, explainable machine-learning classifiers can match or beat deep models on dysarthria severity grading, at least on balanced accuracy, while returning feature-level explanations clinicians can inspect.","The most severe class—the one missed by the tested deep models—becomes the best-recalled class (18/18 in the test set), which matters for screening and therapy planning.","The top-ranked features all come from Structural Prosody, suggesting that timing and rhythm carry the most severity signal among the proposed features.","Because the features rely on ASR output rather than manual transcription, the approach could reduce the annotation burden of prosodic analysis if it transfers to other languages and datasets.","Combining waveform and SR-features did not improve balanced accuracy, so the advantage is not simply a matter of feature count."],"supporting_citations":[{"why":"Supplies the pause-token fine-tuning method that lets Whisper detect pause locations and durations.","marker":"[25]"},{"why":"Defines Whisper, the ASR model the paper fine-tunes into DysarthricWhisper.","marker":"[24]"},{"why":"Provides the clinically defined dysarthria subsystems that inspire the two SR-feature families.","marker":"[22]"},{"why":"Defines the waveform-feature baseline set the proposed features are compared against.","marker":"[33]"},{"why":"Supplies the AutoML classifier used for all machine-learning baselines and the proposed features.","marker":"[31]"},{"why":"Defines BERT score, the semantic correctness feature used in Pronunciation Correctness.","marker":"[26]"},{"why":"Supports the choice of BERT score as a measure of ASR errors in disordered speech.","marker":"[28]"},{"why":"Provides the wav2vec 2.0 representation used as a DNN baseline.","marker":"[35]"},{"why":"Provides a Whisper-feature DNN baseline with a ResNet classifier.","marker":"[36]"},{"why":"Defines the Autumn paragraph reading task, the standard stimulus for severity evaluation in the dataset.","marker":"[23]"}],"fun_headline_variants":["ASR edits and pauses rank dysarthria severity","Whisper transcript gaps reveal dysarthria severity","Mismatched ASR text predicts dysarthria severity","Dysarthria severity via speech recognition errors","ASR features hit 83.7% balanced accuracy in dysarthria"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the utterance-level 8:1:1 data split keeps training and test independent; if utterances from the same patient appear in both, the reported balanced accuracy could reflect speaker identity rather than severity.","fun_headline_variants_meta":{"raw":{"variants":["ASR edits and pauses rank dysarthria severity","Whisper transcript gaps reveal dysarthria severity","Mismatched ASR text predicts dysarthria severity","Dysarthria severity via speech recognition errors","ASR features hit 83.7% balanced accuracy in dysarthria"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000676,"raw_usage":{"total_tokens":3067,"prompt_tokens":928,"completion_tokens":2139,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":544,"completion_tokens_details":{"reasoning_tokens":2058}},"tokens_in":544,"tokens_out":2139,"duration_ms":16265,"temperature":1.0,"reasoning_tokens":2058,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T22:06:25.512309+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same feature extraction and AutoML pipeline on a speaker-disjoint split of the same dataset, so that no patient contributes utterances to both training and test. If balanced accuracy drops to the waveform baseline or below, the method's advantage is an artifact of speaker leakage; if it stays near 83.72%, the ASR-feature claim survives.","supporting_citations":[{"cited_title":"Dysarthric Speech Classification Using Glottal Features Computed from Non-words, Words and Sentences,","cited_arxiv_id":null,"evidence_quote":"Defines Whisper, the ASR model the paper fine-tunes into DysarthricWhisper."},{"cited_title":"Inspired by these attributes, we designed two main categories of fea- tures: Pronunciation Correctness, and Structural Prosody","cited_arxiv_id":null,"evidence_quote":"Provides the clinically defined dysarthria subsystems that inspire the two SR-feature families."},{"cited_title":"Phonological features in discrimina- tive classification of dysarthric speech,","cited_arxiv_id":null,"evidence_quote":"Supplies the pause-token fine-tuning method that lets Whisper detect pause locations and durations."},{"cited_title":"Inappropriate pause detection in dysarthric speech using large-scale speech recognition,","cited_arxiv_id":null,"evidence_quote":"Defines the waveform-feature baseline set the proposed features are compared against."},{"cited_title":"Dysarthria evaluation,","cited_arxiv_id":null,"evidence_quote":"Supplies the AutoML classifier used for all machine-learning baselines and the proposed features."},{"cited_title":"Neurospeech: An open-source software for parkinson’s speech analysis,","cited_arxiv_id":null,"evidence_quote":"Defines BERT score, the semantic correctness feature used in Pronunciation Correctness."},{"cited_title":"Speech Intelligibility of Dysarthric Speech: Human Scores and Acoustic-Phonetic Features,","cited_arxiv_id":null,"evidence_quote":"Supports the choice of BERT score as a measure of ASR errors in disordered speech."},{"cited_title":"BERT: pre-training of deep bidi- rectional transformers for language understanding,","cited_arxiv_id":null,"evidence_quote":"Provides the wav2vec 2.0 representation used as a DNN baseline."},{"cited_title":"Grad-cam: Visual explanations from deep net- works via gradient-based localization,","cited_arxiv_id":null,"evidence_quote":"Defines the Autumn paragraph reading task, the standard stimulus for severity evaluation in the dataset."}],"review_version":1}