{"id":"64d6d670-ee82-4e8e-a14b-0cc39404ddb3","arxiv_id":"2412.12111","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Combining language-universal and language-specific acoustic features in XGBoost improves multilingual dysarthria severity classification by 7.33% relative over a language-universal-only baseline.","lead":"This PhD thesis applies voice biomarkers and machine learning to classify dysarthria severity in English, Korean, and Tamil, adding language-specific features on top of shared ones. The method reports small accuracy gains over standard multilingual baselines, but the experiments have unresolved evaluation issues.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Feature selection in §7.3.1 is not demonstrably nested inside LOSO; test-speaker information may select the optimal feature sets, inflating the reported 7.33% gain.","rationale":"The reader's weakest_assumption identifies exactly the load-bearing issue: Section 7.3.1 describes feature importance under LOSO but then uses iterative accuracy-based elimination to select optimal per-language feature sets, without stating that this selection is nested inside each training fold. This is not a manufactured concern; the paper's own Section 7.5.2 emphasizes preventing data leakage, yet the described protocol does not show nested selection. The reported 7.33% relative improvement over the intersection baseline is the central quantitative claim, so any test-set influence on feature subsets directly threatens it. The Tamil data-ratio selection in Section 7.5.1 is a second, compounding form of test-set selection: the authors explicitly chose 80% because it yielded the highest performance. However, the feature-selection protocol is the more fundamental issue because it affects all reported multilingual results, including the baselines derived from Table 7.2. I agree with the reader's CONDITIONAL verdict; no verdict change is needed, but the concrete nested-selection check should be run before accepting the improvement as stated. The paper has real strengths: public datasets, two-tier statistical/clinical validation, and a clear comparison to prior work, so the concern is about evaluation protocol, not about the underlying hypothesis being unreasonable.","tokens_in":53519,"tokens_out":4766,"duration_ms":47689,"concrete_test":"Re-run the Section 7.3.1 feature selection inside each LOSO training fold: for each held-out speaker, compute XGBoost gain importance and the backward-elimination accuracy using only training speakers' utterances, selecting the feature subset by internal cross-validation on that training fold; then train on the selected features and evaluate on the held-out speaker. Compare the average weighted F1 and the relative improvement over intersection/union. If the proposed method's advantage over the intersection baseline drops from 7.33% to a value within the 0.93% gap observed over the union baseline, the reported gain is largely an artifact of feature-selection leakage.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Abstract; §7.4.2) is that combining language-universal and language-specific features yields a 7.33% relative improvement over universal-only multilingual baselines, at 71.74% average weighted F1. Section 7.3.1 computes XGBoost gain importance under LOSO and then 'systematically remove[s] the least important feature and reassess[es] classification accuracy' to pick each language's optimal set (Table 7.2). As described, the importance scores are averaged across all LOSO folds, and the backward-elimination accuracy is reassessed on the same corpus; there is no statement that the per-fold held-out speaker is excluded during feature-subset selection. If subset selection uses the full LOSO predictions, the optimal feature sets are chosen with knowledge of the test speakers, so the per-language features in the proposed method (and the derived intersection/union baselines) are selected on the test set. The reported 71.74% and the 7.33% relative gain are then optimistic. The discussion in §7.5.2 claims to prevent data leakage with 'LOSO ... for feature importance scoring and selection,' but scoring under LOSO is not the same as nesting selection inside each training fold. This is the load-bearing weak point: the qualitative claim that language-specific features help may survive, but the headline improvement is not established without nested selection. The §7.5.1 choice of 80% Tamil data based on highest test performance compounds the optimism, but the feature-selection leak alone is sufficient to cast doubt.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript, based on a PhD dissertation, proposes a multilingual dysarthria severity classification method for English (TORGO), Korean (QoLT), and Tamil (SSNCE). It first analyzes paralinguistic features and Goodness-of-Pronunciation scores, then validates a clinical-knowledge-driven feature set through statistical and clinical criteria, and finally proposes an XGBoost-based classifier that combines language-universal and language-specific features. The central quantitative claim is an average weighted F1 of 71.74% and a 7.33% relative improvement over a language-universal-only intersection baseline.","tokens_in":53798,"tokens_out":5964,"duration_ms":54285,"significance":"If the evaluation were fully valid, this would be a useful contribution: it addresses a real gap in multilingual dysarthria assessment, uses three typologically distinct public datasets, and provides a clinically motivated feature set with per-language analysis. Strengths include the two-tier statistical/clinical validation, the comparison against eGeMAPS and MFCC baselines, the detailed per-language tables, and the explicit discussion of limitations such as the default missing-value handling in XGBoost. However, the headline quantitative claims rest on evaluation choices that are not fully nested and on a data-ratio selection made after observing test performance; the exact magnitudes of the reported gains are therefore not established by the current experiments.","major_comments":[{"comment":"The feature-selection procedure is not demonstrably nested inside the LOSO folds. The text states that feature importance scores are averaged across LOSO folds and that the least important feature is then removed and classification accuracy 'reassessed,' but it does not state that the reassessment is performed separately inside each training fold before evaluating the held-out speaker. As described, the optimal per-language feature sets in Table 7.2 and the proposed-method row of Table 7.5 are selected using information from the same speakers that are later reported as test speakers. This is a form of test-set leakage and would inflate the reported 71.74% average F1 and the 7.33% relative gain. A nested selection—choosing the subset inside each training fold and applying only that fold's subset to the held-out speaker—must be reported for the claimed improvement to be credible.","section":"§7.3.1, Tables 7.2 and 7.5"},{"comment":"The distance-to-healthy normalization and the feature validation step are fitted on the full corpus before the LOSO split. The healthy-control mean and standard deviation in Eq. (7.1) are computed over all healthy speakers, including those later held out in the classification experiments; likewise, the Kruskal-Wallis and Kendall validation in Section 6.3.1 uses all speakers to decide which features are retained. Even if the later classifier training is LOSO, the feature representation and the candidate feature set have already seen the test speakers. This is a second, independent source of optimistic bias in the reported results.","section":"§7.2.2, Eq. (7.1), and §6.3.1"},{"comment":"The choice of the 80% Tamil utterance ratio is made after observing classification performance on the evaluation set. The paper states that 'the highest classification performance was achieved using 80% of the utterances, forming the basis for the results presented in this paper.' Because the data ratio is selected on the same test data, the reported 71.74% average is a maximum over data-ratio choices rather than a fixed-procedure estimate. This compounds the optimism from the feature-selection issue.","section":"§7.5.1, Figure 7.4"},{"comment":"The reported relative improvements do not match Table 7.5. From Table 7.5, the proposed method (71.74) versus the intersection baseline (66.47) is a relative gain of 7.93%, not 7.33%, and versus the union baseline (70.68) it is 1.50%, not 0.93%. In addition, the claim that the proposed method 'significantly outperforms' the baselines is not supported by any significance test; with only 15, 80, and 30 speakers in the three datasets, differences of this size could be within sampling variability. The manuscript should either correct the reported percentages or clarify which experimental setting they come from, and it should provide significance testing or confidence intervals for the headline comparison.","section":"Abstract and §7.4.2"}],"minor_comments":[{"comment":"The entropy formula appears to have a typo: the logarithm in the sum uses P(p|F) instead of P(q|F), so the equation should be sH(p) = -Σ_q P̄(q|F) log P̄(q|F).","section":"§5.1.3, Eq. (5.6)"},{"comment":"The text reports Korean FCR τ = 0.14 and VAI τ = 0.44, but Table 6.3 lists FCR τ = 0.44 and VAI τ = -0.44; the signs and values should be reconciled.","section":"§6.4.1"},{"comment":"The worked example contains '2/5100 = 40.00%' and '5/8100 = 62.50%'; these should read 2/5 × 100 and 5/8 × 100, respectively.","section":"§6.2.2"},{"comment":"The header 'T amil T amil' contains a typo and should read 'Tamil Tamil'.","section":"Table 6.5"},{"comment":"The sentence 'the relative increase from monolingual to multilingual experiments was much lower than in the previous study (18% vs. 6.85%)' is unclear; the manuscript should specify what quantities these percentages refer to.","section":"§7.5.2"},{"comment":"As written, the distance metric assigns smaller values to larger deviations from the healthy mean (for |f_i - μ_h| > σ_h, the value is σ_h / |f_i - μ_h|, which decreases as the deviation grows), which conflicts with the description of measuring 'degree of acoustic atypicality.' Either the formula or the interpretation should be revised.","section":"§7.2.2, Eq. (7.1)"}],"recommendation":"major_revision","confidential_remarks":"The manuscript appears to be a lightly revised PhD thesis posted on arXiv. If it is being considered as a journal article, the substantial overlap with the author's prior conference papers (Yeo et al. 2022a, 2022b, 2023b) should be disclosed in a prior-publication statement. The editorial question is separate from the technical evaluation above."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe useful core is the three-language study itself: clinically validated feature sets for English, Korean, and Tamil, a careful monolingual/cross-lingual/multilingual comparison, and a sensible argument that language-specific features matter. The GoP chapter with uncertainty quantification is real work, and the appendix tables give enough detail to re-run much of it. I came away convinced the qualitative direction is right; I am not convinced by the headline number.\n\nThe claimed 7.33% relative gain over the language-universal-only baseline depends on feature subsets chosen in §7.3.1. Importance is computed under LOSO, but the iterative backward elimination then \"reassess[es] classification accuracy\" on the same corpus. The text does not say the held-out speaker is excluded when the subset is chosen. If selection uses full LOSO predictions, the optimal per-language sets (and the intersection/union baselines derived from them) have seen the test speakers, so 71.74% and the 7.33% gain are optimistic. The §7.5.2 claim that LOSO prevents leakage is not enough; scoring under LOSO is not the same as nesting selection inside each training fold. This needs to be fixed or the claim moderated.\n\nTwo smaller issues. The 80% Tamil ratio is chosen because it gave the highest performance (§7.5.1), which is a post hoc selection on the test data. And there are no error bars or significance tests, so \"significantly outperforms\" is doing work it cannot support. The gain over the union baseline is only 0.93%; that piece is consistent with masking-as-missing being a modest implementation choice rather than a breakthrough.\n\nThe thesis is honest in places: it flags XGBoost's default missing-value handling as a limitation and acknowledges that its own earlier work had data leakage. That gives me confidence the author can engage with a critique. The self-citations are appropriate because the thesis explicitly extends those papers.\n\nWho is this for? People working on multilingual or low-resource dysarthria assessment, and anyone teaching evaluation hygiene in clinical ML. A serious referee should engage with it, mainly to demand nested feature selection and a justified Tamil-data rule before the quantitative claims are accepted. The qualitative story will likely survive; the numbers, as stated, will not.\n\nI would send it out for review rather than desk-reject.","headline":"Solid multilingual dysarthria thesis whose qualitative claim is plausible, but the headline 7.33% gain is not established because feature selection is not demonstrably nested inside LOSO.","tokens_in":54364,"tokens_out":2165,"would_cite":false,"duration_ms":22585,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A multilingual dysarthria severity classifier that combines language-universal and language-specific voice biomarkers outperforms universal-features-only models, gaining a 7.33% relative improvement in average F1 across English, Korean…","keywords":["dysarthric speech","multilingual speech analysis","automatic dysarthria severity classification","voice biomarkers","language-specific features","XGBoost","Goodness of Pronunciation"],"falsifier":"Re-run the iterative feature-selection step inside each Leave-One-Speaker-Out training fold, so that each test speaker contributes nothing to choosing the features, and compare the resulting average F1 with 71.74% and the relative gain over the intersection baseline; if the gain shrinks or reverses, the central claim is an artifact of selection leakage.","tokens_in":53254,"feed_emoji":"🗣️","tokens_out":8012,"duration_ms":85105,"temperature":0.7,"pith_summary":"This thesis claims that multilingual dysarthria severity classification is held back by reliance on language-universal acoustic features, because languages differ in phoneme inventories and rhythmic structure. It builds a clinically interpretable feature set of voice quality, pronunciation, and prosody measures, validates which features matter for English, Korean, and Tamil, and then combines each language's optimal features into one classifier. The resulting model reaches an average weighted F1 of 71.74% across three languages, a 7.33% relative improvement over the universal-features-only baseline, while matching the best monolingual models.","feed_headline":"Dysarthria grading gains 7.33% from language-specific features","feed_subtitle":"Universal-plus-language-specific biomarkers beat universal-only methods across English, Korean, and Tamil.","key_machinery":"The load-bearing mechanism is a clinical-knowledge-driven feature set whose members are screened by a two-step validation (statistical significance plus clinical directionality), then converted into distances from the healthy-speaker distribution, and finally selected per language using XGBoost gain importance. The classifier is XGBoost, which absorbs the per-language feature sets jointly by treating features not selected for a language as missing values, so universal features share training data while language-specific features do not interfere. A distance metric scales each feature by its deviation from the healthy mean in units of the healthy standard deviation, turning raw acoustic values into a measure of atypicality.","core_discovery":"The central discovery is that the features that best separate dysarthria severity levels are not the same across languages; only pronunciation-accuracy and vowel-space measures (CRR, PRR, median F0, and related features) are shared by all three languages, while voice quality and rhythm features differ. A multilingual XGBoost classifier that feeds each language its own optimal feature set, treating features irrelevant to a given language as missing, outperforms the conventional intersection (universal-only) and union (all-features) approaches. With this design, the average weighted F1 across English, Korean, and Tamil reaches 71.74%, a 7.33% relative gain over the universal-only baseline, and the multilingual model is competitive with the best monolingual classifiers.","pith_inferences":["The success of the distance-to-healthy representation suggests that a multilingual system could be bootstrapped for a new language by computing distances using healthy-speaker statistics from a closely related language, then fine-tuning with very few local dysarthric recordings.","If the nested-selection caveat holds, the largest benefit may come not from any single feature but from the decision to treat per-language feature sets as missing values, a recipe that could generalize to other clinical speech tasks such as stuttering or aphasia assessment.","The fact that only pronunciation-related features were universal across all three languages hints that future multilingual models should prioritize phoneme-level supervision and cross-lingual phonetic encoders rather than generic acoustic features."],"forward_implications":["Multilingual dysarthria assessment can match the accuracy of language-specific monolingual models without sacrificing coverage, since the proposed method reaches 71.74% average F1 versus 71.72% for the best monolingual runs.","Universal-features-only multilingual models are provably leaving performance on the table; the intersection baseline trails the proposed method by 7.33% relative.","Clinically validated feature sets are small (7–16 per language) and interpretable, so the same pipeline could be adapted to other languages by re-running validation and selection on local data.","The distance-to-healthy representation makes the classifier robust to variations in loudness or duration, because each feature is normalized by healthy-speaker spread.","Features shared across languages, such as phoneme recognition rates and vowel space, are candidates for cross-lingual transfer to low-resource languages."],"supporting_citations":[{"why":"Prior multilingual dysarthria classification on the same three datasets; the baseline whose data-leakage issues this thesis corrects and whose F1 scores the proposed method is compared against.","marker":"Yeo et al. (2022a)"},{"why":"TORGO English dysarthric speech dataset used for the English experiments.","marker":"Rudzicz et al. (2011)"},{"why":"QoLT Korean dysarthric speech dataset used for the Korean experiments.","marker":"Choi et al. (2012)"},{"why":"SSNCE Tamil dysarthric speech dataset used for the Tamil experiments.","marker":"TA et al. (2016)"},{"why":"XLS-R cross-lingual self-supervised model fine-tuned to extract phoneme-recognition-rate features (CRR, VRR, PRR).","marker":"Babu et al. (2022)"},{"why":"Source of the distance-from-healthy-distribution metric that converts acoustic features into atypicality scores.","marker":"Zusag et al. (2023)"},{"why":"Montreal Forced Aligner used to obtain phoneme alignments for pronunciation and vowel-space features.","marker":"McAuliffe et al. (2017)"}],"fun_headline_variants":["Language-specific features boost dysarthria severity scoring by 7.33%","Multilingual dysarthria model gains 7.33% with tailored features","Tailored voice biomarkers outperform universal ones in dysarthria grading","Language-specific voice features improve dysarthria severity classification","7.33% relative gain for dysarthria grading using per-language features"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole gain rests on per-language feature sets being chosen without ever looking at the held-out test speakers; if the feature-selection step was run on the full corpus including test speakers, the 71.74% average F1 and the 7.33% relative improvement are optimistic.","fun_headline_variants_meta":{"raw":{"variants":["Language-specific features boost dysarthria severity scoring by 7.33%","Multilingual dysarthria model gains 7.33% with tailored features","Tailored voice biomarkers outperform universal ones in dysarthria grading","Language-specific voice features improve dysarthria severity classification","7.33% relative gain for dysarthria grading using per-language features"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000776,"raw_usage":{"total_tokens":3365,"prompt_tokens":814,"completion_tokens":2551,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":430,"completion_tokens_details":{"reasoning_tokens":2458}},"tokens_in":430,"tokens_out":2551,"duration_ms":19327,"temperature":1.0,"reasoning_tokens":2458,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T05:09:51.839913+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the iterative feature-selection step inside each Leave-One-Speaker-Out training fold, so that each test speaker contributes nothing to choosing the features, and compare the resulting average F1 with 71.74% and the relative gain over the intersection baseline; if the gain shrinks or reverses, the central claim is an artifact of selection leakage.","supporting_citations":[{"cited_title":", author Namasivayam, A.K","cited_arxiv_id":null,"evidence_quote":"TORGO English dysarthric speech dataset used for the English experiments."},{"cited_title":", author Wagner, L","cited_arxiv_id":null,"evidence_quote":"Source of the distance-from-healthy-distribution metric that converts acoustic features into atypicality scores."}],"review_version":1}