REVIEW 4 major objections 5 minor 36 references
Multimodal Assessment of Speech Impairment in ALS Using Audio-Visual and Machine Learning Approaches
T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read The paper's central claim is that combining audio and video features improves machine-learning prediction of clinician-rated speech impairment in ALS, with the best model reaching a mean error of 0.93 on a 5-25 severity scale.
desk verdict Small, transparent pilot study; the multimodal gain claim is not statistically supported, but the nested cross-validation and honest reporting make it worth refereeing. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The mechanism is feature fusion under nested leave-one-subject-out cross-validation. Each repetition of the sentence 'Buy Bobby a puppy' is reduced to acoustic measures (fundamental frequency, jitter, shimmer, harmonics-to-noise ratio, durations, word error rate, dynamic time warping) and kinematic measures from 68 facial landmarks (mouth and jaw motion distances, velocities, asymmetry, mouth-area and eccentricity ranges, normalized by intercanthal distance). These feed SVR, MLP, and XGB regressors whose hyperparameters are tuned on the inner loop, with one participant's repetitions held out as the test set each iteration, and the target is the mean of two speech-language pathologists' total orofacial impairment scores.
What would settle it
Run the same protocol on a pre-registered cohort with at least 50 participants per group; if the audio-only regressor's mRMSE is not worse than the audio-video regressor's mRMSE in a paired significance test at p<0.05, the claim that integrating video improves speech impairment assessment is false.
Extended reading notes
Core claim
The central claim is that a multimodal audio-video pipeline can predict clinician-rated speech impairment severity in ALS more accurately than audio-only or video-only pipelines. In the paper's comparison, the best model is a support vector regression trained on the union of 18 audio features and 15 video features, achieving an mRMSE of 0.93 overall, against 0.99 for the best audio-only model and 1.21 for the best video-only model. The authors interpret the lower error and the visually aligned predictions in the multimodal condition as suggesting that acoustic and kinematic features carry complementary information about bulbar dysfunction.
Load-bearing premise
The whole multimodal-enhancement conclusion rests on 17 participants (9 healthy, 8 with ALS) and on a 0.06 difference in average error that the paper's own Friedman test does not find statistically significant.
Editorial extensions
If this is right
- If the reported accuracy holds, clinicians could obtain a quantitative severity estimate from a short smartphone recording instead of relying only on subjective rating scales.
- Frequent home-based recordings could track bulbar decline over time and support early detection of speech deterioration in ALS.
- The same regression pipeline could be adapted to other neurological conditions that affect orofacial movement and speech.
- The finding that video alone is less informative at low impairment levels suggests that multimodal monitoring may be most valuable once bulbar involvement progresses.
Reading between the lines
- A caution the authors state but do not resolve: their own Friedman test gives p=0.092, so the superiority of the multimodal model over audio-only is not statistically established in this sample.
- Because participant repetitions are treated as separate instances, the reported mRMSE likely underestimates the true error for new individuals; a properly powered subject-level analysis would be a natural next test.
- The video-only underperformance may reflect the dataset's mild impairment range; if future cohorts include more severe bulbar involvement, the contribution of kinematic features could change.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper investigates automatic regression of clinician-assigned speech impairment scores in ALS from audio-only, video-only, and audio-video (multimodal) feature sets. Using the Toronto NeuroFace dataset, the authors extract 18 acoustic and 15 kinematic features from repetitions of the sentence "Buy Bobby a puppy," and train SVR, MLP, and XGB regressors with nested leave-one-subject-out cross-validation. The reported best result is an mRMSE of 0.93 for multimodal SVR, compared with 0.99 for audio-only XGB and 1.21 for video-only XGB. The abstract and conclusions state that integrating audio-video analysis enhances speech impairment assessment. The paper also reports, however, that the Friedman test over the nine model-modality combinations was not statistically significant (p=0.092, chi-square=13.6267, df=8).
Significance. If the multimodal advantage were statistically established, the work would offer a useful, low-cost, non-invasive tool for remote monitoring of bulbar dysfunction in ALS. The methodological backbone is sound in several respects: the nested leave-one-subject-out cross-validation avoids subject-level data leakage; the hyperparameter grid searches are reported in detail; per-participant RMSE and coefficient of variation are provided, giving a more granular view than a single aggregate score; and the authors frankly acknowledge methodological limitations such as small sample size and non-identical participant sets across modalities. The main weakness is that the paper's central claim, that adding video to audio improves prediction, rests on a non-significant statistical comparison and on an internal inconsistency about which model achieved the best result. The study is better framed as an exploratory feasibility analysis than as evidence of multimodal enhancement.
major comments (4)
- [Section 3, Table 1 and statistical analysis] The paper's own Friedman test (p=0.092, chi-square=13.6267, df=8) shows no statistically significant effect of modality or model, yet the abstract and Section 5 conclude that integrating audio-video analysis enhances assessment. The observed mRMSE differences (0.93, 0.99, 1.21) are therefore within the range that could plausibly arise from noise. No confidence intervals or effect sizes are reported for the mRMSE values. To support the multimodal-enhancement claim, the authors should either provide paired comparisons on identical subjects/instances with appropriate correction for multiple comparisons, or explicitly reframe the results as exploratory and remove the causal-sounding language from the abstract and conclusions.
- [Abstract vs. Section 3] The abstract states that the best performance was achieved by the extreme boosting machine regressor with multimodal features, but Section 3 and Table 1 report that the best multimodal model is SVR, with XGB achieving mRMSE 1.06 in the audio-video condition. This is not a trivial wording discrepancy: it changes which model-family result is being advertised. The abstract and results must be aligned, and if the XGB statement is a typo it should be corrected, because the current inconsistency weakens confidence in the reported headline numbers.
- [Section 3, non-identical participant sets and post-hoc selection] The audio and multimodal analyses use 9 HC and 8 ALS participants (153 and 151 instances), while the video analysis uses 11 HC and 9 ALS participants (202 instances). The headline comparison of 0.93 (multimodal SVR) versus 0.99 (audio XGB) is therefore not a paired comparison on the same data. In addition, the best model within each modality was selected after inspecting all nine model-modality combinations, without any correction for multiple comparisons. The claimed multimodal advantage is thus a post-hoc selection on different participant sets. The authors should re-run the comparison on the common subset of participants and instances, or explicitly quantify the uncertainty of the difference (e.g., bootstrap confidence intervals on the mRMSE difference).
- [Section 3 and Section 4, interpretation of Figure 2] The Results state that ALS participants with higher real scores have predicted scores lower than the actual ones, whereas the Discussion states that participants with more severe impairment tended to have predictions above the perfect prediction line. These two statements are contradictory: a prediction above the line is higher than the actual score, not lower. This inconsistency affects the substantive interpretation of whether the model under- or over-predicts severe impairment and needs to be resolved by checking the figure axes and correcting the text.
minor comments (5)
- [Section 2.1 and References] The Toronto NeuroFace dataset is cited as reference [21], but reference [21] is Tavazzi et al., a systematic review on ALS progression prediction; the actual dataset paper appears to be reference [22] (Bandini et al., A New Dataset for Facial Motion Analysis in Individuals With Neurological Disorders). Please correct the citation.
- [References] Reference [17] (Guarin et al.) appears to be missing from the citation list in the text: the text jumps from [16] to [18] in the Introduction, and [17] is never cited. Every list entry should be cited in the text.
- [Throughout] There are several typographical errors, including a myotrophic lateral sclerosis, an d, distribuition, the we applied, form Table 1, and SPLs where SLPs is meant. These should be corrected in a final proofreading pass.
- [Section 2.4.1] The hyperparameter grid notation C = [10-1, 1, 10, 102, ...] should be written with superscripts as 10^{-1}, 10^2, and so on, to avoid confusion with subtraction.
- [Section 5] The conclusion repeats the claim that multimodal integration enhances assessment, but it does not mention the non-significant Friedman result from Section 3. The conclusion should be made consistent with the statistical evidence or should explicitly frame the finding as exploratory.
Circularity Check
No significant circularity: the regression target is externally anchored to SLP ratings, and features are independently measured acoustic/kinematic signals.
full rationale
The derivation chain is not circular. The target variable is the mean of the two speech-language pathologists' total scores (Section 2.3: 'The target value to be estimated with our proposed approach was the mean value of the total scores assigned by the two SLPs, which ranges between 5 and 25'), while the features are acoustic and kinematic measurements extracted from the recordings using external tools such as Parselmouth/Praat, the Vosk speech recognizer, the SFD face detector, and the FAN face-alignment network (Sections 2.2-2.3). No feature is defined in terms of the SLP score, and the SLP score is not derived from the features. The regressors are evaluated with nested leave-one-subject-out cross-validation, with hyperparameters tuned only on training folds and held-out participants used as test sets (Section 2.4), so the reported mRMSE values are genuine out-of-sample predictions rather than fitted inputs relabeled as predictions. The author's self-citations to prior Bandini et al. work on video-based facial analysis and the Toronto NeuroFace dataset provide data and methodology, not a load-bearing uniqueness theorem or an ansatz that smuggles in the conclusion; the clinician ratings in the dataset are external evidence. The paper's internal inconsistency between the abstract (extreme boosting machine as best multimodal model) and Section 3 (SVR as best), together with the non-significant Friedman test (p=0.092, χ²=13.6267, df=8) and the non-identical participant/instance sets across modalities, are statistical and reporting weaknesses that undermine the multimodal-enhancement conclusion, but they are not circularity: the weakness is that the evidence is too noisy to support the claim, not that the claim is equivalent to its inputs by construction. No step in the paper's derivation reduces Eq. X to Eq. Y by definition or depends on an unverified self-citation chain, so the appropriate circularity score is 0.
Assumptions & free parameters
free parameters (1)
- Model hyperparameters (SVR C, epsilon, kernel; MLP architecture, learning rate, activation; XGB n_estimators… =
Selected by grid search in the inner leave-one-subject-out cross-validation
assumptions (4)
- domain assumption The averaged total score from two speech-language pathologists is a valid interval-scale gold standard for speech impairment severity.
- domain assumption The "Buy Bobby a puppy" sentence repetition task adequately captures bulbar speech impairment.
- domain assumption Manual segmentation of sentence onset and offset, performed by visually inspecting audio and video, is accurate and consistent across participants.
- standard math Nested leave-one-subject-out cross-validation produces unbiased generalization estimates on this small dataset.
Cite this review
Pith. "Pith review of Multimodal Assessment of Speech Impairment in ALS Using Audio-Visual and Machine Learning Approaches." pith.science (2026). https://pith.science/paper/MAFAX24Z
@misc{pith2026250521093,
author = {Pith},
title = {Pith review of: Multimodal Assessment of Speech Impairment in ALS Using Audio-Visual and Machine Learning Approaches},
year = {2026},
howpublished = {\url{https://pith.science/paper/MAFAX24Z}},
note = {Machine review of arXiv:2505.21093}
}
read the original abstract
The analysis of speech in individuals with amyotrophic lateral sclerosis is a powerful tool to support clinicians in the assessment of bulbar dysfunction. However, current methods used in clinical practice consist of subjective evaluations or expensive instrumentation. This study investigates different approaches combining audio-visual analysis and machine learning to predict the speech impairment evaluation performed by clinicians. Using a small dataset of acoustic and kinematic features extracted from audio and video recordings of speech tasks, we trained and tested some regression models. The best performance was achieved using the extreme boosting machine regressor with multimodal features, which resulted in a root mean squared error of 0.93 on a scale ranging from 5 to 25. Results suggest that integrating audio-video analysis enhances speech impairment assessment, providing an objective tool for early detection and monitoring of bulbar dysfunction, also in home settings.
Reference graph
Works this paper leans on
-
[1]
Introduction Amyotrophic lateral sclerosis (ALS) is a rare neurodegenerative disease affecting 4.42 per 100,000 people globally, with an estimated survival time of 3 to 5 years after the disease onset [1], [2]. However, survival time can vary depending on several factors including the age and site of onset (e.g., bulbar or spinal) and the rate of disease ...
-
[2]
Methods 2.1. Dataset description In this study, we used the Toronto NeuroFace dataset [21], which includes 11 individuals with ALS (6 females, age range 45-75 years) and 11 individuals with no speech and orofacial impairments as a control group (HC) (4 females, age range 33- 78 years). The dataset contains a udio and video recordings of participants perfo...
-
[3]
Results For the multimodal approach, we considered only participants who had both audio and video recordings. Additionally, due to some inconsistencies in the number of repetitions between the two modalities for some participants, we carefully removed instances that were present in only one modality. As a result, the number of participants included in the...
-
[4]
Discussion In this study, we compared different approaches for predicting speech impairment scores in individuals with ALS. We aimed to investigate whether a multimodal approach based on audio and video features is feasible and can further improve clinical outcome predictions. The overall mRMSE of 0.93 achieved by the SVR regressor is a promising result f...
-
[5]
Conclusions In this work, we presented three different approaches to highlight the most suitable method for implementing a low - cost, non -invasive, and objective evaluation of speech impairment based on audio-video recordings. In particular, the best regression model was identified with the SVR, trained using multimodal features, achieving a mRMSE of 0....
-
[6]
The research in this paper uses the Toronto NeuroFace Dataset collected by Dr
Acknowledgements The financial support of AriSLA – Fondazione Italiana di ricerca per la SLA is acknowledged ( Project MIMOSA - Multimodal Intelligent Methods for Orofacial and Speech Assessment to predict ALS bulbar decline). The research in this paper uses the Toronto NeuroFace Dataset collected by Dr. Yana Yunusova and the Vocal Tract Visualization and...
-
[7]
L. Xu et al., «Global variation in prevalence and incidence of amyotrophic lateral sclerosis: a systematic review and meta - analysis», J. Neurol., vol. 267, fasc. 4, pp. 944 –953, apr. 2020, doi: 10.1007/s00415-019-09652-y
-
[8]
R. H. Brown e A. Al -Chalabi, «Amyotrophic Lateral Sclerosis», N. Engl. J. Med., vol. 377, fasc. 2, pp. 162 –172, lug. 2017, doi: 10.1056/NEJMra1603471
Show all 36 references
-
[9]
Bandini, J
A. Bandini, J. R. Green, B. Taati, S. Orlandi, L. Zinman, e Y. Yunusova, «Automatic Detection of Amyotrophic Lateral Sclerosis (ALS) from Video -Based Analysis of Facial Movements: Speech and Non-Speech Tasks», in 2018 13th IEEE International Conference on Automatic Face & Ges...
2018
-
[10]
S. H. Felgoise , V. Zaccheo, J. Duff, e Z. Simmons, «Verbal communication impacts quality of life in patients with amyotrophic lateral sclerosis», Amyotroph. Lateral Scler. Front. Degener., vol. 17, fasc. 3 –4, pp. 179 –183, mag. 2016, doi: 10.3109/21678421.2015.1125499
2016
-
[11]
K. M. Allison, Y. Yunusova, T. F. Campbell, J. Wang, J. D. Berry, e J. R. Green, «The diagnostic utility of patient-report and speech- language pathologists’ ratings for detecting the early onset of bulbar symptoms due to ALS», Amyotroph. Lateral Scler. Front. Degener., vol. 1...
2017
-
[12]
R. A. Smith et al., «Assessment of bulbar function in amyotrophic lateral sclerosis: validation of a self ‐report scale (Center for Neurologic Study Bulbar Function Scale)», Eur. J. Neurol., vol. 25, fasc. 7, p. 907, lug. 2018, doi: 10.1111/ene.13638
2018 doi
-
[13]
Lynch, «Pathogenesis and presentation of ALS: examining reasons for delayed diagnosis and identifying opportunities for improvement», Am
K. Lynch, «Pathogenesis and presentation of ALS: examining reasons for delayed diagnosis and identifying opportunities for improvement», Am. J. Manag. Care, vol. 29, fasc. Suppl 7, pp. S104–S111, giu. 2023, doi: 10.37765/ajmc.2023.89390
2023
-
[14]
Yunusova, E
Y. Yunusova, E. K. Plowman, J. R. Green, C. Barnett, e P. Bede, «Clinical Measures of Bulbar Dysfunction in ALS», Front. Neurol., vol. 10, p. 106, feb. 2019, doi: 10.3389/fneur.2019.00106
2019
-
[15]
Richter et al., «Remote Assessment for ALS using Multimodal Dialog Agents: Data Quality, Feasibility and Task Compliance», in INTERSPEECH 2023, ISCA, ago
V. Richter et al., «Remote Assessment for ALS using Multimodal Dialog Agents: Data Quality, Feasibility and Task Compliance», in INTERSPEECH 2023, ISCA, ago. 2023, pp. 5441 –5445. doi: 10.21437/Interspeech.2023-2115
2023 doi
-
[16]
J. R. Green et al., «Bulbar and speech motor assessment in ALS: Challenges and future directions», Amyotroph. Lateral Scler. Front. Degener., vol. 14, fasc. 7 –8, pp. 494–500, dic. 2013, doi: 10.3109/21678421.2013.817585
2013
-
[17]
Neumann, H
M. Neumann, H. Kothare, e V. Ramanarayanan, «Multimodal speech biomarkers for remote monitoring of ALS disease progression», Comput. Biol. Med., vol. 180, p. 108949, set. 2024, doi: 10.1016/j.compbiomed.2024.108949
2024
-
[18]
Neumann, H
M. Neumann, H. Kothare, J. Liscombe, E. C. L. Leschly, O. Roesler, e V. Ramanarayanan, «Multimodal Digital Biomarkers for Longitudinal Tracking of Speech Impairment Severity in ALS: An Investigation of Clinically Important Differences», in Interspeech 2024 , ISCA, set. 2024, p...
2024 doi
-
[19]
Dubbioso et al., «Voice signals database of ALS patients with different dysarthria severity and healthy controls», Sci
R. Dubbioso et al., «Voice signals database of ALS patients with different dysarthria severity and healthy controls», Sci. Data, vol. 11, fasc. 1, p. 800, lug. 2024, doi: 10.1038/s41597-024-03597-2
2024 doi
-
[20]
G. M. Stegmann et al., «Early detection and tracking of bulbar changes in ALS via frequent and remote speech analysis», Npj Digit. Med., vol. 3, fasc. 1, p. 132, ott. 2020, doi: 10.1038/s41746- 020-00335-x
2020 doi
-
[21]
L. E. R. Simmatis, J. Robin, M. J. Spilka, e Y. Yunusova, «Detecting bulbar amyotrophic lateral sclerosis (ALS) using automatic acoustic analysis», Biomed. Eng. Online, vol. 23, fasc. 1, p. 15, feb. 2024, doi: 10.1186/s12938-023-01174-z
2024 doi
-
[22]
Bandini, J
A. Bandini, J. Green, B. Richburg, e Y. Yunusova, «Automatic Detection of Orofacial Impairment in Stroke», in Interspeech 2018, ISCA, set. 2018, pp. 1711 –1715. doi: 10.21437/Interspeech.2018-2475
2018 doi
-
[23]
D. L. Guarin, B. Taati, A. Abrahao, L. Zinman, e Y. Yunusova, «Video-Based Facial Movement Analysis in the Assessment of Bulbar Amyotrophic Lateral Sclerosis: Clinical Validation», J. Speech Lang. Hear. Res., vol. 65, fasc. 12, pp. 4667 –4678, dic. 2022, doi: 10.1044/2022_JSLH...
2022 doi
-
[24]
Neumann, H
M. Neumann, H. Kothare, e V. Ramanarayanan, «Combining Multiple Multimodal Speech Features into an Interpretable Index Score for Capturing Disease Progression in Amyotrophic Lateral Sclerosis», Interspeech, vol. 2023, pp. 2353–2357, ago. 2023, doi: 10.21437/interspeech.2023-2100
2023 doi
-
[25]
Simmatis, T
L. Simmatis, T. Pommeé, e Y. Yunusova, «Multimodal Assessment of Bulbar Amyotrophic Lateral Sclerosis (ALS) Using a Novel Remote Speech Assessment App», in INTERSPEECH 2023, ISCA, ago. 2023, pp. 2378 –2382. doi: 10.21437/Interspeech.2023-2093
2023 doi
-
[26]
A. Guazzo et al., «Predicting clinical events characterizing the progression of amyotrophic lateral sclerosis via machine learning approaches using routine visits data: a feasibility study», BMC Med. Inform. Decis. Mak., vol. 24, fasc. S4, p. 318, ott. 2024, doi: 10.1186/s1291...
2024 doi
-
[27]
Tavazzi et al., «Artificial intelligence and statistical methods for stratification and prediction of progression in amyotrophic lateral sclerosis: A systematic review», Artif
E. Tavazzi et al., «Artificial intelligence and statistical methods for stratification and prediction of progression in amyotrophic lateral sclerosis: A systematic review», Artif. Intell. Med., vol. 142, p. 102588, ago. 2023, doi: 10.1016/j.artmed.2023.102588
2023
-
[28]
Bandini et al., «A New Dataset for Facial Motion Analysis in Individuals With Neurological Disorders», IEEE J
A. Bandini et al., «A New Dataset for Facial Motion Analysis in Individuals With Neurological Disorders», IEEE J. Biomed. Health Inform., vol. 25, fasc. 4, pp. 1111 –1119, apr. 2021, doi: 10.1109/JBHI.2020.3019242
2021
-
[29]
Zhang, X
S. Zhang, X. Zhu, Z. Lei, H. Shi, X. Wang, e S. Z. Li, «S^3FD: Single Shot Scale -Invariant Face Detector», in 2017 IEEE International Conference on Computer Vision (ICCV), Venice: IEEE, ott. 2017, pp. 192–201. doi: 10.1109/ICCV.2017.30
2017 doi
-
[30]
Bulat e G
A. Bulat e G. Tzimiropoulos, «How Far are We from Solving the 2D & 3D Face Alignment Problem? (and a Dataset of 230,000 3D Facial Landmarks)», in 2017 IEEE International Conference on Computer Vision (ICCV), Venice: IEEE, ott. 2017, pp. 1021 –
2017
-
[32]
Jadoul, B
Y. Jadoul, B. Thompson, e B. De Boer, «Introducing Parselmouth: A Python interface to Praat», J. Phon., vol. 71, pp. 1–15, nov. 2018, doi: 10.1016/j.wocn.2018.07.001
2018 doi
-
[33]
Boersma, «Praat, a system for doing phonetics by computer», Glot International, vol
P. Boersma, «Praat, a system for doing phonetics by computer», Glot International, vol. 5, no. 9/10, pp. 341–345, 2001
2001
-
[34]
J. R. Duffy, «Motor Speech Disorders: Clues to Neurologic Diagnosis»
-
[35]
C. R. Watts e M. Vanryckeghem, «Laryngeal dysfunction in Amyotrophic Lateral Sclerosis: a review and case report», BMC Ear Nose Throat Disord., vol. 1, fasc. 1, p. 1, nov. 2001, doi: 10.1186/1472-6815-1-1
2001 doi
-
[36]
C. Oh, R. J. Morris, e X. Wang, «Prosody Analysis as a Tool for Differential Diagnosis of Cognitive Impairment», presentato al 183rd Meeting of the Acoustical Society of America, Nashville, Tennessee, 2022, p. 060001. doi: 10.1121/2.0001709
2022 doi
-
[1030]
doi: 10.1109/ICCV.2017.116
2017 doi
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.