REVIEW 4 major objections 4 minor 32 references
Evaluating the Usefulness of Non-Diagnostic Speech Data for Developing Parkinson's Disease Classifiers
T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Speech recorded for a turn-taking study, not for diagnosis, can train a Parkinson's disease classifier almost as accurately as a dedicated clinical corpus, and transfers better across datasets.
desk verdict Honest feasibility study with a new mid-level PD speech dataset; the central 'as useful' claim is weakened by a disclosed but unrepaired speaker-leakage issue in model selection. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The classification model is an end-to-end network built on the WavLM Base self-supervised encoder, whose frame-level features are aggregated by a weighted sum with attention pooling and passed through two fully connected layers with ReLU activations before a sigmoid output; it is trained in nested 10-fold speaker-independent cross-validation, with five random seeds averaged for stability. The companion machinery is dataset construction: TT-concat randomly concatenates four short utterances per speaker to create longer samples, and TT-balanced selects a subset balanced on gender and disease status; the combination of both, TT-concat-balanced, is the configuration that carries the paper's main result. Preprocessing uses voice activity detection and speaker diarization to remove silence and multi-speaker segments.
What would settle it
Re-run the TT experiments with a strictly speaker-independent inner validation split, for example by placing all recordings of each speaker in the same validation fold, and compare the F1 of TT-concat-balanced with the replicated PC-GITA baseline; if the gap narrows or reverses, the paper's equality claim depends on validation leakage rather than on the data itself.
Extended reading notes
Core claim
The central claim is that a 'mid-level' diagnostic-purpose dataset—speech recorded under controlled conditions for research but not for PD diagnosis—can serve as a basis for PD classification. On the TT dataset, the best configuration (TT-concat-balanced, which concatenates four recordings per speaker and balances participants by gender and status) reaches F1 = 74.19 and ROC-AUC = 84.24, landing inside the range the authors take as promising when set against the replicated PC-GITA baseline (F1 = 77.67). The paper further claims that generalization is asymmetric: a model trained on PC-GITA nearly collapses when tested on TT-concat-balanced (F1 = 0.396, specificity = 0.07, predicting almost every case as PD), whereas a model trained on TT-concat-balanced reaches F1 = 63.53 on PC-GITA. This asymmetry, the authors argue, makes non-diagnostic controlled data at least as useful as diagnostic data for developing a classifier, with better cross-dataset generalization.
Load-bearing premise
The inner validation set used for model selection and early stopping is randomly drawn from the training set and is not speaker-independent, so validation speakers can overlap training speakers and leak speaker identity into the chosen model, inflating the reported within-dataset scores.
Editorial extensions
If this is right
- Non-diagnostic speech corpora collected for other research purposes can be repurposed to train PD classifiers, widening the pool of usable data without new diagnostic recording protocols.
- Concatenating short recordings and balancing gender and disease status are concrete dataset-design decisions that move non-diagnostic data toward diagnostic-grade performance.
- Cross-dataset transfer is asymmetric: training on non-diagnostic dialogue data transfers to a diagnostic corpus better than the reverse, so task type and disease severity deserve attention in future data collection.
- Fold-to-fold variability in PD classification is driven mainly by a small number of consistently misclassified individual speakers, and average confidence score correlates strongly (0.731) with per-speaker accuracy, offering a practical signal for flagging uncertain predictions.
Reading between the lines
- The non-speaker-independent inner validation split is a real threat to the absolute within-dataset numbers; the cross-dataset asymmetry is less exposed to this because test speakers are disjoint, so the relative conclusion is more robust than the absolute one.
- The asymmetry could partly reflect that TT patients are earlier in the disease (time-after-diagnosis 7.5 years versus 11.2 for PC-GITA) rather than only task differences; a severity-matched subset would separate these explanations.
- The strong confidence-score correlation suggests a practical triage rule: route low-confidence predictions to human review, even though average confidence is high, since uncertainty concentrates in the speakers the model gets wrong.
- If strictly speaker-independent validation confirms the result, a clinical pathway opens: existing laboratory recordings made for other studies could support PD screening in resource-limited settings without standardized diadochokinesis or sustained-vowel tasks.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper evaluates whether a non-diagnostic, controlled-setting speech dataset (the new Dutch Turn-Taking dataset, TT) can be used to train a Parkinson's disease classifier, comparing it with the diagnostic-oriented PC-GITA dataset. Using a pretrained WavLM encoder with attention pooling, the authors run nested 10-fold cross-validation on four TT variants (TT, TT-concat, TT-balanced, TT-concat-balanced), replicate PC-GITA with the same pipeline, and perform cross-dataset evaluation. The main claims are that TT-concat-balanced reaches F1=74.19 versus 77.67 for replicated PC-GITA, that concatenation and demographic balancing improve within-dataset performance, that PC-GITA-trained models generalize poorly to TT while TT-trained models generalize better to PC-GITA, and that cross-fold variability is driven by per-speaker differences. The paper includes a failure analysis correlating per-speaker accuracy with confidence scores, disease duration, and self-reported speech scores.
Significance. If the central finding holds, the paper would broaden the pool of usable speech data for PD classification beyond diagnostic-oriented corpora, which is practically valuable for earlier screening and for resource-limited settings. The paper also contributes a new public-facing Dutch PD speech dataset, a careful replication of a strong baseline system, five-seed averaging for stability, and a cross-dataset evaluation that is still rare in this literature. The failure analysis addressing cross-fold variability is a useful methodological addition. However, the central 'as useful as diagnostic data' claim currently rests on a validation protocol that is not fully speaker-independent inside the nested cross-validation, and on differences that are not tested for significance; both need to be resolved before the main conclusion is supported.
major comments (4)
- [Section 3.2 and Table 3] The inner validation split of the nested cross-validation is not speaker-independent, as stated in Section 3.2 ('The validation set is randomly selected from the training set and is not speaker-independent according to the official implementation'). This contradicts the Section 3.1 description of 'nested speaker-independent cross-validation,' and it is load-bearing for the central claim because all within-dataset metrics in Table 3, including the headline TT-concat-balanced F1=74.19, are produced by this pipeline. With roughly 143 recordings per TT participant, a random inner split places same-speaker recordings in both the training and validation sets, so model selection and early stopping can exploit speaker identity. The bias direction is not guaranteed, but the protocol as executed is not the fully speaker-independent evaluation that the 'as useful' comparison to PC-GITA requires. The authors should re-run the TT experiments with a speaker-grouped inner validation split, or clearly report the difference in results under both protocols.
- [Section 4.1 and Section 5] The conclusion that 'a dataset like TT can be as useful as traditional diagnostic datasets' is based on a comparison of mean F1 values (74.19 versus 77.67) without any significance test, and the reported standard deviations across folds are large (around 10 points). Differences among TT variants are similarly within one standard deviation of each other, so it is unclear whether the observed ordering of configurations is stable. The authors should report paired statistical tests over the ten folds or over the five training runs—for example, a paired bootstrap or Wilcoxon signed-rank test on fold-level F1—before claiming equivalence or superiority. At minimum, the claim should be softened to reflect the overlap in variability.
- [Section 4.1 and Section 5] The phrase 'exhibiting even better generalization ability' is stronger than the evidence supports. The cross-dataset result is a single source-target pair (TT-concat-balanced versus PC-GITA) that differs in language, speech tasks, recording conditions, and time after diagnosis, so the observed asymmetry cannot be attributed to dataset purpose alone. Moreover, the PC-GITA-to-TT row reaches specificity of only 7%, meaning the model predicts nearly all TT speakers as PD; this is not a meaningful form of generalization. The authors should either restrict the conclusion to the observed asymmetry in this specific pair or add supporting evidence from additional datasets and controlled ablation of task and language differences.
- [Section 4.2] The failure analysis reports per-participant Mann-Whitney U tests on 14 speech features without correcting for multiple comparisons, and then interprets the absence of significant differences as evidence that no feature distinguishes low- and high-accuracy participants. With 14 tests, the expected number of false positives is above zero, and the power to detect effects in roughly 20 participants per group is limited. The authors should either apply a multiple-comparison correction or explicitly discuss the power limitation; otherwise the 'no statistically significant differences' statement is potentially misleading.
minor comments (4)
- [Section 3.2] The sentence 'We found that retraining the model with different random seeds causes variable results' is vague; it would be clearer to state the range of F1 or accuracy observed across seeds, not only the average and the statement that the standard deviation is around 1%.
- [Section 4.2] The phrase 'trained and tested on TT model' should read 'trained and tested on the TT dataset'.
- [Table 3] There are minor formatting inconsistencies in the table, such as '84.24 ± 10.7' versus '7.00 ±9.5' and '39.63±8.3'; these should be unified for readability.
- [Section 2.2.2] 'manually verified by human listening' should be 'manually verified through human listening' or 'manually verified by listening'.
Circularity Check
No circularity: the central claims are empirical measurements on held-out speakers and an external dataset, not quantities determined by construction or by self-citation.
full rationale
The paper's central claim, that a non-diagnostic dataset like TT can be as useful as a diagnostic dataset like PC-GITA for developing a PD classifier, is an empirical finding supported by within-dataset accuracy/F1 measurements and cross-dataset generalization experiments. The classifier is an external method taken from La Quatra et al. [7], reused with the official implementation and stated hyperparameter changes; this is methodological borrowing, not circular reasoning. The TT results are computed under speaker-independent nested cross-validation for the outer test folds, and the comparison to PC-GITA is made on a separate, externally collected corpus with its own provided train/test splits. The key comparative numbers (TT-concat-balanced F1=74.19 vs replicated PC-GITA F1=77.67; cross-dataset F1=39.63 for PC-GITA-trained on TT and F1=63.53 for TT-trained on PC-GITA) are measured outcomes, not fitted parameters renamed as predictions. The paper also reports the instability across random seeds and averages over five runs, which does not create circularity. The disclosed non-speaker-independent inner validation split is a methodological weakness that could affect reliability of the model-selection procedure, but it is a correctness risk, not a circular-derivation risk: the reported test metrics still come from held-out speakers and an external dataset. The only self-citations are to a systematic review co-authored by one of the present authors [15] and to the Turn-Taking project itself; these citations provide context and dataset provenance, and neither is load-bearing for the classification results or for the usefulness conclusion. No equation or construction in the paper defines the target result in terms of its inputs, and no fitted value is reported as a prediction. Therefore, no significant circularity is present.
Assumptions & free parameters
free parameters (6)
- Concatenation factor =
4
- Maximum input length for TT =
5 seconds
- Batch size =
16
- Outlier removal threshold =
3 standard deviations
- TT-balanced subset demographics =
30 HC / 30 PD, 15 M / 15 F
- Number of training seeds =
5
assumptions (5)
- domain assumption WavLM Base self-supervised representations capture speech characteristics relevant to Parkinson's disease
- domain assumption The diagnostic labels in TT and PC-GITA are correct
- domain assumption Speaker-independent splits prevent speaker identity leakage
- domain assumption Cross-dataset comparison between TT and PC-GITA is meaningful despite differences in language, tasks, and participant age or disease duration
- standard math Statistical tests (Mann-Whitney U, Pearson correlation) are appropriate for the speaker-level analyses
invented entities (1)
-
Turn-Taking (TT) dataset
independent evidence
Cite this review
Pith. "Pith review of Evaluating the Usefulness of Non-Diagnostic Speech Data for Developing Parkinson's Disease Classifiers." pith.science (2026). https://pith.science/paper/KCSQ2SYC
@misc{pith2026250518722,
author = {Pith},
title = {Pith review of: Evaluating the Usefulness of Non-Diagnostic Speech Data for Developing Parkinson's Disease Classifiers},
year = {2026},
howpublished = {\url{https://pith.science/paper/KCSQ2SYC}},
note = {Machine review of arXiv:2505.18722}
}
read the original abstract
Speech-based Parkinson's disease (PD) detection has gained attention for its automated, cost-effective, and non-intrusive nature. As research studies usually rely on data from diagnostic-oriented speech tasks, this work explores the feasibility of diagnosing PD on the basis of speech data not originally intended for diagnostic purposes, using the Turn-Taking (TT) dataset. Our findings indicate that TT can be as useful as diagnostic-oriented PD datasets like PC-GITA. We also investigate which specific dataset characteristics impact PD classification performance. The results show that concatenating audio recordings and balancing participants' gender and status distributions can be beneficial. Cross-dataset evaluation reveals that models trained on PC-GITA generalize poorly to TT, whereas models trained on TT perform better on PC-GITA. Furthermore, we provide insights into the high variability across folds, which is mainly due to large differences in individual speaker performance.
Figures
Reference graph
Works this paper leans on
-
[1]
Introduction Parkinson’s disease (PD) is the second most common neurodegenerative disorder, affecting over 10 million people worldwide [1, 2]. During the prodromal stages of PD, patients may develop various speech problems up to five years before the onset of significant motor impairments [3, 4, 5]. Speech-based PD detection has attracted attention due to...
-
[2]
Datasets 2.1. Different Levels of Diagnostic-Purpose We start by clarifying our definition of different levels of PD diagnostic-purpose, as the distinction between different datasets in terms of purpose may not be immediately clear. Here we only consider publicly available speech datasets [15] containing arXiv:2505.18722v1 [eess.AS] 24 May 2025 PD labels....
work page Pith review arXiv 2025
-
[3]
Method 3.1. Classification Model We utilized a machine learning classification approach, as motivated by recent advances [6, 7, 23], to explore the potential of PD diagnosis using TT. Based on a literature review on classification performance using the PC-GITA dataset, we selected the classification model proposed in [7] for our experiments, as it is the ...
-
[4]
Experimental Results Table 3 presents a performance comparison across different conditions
Results and Discussion 4.1. Experimental Results Table 3 presents a performance comparison across different conditions. We see that the F1 scores for PC-GITA (Replicated) are in the range of 0.75–0.80. If we take this range to be representative of the scores that would be needed to consider a classifier promising for further development, and ultimate use ...
-
[5]
Conclusion This study examined the feasibility of diagnosing PD using controlled setting speech data not originally intended for PD diagnostics. Our findings indicate that a dataset like TT can be as useful as traditional diagnostic datasets for developing a PD classifier while exhibiting even better generalization ability (RQ1). This expands the range of...
-
[6]
Turn-taking in Dialogue in Populations with Communicative Impairment
Acknowledgments This publication is part of the project Responsible AI for V oice Diagnostics (RAIVD) with file number NGF.1607.22.013 of the research program NGF AiNed Fellowship Grants, which is financed by the Dutch Research Council (NWO). This work was conducted in close collaboration with the project “Turn-taking in Dialogue in Populations with Commu...
-
[7]
Computerized analysis of speech and voice for Parkinson’s disease: A systematic review,
Q. C. Ngo, M. A. Motin, N. D. Pah, P. Drotár, P. Kempster, and D. Kumar, “Computerized analysis of speech and voice for Parkinson’s disease: A systematic review,”Computer Methods and Programs in Biomedicine, vol. 226, p. 107133, 2022
work page 2022
-
[8]
W. Poewe, K. Seppi, C. M. Tanner, G. M. Halliday, P. Brundin, J. V olkmann, A.-E. Schrag, and A. E. Lang, “Parkinson disease,” Nature Reviews Disease Primers, vol. 3, no. 1, pp. 1–21, 2017
work page 2017
Show all 32 references
-
[9]
Treatments for dysarthria in Parkinson’s disease,
S. Pinto, C. Ozsancak, E. Tripoliti, S. Thobois, P. Limousin- Dowsey, and P. Auzou, “Treatments for dysarthria in Parkinson’s disease,”The Lancet Neurology, vol. 3, no. 9, pp. 547–556, 2004
2004
-
[10]
Imprecise vowel articulation as a potential early marker of Parkinson’s disease: effect of speaking task,
J. Rusz, R. Cmejla, T. Tykalova, H. Ruzickova, J. Klempir, V . Majerova, J. Picmausova, J. Roth, and E. Ruzicka, “Imprecise vowel articulation as a potential early marker of Parkinson’s disease: effect of speaking task,”The Journal of the Acoustical Society of America, vol. 13...
2013
-
[11]
Parkinson’s disease subtypes identified from cluster analysis of motor and non-motor symptoms,
J. Mu, K. R. Chaudhuri, C. Bielza, J. de Pedro-Cuesta, P. Larrañaga, and P. Martinez-Martin, “Parkinson’s disease subtypes identified from cluster analysis of motor and non-motor symptoms,”Frontiers in Aging Neuroscience, vol. 9, p. 301, 2017
2017
-
[12]
Speech as a biomarker for disease detection,
C. Botelho, A. Abad, T. Schultz, and I. Trancoso, “Speech as a biomarker for disease detection,”IEEE Access, vol. 12, pp. 184 487–184 508, 2024
2024
-
[13]
Exploiting foundation models and speech enhancement for Parkinson’s disease detection from speech in real-world operative conditions,
M. La Quatra, M. F. Turco, T. Svendsen, G. Salvi, J. R. Orozco- Arroyave, and S. M. Siniscalchi, “Exploiting foundation models and speech enhancement for Parkinson’s disease detection from speech in real-world operative conditions,” inInterspeech 2024, 2024, pp. 1405–1409
2024
-
[14]
Automatic parkinson’s disease detection from speech: Layer selection vs adaptation of foundation models,
T. Purohit, B. Ruvolo, J. R. Orozco-Arroyave, and M. Magimai.- Doss, “Automatic parkinson’s disease detection from speech: Layer selection vs adaptation of foundation models,” inICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP...
2025
-
[15]
A pilot study for speech assessment to detect the severity of Parkinson’s disease: An ensemble approach,
G. C. Oliveira, N. D. Pah, Q. C. Ngo, A. Yoshida, N. B. Gomes, J. P. Papa, and D. Kumar, “A pilot study for speech assessment to detect the severity of Parkinson’s disease: An ensemble approach,”Computers in Biology and Medicine, vol. 185, p. 109565, 2025
2025
-
[16]
Convolutional neural networks and a transfer learning strategy to classify Parkinson’s disease from speech in three different languages,
J. C. Vásquez-Correa, T. Arias-Vergara, C. D. Rios-Urrego, M. Schuster, J. Rusz, J. R. Orozco-Arroyave, and E. Nöth, “Convolutional neural networks and a transfer learning strategy to classify Parkinson’s disease from speech in three different languages,” inProgress in Pattern...
2019
-
[17]
The role of artificial intelligence in reducing healthcare costs and improving operational efficiency,
K. J. Prabhod, “The role of artificial intelligence in reducing healthcare costs and improving operational efficiency,”Quarterly Journal of Emerging Technologies and Innovations, vol. 9, no. 2, pp. 47–59, 2024
2024
-
[18]
The observer effect in a hospital setting - experiences from the observed and the observers,
K. Svensberg, B. G. Kalleberg, L. Mathiesen, Y . Andersson, S. E. Rognan, and S. K. Sporrong, “The observer effect in a hospital setting - experiences from the observed and the observers,” Research in Social and Administrative Pharmacy, vol. 17, no. 12, pp. 2136–2144, 2021
2021
-
[19]
Unveiling early signs of Parkinson’s disease via a longitudinal analysis of celebrity speech recordings,
A. Favaro, A. Butala, T. Thebaud, J. Villalba, N. Dehak, and L. Moro-Velázquez, “Unveiling early signs of Parkinson’s disease via a longitudinal analysis of celebrity speech recordings,”npj Parkinsons Dis., vol. 10, no. 1, p. 207, 2024
2024
-
[20]
Turn-taking in conversations,
J. M. Wiemann and M. L. Knapp, “Turn-taking in conversations,” Communication Theory, pp. 226–245, 2017
2017
-
[21]
Innovative speech-based deep learning approaches for Parkinson’s disease classification: A systematic review,
L. van Gelderen and C. Tejedor-García, “Innovative speech-based deep learning approaches for Parkinson’s disease classification: A systematic review,”Applied Sciences, vol. 14, no. 17, 2024
2024
-
[22]
New Spanish speech corpus database for the analysis of people suffering from Parkinson‘s disease,
J. R. Orozco-Arroyave, J. D. Arias-Londoño, J. F. Vargas-Bonilla, M. C. González-Rátiva, and E. Nöth, “New Spanish speech corpus database for the analysis of people suffering from Parkinson‘s disease,” inProceedings of the Ninth International Conference on Language Resources a...
2014
-
[23]
NeuroVoz: a Castilian Spanish corpus of parkinsonian speech,
J. Mendes-Laureano, J. A. Gómez-García, A. Guerrero-López, E. Luque-Buzo, J. D. Arias-Londoño, F. J. Grandas-Pérez, and J. I. Godino-Llorente, “NeuroVoz: a Castilian Spanish corpus of parkinsonian speech,”Scientific Data, vol. 11, no. 1, p. 1367, 2024
2024
-
[24]
Picture naming,
W. R. Glaser, “Picture naming,”Cognition, vol. 42, no. 1-3, pp. 61–105, 1992
1992
-
[25]
Planning versus comprehension in turn-taking: Fast responders show reduced anticipatory processing of the question,
S. Bögels, M. Casillas, and S. C. Levinson, “Planning versus comprehension in turn-taking: Fast responders show reduced anticipatory processing of the question,”Neuropsychologia, vol. 109, pp. 295–310, 2018
2018
-
[26]
Next speakers plan their turn early and speak after turn-final “go-signals
M. Barthel, A. S. Meyer, and S. C. Levinson, “Next speakers plan their turn early and speak after turn-final “go-signals”,”Frontiers in Psychology, vol. 8, p. 393, 2017
2017
-
[27]
Automatic classification of Parkinson’s disease using wav2vec embeddings at phoneme, syllable, and word levels,
J. D. Gallo-Aristizábal, D. Escobar-Grisales, C. D. Ríos-Urrego, E. Nöth, and J. R. Orozco-Arroyave, “Automatic classification of Parkinson’s disease using wav2vec embeddings at phoneme, syllable, and word levels,” inText, Speech, and Dialogue, 2024, vol. 15049, pp. 313–323
2024
-
[28]
Reproducibility and validity of patient-rated assessment of speech, swallowing, and saliva control in Parkinson’s disease,
J. G. Kalf, G. F. Borm, B. J. de Swart, B. R. Bloem, M. J. Zwarts, and M. Munneke, “Reproducibility and validity of patient-rated assessment of speech, swallowing, and saliva control in Parkinson’s disease,”Archives of Physical Medicine and Rehabilitation, vol. 92, no. 7, pp. ...
2011
-
[29]
Unveiling interpretability in self-supervised speech representations for parkinson’s diagnosis,
D. Gimeno-Gómez, C. Botelho, A. Pompili, A. Abad, and C.-D. Martínez-Hinarejos, “Unveiling interpretability in self-supervised speech representations for parkinson’s diagnosis,”IEEE Journal of Selected Topics in Signal Processing, p. 1–14, 2025
2025
-
[30]
Wavlm: Large-scale self- supervised pre-training for full stack speech processing,
S. Chen, C. Wang, Z. Chen, Y . Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiaoet al., “Wavlm: Large-scale self- supervised pre-training for full stack speech processing,”IEEE Journal of Selected Topics in Signal Processing, vol. 16, no. 6, pp. 1505–1518, 2022
2022
-
[31]
Mann-Whitney U Test,
P. E. McKnight and J. Najab, “Mann-Whitney U Test,”The Corsini Encyclopedia of Psychology, pp. 1–1, 2010
2010
-
[32]
Spectral feature selection for supervised and unsupervised learning,
Z. Zhao and H. Liu, “Spectral feature selection for supervised and unsupervised learning,” inProceedings of the 24th International Conference on Machine Learning, 2007, pp. 1151–1157
2007
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.