{"id":"762e43a8-b55b-4355-a64b-484ed0233a8c","arxiv_id":"2505.18722","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Non-diagnostic dialogue speech, after concatenation and demographic balancing, can perform nearly as well as diagnostic-task speech for Parkinson's disease classification, though validation leakage clouds the result.","lead":"This paper tests whether ordinary conversation recordings, not designed for diagnosis, can be used to train an AI that detects Parkinson's disease from speech. It finds that with some data adjustments, these recordings come close to specialized diagnostic recordings in accuracy.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Inner validation split is not speaker-independent; re-run with speaker-grouped validation is needed to support the 'as useful' claim.","rationale":"The reader's weakest assumption is the non-speaker-independent inner validation set, and I agree this is the most concrete, testable threat to the central claim. The paper's Section 5 states that 'a dataset like TT can be as useful as traditional diagnostic datasets,' supported primarily by the within-dataset F1 of 74.19 for TT-concat-balanced versus 77.67 for the replicated PC-GITA. If inner validation leaks speaker identity, the model-selection step is not speaker-independent, so the reported F1 cannot be taken as clean evidence about unseen-speaker generalization. Unlike the paper's suggestion, the bias may not be inflationary: a too-easy validation set can cause late stopping and overfitting to training speakers, which would depress outer test performance. Either way, the internal inconsistency between Sections 3.1 and 3.2 makes the current numbers unverified. My proposed test directly compares the reported results against a fully speaker-independent inner split; if the F1 is materially unchanged, the concern is resolved and the verdict stands. I also note that the small sample size (30 PD/30 HC in the balanced subset) and the lack of statistical comparison between the TT and PC-GITA F1 values are secondary weaknesses, but the validation leakage is more fundamental because it affects the validity of the evaluation protocol itself. Therefore, the reader's conditional verdict remains appropriate.","tokens_in":8858,"tokens_out":10989,"duration_ms":94361,"concrete_test":"Re-run the four TT conditions using the same code, seeds, and outer 10-fold speaker-independent splits, but construct the inner validation set by holding out a random 20% of training speakers (all their recordings) rather than sampling recordings. Compare the resulting F1 and ROC-AUC for TT-concat-balanced against the reported 74.19/84.24. If the F1 changes by more than the reported ~3-point run-level range (about 1% std across runs plus fold variance), the original numbers are not reproducible under a fully speaker-independent protocol and the 'as useful' claim needs to be revised; if the F1 remains within that range, the leakage concern is empirically settled.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2 discloses that the inner validation set is randomly selected from the training set and is not speaker-independent, contradicting the Section 3.1 claim of 'nested speaker-independent cross-validation.' With roughly 143 recordings per TT participant (Table 2), a random inner split places same-speaker recordings in both training and validation. Early stopping and all model selection decisions therefore exploit speaker identity, undermining the claim that the reported within-dataset results (TT-concat-balanced F1=74.19) reflect generalization to unseen speakers. The bias direction is not guaranteed: overfitting to seen speakers during late stopping could depress, not inflate, unseen-speaker test scores. However, the protocol as executed is not the fully speaker-independent evaluation asserted, and the central 'as useful' conclusion rests directly on these numbers. A re-run with a truly speaker-independent inner split is required before the within-dataset comparison to PC-GITA can be trusted.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper evaluates whether a non-diagnostic, controlled-setting speech dataset (the new Dutch Turn-Taking dataset, TT) can be used to train a Parkinson's disease classifier, comparing it with the diagnostic-oriented PC-GITA dataset. Using a pretrained WavLM encoder with attention pooling, the authors run nested 10-fold cross-validation on four TT variants (TT, TT-concat, TT-balanced, TT-concat-balanced), replicate PC-GITA with the same pipeline, and perform cross-dataset evaluation. The main claims are that TT-concat-balanced reaches F1=74.19 versus 77.67 for replicated PC-GITA, that concatenation and demographic balancing improve within-dataset performance, that PC-GITA-trained models generalize poorly to TT while TT-trained models generalize better to PC-GITA, and that cross-fold variability is driven by per-speaker differences. The paper includes a failure analysis correlating per-speaker accuracy with confidence scores, disease duration, and self-reported speech scores.","tokens_in":9067,"tokens_out":2501,"duration_ms":20477,"significance":"If the central finding holds, the paper would broaden the pool of usable speech data for PD classification beyond diagnostic-oriented corpora, which is practically valuable for earlier screening and for resource-limited settings. The paper also contributes a new public-facing Dutch PD speech dataset, a careful replication of a strong baseline system, five-seed averaging for stability, and a cross-dataset evaluation that is still rare in this literature. The failure analysis addressing cross-fold variability is a useful methodological addition. However, the central 'as useful as diagnostic data' claim currently rests on a validation protocol that is not fully speaker-independent inside the nested cross-validation, and on differences that are not tested for significance; both need to be resolved before the main conclusion is supported.","major_comments":[{"comment":"The inner validation split of the nested cross-validation is not speaker-independent, as stated in Section 3.2 ('The validation set is randomly selected from the training set and is not speaker-independent according to the official implementation'). This contradicts the Section 3.1 description of 'nested speaker-independent cross-validation,' and it is load-bearing for the central claim because all within-dataset metrics in Table 3, including the headline TT-concat-balanced F1=74.19, are produced by this pipeline. With roughly 143 recordings per TT participant, a random inner split places same-speaker recordings in both the training and validation sets, so model selection and early stopping can exploit speaker identity. The bias direction is not guaranteed, but the protocol as executed is not the fully speaker-independent evaluation that the 'as useful' comparison to PC-GITA requires. The authors should re-run the TT experiments with a speaker-grouped inner validation split, or clearly report the difference in results under both protocols.","section":"Section 3.2 and Table 3"},{"comment":"The conclusion that 'a dataset like TT can be as useful as traditional diagnostic datasets' is based on a comparison of mean F1 values (74.19 versus 77.67) without any significance test, and the reported standard deviations across folds are large (around 10 points). Differences among TT variants are similarly within one standard deviation of each other, so it is unclear whether the observed ordering of configurations is stable. The authors should report paired statistical tests over the ten folds or over the five training runs—for example, a paired bootstrap or Wilcoxon signed-rank test on fold-level F1—before claiming equivalence or superiority. At minimum, the claim should be softened to reflect the overlap in variability.","section":"Section 4.1 and Section 5"},{"comment":"The phrase 'exhibiting even better generalization ability' is stronger than the evidence supports. The cross-dataset result is a single source-target pair (TT-concat-balanced versus PC-GITA) that differs in language, speech tasks, recording conditions, and time after diagnosis, so the observed asymmetry cannot be attributed to dataset purpose alone. Moreover, the PC-GITA-to-TT row reaches specificity of only 7%, meaning the model predicts nearly all TT speakers as PD; this is not a meaningful form of generalization. The authors should either restrict the conclusion to the observed asymmetry in this specific pair or add supporting evidence from additional datasets and controlled ablation of task and language differences.","section":"Section 4.1 and Section 5"},{"comment":"The failure analysis reports per-participant Mann-Whitney U tests on 14 speech features without correcting for multiple comparisons, and then interprets the absence of significant differences as evidence that no feature distinguishes low- and high-accuracy participants. With 14 tests, the expected number of false positives is above zero, and the power to detect effects in roughly 20 participants per group is limited. The authors should either apply a multiple-comparison correction or explicitly discuss the power limitation; otherwise the 'no statistically significant differences' statement is potentially misleading.","section":"Section 4.2"}],"minor_comments":[{"comment":"The sentence 'We found that retraining the model with different random seeds causes variable results' is vague; it would be clearer to state the range of F1 or accuracy observed across seeds, not only the average and the statement that the standard deviation is around 1%.","section":"Section 3.2"},{"comment":"The phrase 'trained and tested on TT model' should read 'trained and tested on the TT dataset'.","section":"Section 4.2"},{"comment":"There are minor formatting inconsistencies in the table, such as '84.24 ± 10.7' versus '7.00 ±9.5' and '39.63±8.3'; these should be unified for readability.","section":"Table 3"},{"comment":"'manually verified by human listening' should be 'manually verified through human listening' or 'manually verified by listening'.","section":"Section 2.2.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is an Interspeech camera-ready submission, and the core contribution—a new non-diagnostic Dutch PD speech dataset plus a systematic comparison with a diagnostic corpus—is timely and useful. However, the central 'as useful' claim is currently supported by a leaky inner validation protocol and by point estimates without significance testing. I believe these issues are fixable in revision and do not require rejecting the paper's scope, but they do require a re-run or at least a transparent re-analysis with a speaker-independent inner split. The third major comment about overclaiming cross-dataset generalization is also important for the camera-ready wording."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: this paper introduces a genuinely new resource, the Dutch Turn-Taking (TT) corpus, and asks whether speech collected for turn-taking research, not for diagnosis, can train a PD classifier. That is a sensible and worthwhile question, and the authors do a decent job of scoping it. They compare against PC-GITA using a replicated state-of-the-art model, report cross-dataset generalization, and dig into fold-to-fold variability with a per-speaker failure analysis. The best result, TT-concat-balanced at F1=74.19 versus 77.67 for the replicated PC-GITA, is honestly framed in the results section as touching the lower end of the promising range, though the conclusion overstates it as 'as useful ... while exhibiting even better generalization ability.' The generalization asymmetry is real and interesting: a TT-trained model transfers to PC-GITA at F1=63.53, while the reverse direction collapses to F1=39.6 with near-zero specificity. That asymmetry is worth taking seriously, even if language, task, and disease-stage differences confound it.\n\nThe soft spot is exactly what the paper itself discloses in Section 3.2: the inner validation set used for model selection and early stopping is randomly drawn from the training set and is not speaker-independent. With roughly 143 recordings per TT participant, the same speaker's voice appears on both sides of the inner split. The authors call the protocol 'speaker-independent nested 10-fold cross-validation' in Section 3.1, which is misleading given the next sentence. This does not automatically invalidate the numbers, but it means the within-dataset results are not a clean estimate of unseen-speaker performance, and the paper's central comparison rests on those numbers. A re-run with a speaker-grouped inner split is needed before the 'as useful' claim is solid. I also note that no significance test backs the headline comparison, and the TT corpus is promised but not yet available, which limits independent verification.\n\nCredit where it is due: the paper is transparent about the validation limitation rather than burying it, the failure analysis is a good-faith attempt to explain fold variability, and the authors are careful to report five-seed averages. The citation pattern looks fine; building on La Quatra et al. is reasonable, and the ParkCeleb comparison is appropriately positioned. This is a solid empirical study with one fixable but load-bearing flaw. It deserves a serious referee, and in revision I would require the speaker-independent inner split plus a more measured conclusion. I would not cite it in its current form, but I would cite a corrected version.\n\nFor your reading group: worth a look if you care about speech biomarkers or dataset-provenance questions, but flag the validation issue and discuss whether the cross-dataset asymmetry survives a cleaner protocol.","headline":"Honest feasibility study with a new mid-level PD speech dataset; the central 'as useful' claim is weakened by a disclosed but unrepaired speaker-leakage issue in model selection.","tokens_in":9568,"tokens_out":1501,"would_cite":false,"duration_ms":16204,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Speech recorded for a turn-taking study, not for diagnosis, can train a Parkinson's disease classifier almost as accurately as a dedicated clinical corpus, and transfers better across datasets.","keywords":["Parkinson's disease","speech-based detection","non-diagnostic speech data","turn-taking dataset","cross-dataset generalization","self-supervised speech representations","WavLM","dataset balancing"],"falsifier":"Re-run the TT experiments with a strictly speaker-independent inner validation split, for example by placing all recordings of each speaker in the same validation fold, and compare the F1 of TT-concat-balanced with the replicated PC-GITA baseline; if the gap narrows or reverses, the paper's equality claim depends on validation leakage rather than on the data itself.","tokens_in":8693,"feed_emoji":"🎙️","tokens_out":7808,"duration_ms":59699,"temperature":0.7,"pith_summary":"The paper asks whether speech data collected for purposes other than diagnosis—here, a turn-taking experiment in which Dutch participants answered questions and named pictures—can be used to train a Parkinson's disease (PD) classifier. It argues yes: with the right preprocessing, the best configuration of this non-diagnostic dataset reaches F1 = 74.19, slightly below the replicated PC-GITA diagnostic baseline of 77.67, and models trained on it perform better when tested on PC-GITA than models trained the other way. The paper identifies two dataset characteristics that matter for within-dataset performance: concatenating short recordings into longer samples and balancing participants by gender and disease status. It also shows that the large performance variation across cross-validation folds traces to a few individual speakers who are consistently easy or hard to classify.","feed_headline":"Non-diagnostic speech rivals clinical data for Parkinson's detection","feed_subtitle":"A Dutch turn-taking corpus trained a PD classifier that transfers to a clinical corpus, while the reverse fails.","key_machinery":"The classification model is an end-to-end network built on the WavLM Base self-supervised encoder, whose frame-level features are aggregated by a weighted sum with attention pooling and passed through two fully connected layers with ReLU activations before a sigmoid output; it is trained in nested 10-fold speaker-independent cross-validation, with five random seeds averaged for stability. The companion machinery is dataset construction: TT-concat randomly concatenates four short utterances per speaker to create longer samples, and TT-balanced selects a subset balanced on gender and disease status; the combination of both, TT-concat-balanced, is the configuration that carries the paper's main result. Preprocessing uses voice activity detection and speaker diarization to remove silence and multi-speaker segments.","core_discovery":"The central claim is that a 'mid-level' diagnostic-purpose dataset—speech recorded under controlled conditions for research but not for PD diagnosis—can serve as a basis for PD classification. On the TT dataset, the best configuration (TT-concat-balanced, which concatenates four recordings per speaker and balances participants by gender and status) reaches F1 = 74.19 and ROC-AUC = 84.24, landing inside the range the authors take as promising when set against the replicated PC-GITA baseline (F1 = 77.67). The paper further claims that generalization is asymmetric: a model trained on PC-GITA nearly collapses when tested on TT-concat-balanced (F1 = 0.396, specificity = 0.07, predicting almost every case as PD), whereas a model trained on TT-concat-balanced reaches F1 = 63.53 on PC-GITA. This asymmetry, the authors argue, makes non-diagnostic controlled data at least as useful as diagnostic data for developing a classifier, with better cross-dataset generalization.","pith_inferences":["The non-speaker-independent inner validation split is a real threat to the absolute within-dataset numbers; the cross-dataset asymmetry is less exposed to this because test speakers are disjoint, so the relative conclusion is more robust than the absolute one.","The asymmetry could partly reflect that TT patients are earlier in the disease (time-after-diagnosis 7.5 years versus 11.2 for PC-GITA) rather than only task differences; a severity-matched subset would separate these explanations.","The strong confidence-score correlation suggests a practical triage rule: route low-confidence predictions to human review, even though average confidence is high, since uncertainty concentrates in the speakers the model gets wrong.","If strictly speaker-independent validation confirms the result, a clinical pathway opens: existing laboratory recordings made for other studies could support PD screening in resource-limited settings without standardized diadochokinesis or sustained-vowel tasks."],"forward_implications":["Non-diagnostic speech corpora collected for other research purposes can be repurposed to train PD classifiers, widening the pool of usable data without new diagnostic recording protocols.","Concatenating short recordings and balancing gender and disease status are concrete dataset-design decisions that move non-diagnostic data toward diagnostic-grade performance.","Cross-dataset transfer is asymmetric: training on non-diagnostic dialogue data transfers to a diagnostic corpus better than the reverse, so task type and disease severity deserve attention in future data collection.","Fold-to-fold variability in PD classification is driven mainly by a small number of consistently misclassified individual speakers, and average confidence score correlates strongly (0.731) with per-speaker accuracy, offering a practical signal for flagging uncertain predictions."],"supporting_citations":[{"why":"Supplies the classification model and official code used for all experiments, including the replicated PC-GITA baseline.","marker":"[7]"},{"why":"Defines the PC-GITA dataset, the diagnostic-purpose corpus used as the comparison and cross-dataset test target.","marker":"[16]"},{"why":"Introduces WavLM, the pre-trained self-supervised encoder that provides the frame-level features for the classifier.","marker":"[24]"},{"why":"Provides ParkCeleb, the low-level in-the-wild speech dataset whose limitations motivate the mid-level category and the TT comparison.","marker":"[13]"},{"why":"Reports high cross-fold standard deviations in PD detection, motivating the paper's fold-variability analysis.","marker":"[8]"},{"why":"Defines the ROMP self-reported speech score used in the correlation analysis with per-speaker accuracy.","marker":"[22]"}],"fun_headline_variants":["Non-diagnostic speech rivals clinical for Parkinson's detection","Speech not meant for diagnosis still catches Parkinson's","Turn-taking speech matches clinical for Parkinson's detection","Turn-taking speech outperforms clinical in Parkinson's transfer","Non-clinical speech data on par for Parkinson's detection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The inner validation set used for model selection and early stopping is randomly drawn from the training set and is not speaker-independent, so validation speakers can overlap training speakers and leak speaker identity into the chosen model, inflating the reported within-dataset scores.","fun_headline_variants_meta":{"raw":{"variants":["Non-diagnostic speech rivals clinical for Parkinson's detection","Speech not meant for diagnosis still catches Parkinson's","Turn-taking speech matches clinical for Parkinson's detection","Turn-taking speech outperforms clinical in Parkinson's transfer","Non-clinical speech data on par for Parkinson's detection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001608,"raw_usage":{"total_tokens":6383,"prompt_tokens":904,"completion_tokens":5479,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":520,"completion_tokens_details":{"reasoning_tokens":5403}},"tokens_in":520,"tokens_out":5479,"duration_ms":35279,"temperature":1.0,"reasoning_tokens":5403,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:26:39.759378+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the TT experiments with a strictly speaker-independent inner validation split, for example by placing all recordings of each speaker in the same validation fold, and compare the F1 of TT-concat-balanced with the replicated PC-GITA baseline; if the gap narrows or reverses, the paper's equality claim depends on validation leakage rather than on the data itself.","supporting_citations":[{"cited_title":"Computerized analysis of speech and voice for Parkinson’s disease: A systematic review,","cited_arxiv_id":null,"evidence_quote":"Supplies the classification model and official code used for all experiments, including the replicated PC-GITA baseline."},{"cited_title":"Convolutional neural networks and a transfer learning strategy to classify Parkinson’s disease from speech in three different languages,","cited_arxiv_id":null,"evidence_quote":"Defines the PC-GITA dataset, the diagnostic-purpose corpus used as the comparison and cross-dataset test target."},{"cited_title":"Picture naming,","cited_arxiv_id":null,"evidence_quote":"Introduces WavLM, the pre-trained self-supervised encoder that provides the frame-level features for the classifier."},{"cited_title":"Exploiting foundation models and speech enhancement for Parkinson’s disease detection from speech in real-world operative conditions,","cited_arxiv_id":null,"evidence_quote":"Provides ParkCeleb, the low-level in-the-wild speech dataset whose limitations motivate the mid-level category and the TT comparison."},{"cited_title":"Parkinson disease,","cited_arxiv_id":null,"evidence_quote":"Reports high cross-fold standard deviations in PD detection, motivating the paper's fold-variability analysis."},{"cited_title":"New Spanish speech corpus database for the analysis of people suffering from Parkinson‘s disease,","cited_arxiv_id":null,"evidence_quote":"Defines the ROMP self-reported speech score used in the correlation analysis with per-speaker accuracy."}],"review_version":1}