Pith. sign in

REVIEW 2 major objections 7 minor 25 references

Simple text cleanup beats model fine-tuning on messy neurological Spanish speech, and a new wild corpus makes that gap visible.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

T0 review · grok-4.5

2026-07-12 04:07 UTC pith:E4EODSWQ

load-bearing objection Usable multi-pathology Spanish in-the-wild ASR benchmark; PP beats the FT recipes they actually ran, and that result is real for those setups. the 2 major comments →

arxiv 2607.03207 v1 pith:E4EODSWQ submitted 2026-07-03 cs.CL cs.SD

S-DiverSe: Spanish Diverse Speech

classification cs.CL cs.SD
keywords speech recognitionpathological speechspeech corpusneurological disordersdysarthriaSpanish ASRin-the-wild speechpost-processing
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Automatic speech recognition works well on ordinary Spanish, but speech changed by ALS, Parkinson's, or stroke remains hard. This paper releases S-DiverSe, 3.2 hours of real-world Spanish from 22 speakers with those conditions, each segment transcribed by hand and tagged for sex, disease, and intelligibility. The authors run strong ASR systems on it and try two common fixes: heuristic post-processing of the text output, and fine-tuning the models on other dysarthric or Spanish data. Post-processing reliably cuts word error rate without hurting performance on cleaner clinical sets; fine-tuning improves those cleaner sets but often makes results worse on S-DiverSe. The point is that today's adaptation recipes do not close the gap to uncontrolled neurological Spanish, so the field needs dedicated wild benchmarks rather than more of the same clinical or read-speech data.

Core claim

On the new in-the-wild Spanish neurological corpus S-DiverSe, rule-based text post-processing that removes repeated characters, words, and phrases lowers word error rate more reliably than full or LoRA fine-tuning of Whisper or Voxtral, which improve in-domain TORGO and NeuroVoz scores but fail to transfer and can push error rates above 100 percent.

What carries the argument

S-DiverSe itself—the 3.2-hour, 444-segment evaluation set with pathology, sex, and intelligibility labels—plus the direct head-to-head of the three-step heuristic post-processor against the four fine-tuning recipes (full, full-LoRA, encoder-only, encoder-LoRA) trained on TORGO, NeuroVoz, and Common Voice combinations.

Load-bearing premise

The particular fine-tuning recipes, learning rates, epochs, and data mixes the authors tried are representative enough of what parameter adaptation can do, so that post-processing's win can be blamed on a real domain gap rather than under-tuned adaptation.

What would settle it

A carefully tuned or larger-scale fine-tuning run on the same or expanded pathological Spanish data that beats the post-processing-only baseline on the full S-DiverSe set without collapsing on TORGO or NeuroVoz.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • S-DiverSe becomes a public Spanish evaluation benchmark that exposes where current ASR systems fail on uncontrolled dysarthric speech.
  • Practitioners should prefer lightweight text cleanup over fine-tuning when the target is wild neurological Spanish outside the training distribution.
  • Future data collection must deliberately cover more stroke and female speakers, because the present wild-sourced mix is male- and ALS-heavy.
  • Clean read speech and hospital monologues cannot substitute for in-the-wild pathological Spanish when measuring real generalization.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the domain gap is mainly acoustic and stylistic rather than linguistic, multi-condition data augmentation or self-supervised adaptation on unlabeled wild pathological audio may succeed where supervised fine-tuning failed.
  • The same post-processing-versus-fine-tuning comparison could be run on other low-resource languages that currently lack wild neurological corpora, testing whether the robustness pattern is language-independent.
  • Commercial black-box systems that already show lower error on S-DiverSe may already contain the kind of diverse training data the open models lack; releasing even partial metadata about that data would accelerate open progress.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 7 minor

Summary. The paper introduces S-DiverSe, a 3.2-hour in-the-wild Spanish ASR evaluation corpus of 444 manually transcribed segments from 22 speakers with ALS, Parkinson’s disease, and stroke, with metadata for sex, condition, and intelligibility. It reports baselines for Whisper-large-v3, Voxtral-Mini, omniASR CTC 1B v2, and ElevenLabs Scribe v2, and compares heuristic text post-processing against several fine-tuning recipes (full FT, encoder FT, and LoRA variants) trained on TORGO, NeuroVoz, and Spanish Common Voice. The central empirical claim is that rule-based post-processing is more robust than the tested parameter-updating methods for out-of-domain neurological Spanish speech: PP lowers S-DiverSe WER without harming in-domain scores, while fine-tuning improves TORGO/NeuroVoz but fails to generalize (and can catastrophically degrade) on S-DiverSe.

Significance. The work addresses a clear resource gap: publicly available Spanish pathological speech is scarce and mostly controlled/clinical, so an in-the-wild multi-condition evaluation set with human transcripts is a useful community contribution even at modest scale. Strengths include transparent composition analysis (sex/condition/intelligibility/duration), multi-system baselines with error-type breakdowns, explicit comparison to TORGO and NeuroVoz, and open release of annotations and video links. The PP-versus-FT contrast in Table 2 is concrete and actionable for practitioners. The finding is scoped to the tested adaptation recipes rather than a universal proof that fine-tuning cannot work, but it still motivates dedicated in-the-wild Spanish pathological benchmarks and more careful domain-adaptation practice.

major comments (2)
  1. §3.3–3.4 and §4.2/§5: The conclusion that heuristic post-processing is more robust than fine-tuning, and that parameter updates “fail to generalize … regardless of language composition,” rests on a narrow adaptation search (fixed LRs/epochs, LoRA r=8/α=16, three data mixes, NeuroVoz oversampled ×3). Table 2 supports the claim for these recipes, but the manuscript should more tightly scope the claim to the tested strategies and state that broader adaptation (e.g., LR/rank sweeps, longer training, multi-task or continued pretraining objectives) remains unexplored. Without that qualification, the domain-gap interpretation is stronger than the experimental design strictly warrants.
  2. Table 2 and §4.1–4.2: WER differences that drive the central PP-vs-FT claim (e.g., Whisper PP 22.01% vs FFT+PP 125.68% on S-DiverSe; Voxtral PP 23.73% vs several FT variants) are reported as point estimates only. With 444 segments and highly skewed pathology/intelligibility strata, bootstrap CIs or paired significance tests on total and per-pathology WER would make the robustness claim more rigorous and show whether smaller gaps (e.g., E-LoRA vs PP-only) are stable.
minor comments (7)
  1. §2.2: Weighted Cohen’s κ=0.38 (fair) for intelligibility is reported with 74% adjacent agreement; consider adding a short note on how this uncertainty should affect interpretation of Figure 4 (WER by intelligibility), or reporting results with collapsed adjacent bins as a sensitivity check.
  2. Table 1 / Figure 1: The strong male (87.4%) and ALS (78.1%) imbalance is acknowledged as reflecting in-the-wild availability; a one-sentence caution that per-pathology and per-sex WERs (especially stroke and female) are underpowered would help readers avoid over-reading those columns.
  3. §3.5 / evaluation protocol: Clarify whether filled-pause tokens and <unk> stripping are applied symmetrically to all systems (including Scribe v2 API output), and whether any system-specific normalization was needed for commercial transcripts.
  4. Figure 4 caption/text: The note that “High” is PD-only and “High/Medium” is ALS-only is important; move it into the figure caption so the plot is self-contained.
  5. §2.1 perplexity analysis: Median perplexity 33 (S-DiverSe) vs 49 (Common Voice) is used later to argue domain mismatch; briefly define the salamandra-2b prompting/tokenization setup so the comparison is reproducible.
  6. Presentation: Normalize author-name accents and model names consistently (e.g., Voxtral vs V oxtral, NeuroVoz vs NeuroV oz) across text and Table 2; fix minor spacing/typos in the arXiv rendering of compound names.
  7. Data release: State more explicitly what a third party must do to reconstruct the audio (YouTube availability risk, segment time stamps if any) so the benchmark’s long-term reproducibility expectations are clear.

Circularity Check

0 steps flagged

No circularity: empirical dataset/evaluation paper; WER and adaptation comparisons are measured, not derived by construction from fitted inputs.

full rationale

S-DiverSe is a corpus-release and ASR benchmarking paper. Its central claims (baseline WERs; heuristic post-processing more robust than the reported fine-tuning recipes on out-of-domain S-DiverSe) rest on direct measurement against held-out human orthographic transcripts, not on a derivation chain. Post-processing rules (character/word/phrase deduplication) are fixed heuristics independent of the S-DiverSe score; fine-tuning uses external corpora (TORGO, NeuroVoz, Common Voice) with stated hyperparameters and is evaluated, not defined, on S-DiverSe. No parameter is fitted to a subset of S-DiverSe and then re-reported as a prediction; no uniqueness theorem or ansatz is imported via self-citation to force the result; no known empirical pattern is merely renamed. Using the new corpus as the OOD test set is the purpose of a benchmark paper and does not constitute circular derivation. Score 0 is therefore appropriate.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 1 invented entities

Load-bearing content is empirical: a new annotated YouTube-derived corpus plus standard ASR metrics and adaptation recipes. Free parameters are training/adaptation knobs and the NeuroVoz oversampling factor. Axioms are domain assumptions about diagnosis validity, intelligibility labeling, and that the chosen FT grid represents adaptation. The main invented entity is the S-DiverSe corpus itself as an evaluation object; no new physical or theoretical entities.

free parameters (4)
  • NeuroVoz oversampling factor = 3
    NeuroVoz oversampled by 3 when combined with TORGO to compensate language imbalance; chosen by authors, affects FT training distribution.
  • LoRA rank r and alpha = r=8, α=16, dropout=0.1
    r=8, α=16, dropout=0.1 fixed for LoRA strategies; hand-chosen adaptation capacity.
  • Fine-tuning learning rates and epochs = strategy-specific (1e-5/3, 2e-5/5, 3e-4/10)
    FFT lr=1e-5/3 epochs; EFT lr=2e-5/5 epochs; LoRA lr=3e-4/10 epochs with early stopping patience 5—settings that strongly influence whether FT collapses on OOD.
  • PP max word length and dedup rules = 15-char threshold; consecutive word/phrase collapse
    Words >15 chars with internal repetition collapsed; consecutive word/phrase deduplication—heuristic thresholds that drive the PP gains.
axioms (4)
  • domain assumption Self-reported diagnoses plus video metadata and perceptual non-normative speech suffice to label ALS/PD/stroke for ASR evaluation (not clinical inference).
    Stated in §2.2 sourcing and §4.3 limitations; underpins pathology-stratified WER.
  • domain assumption A 1–5 intelligibility scale based on perceived transcription effort is a valid analysis axis despite fair IAA (κ=0.38).
    §2.2 annotation protocol and decision to keep original labels after IAA study.
  • ad hoc to paper S-DiverSe must not be split for training because strata would be too skewed; it is evaluation-only.
    §3 experimental setup; shapes the OOD-only use of the corpus.
  • domain assumption Standard WER after lowercasing, punctuation strip, <unk> removal, and digit-to-word conversion is the right success metric, with filled pauses counted as tokens.
    §3.5 evaluation protocol used for all claims.
invented entities (1)
  • S-DiverSe corpus independent evidence
    purpose: Provide the first multi-condition in-the-wild Spanish neurological speech benchmark with transcripts and metadata for ASR evaluation.
    New resource constructed by the authors from YouTube; independent of prior NeuroVoz/TORGO content though comparable in purpose.

pith-pipeline@v1.1.0-grok45 · 13528 in / 3523 out tokens · 38383 ms · 2026-07-12T04:07:08.563684+00:00 · methodology

0 comments
read the original abstract

Automatic speech recognition (ASR) has advanced remarkably for standard speech, yet speech affected by neurological conditions remains a challenge. We present S-DiverSe (Spanish Diverse Speech), a corpus of 3.2 hours of in-the-wild Spanish speech from 22 speakers with amyotrophic lateral sclerosis, Parkinson's disease, and stroke. The dataset contains 444 manually transcribed audio segments with metadata on speaker sex, disease type, and intelligibility. S-DiverSe is designed to support ASR evaluation and development for neurologically affected Spanish speech. We describe the dataset, analyze its composition, and report baseline ASR results alongside initial adaptation experiments. Our findings reveal that heuristic text post-processing is more robust than fine-tuning for out-of-domain neurological Spanish speech. This underscores the need for dedicated in-the-wild Spanish benchmarks.

Figures

Figures reproduced from arXiv: 2607.03207 by Ana Mart\'inez, Fernando Iba\~nez, Fernando L\'opez, Iv\'an Alonso, Jordi Luque, Pablo G\'omez, Santosh Kesiraju.

Figure 1
Figure 1. Figure 1: Sex distribution by pathological condition [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: shows the distribution of samples by condition and intelligibility. ALS and stroke exhibit unimodal distributions skewed toward medium-to-low intelligibility, while PD displays a bimodal pattern. High High/Medium Medium Medium/Low Low Intelligibility Level ALS PD STROKE Pathological Condition 0 34 6 137 75 82 0 3 26 0 0 0 0 64 17 0 20 40 60 80 100 120 Number of Samples [PITH_FULL_IMAGE:figures/full_fig_p0… view at source ↗
Figure 3
Figure 3. Figure 3: Audio Duration Distribution. Transcript linguistic complexity was estimated using salamandra-2b [10], a Castilian Spanish open-weight LLM, yielding a median perplexity of 33, indicating moderate linguistic simplicity. As a reference point, we computed the median perplexity of the Spanish Common Voice v24.0 [11] training set, which yielded 49. 2.2. Source and annotation Audio was sourced from in-the-wild Yo… view at source ↗
Figure 4
Figure 4. Figure 4: shows the expected inverse relationship between intelligibility and WER. The “High” bin comprises PD speech exclusively, while the “High/Medium” bin comprises uniquely ALS samples. This makes these bins not directly comparable to others. At low intelligibility, Scribe v2 maintains relative robustness while open-weight models degrade substantially. High High/Medium Medium Medium/Low Low Intelligibility Leve… view at source ↗
Figure 5
Figure 5. Figure 5: WER by ASR model decomposed by error type. 4.2. Adapted models PP is the most robust adaptation strategy on S-DiverSe: it con￾sistently reduces WER for both models without harming in￾domain performance on TORGO, confirming that a substantial share of errors is correctable at the text level. Fine-tuning con￾sistently improves in-domain performance but fails to general￾ize to out-of-domain data. FFT+PP yield… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

25 extracted references · 4 linked inside Pith

  1. [1]

    However, they con- tinue to face substantial challenges when applied to real-world speech scenarios [2]

    Introduction Automatic speech recognition (ASR) has improved markedly in recent years, with state-of-the-art models achieving low word error rates on standard benchmarks [1]. However, they con- tinue to face substantial challenges when applied to real-world speech scenarios [2]. One of these challenges is the recogni- tion of speech produced by individual...

  2. [2]

    mmm”, “eh

    S-DiverSe: Spanish Diverse Speech S-DiverSe is designated for speech recognition tasks. It com- prises 3.2 hours of human-transcribed speech, extracted from in-the-wild recordings. It features 22 unique speakers with ALS, PD, and post-stroke aftereffects. The corpus contains 444 audio segments of variable duration, and provides metadata for speaker sex, c...

  3. [3]

    Experimental setup S-DiverSe is used exclusively for ASR evaluation. The fac- tor preventing further partitioning is not its modest size but its variability: it spans three pathologies, multiple intelligibil- ity levels, and heterogeneous acoustic conditions, so any train- ing split would be too skewed per stratum for reliable adapta- tion. Following the ...

  4. [4]

    High” bin comprises PD speech exclusively, while the “High/Medium

    Results and discussion Table 2 summarizes WER on NeuroV oz, TORGO, and S- DiverSe for all systems and adaptation settings. 4.1. Baseline results Among open-weight models,omniASR CTC 1B v2achieves the lowest WER on S-DiverSe, while V oxtral-Mini leads on NeuroV oz and Whisper-large-v3 on TORGO. No single open- weight model dominates across all corpora, sug...

  5. [5]

    The dataset contains 3.2 hours of in-the-wild segments from 22 speakers with ALS, Parkinson’s disease, and stroke

    Conclusion We present S-DiverSe, the first Spanish corpus of neurologi- cally affected speech spanning multiple diseases. The dataset contains 3.2 hours of in-the-wild segments from 22 speakers with ALS, Parkinson’s disease, and stroke. Our experiments re- veal that current ASR systems struggle under the heterogeneous captured conditions. Heuristic text p...

  6. [6]

    Generative AI Use Disclosure We used a generative AI tool to paraphrase and polish portions of the manuscript to improve readability and grammar

  7. [7]

    Linguistics, Artificial Intelligence and Lan- guage and Speech Technologies: from Research to Applica- tions

    Acknowledgments We would like to thank Irene Gordo Bernat, speech-language pathologist, for her valuable insights and assistance with speech analysis and data selection. This project has been partially funded by the European Union’s Horizon 2020 RIA ELO- QUENCE project (Grant Agreement No. 101135916). Views and opinions expressed are, however, those of th...

  8. [8]

    Automatic speech recognition: A survey of deep learning techniques and ap- proaches,

    H. Ahlawat, N. Aggarwal, and D. Gupta, “Automatic speech recognition: A survey of deep learning techniques and ap- proaches,”International Journal of Cognitive Computing in En- gineering, 2025

  9. [9]

    Asr benchmarking: Need for a more representative conversational dataset,

    G. Maheshwari, D. Ivanov, T. Johannet, and K. El Haddad, “Asr benchmarking: Need for a more representative conversational dataset,” inICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2025, pp. 1–5

  10. [10]

    The torgo database of acoustic and articulatory speech from speakers with dysarthria,

    F. Rudzicz, A. K. Namasivayam, and T. Wolff, “The torgo database of acoustic and articulatory speech from speakers with dysarthria,”Language resources and evaluation, vol. 46, no. 4, pp. 523–541, 2012

  11. [11]

    Dysarthric speech database for universal access research

    H. Kim, M. Hasegawa-Johnson, A. Perlman, J. R. Gunderson, T. S. Huang, K. L. Watkin, S. Frameet al., “Dysarthric speech database for universal access research.” inInterspeech, vol. 2008, 2008, pp. 1741–1744

  12. [12]

    Community-supported shared infrastructure in support of speech accessibility,

    M. Hasegawa-Johnson, X. Zheng, H. Kim, C. Mendes, M. Dickin- son, E. Hege, C. Zwilling, M. M. Channell, L. Mattie, H. Hodges et al., “Community-supported shared infrastructure in support of speech accessibility,”Journal of Speech, Language, and Hearing Research, vol. 67, no. 11, pp. 4162–4175, 2024

  13. [13]

    The Interspeech 2025 Speech Accessibility Project Challenge,

    X. Zheng, B. Phukon, J. Na, E. Cutrell, K. J. Han, M. Hasegawa- Johnson, P.-P. Jiang, A. Kuila, C. Lea, B. MacDonald, G. Man- tena, V . Ravichandran, L. Sari, K. Tomanek, C. D. Yoo, and C. Zwilling, “The Interspeech 2025 Speech Accessibility Project Challenge,” inInterspeech 2025, 2025, pp. 3269–3273

  14. [14]

    New spanish speech cor- pus database for the analysis of people suffering from parkinson’s disease

    J. R. Orozco-Arroyave, J. D. Arias-Londo ˜no, J. F. Vargas-Bonilla, M. C. Gonzalez-R ´ativa, and E. N ¨oth, “New spanish speech cor- pus database for the analysis of people suffering from parkinson’s disease.” inLrec, 2014, pp. 342–347

  15. [15]

    Neurovoz: a castillian spanish corpus of parkinsonian speech,

    J. Mendes-Laureano, J. A. G ´omez-Garc´ıa, A. Guerrero-L ´opez, E. Luque-Buzo, J. D. Arias-Londo ˜no, F. J. Grandas-P ´erez, and J. I. Godino-Llorente, “Neurovoz: a castillian spanish corpus of parkinsonian speech,”Scientific Data, vol. 11, no. 1, p. 1367, 2024

  16. [16]

    Neurovoz: a castillian spanish corpus of parkinsonian speech,

    J. Mendes-Laureano, J. A. G ´omez-Garc´ıa, A. Guerrero-L ´opez, E. Luque-Buzo, J. D. Arias-Londo ˜no, F. J. Grandas-P ´erez, and J. I. Godino Llorente, “Neurovoz: a castillian spanish corpus of parkinsonian speech,” Mar. 2024. [Online]. Available: https://doi.org/10.5281/zenodo.10777657

  17. [17]

    Salamandra technical report,

    A. Gonzalez-Agirre, M. P `amies, J. Llop, I. Baucells, S. Da Dalt, D. Tamayo, J. J. Saiz, F. Espu ˜na, J. Prats, J. Aula-Blascoet al., “Salamandra technical report,”arXiv preprint arXiv:2502.08489, 2025

  18. [18]

    Com- mon voice: A massively-multilingual speech corpus,

    R. Ardila, M. Branson, K. Davis, M. Kohler, J. Meyer, M. Hen- retty, R. Morais, L. Saunders, F. Tyers, and G. Weber, “Com- mon voice: A massively-multilingual speech corpus,” inProceed- ings of the twelfth language resources and evaluation conference, 2020, pp. 4218–4222

  19. [19]

    Robust speech recognition via large-scale weak supervision,

    A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” inInternational conference on machine learning. PMLR, 2023, pp. 28 492–28 518

  20. [20]

    V oxtral,

    A. H. Liu, A. Ehrenberg, A. Lo, C. Denoix, C. Barreau, G. Lam- ple, J.-M. Delignon, K. R. Chandu, P. von Platen, P. R. Mud- direddyet al., “V oxtral,”arXiv preprint arXiv:2507.13264, 2025

  21. [21]

    Ministral 3,

    A. H. Liu, K. Khandelwal, S. Subramanian, V . Jouault, A. Rastogiet al., “Ministral 3,” 2026. [Online]. Available: https://arxiv.org/abs/2601.08584

  22. [22]

    Omnilingual asr: Open-source multilingual speech recognition for 1600+ languages,

    A. Omnilingual, G. Keren, A. Kozhevnikov, Y . Meng, C. Rop- ers, M. Setzler, S. Wang, I. Adebara, M. Auli, C. Baliogluet al., “Omnilingual asr: Open-source multilingual speech recognition for 1600+ languages,”arXiv preprint arXiv:2511.09690, 2025

  23. [23]

    wav2vec 2.0: A framework for self-supervised learning of speech repre- sentations,

    A. Baevski, Y . Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech repre- sentations,”Advances in neural information processing systems, vol. 33, pp. 12 449–12 460, 2020

  24. [24]

    Scribe v2,

    ElevenLabs, “Scribe v2,” Speech-to-Text model, 2026, https: //elevenlabs.io/docs/overview/models

  25. [25]

    Cba-whisper: Curriculum learning-based adalora fine-tuning on whisper for low-resource dysarthric speech recognition,

    T. Tan, X. Chen, X. Le, W. Fan, X. Xia, C. Huang, and J. Lu, “Cba-whisper: Curriculum learning-based adalora fine-tuning on whisper for low-resource dysarthric speech recognition,” inProc. Interspeech 2025, 2025, pp. 3309–3313