Boli: A dataset for understanding stuttering experience and analyzing stuttered speech

Ashita Batra; Mannas Narang; Neeraj Kumar Sharma; Pradip K Das

REVIEW 5 major objections 6 minor 15 references

Boli: A dataset for understanding stuttering experience and analyzing stuttered speech

T0 review · 5 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The Boli dataset provides word-level, timestamped labels of five stutter types in read and spontaneous Indian-language speech, along with questionnaire responses from people who stutter.

desk verdict A genuinely new stuttered-speech resource for Indian languages, but the paper currently makes it impossible to verify the dataset exists or that the labels are trustworthy. read the letter →

arxiv 2501.15877 v3 pith:UKRVQY2M submitted 2025-01-27 cs.HC cs.AI

Ashita Batra , Mannas Narang , Neeraj Kumar Sharma , Pradip K Das This is my paper · ORCID

classification cs.HCcs.AI

keywords stutteredspeechdatasetIndianlanguagesword-levelannotationstuttertypeclassificationreadspontaneousquestionnairestutteringeventdetection

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces Boli, a dataset of stuttered speech from people who stutter in India, designed to support both scientific study and speech technology. The central claim is that Boli fills a gap left by existing stutter datasets, which are mostly English, file-level, or non-Indian, by offering an open-access multilingual Indian-language resource. The dataset pairs read speech (Rainbow Passage) and spontaneous speech (image descriptions) with word-level annotations of five stutter types — blocks, prolongations, interjections, sound repetitions, and word repetitions — and adds questionnaire data on how stuttering affects daily life. If the data are made available as claimed, Boli would give researchers a benchmark for stutter detection and automatic speech recognition evaluation in Indian languages that did not previously exist.

What carries the argument

The load-bearing object is the Boli dataset itself: paired read and spontaneous recordings, word-level stutter-type labels with timestamps, and anonymized questionnaire responses. The word-level annotation scheme is what lets the dataset support event detection rather than merely file-level classification, and the questionnaire connects acoustic phenomena to lived experience. For validation, the paper uses a standard pipeline — mel-frequency cepstral coefficients as audio features, hybrid class balancing, and a five-way classifier trained on an existing English dataset and tested on Boli — to show that the audio admits stutter-type classification.

What would settle it

Try to download the audio from the project site given in the paper and have two independent annotators relabel a random sample of the same files; if the files are not downloadable or the annotators disagree substantially, the dataset's central promise of open access and reliable word-level labels is not yet met.

Watch

Extended reading notes

Core claim

The paper's discovery is the Boli dataset: audio and metadata from 28 people who stutter, selected after screening 67 volunteers, totaling about 2.5 to 2.8 hours of speech. Each participant read part of the Rainbow Passage and described an image, in English and in their mother tongue, spanning Hindi, Telugu, Bengali, Marathi, and Assamese. The recordings carry word-level, time-stamped annotations of five stutter types: blocks, prolongations, interjections, sound repetitions, and word repetitions. The paper also reports that stuttering was less frequent in spontaneous speech (2.02 events per minute) than in read speech (6.76 events per minute), and it validates the audio through a five-class stutter-type classification experiment in which a balanced random forest trained on an existing English dataset reaches an average F1 of 0.87 on Boli's English utterances. It further reports word-error-rate baselines from two automatic speech recognition systems on stuttered Indian English, with Whisper outperforming Wav2Vec2.0, and it summarizes the 25-question responses from 67 participants on self-reported triggers, speech-therapy experience, and social effects of stuttering.

Load-bearing premise

The load-bearing premise is that the audio is actually available for download and that the manual word-level stutter labels are consistent enough to be reused, because the paper provides no repository link, annotation guideline, or inter-annotator agreement figures.

Editorial extensions

If this is right

Word-level labels with timestamps create a training target for stutter-event detection in Indian languages, not just file-level classification.
The paired read/spontaneous design shows stuttering rate drops in spontaneous speech (2.02 vs 6.76 events per minute), so Boli can support studies of speaking-mode effects.
The questionnaire responses tie acoustic events to self-reported triggers and coping strategies, enabling experience-aware speech analysis.
Cross-dataset classification results (average F1 0.87 with a balanced random forest) suggest models trained on English stutter data transfer to Indian English, so Boli could serve as a target benchmark.
ASR word-error-rate baselines for stuttered Indian English give a concrete starting point for improving speech recognition for people who stutter.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural next step is to release annotation guidelines and conduct an inter-annotator agreement study; without those, the word-level labels cannot be independently verified.
The current class imbalance (140 sound repetitions vs 8 interjections) means stutter-type models need explicit balancing, and expansion should oversample rare types and severe cases.
The read/spontaneous paired design transfers naturally to many under-resourced languages and could be paired with a standardized severity instrument to make experiential data comparable across studies.
A first practical check is to verify that the stated project website actually serves the audio, since the paper gives no download link.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The Boli dataset is a real gap-filler: the first resource combining multiple Indian languages, read and spontaneous speech, word-level annotations of five stutter types, and questionnaire data from people who stutter. The scope is modest (28 speakers, ~2.8 hours), but the combination is new, and the authors are honest that they are not aiming for state-of-the-art classification. The descriptive statistics about read versus spontaneous stuttering rates and the questionnaire findings are useful for the community.

The paper's strengths: the collection procedure is explained, the annotation types follow the Sep-28k conventions, and the cross-dataset evaluation (training on Sep-28k, testing on Boli) is a sensible way to get some signal despite the small size. The authors also include the questionnaire summaries, which is unusual and valuable.

The soft spots are real and they matter. First, the paper claims the dataset is open access, but no repository, DOI, license, or download procedure is provided anywhere. The project website is mentioned, but it does not obviously host the data, so as written a reader cannot check whether the dataset exists at all. Second, the word-level annotations have no associated protocol: no annotation guidelines, no number of annotators, no adjudication process, and no inter-annotator agreement. Every downstream result, including Table IV stuttering rates and the classification labels in Table V, rests on labels that cannot be verified. Third, the segmentation into ~5-second clips used for classification is not described, so the technical validation cannot be reconstructed. The F1-scores are also reported on a tiny test set without confidence intervals, so they are illustrative at best.

These issues are fixable, not fatal. If the authors supply a direct data link with a clear license, release annotation guidelines, report inter-annotator agreement, and describe the segmentation, the contribution becomes solid. As is, the paper's central claims are conditional on materials that are absent.

This paper deserves peer review because the dataset fills a genuine gap and the authors have put in real effort. I would send it out with a strong request to make the data accessible and to document label reliability. The reader's conditional verdict is fair, and the stress-test concern about verifiability is accurate and should be the primary request for revision.

For my own work, I would not cite this yet because I cannot access the data, but I would revisit it once the dataset is released. I would bring it to a reading group if the discussion is about dataset construction pitfalls in speech pathology resources.

Referee Report

5 major / 6 minor

Summary. The paper presents Boli, a crowd-sourced multilingual dataset of read and spontaneous speech from people who stutter, together with demographic metadata, responses to a 25-item questionnaire, and word-level annotations of five stutter types. The authors report descriptive statistics on stuttering rates, a cross-dataset stutter-type classification experiment trained on Sep-28k and tested on English Boli clips, ASR word-error-rate comparisons, and qualitative summaries of the questionnaire. The stated goal is to provide an open-access resource for stuttered speech research, particularly for Indian languages.

Significance. If the dataset is made available as described, it would fill a genuine gap: there is currently no Indian-language stuttered speech dataset with word-level annotations and linked experiential questionnaire data. The independent cross-dataset evaluation design is a strength, since the classifiers are trained on external data and tested on Boli, avoiding circularity. The manual collection of both read and spontaneous speech from 28 speakers with stutter is a useful contribution, as are the reported patterns of stuttering rates across speech modes. However, the paper as written does not allow a reader to access the dataset, assess label reliability, or interpret the quantitative validation; these gaps make the contribution contingent.

major comments (5)

[Section II, data collection] The central claim that Boli is an open-access dataset is not verifiable. The manuscript names only the data-collection website (https://project-boli.vercel.app/) and provides no repository URL, DOI, license, checksum, or download procedure. Without a persistent data availability statement, a reader cannot confirm that the dataset exists or use it. Please add a stable repository link or DOI and a clear data-access section, and state the license under which the audio and annotations are released.
[Section II, manual annotation] No annotation protocol is reported: the paper does not state the number of annotators, their training or screening, the annotation unit (word vs. syllable), the annotation tool, or the adjudication procedure. No inter-annotator agreement measure is provided. This matters because Table IV's stuttering rates, the class distribution (SR=140, B=70, PR=41, WR=21, IN=8), Table V's classification labels, and Table VI's ASR verification all depend on the unvalidated manual labels. Please report annotation guidelines and inter-annotator agreement on a representative subset.
[Section III, Table V] The stutter-type classification evaluation is under-specified. The paper does not state the number of Boli test clips per class, how continuous recordings were segmented into the approximately 5-second clips, or how the English-only subset was extracted from the 28 speakers. F1 scores are reported without confidence intervals; with class counts as small as IN=8, values such as 0.99 for interjection are not interpretable. Please report test-set size, per-class support, and bootstrap or equivalent confidence intervals, along with an explicit segmentation rule.
[Section III, Eq. (1)] The hybrid-sampling procedure is not described precisely enough to reproduce. Equation (1) defines only an average of N1 and N2, and the surrounding text does not specify how many samples are drawn from the minority and majority classes, whether replacement is used, or how the balanced dataset is constructed from the Sep-28k training set. Because the paper's main conclusion that random forest is the best classifier rests on the balanced-training comparison, please specify the exact resampling algorithm.
[Section IV, questionnaire analysis] The questionnaire analysis is presented as qualitative summary statements, but the instrument itself is not included, the response distributions are not tabulated, and no statistical test or analysis method is described. For example, the claim that 'language plays a crucial role in stuttering' is not connected to any reported quantitative result. Please add the full questionnaire or a link to it, summary statistics for each item, and a description of how the qualitative findings in Figure 3 were derived.

minor comments (6)

[Section II, Table III] The cross-reference 'Table reftable:demographics' appears to be a LaTeX error; it should be Table III.
[Table II vs. Section II] Table II lists the Boli duration as 2.5 hours, while the text states approximately 2.8 hours of audio; please reconcile these numbers.
[Figure 2] The label 'PRBIN SR WR' in Figure 2 is unclear; it should list the five stutter types as PR, B, IN, SR, WR.
[Section III, ASR evaluation] Table VI reports WER but does not specify the ASR model versions, whether reference transcripts are the manual word-level annotations, how punctuation and casing were handled, or how the concatenated-speech setting was aligned with the manual annotations.
[General] The paper does not report ethics approval, informed consent, or anonymization procedures for the human participants; a dataset paper of this type should include an ethics and consent statement.
[Abstract and index terms] The abstract promises 'severity assessment of stuttering events,' and the index terms include 'Intelligibility assessment,' but the manuscript does not actually model severity or intelligibility; it only reports self-reported severity in Table III. Please align the claims with the content.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: Boli's technical validation trains on the external Sep-28k dataset and tests on Boli, while the questionnaire analysis is descriptive; the dataset claims rest on manual annotation and accessibility, not on self-referential derivation.

full rationale

The paper's central contribution is a new dataset, and its reported analyses do not reduce to their own inputs. The classifier evaluation in Section III is explicitly a cross-dataset validation: 'training on Sep-28k and testing on the Boli dataset (English utterances only).' This means the machine-learning results are not fitted on the data they are evaluated against, so no fitted parameter is being renamed as a prediction. The stutter-type annotations are produced by 'Manual annotation of stuttered speech recordings' and are a data-creation step, not an output of the paper's models. Table IV's stuttering rates are direct event counts divided by recording durations. Section IV's questionnaire summary is descriptive aggregation of self-reported experiences, not derived from the speech signal, so there is no self-definitional link between the questionnaire claims and the stutter labels. There are no equations in the paper that define one reported quantity in terms of another reported quantity in a way that would make a 'prediction' trivially equal to its input. The references to prior datasets and models (Sep-28k, UCLASS, KSoF, LibriStutter, etc.) are external works, not self-citations by the present authors, and no uniqueness theorem is invoked to force a choice. The absence of a direct download link and of inter-annotator agreement statistics is a verifiability and reproducibility concern, not a circularity concern; it does not make the derivation self-referential. Therefore no significant circularity is present, and the appropriate score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claim (a usable dataset) rests on the accuracy of self-identification, manual annotation, and task design. No fitted parameters or invented entities are introduced; the only hand-chosen element is the hybrid sampling formula in the classification experiments, which does not affect the dataset claim itself.

assumptions (4)

domain assumption The five stutter types (block, prolongation, interjection, sound repetition, word repetition) are defined and mutually exclusive as in Table I.
The entire annotation scheme depends on these categories being correct and applicable across languages; no validation of category definitions is provided.
domain assumption Participants self-reported as people who stutter and produced natural, not artificial, stuttering.
Data collection relied on crowd-sourced self-selection and encouragement to relax; there is no verification that the recordings reflect natural stuttering.
domain assumption Manual annotations of stutter events and timestamps are accurate.
The dataset's value depends on annotation quality; the paper does not report inter-annotator agreement or a second-pass verification protocol.
domain assumption Read speech and spontaneous speech tasks elicit representative stuttering behavior.
The use of the Rainbow Passage and image description assumes these tasks capture typical stuttering patterns across participants.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Boli: A dataset for understanding stuttering experience and analyzing stuttered speech." pith.science (2026). https://pith.science/paper/UKRVQY2M

@misc{pith2026250115877,
  author       = {Pith},
  title        = {Pith review of: Boli: A dataset for understanding stuttering experience and analyzing stuttered speech},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UKRVQY2M}},
  note         = {Machine review of arXiv:2501.15877}
}

read the original abstract

There is a growing need for diverse, high-quality stuttered speech data, particularly in the context of Indian languages. This paper introduces Project Boli, a multi-lingual stuttered speech dataset designed to advance scientific understanding and technology development for individuals who stutter, particularly in India. The dataset constitutes (a) anonymized metadata (gender, age, country, mother tongue) and responses to a questionnaire about how stuttering affects their daily lives, (b) captures both read speech (using the Rainbow Passage) and spontaneous speech (through image description tasks) for each participant and (c) includes detailed annotations of five stutter types: blocks, prolongations, interjections, sound repetitions and word repetitions. We present a comprehensive analysis of the dataset, including the data collection procedure, experience summarization of people who stutter, severity assessment of stuttering events and technical validation of the collected data. The dataset is released as an open access to further speech technology development.

Figures

Figures reproduced from arXiv: 2501.15877 by the authors.

**Figure 1.** Spectrograms illustrating a few stutter types associated with a male speaker: (a) Sound Repetition (SR), (b) Prolongation [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

**Figure 2.** Proposed methodology for stutter-type classification [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Most stuttered sounds based on information shared [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

15 extracted references · 11 canonical work pages

[1]

International Stammering/ Stut- tering Awareness Day 2022

Redcliffe Labs. International Stammering/ Stut- tering Awareness Day 2022. 2022. Available: https://redcliffelabs.com/myhealth/health/general-health/ international-stammering-stuttering-awareness-day-2022/

work page 2022
[2]

Lea, Colin, et al. ”From user perceptions to technical improvement: En- abling people who stutter to better use speech recognition.” Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 2023, pp. 1–16

work page 2023
[3]

”Enhancing asr for stuttered speech with limited data using detect and pass”

Shonibare, O., Tong, X., & Ravichandran, V . ”Enhancing asr for stuttered speech with limited data using detect and pass”. arXiv preprint arXiv:2202.05396., 2022

arXiv 2022
[4]

”Disordered Speech Data Collection: Lessons Learned at 1 Million Utterances from Project Euphonia.” Interspeech

MacDonald, Robert L., et al. ”Disordered Speech Data Collection: Lessons Learned at 1 Million Utterances from Project Euphonia.” Interspeech. V ol. 2021, 2021, pp. 4833–4837

work page 2021
[5]

”Analysis and tuning of a voice assistant system for dysfluent speech.” arXiv preprint arXiv:2106.11759, 2021

Mitra, Vikramjit, et al. ”Analysis and tuning of a voice assistant system for dysfluent speech.” arXiv preprint arXiv:2106.11759, 2021

arXiv 2021
[6]

”The university college london archive of stuttered speech (uclass)”

Howell, P., Davis, S., & Bartrip, J. ”The university college london archive of stuttered speech (uclass)”. 2009

work page 2009
[7]

”Detecting multiple speech disfluencies using a deep residual network with bidirectional long short- term memory”

Kourkounakis, T., Hajavi, A., & Etemad, A. ”Detecting multiple speech disfluencies using a deep residual network with bidirectional long short- term memory”. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing. pp: 6089–6093, 2020

work page 2020
[8]

”Fluentnet: End-to-end de- tection of stuttered speech disfluencies with deep learning”

Kourkounakis, T., Hajavi, A., & Etemad, A. ”Fluentnet: End-to-end de- tection of stuttered speech disfluencies with deep learning”. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29, pp: 2986– 2999, 2021

work page 2021

Show all 15 references

[9]

P., von Gudenberg, A

Bayerl, S. P., von Gudenberg, A. W., H¨onig, F., N¨oth, E., & Riedhammer, K. ”KSoF: The Kassel state of fluency dataset–a therapy centered dataset of stuttering”. arXiv preprint arXiv:2203.05383,2022

2022 arXiv
[10]

Lea, C., Mitra, V ., Joshi, A., Kajarekar, S.,& Bigham, J. P. ”Sep-28k: A dataset for stuttering event detection from podcasts with people who stutter”. In 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2021, pp: 6798–6802

2021
[11]

B., & MacWhinney, B

Ratner, N. B., & MacWhinney, B. ”Fluency Bank: A new resource for fluency research and practice”. Journal of fluency disorders, 56, pp:69– 80, 2018

2018
[12]

”AS-70: A Mandarin stuttered speech dataset for automatic speech recognition and stuttering event detection”

Gong, R., Xue, H., Wang, L., Xu, X., Li, Q., Xie, L., & Li, M. ”AS-70: A Mandarin stuttered speech dataset for automatic speech recognition and stuttering event detection”. arXiv preprint arXiv:2406.07256, 2024

2024 arXiv
[13]

S., Mahesh, S., Barche, P., Mirishkar, S

Narasinga, V ., Fathima, H., Motepalli, K. S., Mahesh, S., Barche, P., Mirishkar, S. G., & Vuppala, A. ”Enhancing Stuttering Detection: A Syllable-Level Stutter Dataset”. In 2024 International Conference on Signal Processing and Communications (SPCOM), 2024, pp: 1–5

2024
[14]

”wav2vec 2.0: A framework for self-supervised learning of speech representations”

Baevski, A., Zhou, Y ., Mohamed, A., & Auli, M. ”wav2vec 2.0: A framework for self-supervised learning of speech representations”. Advances in neural information processing systems, vol. 33, pp: 12449– 12460, 2020

2020
[15]

W., Xu, T., Brockman, G., McLeavey, C., & Sutskever, I

Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., & Sutskever, I. ”Robust speech recognition via large-scale weak super- vision”. In International conference on machine learning. PMLR, 2023, pp: 28492-28518

2023

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

International Stammering/ Stut- tering Awareness Day 2022

Redcliffe Labs. International Stammering/ Stut- tering Awareness Day 2022. 2022. Available: https://redcliffelabs.com/myhealth/health/general-health/ international-stammering-stuttering-awareness-day-2022/

work page 2022

[2] [2]

Lea, Colin, et al. ”From user perceptions to technical improvement: En- abling people who stutter to better use speech recognition.” Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 2023, pp. 1–16

work page 2023

[3] [3]

”Enhancing asr for stuttered speech with limited data using detect and pass”

Shonibare, O., Tong, X., & Ravichandran, V . ”Enhancing asr for stuttered speech with limited data using detect and pass”. arXiv preprint arXiv:2202.05396., 2022

arXiv 2022

[4] [4]

”Disordered Speech Data Collection: Lessons Learned at 1 Million Utterances from Project Euphonia.” Interspeech

MacDonald, Robert L., et al. ”Disordered Speech Data Collection: Lessons Learned at 1 Million Utterances from Project Euphonia.” Interspeech. V ol. 2021, 2021, pp. 4833–4837

work page 2021

[5] [5]

”Analysis and tuning of a voice assistant system for dysfluent speech.” arXiv preprint arXiv:2106.11759, 2021

Mitra, Vikramjit, et al. ”Analysis and tuning of a voice assistant system for dysfluent speech.” arXiv preprint arXiv:2106.11759, 2021

arXiv 2021

[6] [6]

”The university college london archive of stuttered speech (uclass)”

Howell, P., Davis, S., & Bartrip, J. ”The university college london archive of stuttered speech (uclass)”. 2009

work page 2009

[7] [7]

”Detecting multiple speech disfluencies using a deep residual network with bidirectional long short- term memory”

Kourkounakis, T., Hajavi, A., & Etemad, A. ”Detecting multiple speech disfluencies using a deep residual network with bidirectional long short- term memory”. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing. pp: 6089–6093, 2020

work page 2020

[8] [8]

”Fluentnet: End-to-end de- tection of stuttered speech disfluencies with deep learning”

Kourkounakis, T., Hajavi, A., & Etemad, A. ”Fluentnet: End-to-end de- tection of stuttered speech disfluencies with deep learning”. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29, pp: 2986– 2999, 2021

work page 2021