Pith. sign in

REVIEW 4 major objections 6 minor 38 references

Clinical Annotations for Automatic Stuttering Severity Assessment

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper proposes a multi-dimensional, clinically aligned annotation scheme for stuttering severity — covering disfluency type, secondary behavior, and tension — and applies it to audiovisual recordings of adults who stutter, yielding a…

desk verdict A genuinely useful multimodal stuttering annotation resource, but the self-referential gold standard and the deliberately hard test set mean the reported F1 scores are group-conformity numbers, not clinical ground truth. read the letter →

arxiv 2506.00644 v1 pith:3PZ4ZBHG submitted 2025-05-31 cs.CL cs.AI

Ana Rita Valente , Rufael Marew , Hawau Olamide Toyin , Hamdan Al-Ali , Anelise Bohnen , Inma Becerra , Elsa Marta Soares , Goncalo Leal
show 1 more author
Hanan Aldarmaki
This is my paper
classification cs.CLcs.AI
keywords stutteringseverityassessmentdisfluencyannotationclinicalspeech-languagepathologymulti-modalinter-annotatoragreementconsensusgoldstandardsecondarybehaviorstensionscoring
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that clinical stuttering severity assessment cannot be supported by existing audio-only, non-expert annotations, and that a multi-dimensional scheme is needed that captures disfluency type, secondary behaviors, and tension together. It proposes such a scheme, grounded in established clinical taxonomies, and applies it to 66 audiovisual recordings of adults who stutter, annotated independently by three speech-language pathologists. It also constructs a consensus-based test set in which disagreed-upon stuttering events were re-reviewed and resolved, and uses it to score individual annotators and aggregation methods. The reported inter-annotator agreement and baseline model results quantify how hard the task is, and the released labels are intended to let automatic stuttering assessment be trained and evaluated against clinically meaningful targets.

What carries the argument

The carrier of the argument is the annotation scheme itself: each stuttering span receives a primary disfluency type from a standard behavioral taxonomy of stuttering-like disfluencies, a secondary-behavior category (verbal, facial grimace, head movement, extremity movement), and a tension level on a 0-3 scale; annotators mark span boundaries using the acoustic waveform alongside video and transcript in a unified multi-modal annotation tool. The consensus process is the second load-bearing mechanism: a combined file with a disagreement tier focuses discussion on disputed events, and the resolved labels form the gold test set. Aggregation baselines (majority vote and distance-based selection methods) and segment-level evaluation metrics supply the quantitative frame that lets annotator quality and model performance be compared against the gold labels.

What would settle it

If a fresh panel of independently trained clinicians, who did not participate in the original consensus, annotates the same test files and their majority labels disagree substantially with the published gold labels (for example, macro F1 well below the original annotators' scores), the claim that the test set is a reliable consensus gold standard would be refuted.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central discovery is that a clinically valid stuttering annotation can be operationalized as a triple of labels per stuttering moment — primary disfluency type from a behavioral taxonomy, secondary behavior category, and a 0-3 tension score — and applied across a public corpus of adult stuttered speech at scale, yielding 1,654 reading spans and 4,037 interview spans. The authors show that expert clinicians disagree substantially, especially on tension, and that a structured consensus process can produce gold-standard labels for a test set. Against those labels, individual clinicians achieve macro F1 scores between about 0.67 and 0.79, simple aggregation methods do not consistently beat the best annotator, and baseline machine-learning models reach 0.95 F1 for detecting any stuttering event but much lower scores for specific types, with audio alone best for primary disfluencies and video or multi-modal input best for secondary behaviors. The claim being established is that this resource, with its multiple dimensions and consensus labels, is a necessary step toward automatic systems that assess severity rather than merely detect disfluency.

Load-bearing premise

The gold-standard test labels come from consensus among the same three clinicians whose individual annotations are being evaluated, and the test files were chosen because they contained the most disagreements between two of those clinicians; the assumption is that this internal consensus is a valid external ground truth for measuring annotation quality and model performance.

Editorial extensions

If this is right

  • Automatic stuttering assessment can move from binary disfluency detection toward clinically meaningful outputs: type, secondary behavior, and tension labels for each stuttering moment.
  • The consensus test set gives a reproducible target for comparing future annotators and models, so reported F1 scores across studies become comparable.
  • The multimodal baseline results indicate that fusing audio and video improves detection of any stuttering event and of secondary behaviors, while audio-only remains stronger for primary disfluency typing; future systems should decide per-label modality.
  • Because agreement improved from the reading section to the later-annotated interview section, iterative discussion rounds on small batches may be an effective protocol for maintaining expert label reliability in larger annotation efforts.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the released labels are used as training targets, the very low inter-annotator agreement on tension means models trained on tension scores will inherit noisy supervision; an ordinal or anchor-based relabeling of tension may be needed before it can be predicted reliably.
  • Because the gold labels were produced by the same three annotators whose individual labels are scored, and the test files were deliberately selected from high-disagreement cases, the published annotator F1 numbers likely overstate how well a new, independent clinician would match the gold; an external validation panel would settle this.
  • A natural extension the authors do not build is to combine the annotated spans into clinical severity indices — percentage of stuttered syllables, average tension, and secondary-behavior counts — which would directly test whether the scheme predicts therapist severity ratings.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes a multi-dimensional annotation scheme for stuttering severity assessment on FluencyBank, in which three expert speech-language pathologists annotate stuttering moments with disfluency type (LBDL), secondary behaviors (SSI-4), and a tension scale. The authors report inter-annotator agreement, construct a consensus-based 'gold standard' test set from files with the most disagreements, evaluate individual annotators and aggregation methods against that gold, and provide audio, video, and multimodal baselines. The stated goal is to make the annotations publicly available to enable clinically aligned automatic stuttering assessment.

Significance. If the dataset and the consensus test set are valid and released, this would be a valuable resource. Existing stuttering annotation efforts either lack visual dimensions, rely on non-expert annotators, or provide only primary disfluency types; a clinically informed multi-dimensional annotation effort involving expert SLPs would be a useful contribution. The paper is transparent about its methodology, uses established taxonomies, and reports detailed annotation processes and challenges. However, the central validity of the gold standard is questionable because it is derived from the same annotators it is used to evaluate, and the tension dimension shows very low agreement despite being a core part of the claimed multi-dimensional scheme. These issues need to be addressed before the resource can be used as a reliable benchmark.

major comments (4)
  1. [4.3, Table 3] The gold standard used to evaluate annotators in Table 3 is produced by the same three annotators whose individual labels are scored against it, and the test set is deliberately selected from files where the two most experienced annotators had the most disagreements. For any instance where all three annotators initially agreed, an annotator's label is identical to the gold label, and for disagreements the gold is the outcome of discussion among those same annotators. The reported F1 scores therefore measure consistency with the group's discussion process rather than independent clinical accuracy, and they are likely inflated by shared training, group dynamics, and selection effects. These scores cannot be interpreted as typical annotator performance on FluencyBank or in clinical practice. The authors should validate the gold standard against an external reference (e.g., a fourth expert not involved in the original annotation), report agreement with that reference, and discuss how the deliberate choice of high-disagreement files affects the test set's representativeness for model evaluation.
  2. [3.1, 4.2] Tension is presented as one of the three core annotation dimensions (Section 3.1, Table 1) and is part of the claim that the scheme aligns with clinical practice, yet the inter-annotator agreement for tension is very low (Krippendorff's alpha = 0.18, KS = 0.38, sigma = 0.34) in Section 4.2. No analysis, baseline, or validation is provided for the tension dimension, and it is absent from the test-set evaluation in Tables 3 and 4. Because a central contribution is a comprehensive multi-dimensional scheme, the unusably low agreement for this dimension substantially weakens the claim. The authors should either provide evidence that tension annotations are reliable after the consensus process (e.g., agreement on the gold test set for tension), refine the tension annotation protocol, or clearly delimit the contribution to the other two dimensions.
  3. [Table 2] The counts in Table 2 are internally inconsistent. The sum of the primary-type rows (SR 190 + ISR 143 + MUR 94 + P 93 + B 265 + None 25) is 810, not the reported total of 732. Additionally, the secondary-behavior columns do not sum to the row totals; for example, the SR row lists 23 + 114 + 38 + 1 + 53 = 229 secondary-behavior occurrences for 190 SR events. The table should clarify whether multiple secondary behaviors can be labeled per event, and the totals must be corrected, since this table is the quantitative description of the test set.
  4. [1, 6] The central artifact of the paper, the annotations themselves, is not actually available: the abstract states the annotations 'will be made publicly available' and the conclusion says they 'will be released,' but no data link is provided. For a dataset contribution, reviewers and readers need access to the annotations and the annotation manual to verify the scheme and reproduce the analysis. Please provide an anonymous download link (or a clear availability statement with the actual repository) as part of the manuscript or supplementary materials.
minor comments (6)
  1. [5, Table 4] The baseline F1 scores in Table 4 are reported without standard deviations or statistical significance tests. Given the small differences between models (e.g., the Any-class F1 values ranging from 0.90 to 0.95) and the use of overlapping 5-second segments, the claim that the multi-modal approach is best for the Any class and generally better for secondary behaviors is not statistically supported. Please report means and variances over multiple runs or seeds.
  2. [4.2] The text states that 'we see higher IAA scores in the interview section which was annotated after the annotation and discussion of the reading section,' but per-section agreement scores are not reported anywhere. Please provide the relevant IAA values for the reading and interview sections separately to substantiate this observation.
  3. [4.3, Table 2] The row label 'None' for primary disfluency type is not defined in Section 3.1. Please clarify what a 'None' primary type denotes (e.g., a secondary behavior observed without a stuttering moment) and how such cases are handled in the span-based annotation scheme.
  4. [3.1] The annotation manual is described in detail but is not provided. Including the manual as supplementary material would greatly improve reproducibility and allow other clinicians to apply the same scheme.
  5. [Table 1] 'Mutli-dimensional' is a typo for 'Multi-dimensional'.
  6. [4.3, 5] The term 'test set' is used for the consensus evaluation set of Section 4.3 and later for model evaluation in Section 5. Please clarify whether the baseline evaluations in Table 4 were performed on the consensus gold test set only, or on the full annotated dataset, and specify the train/dev/test split used.

Circularity Check

1 steps flagged · score 2.0 of 10

No derivation reduces to fitted parameters or imported uniqueness; the only circular element is the self-referential consensus gold standard, which is disclosed in Section 4.3 and limits the benchmark evaluation but does not invalidate the dataset contribution.

  1. self definitional [Section 4.3 (Test Set with Consensus Annotations) and Table 3 caption]
    "Each disagreement was resolved through multiple visualizations of the stuttering event and in-depth discussions until consensus was reached on the final ‘gold standard’ annotations. ... F1 score for each annotator and aggregation method measured against gold labels across the classes using segment based evaluation as described in [29]"

    The gold labels are produced by the same three annotators whose individual labels are then evaluated against them. For any span on which all three annotators agreed, each annotator's label and the gold label coincide by construction. For spans in the disagreement tier, the gold label is the group's negotiated outcome, so an annotator's F1 measures distance from the consensus that they helped form, not agreement with an independent clinical reference. The test files were also selected because the two most experienced annotators disagreed most on disfluency types, so Tables 3-4 report performance on a deliberately hard, non-representative subset.

full rationale

The paper makes no theoretical prediction that reduces to fitted parameters or to an imported uniqueness theorem. The annotation categories are explicitly taken from independent clinical sources: LBDL for disfluency types, Riley's SSI-4 for secondary behaviors, and Boey et al. for tension. Those sources are external to this paper, so the multi-dimensional scheme has independent content. The main circular element is the evaluation of annotators (Table 3) and models (Table 4) against a gold standard produced by the same annotators via consensus. This makes the benchmark a measure of intra-group consistency rather than validation against an independent criterion, and the selection of highest-disagreement files makes the numbers non-representative. Because this limitation is disclosed in Section 4.3 and the annotation scheme is not derived from the gold labels, the paper is not a case of fitted input being renamed as a prediction; it is a methodological self-reference that lowers the evidentiary value of the reported F1 scores but does not collapse the central dataset contribution.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces no fitted parameters or new theoretical entities. It relies on existing clinical taxonomies and on the assumptions that the FluencyBank videos are adequate for visual annotation, that the SLP annotators are expert, and that their consensus is a valid gold standard.

assumptions (4)
  • domain assumption FluencyBank audiovisual recordings are adequate for visual secondary-behavior annotation (face and shoulder girdle visible).
    Section 4.1 describes camera positioning; if video quality or framing is insufficient, secondary behavior labels are unreliable.
  • domain assumption The LBDL, SSI-4, and Boey et al. taxonomies are valid and reliable clinical standards for stuttering assessment.
    Section 3.1 bases the annotation scheme on these instruments without independent validation on this dataset.
  • domain assumption The three SLP annotators are sufficiently expert to provide ground-truth labels.
    Section 3.1 states 2 to 40 years of experience, but no formal certification or external benchmark is provided.
  • ad hoc to paper Consensus among the three annotators after discussion yields a correct gold standard for evaluation.
    Section 4.3 uses consensus to create test labels without validation against clinical severity scores or independent judges.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Clinical Annotations for Automatic Stuttering Severity Assessment." pith.science (2026). https://pith.science/paper/3PZ4ZBHG

@misc{pith2026250600644,
  author       = {Pith},
  title        = {Pith review of: Clinical Annotations for Automatic Stuttering Severity Assessment},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3PZ4ZBHG}},
  note         = {Machine review of arXiv:2506.00644}
}
read the original abstract

Stuttering is a complex disorder that requires specialized expertise for effective assessment and treatment. This paper presents an effort to enhance the FluencyBank dataset with a new stuttering annotation scheme based on established clinical standards. To achieve high-quality annotations, we hired expert clinicians to label the data, ensuring that the resulting annotations mirror real-world clinical expertise. The annotations are multi-modal, incorporating audiovisual features for the detection and classification of stuttering moments, secondary behaviors, and tension scores. In addition to individual annotations, we additionally provide a test set with highly reliable annotations based on expert consensus for assessing individual annotators and machine learning models. Our experiments and analysis illustrate the complexity of this task that necessitates extensive clinical expertise for valid training and evaluation of stuttering assessment models.

Figures

Figures reproduced from arXiv: 2506.00644 by the authors.

Figure 1
Figure 1. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

38 extracted references · 32 canonical work pages

  1. [1]

    loss of control

    Introduction Stuttering is a neurodevelopmental and multidimensional com- munication disorder that begins early in speech and language development [1, 2, 3]. It is characterized by involuntary disrup- tions in speech fluency, such as repetitions, prolongations, and blocks, which are inconsistent and variable [2, 4, 5]. In addition to speech disruptions, s...

    work page

  2. [2]

    FluencyBank is a com- ponent of the broader TalkBank project, which serves as a com- prehensive resource for studying fluency disorders

    Related Work Research on automatic stuttering detection, also called disflu- ency detection, has been facilitated by several publicly avail- able data sets that include audio or video recordings of adults who stutter.FluencyBank[15] is a database that includes au- dio/video recordings from various participants, including chil- dren and adults who stutter ...

    arXiv 2025

  3. [3]

    The combination of these quantitative behavioral measures (i.e., frequency, dura- tion, and severity of secondary behaviors) is used to determine stuttering severity [5, 13]

    Annotation Methodology Stuttering moments, defined as interruptions in speech that rep- resent the observable aspect of the stuttering disorder, can be classified based on frequency, duration, disfluency type, and the type/severity of secondary behaviors [2]. The combination of these quantitative behavioral measures (i.e., frequency, dura- tion, and sever...

    work page

  4. [4]

    disagreement

    Analysis of Annotated Data 4.1. Data Sources All audiovisual samples were sourced from the FluencyBank database. In the audiovisual recordings, participants were po- sitioned facing the camera, allowing for clear visualization of both the face and the shoulder girdle.Reading:A total of 30 audiovisual reading samples from Adults Who Stutter (AWS), with a c...

    work page

  5. [5]

    We split the clips into 5-second segments with a 2-second overlap window

    Baselines In this section, we describe basic experiments performed on the given dataset as baselines for future research2. We split the clips into 5-second segments with a 2-second overlap window. We aggregate the labels of the segments in two stages: first, we use the labels from the best aggregation method (MAJ) described in 4.5. For each segment, we th...

    work page

  6. [6]

    Our analysis and baseline results illustrate the complexity of the task that necessitates further investigations

    Conclusion We presented a clinically annotated dataset for stuttering sever- ity assessment. Our analysis and baseline results illustrate the complexity of the task that necessitates further investigations. The annotations and baseline scripts will be released to encour- age researchers to explore this clinical application beyond stan- dard disfluency det...

    work page

  7. [7]

    Understanding the speaker’s experience of stuttering can improve stuttering therapy,

    S. E. Tichenor, C. Herring, and J. S. Yaruss, “Understanding the speaker’s experience of stuttering can improve stuttering therapy,” Topics in language disorders, vol. 42, no. 1, pp. 57–75, 2022

    work page 2022

  8. [8]

    W. H. Manning and A. DiLollo,Clinical decision making in flu- ency disorders. Plural Publishing, 2023

    work page 2023

Show all 38 references
  1. [9]

    Bloodstein, N

    O. Bloodstein, N. B. Ratner, and S. B. Brundage,A handbook on stuttering. Plural Publishing, 2021, vol. 1

    2021

  2. [10]

    Epidemiology of stuttering: 21st cen- tury advances,

    E. Yairi and N. Ambrose, “Epidemiology of stuttering: 21st cen- tury advances,”Journal of fluency disorders, vol. 38, no. 2, pp. 66–87, 2013

    2013

  3. [11]

    Application of the icf in fluency disorders,

    J. S. Yaruss, “Application of the icf in fluency disorders,” inSemi- nars in speech and language, vol. 28, no. 04. © Thieme Medical Publishers, 2007, pp. 312–322

    2007

  4. [12]

    Guitar,Stuttering: An integrated approach to its nature and treatment

    B. Guitar,Stuttering: An integrated approach to its nature and treatment. Lippincott Williams & Wilkins, 2019

    2019

  5. [13]

    Long-term consequences of child- hood bullying in adults who stutter: Social anxiety, fear of nega- tive evaluation, self-esteem, and satisfaction with life,

    G. W. Blood and I. M. Blood, “Long-term consequences of child- hood bullying in adults who stutter: Social anxiety, fear of nega- tive evaluation, self-esteem, and satisfaction with life,”Journal of fluency disorders, vol. 50, pp. 72–84, 2016

    2016

  6. [14]

    Event-and interval-based measurement of stuttering: a review,

    A. R. S. Valente, L. M. Jesus, A. Hall, and M. Leahy, “Event-and interval-based measurement of stuttering: a review,”International Journal of Language & Communication Disorders, vol. 50, no. 1, pp. 14–30, 2015

    2015

  7. [15]

    The impact of stuttering on the quality of life in adults who stutter,

    A. Craig, E. Blumgart, and Y . Tran, “The impact of stuttering on the quality of life in adults who stutter,”Journal of fluency disorders, vol. 34, no. 2, pp. 61–71, 2009

    2009

  8. [16]

    Stuttering and the international classification of functioning, disability, and health (icf): An up- date,

    J. S. Yaruss and R. W. Quesal, “Stuttering and the international classification of functioning, disability, and health (icf): An up- date,”Journal of communication disorders, vol. 37, no. 1, pp. 35– 52, 2004

    2004

  9. [17]

    A comprehensive view of stut- tering: Implications for assessment and treatment,

    C. Coleman and J. Scott Yaruss, “A comprehensive view of stut- tering: Implications for assessment and treatment,”Perspectives on School-Based Issues, vol. 15, no. 2, pp. 75–80, 2014

    2014

  10. [18]

    Yairi and C

    E. Yairi and C. H. Seery,Stuttering: Foundations and clinical applications. Plural publishing, 2023

    2023

  11. [19]

    Development of assessment tools to evaluate adults with fluency disorders,

    A. R. dos Santos Valente, “Development of assessment tools to evaluate adults with fluency disorders,” Ph.D. dissertation, Uni- versidade de Aveiro (Portugal), 2018

    2018

  12. [20]

    Riley and K

    G. Riley and K. Bakker,Stuttering severity instrument. Pro-ed, 2009

    2009

  13. [21]

    R. B. Gillam, K. J. Logan, and N. A. Pearson,TOCS: Test of child- hood stuttering. Pro-Ed Austin, 2009

    2009

  14. [22]

    Fluency bank: A new resource for fluency research and practice,

    N. B. Ratner and B. MacWhinney, “Fluency bank: A new resource for fluency research and practice,”Journal of fluency disorders, vol. 56, pp. 69–80, 2018

    2018

  15. [23]

    Stuttering moments

    and classifications used in specific stuttering assessment tools, such as the Stuttering Severity Instrument [13]. In this work, we combine three types of clinical classification systems for disfluencies, secondary behaviors, and tension, to enable comprehensive stuttering sev...

    2009

  16. [24]

    Sep-28k: A dataset for stuttering event detection from podcasts with people who stutter,

    C. Lea, V . Mitra, A. Joshi, S. Kajarekar, and J. P. Bigham, “Sep-28k: A dataset for stuttering event detection from podcasts with people who stutter,” 2021. [Online]. Available: https://arxiv.org/abs/2102.12394

    2021 arXiv

  17. [25]

    Identification of primary and collateral tracks in stuttered speech,

    R. Riad, A.-C. Bachoud-L ´evi, F. Rudzicz, and E. Dupoux, “Identification of primary and collateral tracks in stuttered speech,” inProceedings of the Twelfth Language Resources and Evaluation Conference, May 2020. [Online]. Available: https://aclanthology.org/2020.lrec-1.208

    2020

  18. [26]

    The uclass archive of stut- tered speech

    P. Howell, S. Davis, and J. Bartrip, “The uclass archive of stut- tered speech.”Journal of speech, language, and hearing research : JSLHR, 03 2009

    2009

  19. [27]

    Fluentnet: End-to- end detection of stuttered speech disfluencies with deep learning,

    T. Kourkounakis, A. Hajavi, and A. Etemad, “Fluentnet: End-to- end detection of stuttered speech disfluencies with deep learning,” IEEE/ACM Transactions on Audio, Speech, and Language Pro- cessing, vol. 29, pp. 2986–2999, 2021

    2021

  20. [28]

    Ksof: The kassel state of fluency dataset – a therapy centered dataset of stuttering,

    S. P. Bayerl, A. W. von Gudenberg, F. H ¨onig, E. N ¨oth, and K. Riedhammer, “Ksof: The kassel state of fluency dataset – a therapy centered dataset of stuttering,” 2022

    2022

  21. [29]

    A longitudinal study of stuttering in children: A preliminary report,

    E. Yairi and N. Ambrose, “A longitudinal study of stuttering in children: A preliminary report,”Journal of Speech, Language, and Hearing Research, vol. 35, no. 4, pp. 755–760, 1992

    1992

  22. [30]

    The lexicon of stutter- ing,

    A. Packman, M. Onslow, and K. Bryant, “The lexicon of stutter- ing,” inProceedings of the Fifth Oxford Dysfluency Conference. KL Baker Leicester, UK, 2000, pp. 53–60

    2000

  23. [31]

    The lidcombe behav- ioral data language of stuttering,

    K. Teesson, A. Packman, and M. Onslow, “The lidcombe behav- ioral data language of stuttering,”Journal of Speech, Language, and Hearing Research, 2003

    2003

  24. [32]

    Elan: A professional framework for multimodality research,

    P. Wittenburg, H. Brugman, A. Russel, A. Klassmann, and H. Sloetjes, “Elan: A professional framework for multimodality research,” in5th international conference on language resources and evaluation (LREC 2006), 2006, pp. 1556–1559

    2006

  25. [33]

    Characteristics of stuttering-like disfluencies in dutch-speaking children,

    R. A. Boey, F. L. Wuyts, P. H. Van de Heyning, M. S. De Bodt, and L. Heylen, “Characteristics of stuttering-like disfluencies in dutch-speaking children,”Journal of fluency disorders, vol. 32, no. 4, pp. 310–329, 2007

    2007

  26. [34]

    Measuring annotator agreement generally across complex structured, multi-object, and free-text annotation tasks,

    A. Braylan, O. Alonso, and M. Lease, “Measuring annotator agreement generally across complex structured, multi-object, and free-text annotation tasks,” inProceedings of the ACM Web Conference 2022. ACM, pp. 1720–1730. [Online]. Available: https://dl.acm.org/doi/10.1145/3485447.3512242

    2022

  27. [35]

    A general model for aggregating annotations across simple, complex, and multi-object annotation tasks,

    A. Braylan, M. Marabella, O. Alonso, and M. Lease, “A general model for aggregating annotations across simple, complex, and multi-object annotation tasks,” vol. 78, pp. 901–973. [Online]. Available: https://jair.org/index.php/jair/article/view/14388

  28. [36]

    Metrics for polyphonic sound event detection,

    A. Mesaros, T. Heittola, and T. Virtanen, “Metrics for polyphonic sound event detection,”Applied Sciences, vol. 6, no. 6, 2016. [Online]. Available: https://www.mdpi.com/2076-3417/6/6/162

    2016

  29. [37]

    wav2vec 2.0: A framework for self-supervised learning of speech repre- sentations,

    A. Baevski, Y . Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech repre- sentations,”Advances in neural information processing systems, vol. 33, pp. 12 449–12 460, 2020

    2020

  30. [38]

    Vivit: A video vision transformer,

    A. Arnab, M. Dehghani, G. Heigold, C. Sun, M. Lu ˇci´c, and C. Schmid, “Vivit: A video vision transformer,” inProceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 6836–6846

    2021

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.