REVIEW 3 major objections 5 minor 41 references
Automatic classification of stop realisation with wav2vec2.0
T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read This paper claims that wav2vec2.0, fine-tuned on labelled stop tokens, can automatically classify whether a stop consonant has a burst with high accuracy in English and Japanese, and that the predicted labels reproduce the conditioning…
desk verdict Useful applied methods paper: the cross-lingual data-efficiency result is new and believable, but the within-corpus token-level evaluation overstates the robustness claim. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The mechanism is transfer learning with wav2vec2.0, a self-supervised model whose convolutional feature encoder and transformer context network are pre-trained on large amounts of unlabelled speech and then fine-tuned with a small classification head on a binary burst-presence label. The paper's key move is to use a model already trained for burst detection on Japanese as the starting point for English, which turns a requirement of 20k+ hand-labelled tokens into 500–1k. The input to the classifier is the waveform segment of each stop, extracted with a short context window, and the label is defined by whether a separate closure and burst can be observed.
What would settle it
Have two or more independent phoneticians re-annotate a random subset of the SPADE validation stops and compute agreement with the first author's labels; if agreement is low, the reported 88% accuracy is measured against unstable ground truth. Alternatively, apply the trained model to a corpus with independently verified burst labels from a language or dialect outside the training set and check whether accuracy stays near 88%.
Extended reading notes
Core claim
The central claim is that wav2vec2.0 models, after fine-tuning on a few tens of thousands of labelled stop tokens, classify stop burst presence with accuracy comparable to a dedicated phonetic classifier and to human annotation, and that this transfers across languages and across corpus conditions. On the Japanese CSJ-C, all four base models reach 0.94 accuracy and 0.86–0.90 F1. On the English SPADE data, accuracy peaks at 87–88% only with at least 20k training tokens, but fine-tuning on top of a CSJ-C-trained model reaches similar performance with 500–1k tokens. The paper also claims that when the trained models are used to annotate held-out stops, the resulting patterns—higher burst probability for voiced stops and increasing burst probability with duration—match the hand-annotated patterns, with differences of a few percentage points and a systematic overestimate of 7–8% for long stops.
Load-bearing premise
The manual labels for the 55,628 English stops, produced by a single annotator without a reported inter-annotator reliability check, are treated as ground truth; if those labels are biased or inconsistent, every accuracy and pattern comparison inherits that bias.
Editorial extensions
If this is right
- Researchers can annotate stop burst presence in new corpora with a wav2vec2.0 model pre-trained on CSJ-C plus a few hundred hand-labelled tokens, rather than labelling tens of thousands of stops by hand.
- Automatic annotations can be used directly in studies of stop lenition, since they reproduce the same voicing and duration conditioning patterns as manual annotations.
- The approach extends across languages and recording conditions, from clean monologues to heterogeneous multi-corpus collections.
- The CSJ-C-trained checkpoints and code are released, so the reported accuracies are reproducible and the models are reusable.
Reading between the lines
- The same bootstrap recipe—train on a clean curated corpus, then fine-tune on a few hundred tokens from a messy target corpus—could plausibly transfer to other variable phenomena such as glottalisation, flapping, or vowel reduction, whenever a binary or categorical label can be defined.
- The reported 7–8% overestimate of burst probability for long stops suggests that studies using automatic burst labels to model duration effects should either include annotation type as a covariate or restrict analysis to the range where predictions match hand labels.
- Because the voicing pattern (higher burst probability for voiced stops) is found in both manual and predicted labels, the model is learning the corpus annotation convention, not an independent phonetic truth; the same pipeline could be used to audit annotation conventions across corpora.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper trains wav2vec2.0 models (base-960h, large-960h, lv60, xlsr) to classify stop burst presence/absence in two datasets: a clean, hand-corrected Japanese corpus (CSJ-C, Experiment 1) and a large, mostly uncorrected multi-corpus English dataset (SPADE, Experiment 2). The authors report high accuracy/F1 on held-out test/validation tokens (about 91-94% for CSJ-C and 87-88% for SPADE with sufficient training data), show that fine-tuning a CSJ-C-trained model on only 500-1,000 SPADE tokens can approach the performance of models trained on 20k-40k SPADE tokens, and compare predicted burst probabilities with manual annotations in a GAMM analysis of phonological voicing and stop duration, concluding that the automatic annotations replicate patterns found in manual annotations. The paper positions the method as a reusable, scalable tool for annotating variable phonetic phenomena.
Significance. If the central claims hold, this is a practically valuable contribution: it would give phonetic researchers a freely available, cross-linguistic tool for automatic stop-realisation annotation, and the finding that very small amounts of target-domain data suffice after cross-corpus pretraining has clear practical payoff. The paper's strengths include held-out evaluation with bootstrap confidence intervals (Figure 2), the use of a large and varied multi-corpus dataset (SPADE), and the public release of models and code. The main results are empirical rather than derivational, so the assessment rests on the adequacy of the evaluation design and label quality; both need tightening before the headline robustness/generalisability claim is supported.
major comments (3)
- [Section 2.1-2.2 and 3.1-3.2] The central generalisation claim is not yet supported because neither experiment uses a speaker-disjoint or corpus-disjoint split. For CSJ-C, the paper reports a random subset of 40k stops split 80/20 without stating whether any of the 137 speakers appear in both training and test. For SPADE, the 11k validation tokens are 'pseudo-randomly selected (containing some stops from each corpus)' from the same 55,628 manually annotated stops used to create the training subsets, and the paper does not report the number of speakers or corpora in each split. If speakers or recording conditions are shared between training and validation, wav2vec2.0 can exploit speaker- or channel-specific acoustic properties of bursts, inflating the reported 87-88% SPADE accuracy and 91-94% CSJ-C accuracy. The abstract claims robustness 'across both finely-curated and unprepared speech corpora,' but this requires evaluation on held-out speakers and, ideally, held-out corpora. Please report speaker- and corpus-disjoint evaluations, or at minimum quantify the overlap and show that performance is stable when it is removed.
- [Section 3.1] All 55,628 SPADE labels were produced by a single annotator (the first author) with no inter-annotator reliability check reported, and the CSJ-C burst labels are treated as reliable without reporting agreement metrics. Since the accuracy/F1 numbers, data-efficiency curves, and pattern comparisons (Section 4) are all measured against these labels, any systematic annotator bias or inconsistency is inherited by every quantitative claim. This is not a circularity problem, because the validation tokens are not used to fit the predictions, but it is a trustworthiness problem for the ground truth. Please report inter-annotator agreement on a subset (e.g., Cohen's kappa or percentage agreement), or at minimum describe the annotation protocol and any quality-control checks used.
- [Section 4] The RQ3 pattern analysis compares hand annotations with predictions from two CSJ-C-pre-trained models on the same 11k SPADE validation tokens. The GAMM includes by-corpus random smooths and by-speaker random slopes, but the paper does not state the number of speakers/corpora in the validation set or how they are distributed relative to the training data. If the validation speakers are also present in the training subsets, the pattern comparison could reflect speaker-specific learning rather than a generalisable burst/no-burst distinction. This matters because the paper uses the pattern replication as evidence that the models are capturing the same phonetic variability as manual annotation. Please report speaker/corpus counts for the validation set and, if possible, rerun the Section 4 comparison on a speaker-disjoint validation set.
minor comments (5)
- [Table 1] The F1 values are typeset as '0 .9' and '0 .86'; these should be corrected to '0.90' and '0.86'.
- [Figure 1 caption] The caption contains stray symbols: 'right]))' should read 'right)' or 'right', and the references to left/middle/right panels should be cleaned up.
- [Section 2.2] The text says the models' predictive accuracies 'align closely to the ~97%' reported in [9], but Table 1 reports 91-94% accuracy; please clarify whether accuracy or F1 is being compared, and justify the phrase 'closely' given the 3-6 percentage point gap.
- [Section 3.1] The paper says 1.13 million stops were extracted and 55,628 were manually annotated, but the sampling procedure is only described as 'approximately 5% of each corpus's voiced and voiceless stops, up to a maximum of 1,000 stops'; please state whether the 55,628 are a random sample and whether the 11k validation set is disjoint from the training subsets.
- [Figure 2] The x-axis labels are difficult to read and the model names are not all identifiable in the printed figure; please enlarge the labels and add a clear legend.
Circularity Check
No circularity: the burst classifiers are evaluated on held-out tokens, and pattern replication is an out-of-sample empirical result.
full rationale
The paper's derivation chain is a standard supervised-learning pipeline. Manual burst-presence labels are extracted from CSJ-C and SPADE (Secs. 2.1, 3.1); wav2vec2.0 models are fine-tuned on these labels (Secs. 2.2, 3.2); held-out tokens are then scored for accuracy and F1, and the resulting predictions are compared to manual labels in a GAM (Sec. 4). There is no equation in the paper that defines a predicted quantity in terms of the same quantity being claimed as an output. The CSJ-C test set is an 80/20 split of the training pool, and the SPADE validation set of 11k stops is pseudo-randomly selected from the training corpus distribution; because those tokens are not used to fit the model, the reported accuracy and pattern replication are out-of-sample empirical results. Manual annotations are the training target and also the evaluation reference, but that is the usual supervised-evaluation setup, not a definitional equivalence. Self-citations to the SPADE dataset [28], the CSJ RDB [25], and the Montreal Forced Aligner [1] provide data-generation provenance, not an argument that forces the classification outcome. The speaker-overlap limitation raised by a skeptical reader is a generalization and validity concern, not circularity.
Assumptions & free parameters
assumptions (4)
- domain assumption Pre-trained wav2vec2.0 representations carry enough acoustic-phonetic detail for fine-grained stop classification
- domain assumption CSJ-C hand-corrected burst presence labels are reliable ground truth
- domain assumption SPADE forced-aligned boundaries are accurate enough that adding 10 ms context yields a valid stop interval for annotation and classification
- domain assumption The first author's SPADE burst annotations are self-consistent and match the CSJ-C annotation definition
Cite this review
Pith. "Pith review of Automatic classification of stop realisation with wav2vec2.0." pith.science (2026). https://pith.science/paper/ECTPKZCO
@misc{pith2026250523688,
author = {Pith},
title = {Pith review of: Automatic classification of stop realisation with wav2vec2.0},
year = {2026},
howpublished = {\url{https://pith.science/paper/ECTPKZCO}},
note = {Machine review of arXiv:2505.23688}
}
read the original abstract
Modern phonetic research regularly makes use of automatic tools for the annotation of speech data, however few tools exist for the annotation of many variable phonetic phenomena. At the same time, pre-trained self-supervised models, such as wav2vec2.0, have been shown to perform well at speech classification tasks and latently encode fine-grained phonetic information. We demonstrate that wav2vec2.0 models can be trained to automatically classify stop burst presence with high accuracy in both English and Japanese, robust across both finely-curated and unprepared speech corpora. Patterns of variability in stop realisation are replicated with the automatic annotations, and closely follow those of manual annotations. These results demonstrate the potential of pre-trained speech models as tools for the automatic annotation and processing of speech corpus data, enabling researchers to 'scale-up' the scope of phonetic research with relative ease.
Figures
Reference graph
Works this paper leans on
-
[1]
Introduction Over the past twenty years, a number of tools for the automatic and semi-automatic annotation of speech data have been re- leased, including for the time-alignment of phonetic transcrip- tions to audio (‘forced aligners’) [1, 2, 3] and the labelling of variable phonetic phenomena [4, 5, 6], both substantially re- ducing the time required for ...
-
[2]
Experiment 1 The goal of Experiment 1 is to test whether wav2vec2.0 can be trained to accurately predict the presence or absence of a stop closure and burst in the ‘best-case’ spontaneous speech con- text: speech that is cleanly-recorded and stylistically largely ho- mogenous, and whose segmental annotations have been hand- corrected after alignment. 2.1....
work page Pith review arXiv 2025
-
[3]
Experiment 2 Having demonstrated that high predictive performance can be achieved with clean, manually-corrected speech data, Experi- ment 2 explores the performance of these models on more vari- able speech – both in speech style and recording conditions – with no manual correction following forced alignment. 3.1. Data The data for Experiment 2 comes fro...
work page 2000
-
[4]
We consider the patterns of stop burst probability as a function of 1
Analysis of predicted stop patterns Having demonstrated that wav2vec2.0 models can achieve good performance with both curated and non-curated data (RQ1, RQ2), we now consider the extent to which patterns from pre- dicted stops follow those from hand-annotations (RQ3), where predictions from these models are used in a hypothetical study of variable stop re...
-
[5]
Discussion The goal of this study was to determine whether self-supervised speech models, such as wav2vec2.0, can be utilised as a tool for automatically annotating the variable realisation of stops, to allow for both the study of variable lenition in its own right [10, 31], and further downstream annotations. Specifically, we are interested in whether su...
-
[6]
Computational resources were provided by the Digital Research Alliance of Canada
Acknowledgements The authors thank the SPADE Data Guardians, Rachel Macdon- ald, Michael McAuliffe, and Vanna Willerton. Computational resources were provided by the Digital Research Alliance of Canada. This research was supported by a T-AP Digging into Data award in the form of the following grants: ESRC Grant #ES/R003963/1, NSERC/CRSNG Grants #RGPDD-501...
work page 2023
-
[7]
Montreal forced aligner [computer program],
M. McAuliffe, M. Scolof, S. Mihuc, M. Wagner, and M. Son- deregger, “Montreal forced aligner [computer program],” 2017, https: //montrealcorpustools.github.io/Montreal-Forced-Aligner/
work page 2017
-
[8]
FA VE (Forced Alignment and Vowel Extraction) program suite v1.2.2 10.5281/zenodo.22281,
I. Rosenfelder, J. Fruehwald, K. Evanini, S. Seyfarth, K. Gorman, H. Prichard, and J. Yuan, “FA VE (Forced Alignment and Vowel Extraction) program suite v1.2.2 10.5281/zenodo.22281,” 2014
Show all 41 references
-
[9]
LaBB-CAT: an annotation store,
R. Fromont and J. Hay, “LaBB-CAT: an annotation store,” inAus- tralasian Lang. Tech. Workshop, vol. 113, 2012, pp. 113–117
2012
-
[10]
From categories to gradience: Auto-coding sociophonetic variation with random forests,
D. Villarreal, L. Clark, J. Hay, and K. Watson, “From categories to gradience: Auto-coding sociophonetic variation with random forests,” LabPhon, vol. 11, pp. 1–31, 2020
2020
-
[11]
Considering performance in the au- tomated and manual coding of sociolinguistic variables: Lessons from variable (ING),
T. Kendall, C. Vaughn, C. Farrington, K. Gunter, J. McLean, C. Tacata, and S. Arnson, “Considering performance in the au- tomated and manual coding of sociolinguistic variables: Lessons from variable (ING),” Front. A.I, vol. 4, 2021
2021
-
[12]
Using wav2vec 2.0 for phonetic classifi- cation tasks: methodological aspects,
L. Kim and C. Gendrot, “Using wav2vec 2.0 for phonetic classifi- cation tasks: methodological aspects,” inProc. Interspeech, 2024, pp. 1530–1534
2024
-
[13]
Corpus phonetics,
M. Liberman, “Corpus phonetics,” Annual Review of Linguistics, vol. 5, pp. 91–107, 2018
2018
-
[14]
Segmental and prosodic effects on intervocalic voiced stop reduction in connected speech,
D. Bouavichith and L. Davidson, “Segmental and prosodic effects on intervocalic voiced stop reduction in connected speech,” Pho- netica, vol. 70, pp. 182–206, 2013
2013
-
[15]
Extreme stop allophony in Mixtec spontaneous speech: data, word prosody, and modelling,
C. DiCanio, W.-R. Chen, J. Benn, J. D. Amith, and R. C. Garc ´ıa, “Extreme stop allophony in Mixtec spontaneous speech: data, word prosody, and modelling,”J. Phon, vol. 92, 2022
2022
-
[16]
Lavoie, Consonant Strength: Phonological Patterns and Pho- netic Manifestations
L. Lavoie, Consonant Strength: Phonological Patterns and Pho- netic Manifestations. New York: Garland, 2001
2001
-
[17]
Phonetic variability of stops and flaps in spontaneous and careful speech,
N. Warner and B. V . Tucker, “Phonetic variability of stops and flaps in spontaneous and careful speech,” JASA, vol. 130, pp. 1606–1617, 2011
2011
-
[18]
AutoVOT: A tool for automatic measurement of voice onset time using discriminative structured prediction,
J. Keshet, M. Sonderegger, and T. Knowles, “AutoVOT: A tool for automatic measurement of voice onset time using discriminative structured prediction,” 2014, version 0.94. [Online]. Available: https://github.com/mlml/autovot/
2014
-
[19]
Dr. VOT: Measuring posi- tive and negative voice onset time in the wild,
Y . Shrem, M. Goldrick, and J. Keshet, “Dr. VOT: Measuring posi- tive and negative voice onset time in the wild,”Proc. Interspeech, pp. 629–633, 2019
2019
-
[20]
Robust speech recognition via large- scale weak supervision,
A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large- scale weak supervision,” 2022. [Online]. Available: https: //arxiv.org/abs/2212.04356
2022 arXiv
-
[21]
WavLM: Large-scale self-supervised pre-training for full stack speech processing,
S. Chen, C. Wang, Z. Chen, Y . Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiao, J. Wu, L. Zhou, S. Ren, Y . Qian, Y . Qian, J. Wu, M. Zeng, X. Yu, and F. Wei, “WavLM: Large-scale self-supervised pre-training for full stack speech processing,” IEEE J. Sel. Top. Sig. ...
2022
-
[22]
wav2vec: Unsupervised pre-training for speech recognition,
S. Schneider, A. Baevski, R. Collobert, and M. Auli, “wav2vec: Unsupervised pre-training for speech recognition,” CoRR, vol. abs/1904.05862, 2019. [Online]. Available: http: //arxiv.org/abs/1904.05862
1904 arXiv
-
[23]
wav2vec 2.0: A framework for self-supervised learning of speech representations,
A. Baevski, H. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” CoRR, vol. abs/2006.11477, 2020. [Online]. Available: https://arxiv.org/abs/2006.11477
2006 arXiv
-
[24]
Comparison of wav2vec 2.0 models on three speech processing tasks,
M. Kune ˇsov´a, Z. Zaj´ıc, L. ˇSm´ıdl, and M. Karafi ´at, “Comparison of wav2vec 2.0 models on three speech processing tasks,” Int. J. Speech Tech, vol. 27, pp. 847–859, 2024
2024
-
[25]
Wav2vec behind the scenes: How end2end models learn phonet- ics,
T. tom Dieck, P. A. P´erez-Toro, T. Arias, E. Noeth, and P. Klumpp, “Wav2vec behind the scenes: How end2end models learn phonet- ics,” in Interspeech, 2022, pp. 5130–5134
2022
-
[26]
Opening the black box of wav2vec feature encoder,
K. Choi and E. J. Yeo, “Opening the black box of wav2vec feature encoder,” 2022. [Online]. Available: https://arxiv.org/abs/ 2210.15386
2022 arXiv
-
[27]
Human-like linguistic biases in neural speech models: Phonetic categorization and phonotactic constraints in wav2vec2.0,
M. de Heer Kloots and W. Zuidema, “Human-like linguistic biases in neural speech models: Phonetic categorization and phonotactic constraints in wav2vec2.0,” in Interspeech, 2024, pp. 4593–4597
2024
-
[28]
Spontaneous speech corpus of Japanese,
K. Maekawa, H. Koiso, S. Furui, and H. Isahara, “Spontaneous speech corpus of Japanese,” in Proc. 2nd LREC, vol. 2, 2000, pp. 946–952
2000
-
[29]
Performance of segmental and prosodic labeling of spontaneous speech,
H. Kikuchi and K. Maekawa, “Performance of segmental and prosodic labeling of spontaneous speech,” inProceedings of ISCA & IEEE Workshop on Spontaneous Speech Processing and Recog- nition. Tokyo: Tokyo Institute of Technology, 2003
2003
-
[30]
Weakening of stop articulation in Japanese voiced plosives,
K. Maekawa, “Weakening of stop articulation in Japanese voiced plosives,” J. Phon. Soc. Japan, vol. 22, pp. 21–34, 2018
2018
-
[31]
Design and development of an RDB version of the Corpus of Spontaneous Japanese,
H. Koiso, Y . Den, K. Nishikawa, and K. Maekawa, “Design and development of an RDB version of the Corpus of Spontaneous Japanese,” in Proc. 9th Int. Conf. Lang. Res. Eval. , 2014, pp. 1471–1476
2014
-
[32]
Huggingface’s Transformers: State-of-the- art natural language processing,
T. Wolf, L. Debut, V . Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y . Jernite, J. Plu, C. Xu, T. L. Scao, S. Gugger, M. Drame, Q. Lhoest, and A. M. Rush, “Huggingface’s Transformers: State-...
2020 arXiv
-
[33]
Angelo Canty and B. D. Ripley, boot: Bootstrap R (S-Plus) Func- tions, 2024, r package version 1.3-31
2024
-
[34]
Managing data for integrated speech corpus anal- ysis in SPeech Across Dialects of English (SPADE),
M. Sonderegger, J. Stuart-Smith, M. McAuliffe, R. Macdonald, and T. Kendall, “Managing data for integrated speech corpus anal- ysis in SPeech Across Dialects of English (SPADE),” in Open Handbook of Linguistic Data Management . Cambridge: MIT Press, 2022
2022
-
[35]
Praat: doing phonetics by computer [computer program] (version 6.3.09),
P. Boersma and D. Weenink, “Praat: doing phonetics by computer [computer program] (version 6.3.09),” 2023. [Online]. Available: http://www.praat.org/
2023
-
[36]
Corpus of Spontaneous Japanese documentation: phone information,
M. Fujimoto, H. Kikuchi, and K. Maekawa, “Corpus of Spontaneous Japanese documentation: phone information,” National Institute for Japanese Language and Linguistics, Tokyo, Tech. Rep. 6, 2006. [Online]. Available: https: //pj.ninjal.ac.jp/corpus center/csj/k-report-f/06.pdf
2006
-
[37]
Lenition, perception and neutralisation,
J. Katz, “Lenition, perception and neutralisation,” Phonology, vol. 33, pp. 43–85, 2016
2016
-
[38]
The causal structure of lenition: a case for the causal precedence of durational shortening,
U. Cohen Priva and E. Gleason, “The causal structure of lenition: a case for the causal precedence of durational shortening,” Lan- guage, vol. 96, pp. 413–448, 2020
2020
-
[39]
Fast stable restricted maximum likelihood and marginal likelihood estimation of semiparametric generalized lin- ear models,
S. N. Wood, “Fast stable restricted maximum likelihood and marginal likelihood estimation of semiparametric generalized lin- ear models,”J. Royal. Stat. Soc. (B), vol. 73, no. 1, pp. 3–36, 2011
2011
-
[40]
[Online]
R Core Team, R: A Language and Environment for Statistical Computing, R Foundation for Statistical Computing, Vienna, Austria, 2023. [Online]. Available: https://www.R-project.org/
2023
-
[41]
R. V . Lenth, emmeans: Estimated Marginal Means, aka Least-Squares Means, 2023, R package version 1.9.0. [Online]. Available: https://CRAN.R-project.org/package=emmeans
2023
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.