Pith. sign in

REVIEW 2 major objections 5 minor 42 references

Multi-Task Self-Supervised Learning for Disfluency Detection

T0 review · 2 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Pre-training a Transformer on pseudo-disfluent news text—random repetitions and insertions—lets disfluency detection match earlier systems with under 1% of the labeled data and reduce error by 21% on Switchboard.

desk verdict A well-ablated pre-training recipe for disfluency detection whose core transfer result is credible, but the headline SOTA/21% claim is overstated because the comparison table omits two strong recent baselines and no significance testing is reported. read the letter →

arxiv 1908.05378 v2 pith:XKC4HHE4 submitted 2019-08-15 cs.CL

Shaolei Wang , Wanxiang Che , Qi Liu , Pengda Qin , Ting Liu , William Yang Wang This is my paper · ORCID
classification cs.CL
keywords disfluencydetectionself-supervisedlearningmulti-taskpre-trainingpseudotrainingdatatransformerencoderefficiencySwitchboardcorpus
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's thesis is that a Transformer can learn what disfluencies look like without any human labels, by practicing on fake disfluencies made from clean text. The authors generate pseudo-disfluent sentences by repeating or inserting short word sequences of length 1 to 6 at one to three positions in unlabeled news sentences, plus a deletion variant for a companion task. They then pre-train on two objectives jointly: tagging the injected noise words, and classifying which sentence in a pair is fluent. After fine-tuning on gold disfluency data from English Switchboard, they report that 1,000 labeled sentences are enough to match earlier systems trained on the full corpus, and that full-data fine-tuning lowers error by 21% over previous methods. If this holds, expensive annotation is no longer the main barrier to building disfluency detectors.

What carries the argument

The machinery is joint multi-task pre-training over pseudo-disfluent sentences. Pseudo-disfluencies are produced by Repetition(k) and Inserting(k) perturbations—repeating or inserting an m-gram of length 1 to 6 at a random position—applied at one to three positions per sentence; a Delete(k) perturbation is used for the classification task. The tagging head labels each token as original or added on the corrupted sentence, while the classification head labels a sentence pair as add0, add1, del0, or del1 depending on which sentence was corrupted and how. A shared Transformer encoder and embedding layer are trained on the sum of both losses, and after pre-training the tagging head and encoder are fine-tuned directly on gold disfluency data without the classification head. The sentence-level classification task is designed to inject grammaticality information that helps the token-level tagging task.

What would settle it

Train the same model on the same pseudo-disfluent news data, then fine-tune on 1,000 labeled sentences from a held-out conversational speech corpus that was never seen in pre-training or development, and compare against a randomly initialized Transformer of the same size fine-tuned on those same 1,000 sentences with the same hyperparameter budget; if the pre-trained model does not beat the random-initialization baseline by a large margin, the claimed transfer from synthetic corruptions to real disfluencies is not general.

Watch

Extended reading notes

Core claim

The central claim is that the distribution gap between synthetic corruptions and real speech repairs is bridgeable: a model pre-trained to undo random insertions in news text transfers to detecting genuine reparandum regions in conversational transcripts. The paper reports 90.2% F1 on the English Switchboard test set with full supervised fine-tuning, compared with 87.5% for the previous transition-based system, a 21% relative error reduction; with only 1,000 labeled sentences it reports 86.1% F1, above the previous system trained on the full dataset. The authors also show that both self-supervised tasks contribute, that the classification task adds sentence-level grammatical information to the tagging task, and that performance grows with pseudo-data size.

Load-bearing premise

The load-bearing premise is that random repetitions and insertions in written news text are structurally similar enough to real speech repairs that what the model learns on them transfers to human-annotated conversational disfluencies; if that similarity fails, the reported gains would not reproduce outside the Switchboard setup.

Editorial extensions

If this is right

  • A disfluency detector for a new domain or language could be built from a few hundred labeled sentences plus unlabeled text, bypassing the traditional annotation bottleneck.
  • Scaling up the unlabeled pseudo-data improves downstream F1, and the benefit is largest exactly when labeled data is scarce.
  • Simply mixing pseudo-disfluent sentences into supervised training hurts baseline systems; the two-stage pre-train-then-fine-tune ordering is what makes the pseudo-data useful.
  • Combining this pre-trained representation with BERT yields better results than either alone, indicating the two capture complementary information for disfluency detection.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same auto-corruption recipe should be portable to other text-repair and fluency tasks—grammatical error correction, spell normalization, punctuation restoration—where clean text is abundant and corruptions can be generated mechanically.
  • Because the perturbations are contiguous repeats and insertions, the method likely underrepresents disfluencies that involve long-range restructuring or replacement of whole phrases; a testable extension would add substitution or shuffle perturbations and check whether non-repetition F1 rises further.
  • The small-data result suggests a practical data-collection strategy shift: annotate a few hundred utterances for a target transcript style, then let unlabeled monolingual text carry the structural prior; whether that transfers across speaking styles and noise conditions is a question the paper does not settle.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes a self-supervised pre-training method for disfluency detection. Unlabeled English news text is corrupted by repeating or inserting m-grams at one to three positions, yielding pseudo-disfluent sentences. A Transformer encoder is pre-trained jointly on (i) token-level tagging of added words and (ii) sentence-pair classification that distinguishes original sentences from corrupted ones. The pre-trained tagging model is then fine-tuned on the Switchboard disfluency detection corpus. The central empirical claims are that the full-data model reaches 90.2 F1, a 21% relative error reduction over previous systems, and that with 1000 sentences it reaches 86.1 F1, competitive with full-supervision baselines. Ablations show that both self-supervised tasks contribute, that larger models help, and that direct data mixing hurts, supporting the pre-training mechanism.

Significance. The data-efficiency result, if valid, is practically significant because disfluency annotation is expensive. The experimental design is thoughtful: the random-initialization Transformer baseline and the direct-mixing comparison in Figure 4(c) control for architecture and data-volume confounds, and the pseudo-label generation is algorithmic rather than circular. The paper's main weaknesses are empirical: the headline comparison omits recent cited baselines and the central claims have no variance estimates. The method itself is simple and reproducible in principle, which strengthens the contribution.

major comments (2)
  1. [Experiment, Table 3] The headline claim "significantly outperforms previous methods, reducing the error by 21%" is computed against a baseline list ending with Wang et al. (2017). The Related Work section explicitly cites later systems, Jamshid Lou et al. (2018) and Zayats and Ostendorf (2019), which are absent from Table 3. Because the abstract's claim is relative to "previous methods," the omission is load-bearing: if either omitted system reaches an F1 near or above 90.2 on the same split, the 21% relative error reduction shrinks or disappears. Please add these systems (or equivalent recent published numbers on the same split) to Table 3, or state clearly why they cannot be compared.
  2. [Experiment, Training Details and Table 5] No significance estimates are reported for Tables 2, 3, 4, or 6. Table 5 reports averages over three random restarts on the development set only, and for the main results there is no indication of run-to-run variance. The word "significantly" in the abstract and conclusion is therefore unsupported as a statistical claim. Please report mean and standard deviation over at least three restarts for the main comparisons, and, if the word is intended colloquially, replace it with a non-statistical qualifier or provide an actual significance test.
minor comments (5)
  1. [Experiment, Baselines] There is a typo in the baseline description: "human-annotated disfleuncy detection data" should read "human-annotated disfluency detection data."
  2. [Proposed Approach, Sentence Classification Task] The label set {add0, add1, del0, del1} is only partially defined; the text defines add0 and del0 but does not explicitly say that add1 and del1 correspond to the second input sentence being the corrupted one. Please define all four labels.
  3. [Experiment, Settings] The paper states that "um" and "uh" tokens are discarded and that "you know" and "i mean" are merged, following Honnibal and Johnson (2014). It should be stated explicitly whether the same preprocessing was applied to every baseline in Table 3, since differences in tokenization can affect F1 and hence the 21% claim.
  4. [Experiment, Settings] Please state whether the unlabeled WMT news data was filtered to remove any sentences overlapping with the Switchboard train/dev/test sets. Although the domains differ, a leakage check would make the data-efficiency claim cleaner.
  5. [Experiment, Performance on English Switchboard] No code or model release is provided for the proposed method; the link to the transition-based baseline code is not sufficient for reproducing the main result. Consider releasing the pre-training and fine-tuning code.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: pseudo-labels are generated from WMT news data, and final evaluation is against externally annotated Switchboard gold labels.

full rationale

The derivation chain is: (1) generate pseudo-disfluent sentences from fluent WMT news text by random repetition, insertion, or deletion; (2) pre-train a transformer on automatically labeled tagging and sentence-classification examples derived from those pseudo-disfluent sentences; (3) fine-tune on gold Switchboard disfluency annotations; (4) evaluate on the standard held-out Switchboard test set. The pseudo-labels are a deterministic function of the news corpus, not of the target Switchboard labels, and no parameter is fitted to the test set. The tagging and classification objectives are intentionally similar to the target task, but the paper explicitly acknowledges the distribution mismatch, stating that 'the training goal is to keep the generated sentences fluent by deleting disfluent words, which matches the goal of disfluency detection.' That disclosure makes the task similarity a transparent design choice rather than a hidden equivalence. The transition-based baseline from Wang et al. (2017) is prior work by the first author, but it is used only as a reproduced benchmark, not as justification for the proposed method's validity. The omission of more recent cited systems and the absence of significance tests are concerns about empirical support and comparison completeness, not circularity. No load-bearing step reduces, by construction or by self-citation, to its own inputs, so the appropriate finding is no significant circularity.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

No new theoretical entities are introduced; the contribution is an empirical training procedure. Free parameters are conventional hyperparameters and data-generation choices, and the axioms are domain assumptions about the dataset, the pseudo-data distribution, and the architecture.

free parameters (5)
  • Pseudo-data perturbation range (m-gram length and number of positions) = m = 1 to 6, positions = 1 to 3
    Chosen by hand; directly controls the distribution of synthetic disfluencies and therefore what the tagging task learns.
  • Pre-training data mix ratio = 30% tagging, 70% classification
    Hand-set mini-batch composition; affects how much sentence-level versus token-level signal the shared encoder receives.
  • Model architecture size = 6 layers, 512 hidden, 8 heads
    Selected via dev-set ablation (Table 5); the final size is a free choice, not derived from theory.
  • Pre-training hyperparameters = lr=1e-4, batch=256, epochs=30, stream=128
    Hand-tuned on development data; not motivated by analysis.
  • Fine-tuning hyperparameters = lr=1e-5, batch=32, epochs=20
    Chosen on dev; the 20-epoch cap and small batch could influence the low-data result.
assumptions (5)
  • domain assumption Switchboard annotation scheme (reparandum, interregnum, repair) is the correct target for disfluency detection, and the model only labels reparandums.
    Established by Shriberg (1994) and used by all baselines; not re-derived in this paper. Invoked in the Introduction and Experiment sections.
  • domain assumption WMT2017 news sentences are fluent and can serve as clean source sentences for generating pseudo-disfluent data.
    The entire pre-training corpus relies on this; news text may contain headline fragments or non-fluent constructions. Stated in the Proposed Approach section.
  • domain assumption Pseudo-disfluencies generated by repetition, insertion, or deletion of m-grams resemble gold disfluencies closely enough for transfer.
    The paper acknowledges a distribution mismatch but assumes it does not harm training. This is the weakest load-bearing premise; see the Tagging Task section.
  • domain assumption A transformer encoder with GELU, dropout 0.1, and Adam is a capable sequence tagger for this task.
    Borrowed from Vaswani et al. (2017) and BERT-style pre-training; no new theory is developed in the paper.
  • domain assumption Token-level F1 computed after removing 'um' and 'uh' and merging 'you know' and 'i mean' is comparable across all compared systems.
    Preprocessing choices affect task difficulty; the paper assumes all systems are affected equally, but no cross-check is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Multi-Task Self-Supervised Learning for Disfluency Detection." pith.science (2026). https://pith.science/paper/XKC4HHE4

@misc{pith2026190805378,
  author       = {Pith},
  title        = {Pith review of: Multi-Task Self-Supervised Learning for Disfluency Detection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XKC4HHE4}},
  note         = {Machine review of arXiv:1908.05378}
}
read the original abstract

Most existing approaches to disfluency detection heavily rely on human-annotated data, which is expensive to obtain in practice. To tackle the training data bottleneck, we investigate methods for combining multiple self-supervised tasks-i.e., supervised tasks where data can be collected without manual labeling. First, we construct large-scale pseudo training data by randomly adding or deleting words from unlabeled news data, and propose two self-supervised pre-training tasks: (i) tagging task to detect the added noisy words. (ii) sentence classification to distinguish original sentences from grammatically-incorrect sentences. We then combine these two tasks to jointly train a network. The pre-trained network is then fine-tuned using human-annotated disfluency detection training data. Experimental results on the commonly used English Switchboard test set show that our approach can achieve competitive performance compared to the previous systems (trained using the full dataset) by using less than 1% (1000 sentences) of the training data. Our method trained on the full dataset significantly outperforms previous methods, reducing the error by 21% on English Switchboard.

Figures

Figures reproduced from arXiv: 1908.05378 by the authors.

Figure 1
Figure 1. A sentence from the English Switchboard corpus with disfluencies annotated. RM=Reparandum, IM=Interregnum, RP=Repair. The preceding RM is cor￾rected by the following RP. Type Annotation repair [ I just + I ] enjoy working repair [ we want + {well} in our area we want ] to repetition [it’s + {uh} it’s ] almost like restart [ we would like + ] let’s go to the [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Illustration of our proposed methods [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Model structure. The parameters of input embed [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: (a) Plot showing the impact of pseudo training data size to disfluency detection. (b) Plot showing the impact of human￾annotated data size when fine-tuning. (c) Plot showing the effectiveness of pre-training compared with the baseline methods of directly merging gold t…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

42 extracted references · 40 canonical work pages

  1. [3]

    Journal of machine learning research 3(Feb):1137–1155

    A neural probabilistic language model. Journal of machine learning research 3(Feb):1137–1155

    work page

  2. [7]

    Proceedings of NAACL

    Bert: Pre-training of deep bidirectional transformers for language understanding. Proceedings of NAACL

    work page

  3. [8]

    Dong, Q.; Wang, F.; Yang, Z.; Chen, W.; Xu, S.; and Xu, B. 2019. Adapting translation models for transcript disfluency detection. In Proceedings of AAAI

    work page 2019

  4. [9]

    Ferguson, J.; Durrett, G.; and Klein, D. 2015. Dis- fluency detection with a semi-markov model and prosodic features. In Proceedings of NAACL, 257–262

    work page 2015

  5. [10]

    Fernando, B.; Bilen, H.; Gavves, E.; and Gould, S

    work page

  6. [11]

    In Proceedings of CVPR , 3636– 3645

    Self-supervised video representation learning with odd-one-out networks. In Proceedings of CVPR , 3636– 3645

    work page

  7. [14]

    In icassp, 517–520

    Switchboard: Telephone speech corpus for research and development. In icassp, 517–520. IEEE

    work page

  8. [17]

    Hough, J., and Schlangen, D. 2015. Recurrent neural networks for incremental disfluency detection. In INTER- SPEECH

    work page 2015

Show all 42 references
  1. [20]

    Liu, T.; Cui, Y .; Yin, Q.; Zhang, W.; Wang, S.; and Hu, G. 2016. Generating and exploiting large-scale pseudo training data for zero pronoun resolution. arXiv preprint arXiv:1606.01603

    2016 arXiv

  2. [21]

    J., and Johnson, M

    Lou, P. J., and Johnson, M. 2017. Disfluency detec- tion using a noisy channel model and a deep neural language model. Proceedings of ACL

    2017

  3. [22]

    Mart ´ınez Alonso, H., and Plank, B. 2017. When is multitask learning effective? semantic sequence prediction under varying data conditions. In Proceedings of EACL. [2013a] Mikolov, T.; Chen, K.; Corrado, G.; and Dean, J. 2013a. Efficient estimation of word representations in v...

    2017

  4. [24]

    Peng, H.; Thomson, S.; and Smith, N. A. 2017. Deep multitask learning for semantic dependency parsing. InACL

    2017

  5. [25]

    Peters, M.; Neumann, M.; Iyyer, M.; Gardner, M.; Clark, C.; Lee, K.; and Zettlemoyer, L. 2018. Deep contex- tualized word representations. In Proceedings of NAACL

    2018

  6. [26]

    Qian, X., and Liu, Y . 2013. Disfluency detection us- ing multi-step stacked learning. In HLT-NAACL, 820–825

    2013

  7. [27]

    Radford, A.; Narasimhan, K.; Salimans, T.; and Sutskever, I. 2018. Improving language understanding with unsupervised learning. Technical report, Technical report, OpenAI

    2018

  8. [28]

    S., and Tetreault, J

    Rasooli, M. S., and Tetreault, J. R. 2013. Joint parsing and disfluency detection in linear time. In EMNLP, 124– 129

    2013

  9. [30]

    N.; Kaiser, Ł.; and Polosukhin, I

    Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I. 2017. Attention is all you need. In Advances in neural information processing systems, 5998–6008

    2017

  10. [31]

    Wang, X., and Gupta, A. 2015. Unsupervised learn- ing of visual representations using videos. In Proceedings of ICCV

    2015

  11. [32]

    Wang, S.; Che, W.; Zhang, Y .; Zhang, M.; and Liu, T

  12. [33]

    In Proceedings of EMNLP, 2785–2794

    Transition-based disfluency detection using lstms. In Proceedings of EMNLP, 2785–2794

  13. [34]

    Wang, F.; Chen, W.; Yang, Z.; Dong, Q.; Xu, S.; and Xu, B. 2018. Semi-supervised disfluency detection. InCOL- ING

    2018

  14. [35]

    Wang, S.; Che, W.; and Liu, T. 2016. A neural at- tention model for disfluency detection. In Proceedings of COLING

    2016

  15. [36]

    Wu, S.; Zhang, D.; Zhou, M.; and Zhao, T. 2015. Efficient disfluency detection with transition-based parsing. In ACL

    2015

  16. [37]

    Yoshikawa, M.; Shindo, H.; and Matsumoto, Y . 2016. Joint transition-based dependency parsing and disfluency detection for automatic speech recognition texts. InEMNLP

    2016

  17. [38]

    Zayats, V ., and Ostendorf, M. 2018. Robust cross- domain disfluency detection with pattern match networks. arXiv preprint arXiv:1811.07236

    2018 arXiv

  18. [39]

    Zayats, V ., and Ostendorf, M. 2019. Giving attention to the unexpected: Using prosody innovations in disfluency detection. arXiv preprint arXiv:1904.04388

    2019 arXiv

  19. [40]

    Zayats, V .; Ostendorf, M.; and Hajishirzi, H. 2014. Multi-domain disfluency and repair detection. In Proceed- ings of Interspeech

    2014

  20. [41]

    Zayats, V .; Ostendorf, M.; and Hajishirzi, H. 2016. Disfluency detection using a bidirectional lstm. arXiv preprint arXiv:1604.03209

    2016 arXiv

  21. [1992]

    J.; Holliman, E

    Godfrey, J. J.; Holliman, E. C.; and McDaniel, J

  22. [1994]

    Shriberg, E. E. 1994. Preliminaries to a theory of speech disfluencies. Ph.D. Dissertation, Citeseer

    1994

  23. [2001]

    Charniak, E., and Johnson, M. 2001. Edit detec- tion and parsing for transcribed speech. In Proceedings of NAACL

    2001

  24. [2003]

    Bengio, Y .; Ducharme, R.; Vincent, P.; and Jauvin, C

  25. [2004]

    Johnson, M., and Charniak, E. 2004. A tag-based noisy channel model of speech repairs. In Proceedings of ACL

    2004

  26. [2009]

    Georgila, K. 2009. Using integer linear programming for detecting speech disfluencies. In Proceedings of NAACL

    2009

  27. [2010]

    Zwarts, S.; Johnson, M.; and Dale, R. 2010. De- tecting speech repairs incrementally using a noisy channel approach. In Proceedings of COLING, 1371–1378

    2010

  28. [2013]

    Ostendorf, M., and Hahn, S. 2013. A sequential rep- etition model for improved disfluency detection. In Inter- speech

    2013

  29. [2014]

    Honnibal, M., and Johnson, M. 2014. Joint incre- mental disfluency detection and dependency parsing. TACL 2

    2014

  30. [2015]

    Agrawal, P.; Carreira, J.; and Malik, J. 2015. Learning to see by moving. In Proceedings of ICCV, 37–45

    2015

  31. [2016]

    Hendrycks, D., and Gimpel, K. 2016. Bridging non- linearities and stochastic regularizers with gaussian error lin- ear units. arXiv preprint arXiv:1606.08415

    2016 arXiv

  32. [2017]

    Bingel, J., and Søgaard, A. 2017. Identifying ben- eficial task relations for multi-task learning in deep neural networks. In Proceedings of EACL, 164–169

    2017

  33. [2018]

    Jamshid Lou, P.; Anderson, P.; and Johnson, M. 2018. Disfluency detection using auto-correlational neural net- works. In Proceedings of EMNLP, 4610–4619

    2018

  34. [2019]

    Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.