{"id":"f8d24130-75e8-4053-9901-f5bb258932ea","arxiv_id":"1908.05378","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Artificial word insertions and deletions in news text, used as self-supervised pretraining tasks, transfer to human-annotated disfluency detection and cut the labeled-data requirement to 1,000 sentences.","lead":"By adding and deleting words in plain news sentences, the authors train a transformer to spot artificial errors, then fine-tune it on real speech disfluencies. On the Switchboard benchmark this pretraining matches previous systems using 1,000 labeled sentences and improves state-of-the-art F1 from 87.5 to 90.2 with the full training set.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline SOTA/21% claim rests on a comparison that omits recent cited baselines and has no significance estimates; a matched rerun could change or eliminate the claimed advantage.","rationale":"I considered the reader's weakest assumption (pseudo-disfluency distribution mismatch) but do not think it is the most load-bearing: the paper's Table 2 and Figure 4 already demonstrate positive transfer to gold data, so the claim does not require the synthetic distribution to match real disfluencies perfectly. The more direct vulnerability is the SOTA comparison. The abstract's numeric claim is defined relative to previous work, yet Table 3 stops at a 2017 system and omits two later systems that the paper itself cites. Without those numbers, the 21% error reduction could be an artifact of baseline selection. The missing significance testing compounds this: no variance is reported for the main results (Table 5 only gives 3-restart averages for model-size ablation, and BERT tuning selects best-of-3 on dev), so the word 'significantly' is not demonstrated. This does not make the paper valueless—the ablations, the direct-merging control in Figure 4(c), and the BERT combination in Table 7 are real evidence that the self-supervised tasks help—but the headline claim needs the proposed check before it can be accepted. Because the reader already conditioned acceptance on fixing comparison/significance issues, my concern does not move the verdict.","tokens_in":11509,"tokens_out":9230,"duration_ms":98053,"concrete_test":"Add the omitted recent systems to Table 3 using the exact Switchboard split and preprocessing (lower-casing, punctuation/partial-word removal, 'um'/'uh' handling, 'you know'/'i mean' merging). Run all systems, including the proposed one, with at least 10 random seeds and report mean±std and paired bootstrap confidence intervals on token-level F1. If the best included prior system reaches F1 ≥ 90.2, or if the 90.2 vs 87.5 gap is within the confidence interval, the headline '21% error reduction' and 'significantly outperforms' claims should be revised or removed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim ('significantly outperforms previous methods, reducing the error by 21%') is computed in Table 3 against a baseline list ending with Wang et al. (2017), transition-based F1=87.5. The paper's own Related Work cites later systems—Jamshid Lou et al. (2018) auto-correlational networks and Zayats & Ostendorf (2019) prosody-aware models—that are absent from the comparison. If either of these omitted systems has F1 near or above 90.2 on the same split, the 21% relative error reduction (from 12.5% to 9.8% error, measured as 100−F1) shrinks or disappears, and the abstract's 'outperforms previous methods' is unsupported. The paper also reports no error bars or significance tests anywhere, so the word 'significantly' in the abstract is not backed by a statistical test. This is the load-bearing issue because it attacks the headline empirical claim directly, whereas the acknowledged train/test distribution mismatch is an assumption whose effect is already partially evidenced by the positive transfer results.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a self-supervised pre-training method for disfluency detection. Unlabeled English news text is corrupted by repeating or inserting m-grams at one to three positions, yielding pseudo-disfluent sentences. A Transformer encoder is pre-trained jointly on (i) token-level tagging of added words and (ii) sentence-pair classification that distinguishes original sentences from corrupted ones. The pre-trained tagging model is then fine-tuned on the Switchboard disfluency detection corpus. The central empirical claims are that the full-data model reaches 90.2 F1, a 21% relative error reduction over previous systems, and that with 1000 sentences it reaches 86.1 F1, competitive with full-supervision baselines. Ablations show that both self-supervised tasks contribute, that larger models help, and that direct data mixing hurts, supporting the pre-training mechanism.","tokens_in":11734,"tokens_out":5773,"duration_ms":60324,"significance":"The data-efficiency result, if valid, is practically significant because disfluency annotation is expensive. The experimental design is thoughtful: the random-initialization Transformer baseline and the direct-mixing comparison in Figure 4(c) control for architecture and data-volume confounds, and the pseudo-label generation is algorithmic rather than circular. The paper's main weaknesses are empirical: the headline comparison omits recent cited baselines and the central claims have no variance estimates. The method itself is simple and reproducible in principle, which strengthens the contribution.","major_comments":[{"comment":"The headline claim \"significantly outperforms previous methods, reducing the error by 21%\" is computed against a baseline list ending with Wang et al. (2017). The Related Work section explicitly cites later systems, Jamshid Lou et al. (2018) and Zayats and Ostendorf (2019), which are absent from Table 3. Because the abstract's claim is relative to \"previous methods,\" the omission is load-bearing: if either omitted system reaches an F1 near or above 90.2 on the same split, the 21% relative error reduction shrinks or disappears. Please add these systems (or equivalent recent published numbers on the same split) to Table 3, or state clearly why they cannot be compared.","section":"Experiment, Table 3"},{"comment":"No significance estimates are reported for Tables 2, 3, 4, or 6. Table 5 reports averages over three random restarts on the development set only, and for the main results there is no indication of run-to-run variance. The word \"significantly\" in the abstract and conclusion is therefore unsupported as a statistical claim. Please report mean and standard deviation over at least three restarts for the main comparisons, and, if the word is intended colloquially, replace it with a non-statistical qualifier or provide an actual significance test.","section":"Experiment, Training Details and Table 5"}],"minor_comments":[{"comment":"There is a typo in the baseline description: \"human-annotated disfleuncy detection data\" should read \"human-annotated disfluency detection data.\"","section":"Experiment, Baselines"},{"comment":"The label set {add0, add1, del0, del1} is only partially defined; the text defines add0 and del0 but does not explicitly say that add1 and del1 correspond to the second input sentence being the corrupted one. Please define all four labels.","section":"Proposed Approach, Sentence Classification Task"},{"comment":"The paper states that \"um\" and \"uh\" tokens are discarded and that \"you know\" and \"i mean\" are merged, following Honnibal and Johnson (2014). It should be stated explicitly whether the same preprocessing was applied to every baseline in Table 3, since differences in tokenization can affect F1 and hence the 21% claim.","section":"Experiment, Settings"},{"comment":"Please state whether the unlabeled WMT news data was filtered to remove any sentences overlapping with the Switchboard train/dev/test sets. Although the domains differ, a leakage check would make the data-efficiency claim cleaner.","section":"Experiment, Settings"},{"comment":"No code or model release is provided for the proposed method; the link to the transition-based baseline code is not sufficient for reproducing the main result. Consider releasing the pre-training and fine-tuning code.","section":"Experiment, Performance on English Switchboard"}],"recommendation":"major_revision","confidential_remarks":"The main risk to this paper is the incomplete SOTA comparison in Table 3, which directly affects the abstract's central claim. The ablation evidence suggests the method itself is sound, so I am not recommending rejection, but the authors must update the comparison and provide variance estimates before the claims can be accepted as stated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"To be direct: this is a good empirical paper with a real idea, and the headline claim is stronger than the evidence. What's new is the combination of two self-supervised pre-training tasks for disfluency detection—a token-level tagging task that marks randomly inserted/repeated words in pseudo-disfluent news text, and a sentence-pair classification task that decides which of two sentences is fluent. The authors credit earlier work for the pseudo-data generation idea, and the novelty is in the multi-task pre-training itself, which I think is fair. The ablations are the best part. Tagging alone lifts F1 from 78.1 to 87.7 over a random-init transformer on full Switchboard data, classification alone gives 87.0, and the multi-task version reaches 90.2. The control experiment in Figure 4c, showing that directly mixing pseudo-data with gold data hurts while pre-training helps, is the right comparison and it makes the paper's central mechanism credible.\n\nThe soft spots are real but not fatal. The SOTA comparison in Table 3 stops at Wang et al. (2017), even though the paper itself cites Jamshid Lou et al. (2018) and Zayats and Ostendorf (2019) in Related Work. Omitting them makes the '21% error reduction' easy to over-read. Worse, the paper's own BERT-large baseline reaches 90.1 F1 in Table 7, so the method is essentially tied with BERT, not clearly ahead of all prior work. 'Significantly outperforms' in the abstract is not supported by any significance test or error bar. The 1000-sentence subsample is not described, and it isn't even clear how many random restarts, if any, back the main numbers. There is also a small inconsistency: the transition-based baseline is 87.9 in Table 2 but 87.5 in Table 3. No code has been released.\n\nNone of this undermines the core transfer result. The pseudo-data perturbation distribution is admittedly different from real conversational disfluencies, but the paper acknowledges it and the positive fine-tuning results suggest the gap is manageable. I would want to see the missing baselines added, the main results with variance, and the abstract's 'significantly' either justified or softened. If I were handling this as an editor, I would send it to review. The idea is worth a referee's time, and the issues are correctable.\n\nIn short: send it out, but expect revisions on the comparison and the significance claims.","headline":"A well-ablated pre-training recipe for disfluency detection whose core transfer result is credible, but the headline SOTA/21% claim is overstated because the comparison table omits two strong recent baselines and no significance testing is reported.","tokens_in":12282,"tokens_out":3388,"would_cite":true,"duration_ms":32373,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Pre-training a Transformer on pseudo-disfluent news text—random repetitions and insertions—lets disfluency detection match earlier systems with under 1% of the labeled data and reduce error by 21% on Switchboard.","keywords":["disfluency detection","self-supervised learning","multi-task pre-training","pseudo training data","transformer encoder","data efficiency","Switchboard corpus"],"falsifier":"Train the same model on the same pseudo-disfluent news data, then fine-tune on 1,000 labeled sentences from a held-out conversational speech corpus that was never seen in pre-training or development, and compare against a randomly initialized Transformer of the same size fine-tuned on those same 1,000 sentences with the same hyperparameter budget; if the pre-trained model does not beat the random-initialization baseline by a large margin, the claimed transfer from synthetic corruptions to real disfluencies is not general.","tokens_in":11317,"feed_emoji":"🗣️","tokens_out":8142,"duration_ms":74370,"temperature":0.7,"pith_summary":"The paper's thesis is that a Transformer can learn what disfluencies look like without any human labels, by practicing on fake disfluencies made from clean text. The authors generate pseudo-disfluent sentences by repeating or inserting short word sequences of length 1 to 6 at one to three positions in unlabeled news sentences, plus a deletion variant for a companion task. They then pre-train on two objectives jointly: tagging the injected noise words, and classifying which sentence in a pair is fluent. After fine-tuning on gold disfluency data from English Switchboard, they report that 1,000 labeled sentences are enough to match earlier systems trained on the full corpus, and that full-data fine-tuning lowers error by 21% over previous methods. If this holds, expensive annotation is no longer the main barrier to building disfluency detectors.","feed_headline":"Self-supervised pre-training matches disfluency models with 1% of data","feed_subtitle":"Training on corrupted news text cuts Switchboard error by 21 percent.","key_machinery":"The machinery is joint multi-task pre-training over pseudo-disfluent sentences. Pseudo-disfluencies are produced by Repetition(k) and Inserting(k) perturbations—repeating or inserting an m-gram of length 1 to 6 at a random position—applied at one to three positions per sentence; a Delete(k) perturbation is used for the classification task. The tagging head labels each token as original or added on the corrupted sentence, while the classification head labels a sentence pair as add0, add1, del0, or del1 depending on which sentence was corrupted and how. A shared Transformer encoder and embedding layer are trained on the sum of both losses, and after pre-training the tagging head and encoder are fine-tuned directly on gold disfluency data without the classification head. The sentence-level classification task is designed to inject grammaticality information that helps the token-level tagging task.","core_discovery":"The central claim is that the distribution gap between synthetic corruptions and real speech repairs is bridgeable: a model pre-trained to undo random insertions in news text transfers to detecting genuine reparandum regions in conversational transcripts. The paper reports 90.2% F1 on the English Switchboard test set with full supervised fine-tuning, compared with 87.5% for the previous transition-based system, a 21% relative error reduction; with only 1,000 labeled sentences it reports 86.1% F1, above the previous system trained on the full dataset. The authors also show that both self-supervised tasks contribute, that the classification task adds sentence-level grammatical information to the tagging task, and that performance grows with pseudo-data size.","pith_inferences":["The same auto-corruption recipe should be portable to other text-repair and fluency tasks—grammatical error correction, spell normalization, punctuation restoration—where clean text is abundant and corruptions can be generated mechanically.","Because the perturbations are contiguous repeats and insertions, the method likely underrepresents disfluencies that involve long-range restructuring or replacement of whole phrases; a testable extension would add substitution or shuffle perturbations and check whether non-repetition F1 rises further.","The small-data result suggests a practical data-collection strategy shift: annotate a few hundred utterances for a target transcript style, then let unlabeled monolingual text carry the structural prior; whether that transfers across speaking styles and noise conditions is a question the paper does not settle."],"forward_implications":["A disfluency detector for a new domain or language could be built from a few hundred labeled sentences plus unlabeled text, bypassing the traditional annotation bottleneck.","Scaling up the unlabeled pseudo-data improves downstream F1, and the benefit is largest exactly when labeled data is scarce.","Simply mixing pseudo-disfluent sentences into supervised training hurts baseline systems; the two-stage pre-train-then-fine-tune ordering is what makes the pseudo-data useful.","Combining this pre-trained representation with BERT yields better results than either alone, indicating the two capture complementary information for disfluency detection."],"supporting_citations":[{"why":"Supplies the English Switchboard corpus that all experiments are trained and evaluated on.","marker":"Godfrey, Holliman, and McDaniel (1992)"},{"why":"Defines the standard Switchboard train/dev/test split used throughout the evaluation.","marker":"Charniak and Johnson (2001)"},{"why":"Provides the Transformer encoder architecture that the two self-supervised tasks share and that is fine-tuned for detection.","marker":"Vaswani et al. (2017)"},{"why":"The transition-based disfluency detection system used as the strongest baseline and the comparison point for the reported improvements.","marker":"Wang et al. (2017)"},{"why":"Earlier work on pseudo-disfluent sentence generation that motivates the corruption scheme used for the tagging task.","marker":"Dong et al. (2019)"},{"why":"Semi-supervised disfluency detection work cited as the source of inspiration for generating pseudo-disfluent sentences with simple rules.","marker":"Wang et al. (2018)"},{"why":"Provides the BERT model used for comparison and for the combination experiment that shows complementary representations.","marker":"Devlin et al. (2019)"}],"fun_headline_variants":["Disfluency detection matches full training with just 1% data","21% error reduction on Switchboard from self-supervised pre-training","Two self-supervised tasks need 1% data to match disfluency baselines","Corrupted news text pre-trains disfluency detectors, 1% data enough","Self-supervised pre-training: 1% labeled data suffices for disfluency"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that random repetitions and insertions in written news text are structurally similar enough to real speech repairs that what the model learns on them transfers to human-annotated conversational disfluencies; if that similarity fails, the reported gains would not reproduce outside the Switchboard setup.","fun_headline_variants_meta":{"raw":{"variants":["Disfluency detection matches full training with just 1% data","21% error reduction on Switchboard from self-supervised pre-training","Two self-supervised tasks need 1% data to match disfluency baselines","Corrupted news text pre-trains disfluency detectors, 1% data enough","Self-supervised pre-training: 1% labeled data suffices for disfluency"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000593,"raw_usage":{"total_tokens":2745,"prompt_tokens":879,"completion_tokens":1866,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":495,"completion_tokens_details":{"reasoning_tokens":1763}},"tokens_in":495,"tokens_out":1866,"duration_ms":15331,"temperature":1.0,"reasoning_tokens":1763,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:15:44.236789+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same model on the same pseudo-disfluent news data, then fine-tune on 1,000 labeled sentences from a held-out conversational speech corpus that was never seen in pre-training or development, and compare against a randomly initialized Transformer of the same size fine-tuned on those same 1,000 sentences with the same hyperparameter budget; if the pre-trained model does not beat the random-initialization baseline by a large margin, the claimed transfer from synthetic corruptions to real disfluencies is not general.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the standard Switchboard train/dev/test split used throughout the evaluation."},{"cited_title":"N.; Kaiser, Ł.; and Polosukhin, I","cited_arxiv_id":null,"evidence_quote":"Provides the Transformer encoder architecture that the two self-supervised tasks share and that is fine-tuned for detection."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Earlier work on pseudo-disfluent sentence generation that motivates the corruption scheme used for the tagging task."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Semi-supervised disfluency detection work cited as the source of inspiration for generating pseudo-disfluent sentences with simple rules."}],"review_version":1}