Pith. sign in

REVIEW 4 major objections 5 minor 62 references

Large ASR models already encode verbatim and intended transcription styles, and discrete decoder-prefix mode tags can activate either style on command, with English-only training transferring to German zero-shot.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 13:54 UTC pith:3EHNQDVD

load-bearing objection The frozen-embedding mode-tag experiment is a genuinely strong result that shows Whisper's verbatim capability is latent and controllable; the paper overstates two headline claims and one evaluation set is fragile, but it deserves a serious referee. the 4 major comments →

arxiv 2607.18934 v1 pith:3EHNQDVD submitted 2026-07-21 cs.CL

Transcription Policy as a Latent Variable: Activating Controllable Verbatim ASR with Word-Level Timing

Laurin Wagner (1) , Mario Zusag (1) , Bernhard Thallinger (1) ((1) nyra labs) This is my paper
classification cs.CL
keywords verbatim transcriptionintended transcriptiontranscription policydisfluency detectionword-level timingmode tagscross-lingual transferspeech recognition control
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that large speech-recognition models do not need to be taught from scratch how to transcribe disfluent speech; they already contain both a verbatim and a cleaned 'intended' transcription capability, and the real problem is deciding which one to emit. The authors show that prepending small discrete mode tags to the decoder, trained on paired verbatim/intended transcripts of the same audio, turns transcription style into an explicit, controllable choice. Training only the tag embeddings while freezing all other weights lifts German disfluency-event F1 from 10% to 79%, and English-only training transfers to German zero-shot at 94%. They further show that once the output token sequence is stable, supervised cross-attention can yield precise word-level timestamps, beating forced-alignment baselines on disfluent speech, and that a transcript-conditioned 'verbatimize' mode can reconstruct verbatim transcripts from audio plus an intended transcript. If correct, this reframes style and timing as activation problems rather than data or architecture problems, which matters for disfluency detection, clinical and linguistic analysis, and reliable evaluation of conversational ASR.

Core claim

The central claim is that transcription policy—whether the transcript preserves every audible disfluency or only the speaker's intended fluent content—is an uncontrolled latent variable in models trained on heterogeneous data, and that this ambiguity, not missing capability, causes unstable decoding, inflated WER, and ill-defined timestamps. The authors establish this by freezing the entire model and training only 27 new mode-tag embeddings, which raises German disfluency-event F1 from 10% to 79%; full decoder fine-tuning with English-only verbatim data reaches 93.8% German F1 zero-shot, while mode tags improve both verbatim and intended output quality and suppress disfluency leakage into in

What carries the argument

The carrying mechanism is a set of discrete decoder-prefix 'mode tags' ([verbatim 1..3], [sound 1..2], [intended 1..5]) that parameterize the emission policy as a binary choice while leaving encoder representations untouched. Coverage-aware partitioning assigns each training sample only the tags matching its annotation coverage, so missing labels are attributed to data rather than to audio. Paired supervision—the same audio appearing under both verbatim and intended tags—teaches the model to separate what was said from what to emit. For timing, ten cross-attention heads whose raw patterns already correlate with alignments are selected, their averaged distribution is supervised with a cosine

Load-bearing premise

The load-bearing premise is that the paired supervision is faithful—that automatically generated intended transcripts strip disfluencies without altering content and that forced-alignment word boundaries are accurate on the very disfluent speech the method targets; the paper's own Limitations section further notes that the German evaluation set was author-recorded and only a typologically close language was tested, so the zero-shot magnitude is fragile.

What would settle it

Take a set of naturally occurring disfluent utterances with independent human verbatim and intended transcripts; if the frozen-tag F1 gain (10% to 79% German) collapses when the automatically cleaned targets are replaced by human references, or when forced-alignment boundaries are replaced by hand-annotated alignments, then the activation effect is an artifact of the synthetic supervision rather than a genuine mode switch.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Style-matched evaluation becomes possible: on AMI, roughly 60% of Whisper's reported WER comes from style mismatch rather than recognition errors, so verbatim-versus-verbatim and intended-versus-intended comparisons isolate genuine mistakes.
  • Decoding instability on disfluent speech drops: median beam divergence falls from 15.1% to 8.1% in verbatim mode, making transcripts more reproducible.
  • English-only mode-tag training transfers to German zero-shot, implying that verbatim control does not require target-language disfluency data for closely related languages.
  • Supervised cross-attention timing outperforms forced alignment on disfluent speech (102 ms vs 142–200 ms MAE), removing the need for external aligners.
  • Verbatimize can bootstrap high-quality verbatim corpora from intended transcripts plus audio, with rare-word recall rising from 6.8% to 96.1%.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • This suggests that many apparent ASR failures on disfluent or stuttered speech may be control failures rather than recognition failures; an immediate test is whether mode tags also reduce truncation and hallucination rates for stuttered input without new acoustic data.
  • If style is a controllable output axis, the same discrete-tag interface could be extended to other output conventions—punctuation, casing, speaker attributes, or dialect—where pretrained models already contain multiple latent renderings.
  • The frozen-embedding result implies a cheap probe for latent capabilities in any pretrained encoder-decoder: train only a small set of task tokens and measure the jump before committing resources to fine-tuning.
  • A testable extension beyond the paper's scope is to evaluate zero-shot transfer on a typologically distant language (e.g., Japanese or Arabic); if the F1 jump persists, the activation story is genuinely multilingual rather than an artifact of English-German similarity.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper argues that transcription style (verbatim vs. intended) is an uncontrolled latent variable in modern ASR, and that pretrained Whisper models already encode both styles; the missing ingredient is a control signal. The authors introduce discrete decoder-prefix mode tags trained on paired verbatim/intended transcripts, with coverage-aware tag partitioning, a frozen-embedding stage, supervised cross-attention fine-tuning for word-level timing, and a 'verbatimize' task for corpus enrichment. Headline results include: frozen-embedding mode tags raising German disfluency event F1 from 10% to 79% with English-only training; full decoder fine-tuning reaching 94% German eF1; timing MAE of 36 ms on TIMIT and 102 ms on FluencyBank, outperforming forced alignment on disfluent speech; and rare-word recall improving from 6.8% to 96.1% with verbatimize.

Significance. If the activation claim holds, the paper usefully reframes transcription style as an explicit interface rather than an uncontrolled latent variable, and it demonstrates a path to cross-lingual verbatim ASR without target-language verbatim supervision. The frozen-embedding experiment is a clean, interpretable control, and the paper evaluates on several external benchmarks and ships code/data. However, the cross-lingual headline rests partly on a small author-recorded German set, the 'up to 60% WER confounding' claim is not supported by Figure 2, and the TIMIT timing result suffers from selection-on-evaluation leakage. These issues need correction before the strongest claims can be fully credited.

major comments (4)
  1. [§3.3, Figure 2, Abstract] The claim that 'up to 60% of reported WER is attributable to style mismatch' is not supported by the numbers in Figure 2. For AMI, the style share is 11/(11.6+11) ≈ 49% for W→V and 7.7/(7.7+7.7) = 50% for W→I. For TED-LIUM, the figure totals appear to be 5.4% and 7.6%, while the text states 5.7% and 7.3%. Since this statistic is the abstract's primary motivating number, please recompute and report the exact style fractions, and reconcile the text with the figure.
  2. [§5.1, Table 5, Limitations] The German zero-shot activation results (S1 eF1 78.9%, FT0 93.8%) are evaluated on GDS, an author-recorded set of only 202 utterances, and the authors concede in Limitations that the disfluencies 'may be less naturalistic than in the wild.' No confidence intervals are reported despite the small n. The German S1 number is higher than the English S1 number measured on a much larger, more natural set (53.0%), which is suspicious and consistent with an easy evaluation set. Please report confidence intervals, per-disfluency-type F1, and ideally validate on an independent German disfluent corpus or a held-out speaker split. As it stands, the cross-lingual activation claim is fragile.
  3. [§4.3.1, Table 7] The k=10 cross-attention head subset is selected by correlation with TIMIT ground-truth alignments, and TIMIT is then used as an evaluation set in Table 7. This is a selection-on-evaluation leak for the reported 36 ms TIMIT MAE. The FluencyBank and Thorsten results are not affected by this leak, but the TIMIT read-speech number is optimistic. Please select heads on a held-out TIMIT split or use cross-validation, or relegate TIMIT to a development set and make FluencyBank/Thorsten the primary timing evidence.
  4. [§5.2 vs. §3.3] The intended transcripts are generated by GPT-4o with 'reformatting numbers and dates' and 'removing fragments' (§5.2). The statement in §3.3 that verbatim and intended references 'share 100% of content words by construction' is therefore not literally true: reformatting produces different tokens, and WER will count those as content errors. This contaminates the clean content/style decomposition in Figure 2. Please quantify how often such reformatting occurs, or use a minimal cleaning rule that only strips disfluencies without changing surface forms.
minor comments (5)
  1. [Abstract, §4.5, §7.1, Discussion] Stage 1 trains all newly added token embeddings (2 fillers + 12 sound tokens + 10 mode tags + 2 delimiters = 26), not 'only mode-tag embeddings' as stated in the Abstract and §7.1. The Discussion also says '27 new token embeddings.' Please correct the wording and count.
  2. [§3.3] The sentence 'while its content loss is identical (a = 3.8%)' appears to contain a typo ('a' should likely be 'at'). Also, the TED-LIUM reported WER values (5.7%, 7.3%) should be matched to the figure bars.
  3. [§5.2] No manual validation is reported for the GPT-4o-generated intended transcripts on the 40h NSC subset or the larger training corpora. Even if the downstream claims are unaffected, a small human-validation sample would increase confidence in the paired-supervision premise.
  4. [Throughout] Typos and formatting issues: 'arbitarily' in §7.1, 'incosistencies' in §3.3, 'indenpendent' in §3.3, and 'V erbatim' in Table 1 header. Please proofread.
  5. [§7.2] The claim 'for the first time outperforming forced-alignment baselines' is strong. The improvement over CrisperWhisper+s on FluencyBank is 102 ms vs. 122 ms, and no statistical significance is reported. Please soften the claim or provide a more complete comparison with prior attention-based timing methods.

Circularity Check

1 steps flagged

Main derivation is supervised and externally benchmarked; only a localized TIMIT head-selection/evaluation overlap keeps this from a clean 0.

specific steps
  1. fitted input called prediction [Section 4.3.1 (head selection) and Table 7 / Section 7.2 (TIMIT timing evaluation)]
    "We select the k=10 cross-attention heads whose unsupervised attention patterns best correlate with ground-truth alignments on TIMIT [40], adding heads greedily by marginal improvement. ... Table 7: Word-level timing comparison ... TIMIT (EN, read) ... Ours (ah)+s 36"

    The alignment-head subset H is chosen to maximize correlation with TIMIT ground-truth boundaries, and TIMIT is then reported as an evaluation set for the resulting timing model. The TIMIT MAE/F1 numbers therefore evaluate the same criterion used to select H, so they are not an out-of-sample test of that choice; the head identities/count are fitted parameters on the evaluation set. This is a selection-on-test-set reduction local to TIMIT, not a definitional equivalence of the whole timing method: H is fixed before the FluencyBank and Thorsten evaluations, so those remain independent.

full rationale

No definitional circularity was found in the central activation claim. The S1 result (frozen encoder/decoder, only mode-tag embeddings trained) is an empirical result evaluated on external DisfluencySpeech and author-recorded GDS, not an identity with the training loss. The intended transcripts are GPT-4o derivations of verbatim transcripts, which is a synthetic-supervision and evaluation-quality concern rather than a circular derivation. The WER style decomposition (style = WER − CLR) is definitional but used as a measurement description, not as a predicted outcome. Verbatimize's high rare-word recall is partly a prompt-copying effect by design, but the paper frames it as transcript-conditioned insertion, not as acoustic-only recognition. The only concrete reduction-by-fit is the TIMIT head selection described above. The paper's own Limitations (author-recorded German data, MFA-derived timing supervision) weaken evidence strength but are not circular. Self-citations to CrisperWhisper [19] are used as a baseline and design reference, not as the load-bearing proof of the latent-capability claim.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

The central mechanism depends less on free parameters than on the quality of synthetic references (GPT-4o, MFA) and on the closed token inventory. Mode tags and sound tokens are vocabulary items whose behavior is measured in-paper, not unfalsifiable external entities.

free parameters (5)
  • Cross-attention head subset (k=10) = 10 heads selected greedily on TIMIT
    §4.3.1: heads are chosen by correlation with ground-truth TIMIT alignments, and TIMIT is later used as an evaluation set.
  • Attention sharpening temperature tau = 3
    §4.3.2: inference-time scalar chosen to improve boundary sharpness; no independent validation.
  • Timing loss weight lambda = 0.2
    §6.5: weight of L_timing relative to cross-entropy; set by hand.
  • Mode-tag token counts = 3 verbatim + 2 sound + 5 intended tokens
    §4.1.1: manual design; 'multiple prefix tokens provide stronger conditioning' is asserted, not measured.
  • Casing perturbation count = 1–3 content words
    §4.4: hyperparameter for verbatimize; chosen by hand.
axioms (5)
  • domain assumption GPT-4o-generated intended transcripts are sufficiently faithful to serve as intended-mode training targets.
    §5.2: paired intended transcripts are produced by LLM stripping; errors become the target for intended mode and for verbatimize validation.
  • domain assumption MFA word boundaries are accurate enough to supervise cross-attention timing.
    §5.2: 500 h of timing supervision come from MFA, which Table 7 shows degrades on disfluent speech (142 ms MAE).
  • domain assumption The finite canonical token inventory ([uh], [um], 12 sound tags, trailing-hyphen cut-offs) captures all disfluency types of interest.
    §4.2/§6.2: F1 labels and canonicalization assume this closed set; anything else falls into OTHER.
  • domain assumption Edit-distance alignment with repetition disambiguation yields correct disfluency labels.
    §6.2: gold labels are extracted algorithmically from transcript strings, not manually annotated.
  • domain assumption Whisper-medium's pretrained representations contain latent verbatim capability that transfers to German.
    §7.1: the S1 result supports this, but the 'zero-shot German' claim presupposes cross-lingual latent transfer.

pith-pipeline@v1.3.0-alltime-deepseek · 17033 in / 18194 out tokens · 188740 ms · 2026-08-01T13:54:58.509026+00:00 · methodology

0 comments
read the original abstract

Modern ASR models trained on heterogeneously annotated data treat transcription style (verbatim vs. intended) as an uncontrolled latent variable, causing measurable decoding instability, evaluation confounding (up to 60% of reported WER attributable to style mismatch), and unreliable word-level timing. We show that models already encode both styles; the challenge is controlled activation. Using coverage-aware decoder task tokens trained on parallel verbatim/intended transcript pairs, we raise German disfluency F1 from 10% to 79% zero-shot, despite English-only training. Full English-only fine-tuning surpasses all baselines in verbatim accuracy, disfluency detection, and intended-mode quality across both languages. We further introduce supervised cross-attention fine-tuning that improves word-level timestamps on disfluent speech beyond forced-alignment baselines. Finally, we propose verbatimize, a new task enabling scalable creation and enrichment of speech corpora with high-quality canonical verbatim transcriptions.

Figures

Figures reproduced from arXiv: 2607.18934 by Bernhard Thallinger (1) ((1) nyra labs), Laurin Wagner (1), Mario Zusag (1).

Figure 2
Figure 2. Figure 2: presents this decomposition for TED-LIUM and AMI, two widely used benchmarks from the Open ASR Leader￾board [37]. Two findings emerge. First, verbatim and intended reference transcripts of the same audio, which share 100% of content words by construction (subsection 5.2), disagree by 3.7% WER on TED-LIUM and 12.1% on AMI. This disagree￾V→I W→V W→I 0 5 10 15 20 3.7 1.9 3.5 3.8 3.8 Word Error Rate (%) TED-LI… view at source ↗
Figure 3
Figure 3. Figure 3: Decoder prompt structure. Mode tags control tran￾scription style, followed by an optional intended transcript hint (for verbatimize), then standard Whisper tokens. The model outputs time-aligned transcription in the prompted style 4.3. Activating timing: supervised cross-attention With transcription policy resolved and token outputs stable, word-level timing becomes well-defined. Yet cross-attention alignm… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

62 extracted references · 10 linked inside Pith

  1. [1]

    I i [uh] li- like intended transcripts

    Introduction Speech transcription requires a policy decision that most train- ing pipelines leave implicit: should the output preserve what was saidverbatim, including filled pauses, repetitions, self- corrections, cut-offs, and paralinguistic sounds (e.g., “I i [uh] li- like intended transcripts”) or should it produceintendedtext that retains only the sp...

    Pith/arXiv arXiv 2026

  2. [2]

    We highlight the key gaps our work addresses

    Related Work Table 1 summarizes how existing approaches compare on tran- scription style control, timing, and multilingual support. We highlight the key gaps our work addresses. Transcription style.Most ASR systems produce tran- scripts under a single implicit policy, and approaches that target verbatim output typically do so unidirectionally-without bidi...

  3. [3]

    ambiguous zone

    The Transcription Policy Problem Before presenting our method in Section 4, we establish that transcription policy ambiguity is not a theoretical concern but a measurable source of instability, evaluation error, and ill- defined timing in current ASR systems. 3.1. Policy ambiguity under mixed supervision To isolate the effect of annotation inconsistency, ...

  4. [4]

    meeting

    Method We present three mechanisms to unambiguously activate la- tent capabilities in large-scale ASR model: explicit style con- trol via mode tags, supervised cross-attention for word-level timing, and transcript-conditioned disfluency recovery. All ex- periments initialize from the official OpenAI Whisper-medium checkpoint [13]. We extend Whisper’s voca...

  5. [5]

    Data Our training procedure requires paired transcripts: each audio sample needs both a verbatim and an intended reference. 5.1. Corpora Table 3 summarizes all datasets and their roles. For English verbatim data, we use the ICSI Meeting Corpus [41] (72 h), the AMI Meeting Corpus [36] (100 h), CORAAL [42] (150 h), and the National Speech Corpus [43] (2,000...

  6. [6]

    the the the problem

    Experimental Setup 6.1. Evaluation datasets We evaluate three capabilities on separate benchmarks.Dis- fluency detection:DisfluencySpeech [7] (English,n=4,707) and German DisfluencySpeech (German,n=202).Word-level timing:TIMIT [40] (English, read speech), FluencyBank [48] (English, disfluent speech), and Thorsten [49] (German, read speech).Verbatimize:the...

  7. [7]

    t- trr- transformer

    Results We present results organized by the three capabilities activated in Section 4: style control, word-level timing and verbatimize. 7.1. Style control: latent capability and cross-lingual trans- fer Tables 5 and 6 present disfluency detection and intended tran- script quality on English and German. We report our model at two key stages:S1(Stage 1: fr...

  8. [8]

    Discussion and Conclusion Across Sections 7.1–7.3, we find a consistent pattern: pre- trained ASR models often already contain the needed capabil- ity, but require an explicit interface to reliably express it. The frozen-model experiment (S1) is the clearest example: training only 27 new token embeddings, with no updates to the 764M encoder/decoder parame...

  9. [9]

    The experiments were run manually and results were manually verified

    Generative AI Use Disclosure Some code used in the experiments was written with help from a coding assistant (Claude by Anthropic). The experiments were run manually and results were manually verified. Generative AI was also used in the formatting of tables and plots. The paper was manually written. The authors assume full responsibility and accountabilit...

  10. [10]

    Recording for eyes, not echoing to ears: Contextu- alized spoken-to-written conversion of ASR transcripts,

    J. Liu, C. Deng, Q. Zhang, S. Zhou, Q. Chen, H. Yu, and W. Wang, “Recording for eyes, not echoing to ears: Contextu- alized spoken-to-written conversion of ASR transcripts,” inPro- ceedings of AAAI, 2025, demonstrates that downstream NLP tasks benefit from converting verbatim ASR to clean text

    2025

  11. [11]

    Preliminaries to a theory of speech disfluencies,

    E. E. Shriberg, “Preliminaries to a theory of speech disfluencies,” PhD dissertation, University of California, Berkeley, 1994

    1994

  12. [12]

    Clinical applications of disfluency analysis in neurological populations,

    S. Ash and M. Grossman, “Clinical applications of disfluency analysis in neurological populations,”Aphasiology, 2021

    2021

  13. [13]

    A multi-stage feature pipeline on timestamped speech transcriptions for demen- tia assessment,

    B. Thallinger, L. Wagner, T. Bloder, and M. Zusag, “A multi-stage feature pipeline on timestamped speech transcriptions for demen- tia assessment,” inICASSP 2025 - 2025 IEEE International Con- ference on Acoustics, Speech and Signal Processing (ICASSP), 2025, pp. 1–2

    2025

  14. [14]

    Careful whisper – lever- aging advances in automatic speech recognition for robust and interpretable aphasia subtype classification,

    M. Zusag, L. Wagner, and T. Bloder, “Careful whisper – lever- aging advances in automatic speech recognition for robust and interpretable aphasia subtype classification,” inProceedings of INTERSPEECH, 2023, pp. 3933–3937, achieves human-level aphasia classification by combining CTC (acoustic/verbatim) and encoder-decoder (clean) ASR outputs

    2023

  15. [15]

    Emilia: A large-scale, extensive, multilingual, and diverse dataset for speech generation,

    H. He, Z. Shang, C. Wang, X. Li, Y . Gu, H. Hua, L. Liu, C. Yang, J. Li, P. Shi, Y . Wang, K. Chen, P. Zhang, and Z. Wu, “Emilia: A large-scale, extensive, multilingual, and diverse dataset for speech generation,”IEEE Transactions on Audio, Speech and Language Processing, 2025, 101k+ hours of spontaneous speech; models produce more human-like output than ...

    2025

  16. [16]

    DisfluencySpeech: Single- speaker conversational speech dataset with paralanguage,

    K. Wang and D. Herremans, “DisfluencySpeech: Single- speaker conversational speech dataset with paralanguage,”arXiv preprint arXiv:2406.08820, 2024, 10h studio-quality single- speaker dataset derived from Switchboard with filled pauses, dis- course markers, restarts, and non-speech sounds; provides paral- lel verbatim/intended transcripts at three levels ...

    Pith/arXiv arXiv 2024

  17. [17]

    Spontaneous style text-to-speech synthesis with controllable spontaneous behaviors based on language models,

    W. Li, P. Yang, Y . Zhong, Y . Zhou, Z. Wang, Z. Wu, X. Wu, and H. Meng, “Spontaneous style text-to-speech synthesis with controllable spontaneous behaviors based on language models,” arXiv preprint arXiv:2407.13509, 2024, addresses data scarcity for spontaneous speech TTS

    Pith/arXiv arXiv 2024

  18. [18]

    NVSpeech: An integrated and scalable pipeline for human-like speech modeling with paralinguistic vocalizations,

    H. Liao, Q. Ni, Y . Wang, Y . Lu, H. Zhan, P. Xie, Q. Zhang, and Z. Wu, “NVSpeech: An integrated and scalable pipeline for human-like speech modeling with paralinguistic vocalizations,” arXiv preprint arXiv:2508.04195, 2025

    Pith/arXiv arXiv 2025

  19. [19]

    StutterZero and StutterFormer: End-to-end speech con- version for stuttering transcription and correction,

    Q. Xu, “StutterZero and StutterFormer: End-to-end speech con- version for stuttering transcription and correction,”IEEE Access, vol. 13, pp. 208 773–208 787, 2025, converts stuttered wave- forms to fluent waveforms; 24–28% WER reduction vs Whisper- Medium on converted output

    2025

  20. [20]

    From user perceptions to technical improvement: Enabling people who stutter to better use speech recognition,

    C. Lea, Z. Huang, L. Tooley, J. Narain, D. Yee, P. Georgiou, T. D. Tran, J. P. Bigham, and L. Findlater, “From user perceptions to technical improvement: Enabling people who stutter to better use speech recognition,”ACM Transactions on Accessible Comput- ing, vol. 15, no. 2, pp. 1–27, 2023, reports 19.8% WER and 23.8% truncation rate on production ASR for...

    2023

  21. [21]

    J-j-j-just stutter: Benchmarking whisper’s performance disparities on different stuttering patterns,

    C. Sridhar and S. Wu, “J-j-j-just stutter: Benchmarking whisper’s performance disparities on different stuttering patterns,” 08 2025, pp. 3753–3757

    2025

  22. [22]

    Robust speech recognition via large-scale weak su- pervision,

    A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak su- pervision,”International Conference on Machine Learning, pp. 28 492–28 518, 2023

    2023

  23. [23]

    Careless whisper: Speech-to-text hallucination harms,

    A. Koenecke, A. S. G. Choi, K. X. Mei, H. Schellmann, and M. Sloane, “Careless whisper: Speech-to-text hallucination harms,” inProceedings of ACM FAccT, 2024

    2024

  24. [24]

    Inclusive ASR for disfluent speech: Cascaded large- scale self-supervised learning with targeted fine-tuning and data augmentation,

    D. Mujtaba, N. R. Mahapatra, M. Arney, J. S. Yaruss, C. Herring, and J. Bin, “Inclusive ASR for disfluent speech: Cascaded large- scale self-supervised learning with targeted fine-tuning and data augmentation,” inProceedings of INTERSPEECH, 2024, states “a critical barrier to progress is the scarcity of large, annotated disfluent speech datasets”

    2024

  25. [25]

    Speech disfluency detection with contextual representation and data distillation,

    P. Mohapatra, A. Pandey, B. Islam, and Q. Zhu, “Speech disfluency detection with contextual representation and data distillation,” inProceedings of the 1st ACM International Workshop on Intelligent Acoustic Systems and Applications, ser. IASA ’22. New York, NY , USA: Association for Computing Machinery, 2022, p. 19–24. [Online]. Available: https://doi.org...

    arXiv 2022

  26. [26]

    Acous- tically precise hesitation tagging is essential for end-to-end verba- tim transcription systems,

    J.-K. Lin, H.-C. Lu, C.-C. Wang, H.-Y . Lin, and B. Chen, “Acous- tically precise hesitation tagging is essential for end-to-end verba- tim transcription systems,” inarXiv preprint arXiv:2506.04076, 2025, states manual annotation is prohibitively expensive; used LLM labeling at $5 for entire dataset vs high cost of human anno- tators

    Pith/arXiv arXiv 2025

  27. [27]

    Granary: Speech recognition and translation dataset in 25 european languages,

    N. R. Koluguri, M. Sekoyan, G. Zelenfroynd, S. Meister, S. Ding, S. Kostandian, H. Huang, N. Karpov, J. Balam, V . Lavrukhin, Y . Peng, S. Papi, M. Gaido, A. Brutti, and B. Ginsburg, “Granary: Speech recognition and translation dataset in 25 european languages,” 2025. [Online]. Available: https://arxiv.org/abs/2505.13404

    Pith/arXiv arXiv 2025

  28. [28]

    CrisperWhisper: Accu- rate timestamps on verbatim speech transcriptions,

    L. Wagner, B. Thallinger, and M. Zusag, “CrisperWhisper: Accu- rate timestamps on verbatim speech transcriptions,” inProceed- ings of INTERSPEECH, 2024

    2024

  29. [29]

    Adapting an ASR foundation model for spoken language assessment,

    R. Ma, M. Qian, M. J. F. Gales, and K. M. Knill, “Adapting an ASR foundation model for spoken language assessment,” inPro- ceedings of INTERSPEECH, 2023, soft prompt tuning for verba- tim output; English-only, unidirectional, no timing

    2023

  30. [30]

    Prompting Whisper for improved verbatim transcription and end-to-end mis- cue detection,

    G. D. Smith, D. Yee, J. K. Chen, and L. Findlater, “Prompting Whisper for improved verbatim transcription and end-to-end mis- cue detection,”arXiv preprint arXiv:2505.23627, 2025

    Pith/arXiv arXiv 2025

  31. [31]

    WhisperX: Time- accurate speech transcription of long-form audio,

    M. Bain, J. Huh, T. Han, and A. Zisserman, “WhisperX: Time- accurate speech transcription of long-form audio,” inProceedings of INTERSPEECH, 2023

    2023

  32. [32]

    Reverb: Open-source asr and diarization from rev,

    N. Bhandari, D. Chen, M. ´Angel del R´ıo Fern´andez, N. Delworth, J. D. Fox, M. Jett ´e, Q. McNamara, C. Miller, O. Novotn ´y, J. Profant, N. Qin, M. Ratajczak, and J.-P. Robichaud, “Reverb: Open-source asr and diarization from rev,” 2025. [Online]. Available: https://arxiv.org/abs/2410.03930

    Pith/arXiv arXiv 2025

  33. [33]

    On disfluency and non-lexical sound labeling for end- to-end automatic speech recognition,

    P. Mihajlik, Y . Meng, M. K ´ad´ar, J. Linke, B. Schuppler, and K. Mady, “On disfluency and non-lexical sound labeling for end- to-end automatic speech recognition,” inProceedings of INTER- SPEECH, 2024, systematic comparison of disfluency/non-lexical labeling strategies on conversational Hungarian and Austrian Ger- man; distinguishes filled pauses from m...

    2024

  34. [34]

    End-to-end spontaneous speech recognition using disfluency labeling,

    K. Horii, M. Fukuda, K. Ohta, R. Nishimura, A. Ogawa, and N. Kitaoka, “End-to-end spontaneous speech recognition using disfluency labeling,” inProceedings of INTERSPEECH, 2022, joint ASR with disfluency tags for spontaneous speech

    2022

  35. [35]

    Streaming joint speech recogni- tion and disfluency detection,

    H. Futami, E. Tsunoo, K. Shibata, Y . Kashiwagi, T. Okuda, S. Arora, and S. Watanabe, “Streaming joint speech recogni- tion and disfluency detection,” inProceedings of ICASSP, 2023, streaming encoder-decoder joint ASR and disfluency detection (multitask / tag-based variants)

    2023

  36. [36]

    Augmenting auto- matic speech recognition models with disfluency detection,

    R. Amann, Z. Li, B. Bruno, and J. Niehues, “Augmenting auto- matic speech recognition models with disfluency detection,”arXiv preprint arXiv:2409.10177, 2024

    Pith/arXiv arXiv 2024

  37. [37]

    Filler word detection and classification: A dataset and benchmark,

    G. Zhu, J.-P. Caceres, and J. Salamon, “Filler word detection and classification: A dataset and benchmark,” inProceedings of IN- TERSPEECH, 2022, introduces PodcastFillers (145h, 35K fillers) and an ASR+V AD+classifier pipeline for timestamped filler de- tection in podcasts

    2022

  38. [38]

    Weakly-supervised forced alignment of disfluent speech using phoneme-level modeling,

    T. Kouzelis, G. Paraskevopoulos, A. Katsamanis, and V . Kat- souros, “Weakly-supervised forced alignment of disfluent speech using phoneme-level modeling,” inProceedings of INTER- SPEECH, 2023, forced alignment under transcript–audio mis- match due to disfluencies; avoids requiring fully verbatim tran- scripts

    2023

  39. [39]

    End-to-end speech recognition and disfluency removal,

    P. J. Lou and M. Johnson, “End-to-end speech recognition and disfluency removal,” inFindings of the Association for Computa- tional Linguistics: EMNLP 2020, 2020, pp. 2051–2061

    2020

  40. [40]

    Canary-1B- v2 & Parakeet-TDT-0.6B-v3: Efficient and high-performance models for multilingual ASR and AST,

    M. Sekoyan, N. R. Koluguri, N. Tadevosyan, P. Zelasko, T. Bart- ley, N. Karpov, J. Balam, and B. Ginsburg, “Canary-1B- v2 & Parakeet-TDT-0.6B-v3: Efficient and high-performance models for multilingual ASR and AST,”arXiv preprint arXiv:2509.14128, 2025, nVIDIA’s 1B multilingual encoder- decoder ASR trained on 1.7M hours; uses NFA for timestamps due to cros...

    arXiv 2025

  41. [41]

    Unconstrained dysfluency modeling for dysfluent speech transcription and detec- tion,

    J. Lian, C. Feng, N. Farooqi, S. Li, A. Kashyap, C. J. Cho, P. Wu, R. Netzorg, T. Li, and G. K. Anumanchipalli, “Unconstrained dysfluency modeling for dysfluent speech transcription and detec- tion,” inProceedings of the IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), 2023, defines and models dysfluent speech transcription and detecti...

    2023

  42. [42]

    Quantification of stylistic differences in human- and asr-produced transcripts of african american english,

    A. Heuser, T. Kendall, M. Rio, Q. McNamara, N. Bhandari, C. Miller, and M. Jett ´e, “Quantification of stylistic differences in human- and asr-produced transcripts of african american english,” 09 2024, pp. 4538–4542

    2024

  43. [43]

    Toward zero oracle word error rate on the switchboard benchmark,

    A. Faria, A. Janin, K. Riedhammer, and S. Adkoli, “Toward zero oracle word error rate on the switchboard benchmark,” inProc. INTERSPEECH, 2022, pp. 3973–3977

    2022

  44. [44]

    Style-agnostic evalu- ation of asr using multiple reference transcripts,

    Q. McNamara, M. ´A. del R´ıo Fern´andez, N. Bhandari, M. Rata- jczak, D. Chen, C. Miller, and M. Jett ´e, “Style-agnostic evalu- ation of asr using multiple reference transcripts,”arXiv preprint arXiv:2412.07937, 2024

    Pith/arXiv arXiv 2024

  45. [45]

    The AMI meeting corpus,

    J. Carletta, S. Ashby, S. Bourban, M. Flynn, M. Guillemot, T. Hain, J. Kadlec, V . Karaiskos, W. Kraaij, M. Kronenthalet al., “The AMI meeting corpus,” inInternational Conference on Meth- ods and Techniques in Behavioral Research, 2005

    2005

  46. [46]

    Open ASR leaderboard,

    Hugging Face, “Open ASR leaderboard,” https://huggingface.co/ spaces/hf-audio/open asr leaderboard, 2024

    2024

  47. [47]

    AA-WER v2.0: Speech to text accuracy benchmark,

    Artificial Analysis, “AA-WER v2.0: Speech to text accuracy benchmark,” Feb. 2026, accessed: 2026-02-21. [Online]. Available: https://artificialanalysis.ai/articles/aa-wer-v2

    2026

  48. [48]

    Monitoring and self-repair in speech,

    W. J. M. Levelt, “Monitoring and self-repair in speech,”Cogni- tion, vol. 14, no. 1, pp. 41–104, 1983

    1983

  49. [49]

    The DARPA TIMIT acoustic-phonetic continuous speech corpus (TIMIT),

    J. S. Garofolo, L. F. Lamel, W. M. Fisher, J. G. Fiscus, and D. S. Pallett, “The DARPA TIMIT acoustic-phonetic continuous speech corpus (TIMIT),”Linguistic Data Consortium, 1993

    1993

  50. [50]

    The ICSI meeting corpus,

    A. Janin, D. Baron, J. Edwards, D. Ellis, D. Gelbart, N. Mor- gan, B. Peskin, T. Pfau, E. Shriberg, A. Stolckeet al., “The ICSI meeting corpus,” inIEEE International Conference on Acoustics, Speech, and Signal Processing, 2003

    2003

  51. [51]

    The corpus of regional african american language,

    T. Kendall and C. Farrington, “The corpus of regional african american language,” Eugene, OR, 2023. [Online]. Available: https://doi.org/10.7264/1ad5-6t35

    doi:10.7264/1ad5-6t35 2023

  52. [52]

    Building the singapore english national speech corpus,

    J. X. Koh, A. Mislan, K. Khoo, B. Ang, W. Ang, C. Ng, and Y .-Y . Tan, “Building the singapore english national speech corpus,” in Interspeech 2019, 2019, pp. 321–325

    2019

  53. [53]

    Lib- rispeech: An asr corpus based on public domain audio books,

    V . Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Lib- rispeech: An asr corpus based on public domain audio books,” inProceedings of ICASSP, 2015, pp. 5206–5210

    2015

  54. [54]

    Common V oice: A massively-multilingual speech corpus,

    R. Ardila, M. Branson, K. Davis, M. Henretty, M. Kohler, J. Meyer, R. Morais, L. Saunders, F. M. Tyers, and G. We- ber, “Common V oice: A massively-multilingual speech corpus,” inProceedings of the 12th Language Resources and Evaluation Conference, 2020, pp. 4211–4215

    2020

  55. [55]

    V ocalSound: A dataset for improv- ing human vocal sounds recognition,

    Y . Gong, J. Yu, and J. Glass, “V ocalSound: A dataset for improv- ing human vocal sounds recognition,” inIEEE International Con- ference on Acoustics, Speech and Signal Processing (ICASSP), 2022, pp. 151–155

    2022

  56. [56]

    Nonspeech7k dataset: Clas- sification and analysis of human non-speech sound,

    M. M. Rashid, G. Li, and C. Du, “Nonspeech7k dataset: Clas- sification and analysis of human non-speech sound,”IET Signal Processing, vol. 17, no. 6, p. e12233, 2023

    2023

  57. [57]

    Fluency- Bank timestamped: An updated data set for disfluency detection and automatic intended speech recognition,

    A. Romana, M. Niu, M. Perez, and E. Mower Provost, “Fluency- Bank timestamped: An updated data set for disfluency detection and automatic intended speech recognition,”Journal of Speech, Language, and Hearing Research, vol. 67, no. 11, pp. 4203–4215, 2024, provides word-level timestamps and disfluency labels for stuttered speech; enables benchmarking of di...

    2024

  58. [58]

    Thorsten - open german voice (neutral) dataset,

    T. M ¨uller and D. Kreutz, “Thorsten - open german voice (neutral) dataset,” 2021

    2021

  59. [59]

    Hello GPT-4o,

    OpenAI, “Hello GPT-4o,” https://openai.com/index/ hello-gpt-4o/, 2024

    2024

  60. [60]

    Montreal forced aligner: Trainable text-speech align- ment using kaldi,

    M. McAuliffe, M. Socolof, S. Mihuc, M. Wagner, and M. Son- deregger, “Montreal forced aligner: Trainable text-speech align- ment using kaldi,” inProceedings of Interspeech, 2017, pp. 498– 502

    2017

  61. [61]

    Tradition or in- novation: A comparison of modern asr methods for forced align- ment,

    R. Rousso, E. Cohen, J. Keshet, and E. Chodroff, “Tradition or in- novation: A comparison of modern asr methods for forced align- ment,” 09 2024, pp. 1525–1529

    2024

  62. [62]

    Introducing Universal-3 Pro: A new class of speech language model optimized for voice AI,

    AssemblyAI, “Introducing Universal-3 Pro: A new class of speech language model optimized for voice AI,” https://www. assemblyai.com/blog/introducing-universal-3-pro, 2026

    2026