Pith. sign in

REVIEW 4 major objections 4 minor 61 references

This paper shows that timestamp drift in autoregressive ASR can be repaired post-training by editing only timestamp tokens while anchoring the lexicon, so corrected timestamps come natively at decoding time.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 08:28 UTC pith:N2EPANPE

load-bearing objection A genuinely useful diagnostic and a clever lightweight fix for ASR timestamp drift, with the 'no forgetting' half of the claim only partially proven. the 4 major comments →

arxiv 2607.05364 v2 pith:N2EPANPE submitted 2026-07-06 cs.CL cs.AIcs.SD

REDDIT: Correcting Model-Generated Timestamp Drift in ASR without Forgetting via Replay-Based Distribution Editing

Cheng-Kang Chou , Ming-To Chuang , Ke-Han Lu , Chan-Jan Hsu , Hung-yi Lee This is my paper
classification cs.CL cs.AIcs.SD
keywords automatic speech recognitiontimestamp driftnon-speech gapsreplay-based distribution editingcatastrophic forgettingWhispertimestamp tokenspost-training model editing
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Autoregressive ASR that emits timestamps as decoded tokens can produce a plausible transcript while placing it on the wrong part of the audio timeline, especially after long silences. The paper argues this drift is a distinct failure mode, separate from word error and hallucination, and shows it across 15 timestamp-producing systems. It then proposes REDDIT, a two-stage post-training method that corrects timestamp tokens while preserving the recognizer's non-timestamp behavior by replaying the frozen model's own outputs as context and matching its distribution on ordinary text tokens. With 34.9 hours of automatically spliced correction audio and 1.6% of parameters updated, the method raises long-gap temporal overlap on Whisper-tiny from 38.7% to 95.0% and cuts out-of-domain timestamp error from 2752 ms to 223 ms, while avoiding the severe recognition loss that naive timestamp fine-tuning causes. If correct, this means generated timestamps can be repaired natively, with no inference-time aligner, VAD, or DTW.

Core claim

The central claim is that timestamp drift across non-speech gaps is an edit of the time axis, not a failure of speech recognition, and that this axis can be corrected in place: REDDIT overwrites each replayed timestamp token with an exact boundary target derived from a synthetic splicing schedule, trains the student under the frozen model's cached decoder prefix, and uses KL divergence on every non-timestamp position to keep the lexical distribution anchored to the base model. A short second stage re-trains the same objective under the edited prefix to consolidate the corrected transitions. On Whisper-tiny the intervention updates only the last cross-attention and layer norms, using no human

What carries the argument

The load-bearing object is the replay context: the frozen base model's own decoded token sequence, cached once and used as the teacher-forcing prefix for both losses. Timestamp tokens in that replay are replaced by corrected targets, computed from VAD-trimmed speech spans and inserted non-speech gaps with known offsets, and trained with cross-entropy; non-timestamp positions are trained to match the frozen teacher's distribution through KL divergence under the same context. This decouples temporal editing from lexical preservation. Stage 2 swaps the context to the edited prefix and uses the Stage-1 model as teacher.

Load-bearing premise

The load-bearing premise is that matching the frozen teacher's word-output distribution on non-timestamp positions under replayed contexts transfers to free-running decoding on unseen domains—something the paper only probes on small out-of-domain sets, where its own tables show CV-en no-gap MER degrading from 37.0% to 41.3% and ASCEND-zh from 53.9% to 63.5%, and where the KL anchor is measured under teacher-forced contexts rather than multi-step rollout.

What would settle it

Run the REDDIT-trained Whisper-tiny on a large, previously unseen no-gap corpus and measure MER against the base model; if the MER gap exceeds the small degradation reported on the paper's out-of-domain sets, the 'without forgetting' claim fails. More directly, compare the student's non-timestamp token distribution under free decoding (no teacher forcing) with the frozen base's distribution: if that divergence is large while the teacher-forced KL is small, the anchor does not bound free decoding.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Corrected timestamps are produced natively at decoding time, so a REDDIT-processed model needs no VAD, forced alignment, DTW, or attention-head selection.
  • The 34.9-hour, 0.59M-parameter intervention on Whisper-tiny and the 0.43% parameter update on Whisper-large-v3 indicate timestamp repair can be a small post-training step rather than a full retraining.
  • Because correction targets come from splicing offsets, training data require no human transcripts or human timestamp labels; the frozen model supplies pseudo-text and replay distributions.
  • The OOD mixed-gap result (AAS from 2752 ms to 223 ms) implies the correction transfers beyond the synthetic training distribution to unseen gap patterns and domains.
  • The contrast with SFT decoder tuning (41.3% vs 524.2% MER on English Common Voice no-gap) shows that without the non-timestamp KL anchor, timestamp fine-tuning collapses recognition, making forgetting a central part of the timestamp-correction objective.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the decoupling holds beyond Whisper, the replay-edit recipe is a general model-editing pattern: pick one output axis to rewrite, replay the base model's context, and distill everything else. A direct test is applying it to word-level timestamps or to punctuation and speaker-turn tokens.
  • Because the synthetic splice construction is label-free, the pipeline can be pointed at any language or domain the base model already transcribes; a testable extension is multilingual timestamp correction with per-language drift measurement.
  • The fact that only the last cross-attention and layer norms need updating suggests timestamp placement is a localized readout in the decoder; a falsifiable corollary is that drift severity is predictable from cross-attention statistics over silent regions, which could guide where to collect correction data.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper identifies a failure mode in autoregressive ASR systems that emit timestamps as decoded tokens: across long non-speech spans, the decoded timeline can drift while the transcript remains plausible. It introduces a controlled gap/long-gap benchmark across 15 timestamp-producing ASR/audio-language systems and shows that ordinary timestamp-corrected fine-tuning repairs alignment but catastrophically degrades non-target recognition. The proposed method, REDDIT, is a two-stage post-training procedure: Stage 1 edits timestamp targets under a cached base-model replay context while KL-matching the frozen base distribution on non-timestamp positions (Eqs. 6–8); Stage 2 refines under edited-prefix context (Eq. 9), using the Stage-1 checkpoint as the KL teacher. Training data are constructed by VAD-trimming speech spans and inserting non-speech gaps, with exact timestamps derived from the splice offsets. On Whisper-tiny, updating only 1.6% of parameters, REDDIT improves long-gap mIoU from 38.7% to 95.0% and mixed-gap OOD AAS from 2752 ms to 223 ms, while CV-en MER rises to 41.3% compared to 524.2% for full-decoder SFT. The central claim is that timestamp drift can be corrected without ASR forgetting, using no inference-time VAD, forced alignment, or post-processing.

Significance. If the results hold, this is a useful and practical contribution: it offers a parameter-efficient, annotation-free way to repair a model's native timestamp outputs without adding inference-time alignment modules. The controlled synthetic construction with exact reference timestamps is a strength, as is the breadth of the diagnostic benchmark across 15 systems, which cleanly separates temporal drift from lexical hallucination. The paper is also transparent about several limitations, including checkpoint selection by validation behavior and the small single-run OOD evaluation sets. However, the 'without forgetting' half of the central claim is only partially supported by the paper's own numbers: Table VII shows measurable no-gap recognition degradation on CV-en and ASCEND-zh, so the claim needs to be reframed and the retention mechanism tested under free-decoding contexts.

major comments (4)
  1. [Section V-C / Table VII] The paper's headline and conclusion state that REDDIT preserves non-target ASR behavior, but Table VII shows no-gap CV-en MER rising from 37.0% (base) to 41.3% (REDDIT Full) and ASCEND-zh from 53.9% to 63.5%. These are the authors' own numbers on sets of 2,997 and 578 samples, respectively, with no error bars or repeated runs. The abstract's 'preserving CV-en MER at 41.3%' is therefore better described as 'substantially less forgetting than full-decoder SFT' than as absolute preservation. Please report base-relative degradations, confidence intervals, and a direct statistical comparison; otherwise the 'without forgetting' claim is overstated.
  2. [Eqs. (7), (9) / Sec. III-D] The anti-forgetting anchor is a per-position KL divergence computed under teacher-forced contexts: cached replay prefixes in Stage 1 and edited prefixes in Stage 2. It does not constrain the student under its own free-decoding rollouts, where the context includes self-generated timestamp tokens that are neither the cached drifted ones nor the edited targets. Moreover, in Stage 2 the KL teacher is the Stage-1 checkpoint rather than the frozen base, so any Stage-1 drift is not corrected. Since the KL is not evaluated on self-generated contexts, the free-decoding divergence is only indirectly bounded; Table VII's degradation is consistent with this gap. A concrete test would be to compute a rollout-based KL or report OOD no-gap MER with self-generated decoding on longer audio.
  3. [Sec. IV-C] The paper states: 'We select the reported full-pipeline checkpoint from this short refinement trajectory by validation behavior rather than treating a fixed step count as part of the method.' This introduces a best-of-trajectory selection for REDDIT Full, but the manuscript does not define what 'validation behavior' means nor whether the validation set overlaps with the reported Gap/Long-Gap test sets. If the validation set includes any of the reported evaluation splits, the headline numbers are not a fixed-procedure estimate. Please specify the selection rule, the validation data, and the sensitivity of the results to the chosen checkpoint.
  4. [Sec. III-F] Correction examples are pre-filtered to remove 'hallucinations, repetitions, boilerplate, empty or unusable text, and structurally inconsistent timestamps,' retaining examples with exactly 2K well-ordered timestamp tokens. The paper does not report how many cached teacher replays are discarded or the filtering rate per split. If a large fraction of base-model outputs is filtered out, the method may only be demonstrated on already well-structured trajectories, and the '34.9 hours of targeted correction audio' is not reproducible without this rate. Please report filtering counts and characterize the discarded examples.
minor comments (4)
  1. [Table IV caption] The caption states that 'DRIFT AND HALLUCINATION COLUMNS ARE FILE-LEVEL RATES,' but the metric definitions in Sec. IV-D.d describe Drift>5s and Drift>10s as percentages of matched segments. This is inconsistent and should be corrected.
  2. [Abstract / Table VII] The abstract's 'preserving CV-en MER at 41.3%' is misleading without an explicit base of 37.0%; please give the base value as well, e.g., '41.3% vs. 37.0% base and 524.2% SFT decoder.'
  3. [Sec. IV-C] No code, data, or trained checkpoints are linked. Given the synthetic data construction and replay-cache pipelines, a release of the construction code would materially improve reproducibility.
  4. [Sec. V-C] The sentence 'REDDIT Full stays close to the base model on CV-en and ASCEND-en' is too strong given the 4.3-point CV-en and 9.6-point ASCEND-zh degradations in Table VII; consider qualifying with 'closer than the baselines' or reporting a paired significance test.

Circularity Check

0 steps flagged

No significant circularity: timestamp targets are externally derived from splice offsets; retention is assessed on held-out OOD MER, not by construction.

full rationale

REDDIT's derivation chain is not circular. The correction supervision b_i is external: Eq. (11)-(12) compute start/end times directly from inserted non-speech and speech span durations and quantize them via q(·); these targets are not derived from the model's own outputs. L_time (Eq. 6) is a standard cross-entropy to these externally defined timestamp labels under the cached replay context, so the reported mIoU/AAS gains on test_long_gap/test_gap are predictions on disjoint examples (Table I), not refits of training constants. The anti-forgetting component L_text (Eq. 7) regularizes toward the frozen base distribution, but the claim that this preserves recognition is not definitional: the paper tests it on held-out OOD sets (Tables VI-VII) and the results are not the training loss. Indeed, REDDIT Full shows nonzero degradation on no-gap OOD (CV-en MER 37.0→41.3, ASCEND-zh 53.9→63.5), which shows the method does not trivially inherit the teacher's behavior by construction. These numbers are a transfer/robustness weakness, not circularity. The self-referential elements noted in the manuscript—pseudo-text from F_{θ0}, cached-replay pre-filtering for training examples, and the matched synthetic gap splits—are limitations or selection effects; none reduces a headline result to its own input. Other self-citations (e.g., [26],[27],[28]) are contextual/related-work or future-work references and are not load-bearing for the timestamp-correction or retention conclusions. The key assumption that teacher-forced KL under replay/edited-prefix contexts bounds free-decoding divergence is empirically tested rather than asserted by construction, so it falls under correctness risk rather than circularity.

Axiom & Free-Parameter Ledger

4 free parameters · 6 axioms · 0 invented entities

The ledger shows what the paper contributes vs. inherits. Free parameters are the hand-chosen Stage-1 weights (lambda_time=1, lambda_text=5), a validation-behavior checkpoint selection rule that directly shapes the headline Full-pipeline numbers, standard training hyperparameters, and a baseline-only RTF corruption probability. The load-bearing axioms are: VAD boundary accuracy underwrites the 'exact' splice targets (Eq. 11); the cached replay context (Eq. 2-5) proxies inference decoding; the KL anchor (Eq. 7) preserves free-decoding recognition — partially contradicted by Table VII; q(·) quantization assumes ~20 ms timestamp granularity; synthetic gap splicing (Eq. 10) represents real non-speech conditions; and only structurally correct replays are correctable. No invented entities.

free parameters (4)
  • stage1_loss_weights = lambda_time=1, lambda_text=5
    Chosen by hand (Sec. IV-C); no sensitivity analysis reported. The headline tradeoff between timestamp correction (L_time) and retention (L_text) depends on these weights.
  • checkpoint_selection_rule = best full-pipeline checkpoint by validation behavior
    Sec. IV-C: 'We select the reported full-pipeline checkpoint from this short refinement trajectory by validation behavior.' A hand-chosen selection rule that can inflate headline numbers; no seed averaging or typical-step reporting.
  • train_hyperparameters = lr=1e-5, batch=64, 10 warmup steps, AdamW
    Standard post-training hyperparameters (Sec. IV-C); not ablated.
  • rtf_corruption_probability = p_rtf=0.2
    Hyperparameter for the In-Sync-inspired Reduced Teacher Forcing baseline (Sec. IV-B); not part of REDDIT itself.
axioms (6)
  • domain assumption VAD-trimmed speech span boundaries (a_k in Eq. 10) are accurate enough that splice-derived targets tau^s_k, tau^e_k (Eq. 11) are exact reference times.
    Sec. III-F: correction targets are computed from the splicing schedule of VAD-trimmed spans; if the VAD (rVAD, [19]) biases onsets/offsets, the 'exact' supervision inherits that bias. VAD parameters are not reported.
  • domain assumption The cached base-model replay trajectory y_tilde_i (Eq. 2) is a sufficient proxy for the student's inference-time decoding context.
    Sec. III-B: Stage 1 conditions on y_tilde_i,<t and never rolls the student out ('an offline approximation to extreme scheduled sampling'); the learned timestamp transitions depend on this offline context matching deployment.
  • domain assumption Matching the frozen teacher's non-timestamp distribution under replay/edited-prefix contexts (L_text, Eq. 7) preserves recognition under free decoding outside the correction set.
    This is the anti-forgetting premise. Table VII shows partial violations (CV-en 37.0->41.3, ASCEND-zh 53.9->63.5 no-gap MER); no error bars; small OOD sets (214-2997 samples, Table II).
  • domain assumption Timestamp tokens quantize time (Whisper convention, ~0.02 s) and q(·) rounding at Eq. 12 is negligible relative to reported MAE scales (0.06-0.27 s).
    Eq. 12; quantization error ~20 ms is acceptable for the reported metrics, but this holds only for Whisper-style timestamp-token models (Sec. III-A).
  • domain assumption Synthetic gap-spliced audio (Eq. 10: g0 + a1 + g1 + ... + gK) is representative of real non-speech conditions in deployment.
    Sec. IV-A: training and in-domain eval use the same VAD+gap insertion; even OOD gap-stressed sets (Table II) use the same insertion procedure, so no evidence exists for naturally long-form audio with heterogeneous non-speech.
  • domain assumption Consecutive timestamp tokens delimit segment boundaries and each valid replay has exactly 2K timestamps aligned 1:1 with constructed boundaries.
    Sec. III-A/III-F: the method corrects positions, not the number or ordering of segments; replays violating this are discarded by pre-filtering, so structural segmentation errors are out of scope.

pith-pipeline@v1.3.0-alltime-deepseek · 15451 in / 25043 out tokens · 256484 ms · 2026-08-02T08:28:35.135680+00:00 · methodology

0 comments
read the original abstract

Modern autoregressive ASR systems can emit timestamps as decoded tokens, enabling timestamped transcription without frame-level aligners or inference-time post-processing. We show that these generated timestamps can drift across long non-speech spans: the transcript may remain plausible, but the decoded time axis drifts away from the audio. We study this non-speech-induced timestamp drift with self-built gap and long-gap benchmarks across 15 evaluated timestamp-producing ASR and audio-language systems. Naive timestamp-corrected fine-tuning improves alignment but can severely degrade non-target ASR behavior, exposing a forgetting problem. We propose REDDIT(REplay-based Distribution eDITing), a lightweight two-stage post-training framework that corrects timestamps while avoiding this catastrophic forgetting: it first edits timestamp targets under the model's own replayed decoder context while matching the frozen base distribution on non-timestamp tokens, then applies a short edited-prefix refinement stage. In this framework, we construct correction supervision without human transcripts or human timestamp annotations by combining VAD-trimmed speech spans with inserted non-speech gaps and known concatenation offsets. On Whisper-tiny, 34.9 hours of targeted correction audio used and only 1.6% of model parameters updated, raising long-gap mIoU from 38.7% to 95.0% and reducing mixed-gap out-of-domain AAS from 2752 ms to 223 ms while preserving CV-en MER at 41.3% (versus 524.2% for ordinary SFT decoder tuning).

Figures

Figures reproduced from arXiv: 2607.05364 by Chan-Jan Hsu, Cheng-Kang Chou, Hung-yi Lee, Ke-Han Lu, Ming-To Chuang.

Figure 1
Figure 1. Figure 1: Overview of the REDDIT framework. A cached base-model replay sequence serves as the decoder context. Timestamp targets are edited based on [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

61 extracted references · 26 linked inside Pith

  1. [1]

    Robust speech recognition via large-scale weak supervision,

    A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” arXiv preprint arXiv:2212.04356, 2022

    Pith/arXiv arXiv 2022

  2. [2]

    Com- mon V oice: A massively-multilingual speech corpus,

    R. Ardila, M. Branson, K. Davis, M. Henretty, M. Kohler, J. Meyer, R. Morais, L. Saunders, F. M. Tyers, and G. Weber, “Com- mon V oice: A massively-multilingual speech corpus,” arXiv preprint arXiv:1912.06670, 2019

    Pith/arXiv arXiv 1912

  3. [3]

    Word level timestamp generation for automatic speech recognition and translation,

    K. Hu, K. Puvvada, E. Rastorgueva, Z. Chen, H. Huang, S. Ding, K. Dhawan, H. Xu, J. Balam, and B. Ginsburg, “Word level timestamp generation for automatic speech recognition and translation,” arXiv preprint arXiv:2505.15646, 2025

    Pith/arXiv arXiv 2025

  4. [4]

    In-Sync: Adaptation of speech aware large language models for ASR with word level timestamp predictions,

    X. Fan, V . Sunder, S. Thomas, M. Hasegawa-Johnson, B. Kingsbury, and G. Saon, “In-Sync: Adaptation of speech aware large language models for ASR with word level timestamp predictions,” arXiv preprint arXiv:2604.22817, 2026

    Pith/arXiv arXiv 2026

  5. [5]

    Listening between the frames: Bridging temporal gaps in large audio-language models,

    H. Wang, Y . Li, S. Ma, H. Liu, and X. Wang, “Listening between the frames: Bridging temporal gaps in large audio-language models,” arXiv preprint arXiv:2511.11039, 2025

    arXiv 2025

  6. [6]

    Distilling the knowledge in a neural network,

    G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” arXiv preprint arXiv:1503.02531, 2015

    Pith/arXiv arXiv 2015

  7. [7]

    Learning without forgetting,

    Z. Li and D. Hoiem, “Learning without forgetting,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018

    2018

  8. [8]

    Overcoming catastrophic forgetting in neural networks,

    J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, et al., “Overcoming catastrophic forgetting in neural networks,” Proceedings of the National Academy of Sciences, 2017

    2017

  9. [9]

    Gradient episodic memory for continual learning,

    D. Lopez-Paz and M. Ranzato, “Gradient episodic memory for continual learning,” inProc. NeurIPS, 2017

    2017

  10. [10]

    Dark experience for general continual learning: A strong, simple baseline,

    P. Buzzega, M. Boschini, A. Porrello, D. Abati, and S. Calderara, “Dark experience for general continual learning: A strong, simple baseline,” in Proc. NeurIPS, 2020

    2020

  11. [11]

    WhisperX: Time-accurate speech transcription of long-form audio,

    M. Bain, J. Huh, T. Han, and A. Zisserman, “WhisperX: Time-accurate speech transcription of long-form audio,” INTERSPEECH, 2023

    2023

  12. [12]

    Montreal Forced Aligner: Trainable text-speech alignment using Kaldi,

    M. McAuliffe, M. Socolof, S. Mihuc, M. Wagner, and M. Sonderegger, “Montreal Forced Aligner: Trainable text-speech alignment using Kaldi,” INTERSPEECH, 2017

    2017

  13. [13]

    The Kaldi speech recognition toolkit,

    D. Povey, A. Ghoshal, G. Boulianne, L. Burget, O. Glembek, N. Goel, M. Hannemann, et al., “The Kaldi speech recognition toolkit,” inProc. ASRU, 2011

    2011

  14. [14]

    Tradition or inno- vation: A comparison of modern ASR methods for forced alignment,

    R. Rousso, E. Cohen, J. Keshet, and E. Chodroff, “Tradition or inno- vation: A comparison of modern ASR methods for forced alignment,” arXiv preprint arXiv:2406.19363, 2024

    Pith/arXiv arXiv 2024

  15. [15]

    CTC- segmentation of large corpora for German end-to-end speech recogni- tion,

    L. K ¨urzinger, D. Winkelbauer, L. Li, T. Watzel, and G. Rigoll, “CTC- segmentation of large corpora for German end-to-end speech recogni- tion,” arXiv preprint arXiv:2007.09127, 2020

    Pith/arXiv arXiv 2007

  16. [16]

    LLM- ForcedAligner: A non-autoregressive and accurate LLM-based forced aligner for multilingual and long-form speech,

    B. Mu, X. Shi, X. Wang, H. Liu, J. Xu, and L. Xie, “LLM- ForcedAligner: A non-autoregressive and accurate LLM-based forced aligner for multilingual and long-form speech,” arXiv preprint arXiv:2601.18220, 2026

    arXiv 2026

  17. [17]

    CrisperWhisper: Accu- rate timestamps on verbatim speech transcriptions,

    L. Wagner, B. Thallinger, and M. Zusag, “CrisperWhisper: Accu- rate timestamps on verbatim speech transcriptions,” arXiv preprint arXiv:2408.16589, 2024

    Pith/arXiv arXiv 2024

  18. [18]

    Whisper has an internal word aligner,

    S.-L. Yeh, Y . Meng, and H. Tang, “Whisper has an internal word aligner,” arXiv preprint arXiv:2509.09987, 2025

    Pith/arXiv arXiv 2025

  19. [19]

    rV AD: An unsupervised segment-based robust voice activity detection method,

    Z.-H. Tan, A. K. Sarkar, and N. Dehak, “rV AD: An unsupervised segment-based robust voice activity detection method,” arXiv preprint arXiv:1906.03588, 2019

    Pith/arXiv arXiv 1906

  20. [20]

    Investigation of Whisper ASR hallucinations induced by non-speech audio,

    M. Bara ´nski, J. Jasi´nski, J. Bartolewska, S. Kacprzak, M. Witkowski, and K. Kowalczyk, “Investigation of Whisper ASR hallucinations induced by non-speech audio,” ICASSP, 2025

    2025

  21. [21]

    Calm-Whisper: Reduce Whisper hallucination on non-speech by calm- ing crazy heads down,

    Y . Wang, A. Alhmoud, S. Alsahly, M. Alqurishi, and M. Ravanelli, “Calm-Whisper: Reduce Whisper hallucination on non-speech by calm- ing crazy heads down,” arXiv preprint arXiv:2505.12969, 2025

    Pith/arXiv arXiv 2025

  22. [22]

    Lost in transcription, found in distribution shift: Demystifying hallucination in speech foundation models,

    H. Atwany, A. Waheed, R. Singh, M. Choudhury, and B. Raj, “Lost in transcription, found in distribution shift: Demystifying hallucination in speech foundation models,” arXiv preprint arXiv:2502.12414, 2025

    Pith/arXiv arXiv 2025

  23. [23]

    AHa-Bench: Benchmarking audio hallucinations in large audio-language models,

    X. Cheng, D. Fu, C. Wen, S. Yu, Z. Wang, S. Ji, S. Arora, T. Jin, S. Watanabe, and Z. Zhao, “AHa-Bench: Benchmarking audio hallucinations in large audio-language models,” inProc. NeurIPS, 2025

    2025

  24. [24]

    From text metrics to model internals: A study of Whisper ASR hallucination detection,

    J. Jasi ´nski, M. Bara ´nski, J. Bartolewska, M. Witkowski, and K. Kowal- czyk, “From text metrics to model internals: A study of Whisper ASR hallucination detection,” arXiv preprint arXiv:2606.23060, 2026

    Pith/arXiv arXiv 2026

  25. [25]

    Whisper hallu- cination detection and mitigation via hidden representation steering and sparse autoencoders,

    G. Aparin, V . Popov, T. Sadekova, and A. Yermekova, “Whisper hallu- cination detection and mitigation via hidden representation steering and sparse autoencoders,” arXiv preprint arXiv:2606.07473, 2026

    Pith/arXiv arXiv 2026

  26. [26]

    A self-refining framework for enhanc- ing ASR using TTS-synthesized data,

    C.-K. Chou, C.-J. Hsu, H.-L. Chung, L.-H. Tseng, H.-C. Cheng, Y .-K. Fu, K. P. Huang, and H.-Y . Lee, “A self-refining framework for enhanc- ing ASR using TTS-synthesized data,” arXiv preprint arXiv:2506.11130, 2025

    Pith/arXiv arXiv 2025

  27. [27]

    Channel-aware domain-adaptive generative adversarial network for robust speech recognition,

    C.-C. Wang, L.-W. Chen, C.-K. Chou, H.-S. Lee, B. Chen, and H.-M. Wang, “Channel-aware domain-adaptive generative adversarial network for robust speech recognition,” ICASSP, 2025

    2025

  28. [28]

    Speech-IFEval: Evaluating instruction-following and quantifying catastrophic forgetting in speech- aware language models,

    K.-H. Lu, C.-Y . Kuan, and H.-Y . Lee, “Speech-IFEval: Evaluating instruction-following and quantifying catastrophic forgetting in speech- aware language models,” arXiv preprint arXiv:2505.19037, 2025

    Pith/arXiv arXiv 2025

  29. [29]

    Scheduled sampling for sequence prediction with recurrent neural networks,

    S. Bengio, O. Vinyals, N. Jaitly, and N. Shazeer, “Scheduled sampling for sequence prediction with recurrent neural networks,” Advances in Neural Information Processing Systems, 2015

    2015

  30. [30]

    Dynamic-SUPERB Phase-2: A collaboratively expanding benchmark for measuring the capabilities of spoken language models with 180 tasks,

    C.-Y . Huang, W.-C. Chen, S.-W. Yang, A. T. Liu, C.-A. Li, Y .-X. Lin, W.-C. Tseng, et al., “Dynamic-SUPERB Phase-2: A collaboratively expanding benchmark for measuring the capabilities of spoken language models with 180 tasks,” inProc. ICLR, 2025

    2025

  31. [31]

    SUPERB: Speech processing universal performance benchmark,

    S.-W. Yang, P.-H. Chi, Y .-S. Chuang, C.-I. J. Lai, K. Lakhotia, Y . Y . Lin, et al., “SUPERB: Speech processing universal performance benchmark,” inProc. INTERSPEECH, 2021

    2021

  32. [32]

    ML-SUPERB: Multilingual speech universal performance benchmark,

    J. Shi, D. Berrebbi, W. Chen, H.-L. Chung, E.-P. Hu, W.-P. Huang, X. Chang, et al., “ML-SUPERB: Multilingual speech universal performance benchmark,” inProc. INTERSPEECH, 2023

    2023

  33. [33]

    HEAR: Holistic evaluation of audio representations,

    J. Turian, J. Shier, H. R. Khan, B. Raj, B. W. Schuller, C. J. Steinmetz, et al., “HEAR: Holistic evaluation of audio representations,” inProc. NeurIPS Competitions and Demonstrations Track, 2022

    2022

  34. [34]

    AIR-Bench: Benchmarking large audio-language models via generative comprehen- sion,

    Q. Yang, J. Xu, W. Liu, Y . Chu, Z. Jiang, X. Zhou, et al., “AIR-Bench: Benchmarking large audio-language models via generative comprehen- sion,” arXiv preprint arXiv:2402.07729, 2024

    Pith/arXiv arXiv 2024

  35. [35]

    AudioBench: A universal benchmark for audio large language models,

    B. Wang, X. Zou, G. Lin, S. Sun, Z. Liu, W. Zhang, Z. Liu, A. Aw, and N. F. Chen, “AudioBench: A universal benchmark for audio large language models,” arXiv preprint arXiv:2406.16020, 2024

    Pith/arXiv arXiv 2024

  36. [36]

    MMAU: A massive multi-task audio understanding and reasoning benchmark,

    S. Sakshi, U. Tyagi, S. Kumar, A. Seth, R. Selvakumar, O. Nieto, R. Duraiswami, S. Ghosh, and D. Manocha, “MMAU: A massive multi-task audio understanding and reasoning benchmark,” inProc. ICLR, 2025

    2025

  37. [37]

    AudioMarathon: A comprehensive benchmark for long-context audio understanding and efficiency in audio LLMs,

    P. He, Z. Wen, Y . Wang, Y . Wang, X. Liu, J. Huang, Z. Lei, et al., “AudioMarathon: A comprehensive benchmark for long-context audio understanding and efficiency in audio LLMs,” arXiv preprint arXiv:2510.07293, 2025

    arXiv 2025

  38. [38]

    Not in Sync: Unveiling temporal bias in audio chat models,

    J. Yao, S. Liu, Y . Wang, R. Cheng, L. Mei, B. Bi, Z. Xiong, and X. Cheng, “Not in Sync: Unveiling temporal bias in audio chat models,” arXiv preprint arXiv:2510.12185, 2025

    arXiv 2025

  39. [39]

    SpotSound: Enhancing large audio-language models with fine-grained temporal grounding,

    L. Sun, X. Zhou, Z. Li, Y . Zhang, Y . Wang, and W. Xie, “SpotSound: Enhancing large audio-language models with fine-grained temporal grounding,” arXiv preprint arXiv:2604.13023, 2026

    Pith/arXiv arXiv 2026

  40. [40]

    ChronosAudio: A comprehensive long-audio benchmark for evaluating audio-large language models,

    K. Luo, L. Lin, Y . Zhang, M. Aloqaily, J. Tao, D. Wang, et al., “ChronosAudio: A comprehensive long-audio benchmark for evaluating audio-large language models,” arXiv preprint arXiv:2601.04876, 2026

    Pith/arXiv arXiv 2026

  41. [41]

    STAR-Bench: Probing deep spatio-temporal reasoning as audio 4D intelligence,

    Z. Liu, Z. Niu, Q. Xiao, Z. Zheng, R. Yuan, Y . Zang, Y . Cao, et al., “STAR-Bench: Probing deep spatio-temporal reasoning as audio 4D intelligence,” arXiv preprint arXiv:2510.24693, 2025

    arXiv 2025

  42. [42]

    Audio Set: An ontology and human-labeled dataset for audio events,

    J. F. Gemmeke, D. P. W. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, “Audio Set: An ontology and human-labeled dataset for audio events,” inProc. ICASSP, 2017

    2017

  43. [43]

    AudioCaps: Generating captions for audios in the wild,

    C. D. Kim, B. Kim, H. Lee, and G. Kim, “AudioCaps: Generating captions for audios in the wild,” inProc. NAACL-HLT, 2019

    2019

  44. [44]

    Clotho: An audio captioning dataset,

    K. Drossos, S. Lipping, and T. Virtanen, “Clotho: An audio captioning dataset,” inProc. ICASSP, 2020

    2020

  45. [45]

    Clotho- AQA: A crowdsourced dataset for audio question answering,

    S. Lipping, P. Sudarsanam, K. Drossos, and T. Virtanen, “Clotho- AQA: A crowdsourced dataset for audio question answering,” inProc. EUSIPCO, 2022

    2022

  46. [46]

    ToxicTone: A Mandarin audio dataset annotated for toxicity and toxic utterance tonality,

    Y .-X. Luo, Y .-C. Lin, M.-T. Chuang, J.-H. Chen, I.-N. Tsai, P. X. Kiew, Y .-H. Huang, C.-F. Liu, Y .-C. Chen, B.-H. Feng, W. Ren, and H.-Y . Lee, “ToxicTone: A Mandarin audio dataset annotated for toxicity and toxic utterance tonality,” inProc. INTERSPEECH, 2025

    2025

  47. [47]

    How to learn a new language? An efficient solution for self-supervised learning models unseen languages adaption in low-resource scenario,

    S.-H. Wang, Z.-C. Chen, J. Shi, M.-T. Chuang, G.-T. Lin, K.-P. Huang, D. Harwath, S.-W. Li, and H.-Y . Lee, “How to learn a new language? An efficient solution for self-supervised learning models unseen languages adaption in low-resource scenario,” arXiv preprint arXiv:2411.18217, 2024

    Pith/arXiv arXiv 2024

  48. [48]

    Parameter-efficient transfer learning for NLP,

    N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. de Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly, “Parameter-efficient transfer learning for NLP,” inProc. ICML, 2019

    2019

  49. [49]

    LoRA: Low-rank adaptation of large language models,

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “LoRA: Low-rank adaptation of large language models,” inProc. ICLR, 2022

    2022

  50. [50]

    Prefix-tuning: Optimizing continuous prompts for generation,

    X. L. Li and P. Liang, “Prefix-tuning: Optimizing continuous prompts for generation,” inProc. ACL, 2021

    2021

  51. [51]

    The power of scale for parameter-efficient prompt tuning,

    B. Lester, R. Al-Rfou, and N. Constant, “The power of scale for parameter-efficient prompt tuning,” inProc. EMNLP, 2021

    2021

  52. [52]

    BitFit: Simple parameter- efficient fine-tuning for transformer-based masked language-models,

    E. Ben-Zaken, S. Ravfogel, and Y . Goldberg, “BitFit: Simple parameter- efficient fine-tuning for transformer-based masked language-models,” in Proc. ACL, 2022

    2022

  53. [53]

    Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning,

    H. Liu, D. Tam, M. Muqeeth, J. Mohta, T. Huang, M. Bansal, and C. Raffel, “Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning,” inProc. NeurIPS, 2022

    2022

  54. [54]

    ASCEND: A spontaneous Chinese-English dataset for code- switching in multi-turn conversation,

    H. Lovenia, S. Cahyawijaya, G. I. Winata, P. Xu, X. Yan, Z. Liu, R. Frieske, T. Yu, W. Dai, E. J. Barezi, Q. Chen, X. Ma, B. E. Shi, and P. Fung, “ASCEND: A spontaneous Chinese-English dataset for code- switching in multi-turn conversation,” inProc. LREC, 2022

    2022

  55. [55]

    Distil-Whisper: Robust knowledge distillation via large-scale pseudo labelling,

    S. Gandhi, P. von Platen, and A. M. Rush, “Distil-Whisper: Robust knowledge distillation via large-scale pseudo labelling,” arXiv preprint arXiv:2311.00430, 2023

    Pith/arXiv arXiv 2023

  56. [56]

    Qwen2-Audio technical report,

    Y . Chu, J. Xu, Q. Yang, H. Wei, X. Wei, Z. Guo, Y . Leng, Y . Lv, J. He, J. Lin, C. Zhou, and J. Zhou, “Qwen2-Audio technical report,” arXiv preprint arXiv:2407.10759, 2024

    Pith/arXiv arXiv 2024

  57. [57]

    Qwen2.5-Omni technical report,

    J. Xu, Z. Guo, J. He, H. Hu, T. He, S. Bai, K. Chen, J. Wang, Y . Fan, K. Dang, B. Zhang, X. Wang, Y . Chu, and J. Lin, “Qwen2.5-Omni technical report,” arXiv preprint arXiv:2503.20215, 2025

    Pith/arXiv arXiv 2025

  58. [58]

    Qwen3-Omni technical report,

    J. Xu, Z. Guo, H. Hu, Y . Chu, X. Wang, J. He, Y . Wang, X. Shi, T. He, X. Zhu, Y . Lv, Y . Wang, D. Guo, et al., “Qwen3-Omni technical report,” arXiv preprint arXiv:2509.17765, 2025

    Pith/arXiv arXiv 2025

  59. [59]

    MOSS-Audio technical report,

    C. Yang, C. Yu, H. Chen, J. Zhu, J. Chen, K. Chen, W. Wang, Y . Wang, Y . Jiang, Y . Jiang, Z. Lin, Z. Chen, Z. Fei, et al., “MOSS-Audio technical report,” arXiv preprint arXiv:2606.01802, 2026

    Pith/arXiv arXiv 2026

  60. [60]

    VIBEVOICE-ASR technical report,

    Z. Peng, J. Yu, Y . Chang, Z. Wang, L. Dong, Y . Hao, Y . Tu, C. Yang, W. Wang, S. Xu, Y . Sun, H. Bao, W. Xu, et al., “VIBEVOICE-ASR technical report,” arXiv preprint arXiv:2601.18184, 2026

    arXiv 2026

  61. [61]

    DeSTA2.5-Audio: Toward general-purpose large audio language model with self-generated cross-modal alignment,

    K.-H. Lu, Z. Chen, S.-W. Fu, C.-H. H. Yang, S.-F. Huang, C.-K. Yang, C.-E. Yu, C.-W. Chen, W.-C. Chen, C.-Y . Huang, Y .-C. Lin, Y .-X. Lin, C.-A. Fu, et al., “DeSTA2.5-Audio: Toward general-purpose large audio language model with self-generated cross-modal alignment,” arXiv preprint arXiv:2507.02768, 2025

    arXiv 2025