The reviewed record of science sign in
Pith

arxiv: 2607.05364 · v1 · pith:N2EPANPE · submitted 2026-07-06 · cs.CL · cs.AI· cs.SD

REDDIT: Correcting Model-Generated Timestamp Drift in ASR without Forgetting via Replay-Based Distribution Editing

Cheng-Kang Chou , Ming-To Chuang , Ke-Han Lu , Chan-Jan Hsu , Hung-yi Lee This is my paper

Reviewed by Pith2026-07-07 14:37 UTCglm-5.2pith:N2EPANPEopen to challenge →

classification cs.CL cs.AIcs.SD
keywords timestamp driftautoregressive ASRcatastrophic forgettingdistribution editingreplay-based learningnon-speech robustnessWhispermodel editing
0
0 comments X

The pith

Fixing Timestamp Drift in ASR Without Breaking Recognition

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper identifies a specific failure mode in modern autoregressive ASR systems that emit timestamps as decoded tokens: when speech follows long non-speech spans (silence, pauses, background), the model can transcribe the words correctly but place them at the wrong time on the audio timeline. The authors call this 'non-speech-induced timestamp drift' and show it is distinct from hallucination — the words are real, just temporally misplaced. They benchmark 15 timestamp-producing systems and find the problem is widespread. The central contribution is REDDIT (Replay-based Distribution Editing), a two-stage post-training method that corrects timestamp predictions while preserving the model's original text-recognition behavior. The mechanism works as follows: the frozen base model decodes each correction example once, and this trajectory is cached. During training, timestamp token positions in the cached trajectory are overwritten with known-correct boundary targets (derived from synthetic audio construction), while all non-timestamp positions are trained to match the frozen base model's distribution via KL divergence. A short second stage refines the model using the corrected timestamps as decoder context. The method requires no human transcripts or timestamp annotations — correction targets come from concatenating VAD-trimmed speech with inserted non-speech gaps at known offsets. On Whisper-tiny, updating only 1.6% of parameters (0.59M), REDDIT raises long-gap temporal overlap from 38.7% to 95.0% while preserving out-of-domain recognition error at 41.3% MER, compared to 524.2% for ordinary decoder fine-tuning.

Core claim

The key structural insight is that timestamp correction can be decomposed into a token-level distribution-editing problem: at timestamp positions, the student learns corrected targets under a cached replay context; at all other positions, the student is anchored to the frozen teacher's distribution under that same context. This separation prevents catastrophic forgetting because non-timestamp behavior is explicitly preserved rather than left to drift during targeted fine-tuning. The cached replay context — the base model's own decoded trajectory, including its drifted timestamps — serves as a fixed conditioning prefix so that the student learns corrected timestamp transitions conditioned on,

What carries the argument

Cached replay-conditioned distribution editing: freeze the base model's decoded trajectory, overwrite only timestamp token targets with known-correct boundaries, and KL-anchor all non-timestamp positions to the frozen base distribution. Two stages: Stage 1 edits under replay context; Stage 2 refines under edited-prefix context with the Stage-1 checkpoint as KL teacher.

Load-bearing premise

The method assumes that the cached base-model replay trajectory (generated offline) is a good enough proxy for the decoder prefixes the model will actually produce at inference time. If the replay distribution diverges from real deployment-time prefixes, the corrected timestamp transitions may not transfer.

What would settle it

If REDDIT-corrected models still drift on non-speech gaps drawn from distributions different from the training correction set (e.g., music, noise types, or gap patterns not represented in the 34.9 hours of correction audio), the replay-context proxy would be shown insufficient.

Figures

Figures reproduced from arXiv: 2607.05364 by Chan-Jan Hsu, Cheng-Kang Chou, Hung-yi Lee, Ke-Han Lu, Ming-To Chuang.

Figure 1
Figure 1. Figure 1: Overview of the REDDIT framework. A cached base-model replay sequence serves as the decoder context. Timestamp targets are edited based on [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
read the original abstract

Modern autoregressive ASR systems can emit timestamps as decoded tokens, enabling timestamped transcription without frame-level aligners or inference-time post-processing. We show that these generated timestamps can drift across long non-speech spans: the transcript may remain plausible, but the decoded time axis drifts away from the audio. We study this non-speech-induced timestamp drift with self-built gap and long-gap benchmarks across 15 evaluated timestamp-producing ASR and audio-language systems. Naive timestamp-corrected fine-tuning improves alignment but can severely degrade non-target ASR behavior, exposing a forgetting problem. We propose REDDIT(REplay-based Distribution eDITing), a lightweight two-stage post-training framework that corrects timestamps while avoiding this catastrophic forgetting: it first edits timestamp targets under the model's own replayed decoder context while matching the frozen base distribution on non-timestamp tokens, then applies a short edited-prefix refinement stage. In this framework, we construct correction supervision without human transcripts or human timestamp annotations by combining VAD-trimmed speech spans with inserted non-speech gaps and known concatenation offsets. On Whisper-tiny, 34.9 hours of targeted correction audio used and only 1.6% of model parameters updated, raising long-gap mIoU from 38.7% to 95.0% and reducing mixed-gap out-of-domain AAS from 2752 ms to 223 ms while preserving CV-en MER at 41.3% (versus 524.2% for ordinary SFT decoder tuning).

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 5 minor

Summary. The paper identifies non-speech-induced timestamp drift in autoregressive ASR as a distinct failure mode: models can produce plausible transcripts while placing them at wrong temporal positions after long non-speech gaps. The authors propose REDDIT, a two-stage post-training framework that edits timestamp targets under cached base-model replay context while anchoring non-timestamp behavior via KL divergence to the frozen base model, followed by a short edited-prefix refinement stage. Correction data is constructed without human annotation by VAD-trimming speech spans and inserting non-speech gaps with known concatenation offsets. Experiments on Whisper-tiny show long-gap mIoU improving from 38.7% to 95.0% while preserving CV-en MER at 41.3% (versus 524.2% for SFT decoder tuning). The paper also benchmarks 15 timestamp-producing systems and demonstrates scaling to Whisper-large-v3.

Significance. The paper makes several valuable contributions. The diagnostic benchmark across 15 systems (Table IV) is a useful community resource that cleanly separates temporal grounding from transcript quality and hallucination. The method is parameter-efficient (1.6% of Whisper-tiny parameters), requires no human timestamp annotation, and adds no inference-time aligner or post-processing. The ablation design is thorough: Table V properly isolates replay context, KL anchoring, teacher forcing, and edited-prefix refinement. The forgetting-resistant framing is well-motivated, and the two-stage design is a clean application of distribution-editing ideas to the token level. The scaling results on Whisper-large-v3 (Table VIII) are encouraging.

major comments (1)
  1. §IV-A, Tables I–II: Every timestamp evaluation in the paper—both in-domain (test_gap, test_long_gap) and OOD (ASCEND gap stress, CV-en-min gap stress)—is constructed using the same VAD-trim + gap-insertion pipeline (Eq. 10–12) as the 34.9h training set. There is no evaluation on audio with naturally occurring long silences (e.g., meeting recordings, podcasts, audiobooks with real pauses). Real non-speech regions differ acoustically from inserted ones—they may contain background noise, reverberant tails, music, or partial speech—which could change both the base model's replay trajectory and the corrected model's behavior. The headline 95.0% long-gap mIoU may therefore reflect performance on the specific construction pipeline rather than general timestamp correction ability. This is an external validity concern, not an internal inconsistency: the method is sound and the ablations are well-
minor comments (5)
  1. §III-F: The pre-filtering rules for cached teacher replays (removing hallucinations, repetitions, boilerplate, etc.) are described qualitatively but not quantified. Reporting how many examples were filtered out of the raw set would aid reproducibility.
  2. §IV-C: The choice of λ_time=1 and λ_text=5 is not justified. A brief sensitivity analysis or rationale would strengthen the claim that these are not finely tuned.
  3. Table IV: The caption states 'GREEN AND RED MARK BEST AND WORST VALUES' but no color coding is visible in the text rendering. If colors are present in the actual PDF, this is fine; otherwise, consider bold/underline formatting.
  4. §V-D: The claim that 'Stage 2 gives modest additional timing and OOD alignment gains' on Whisper-tiny is supported by Table V, but the Stage-2 training duration is not specified. Stating how many steps or epochs Stage 2 uses would help readers assess the 'short refinement' claim.
  5. No standard deviations or multiple-seed results are reported for the intervention experiments. Given that the differences between REDDIT Stage 1 and REDDIT Full are small on Whisper-tiny (e.g., 94.6% vs 95.0% long-gap mIoU), variance information would help readers judge significance.

Simulated Author's Rebuttal

1 responses · 1 unresolved

We thank the referee for the careful reading and the positive assessment of our contributions. The referee raises a single major concern: all timestamp evaluations—both in-domain and OOD—are constructed using the same VAD-trim + gap-insertion pipeline as the training set, and there is no evaluation on audio with naturally occurring long silences. We address this below.

read point-by-point responses

  1. Referee: Every timestamp evaluation in the paper is constructed using the same VAD-trim + gap-insertion pipeline as the training set. There is no evaluation on audio with naturally occurring long silences. Real non-speech regions differ acoustically from inserted ones and could change both the base model's replay trajectory and the corrected model's behavior. The headline 95.0% long-gap mIoU may reflect performance on the specific construction pipeline rather than general timestamp correction ability.

    Authors: We agree that this is a legitimate external validity concern and that the current manuscript does not test on audio with naturally occurring long silences. The referee is correct that real non-speech regions can contain background noise, reverberant tails, music, or partial speech, and that these acoustic differences could affect both the base model's replay trajectory and the corrected model's behavior. We do not claim that the 95.0% long-gap mIoU generalizes to all naturally occurring silence distributions, and we will make sure the revised manuscript states this limitation explicitly rather than leaving it implicit in the data construction description. We will add a dedicated paragraph in Section IV-A acknowledging that all timestamp evaluations share the same gap-insertion construction and that evaluation on naturally occurring long silences (e.g., meeting recordings, podcasts) remains future work. We will also add a sentence to the abstract and conclusion noting that the evaluation is on controlled gap-insertion benchmarks. That said, we would like to clarify two points about the scope of the concern. First, the OOD evaluations in Tables VI–VII do vary the speech source (ASCEND code-switching data, CommonVoice-en-min) and the non-speech pool is disjoint between training and testing, so the method is not evaluated on identically distributed non-speech audio. The gap-insertion procedure is the same, but the speech content, language, and non-speech segments differ. Second, the method's design does not depend on the inserted gaps being acoustically clean: REDDIT edits timestamp targets under the base model's own replay context, so if the base model produces a different replay trajectory on real non-speech audio, the editing mechanism still applies—it overwrites the 2K- revision: no

standing simulated objections not resolved
  • We cannot add a full evaluation on naturally occurring long-silence audio within this revision cycle because constructing reliable timestamp references for such audio requires either forced alignment (which introduces its own boundary assumptions that conflict with our goal of evaluating native model-generated timestamps) or manual annotation, which is beyond what we can produce in the revision timeframe. We will acknowledge this gap honestly in the revised manuscript and list it as a priority for future work.

Circularity Check

0 steps flagged

No significant circularity found; derivation is self-contained with minor non-load-bearing self-citations.

full rationale

The paper's derivation chain is not circular. Timestamp correction targets (Eq. 11–12) are derived from known audio concatenation offsets, not from model outputs or fitted parameters. The student model must learn to map audio input to correct timestamp tokens without access to the splicing schedule at inference time — this is standard supervised learning, not a definitional reduction. The KL teacher (frozen base model F_θ0) provides independent supervision for non-timestamp positions, and the cached replay context is a fixed input rather than a target being fitted and then reported as prediction. Training and test splits are separate (Table I). The concern that all evaluation uses synthetically constructed gaps is a legitimate external validity limitation (real-world generalization is untested), but it is not circularity: the model is not evaluated on its own training data, and the prediction task does not reduce to the input by construction. Self-citations [26], [28], [46] are to related prior work by overlapping authors but are not load-bearing for the method's derivation — they appear in related work and future work sections, not in the core equations or the central claim. Score 1 reflects these minor self-citations that do not affect the derivation's independence.

Axiom & Free-Parameter Ledger

7 free parameters · 4 axioms · 0 invented entities

The paper introduces no new entities (particles, forces, dimensions, etc.). The free parameters are standard hyperparameters for fine-tuning. The axioms are domain assumptions specific to the ASR timestamp correction problem.

free parameters (7)
  • lambda_time = 1
    Weight for timestamp cross-entropy loss, chosen by the authors.
  • lambda_text = 5
    Weight for non-timestamp KL divergence loss, chosen by the authors.
  • learning_rate = 1e-5
    AdamW learning rate for post-training.
  • p_rtf = 0.2
    Probability of corrupting previous timestamp input in the RTF baseline.
  • KL temperature = 1.0
    Temperature for replay KL divergence.
  • batch_size = 64
    Training batch size.
  • warmup_steps = 10
    Learning rate warmup steps.
axioms (4)
  • domain assumption VAD-trimmed speech spans concatenated with non-speech gaps produce exact timestamp boundaries by construction.
    §III-F, Eq. 10-12. The correctness of edit targets depends on this.
  • domain assumption The cached base-model replay trajectory is a sufficient offline proxy for inference-time decoder prefixes.
    §III-B. The entire Stage-1 design rests on this assumption.
  • domain assumption Updating only last cross-attn + LNs is sufficient to correct timestamp drift without affecting other model behaviors.
    §III-E. The parameter-efficient update scope is a design choice that limits what the model can learn.
  • ad hoc to paper The pre-filtering rules for cached teacher replays (removing hallucinations, repetitions, boilerplate, etc.) produce clean training data.
    §III-F. The filtering rules are described qualitatively but not formalized, and their impact is not ablated.

pith-pipeline@v1.1.0-glm · 19709 in / 2359 out tokens · 183363 ms · 2026-07-07T14:37:02.903045+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

61 extracted references · 61 canonical work pages · 26 internal anchors

  1. [1]

    Robust Speech Recognition via Large-Scale Weak Supervision

    A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” arXiv preprint arXiv:2212.04356, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  2. [2]

    Common Voice: A Massively-Multilingual Speech Corpus

    R. Ardila, M. Branson, K. Davis, M. Henretty, M. Kohler, J. Meyer, R. Morais, L. Saunders, F. M. Tyers, and G. Weber, “Com- mon V oice: A massively-multilingual speech corpus,” arXiv preprint arXiv:1912.06670, 2019

    work page internal anchor Pith review Pith/arXiv arXiv 1912

  3. [3]

    Word Level Timestamp Generation for Automatic Speech Recognition and Translation

    K. Hu, K. Puvvada, E. Rastorgueva, Z. Chen, H. Huang, S. Ding, K. Dhawan, H. Xu, J. Balam, and B. Ginsburg, “Word level timestamp generation for automatic speech recognition and translation,” arXiv preprint arXiv:2505.15646, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  4. [4]

    In-Sync: Adaptation of Speech Aware Large Language Models for ASR with Word Level Timestamp Predictions

    X. Fan, V . Sunder, S. Thomas, M. Hasegawa-Johnson, B. Kingsbury, and G. Saon, “In-Sync: Adaptation of speech aware large language models for ASR with word level timestamp predictions,” arXiv preprint arXiv:2604.22817, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  5. [5]

    Listening between the frames: Bridging temporal gaps in large audio-language models,

    H. Wang, Y . Li, S. Ma, H. Liu, and X. Wang, “Listening between the frames: Bridging temporal gaps in large audio-language models,” arXiv preprint arXiv:2511.11039, 2025

    work page arXiv 2025

  6. [6]

    Distilling the Knowledge in a Neural Network

    G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” arXiv preprint arXiv:1503.02531, 2015

    work page internal anchor Pith review Pith/arXiv arXiv 2015

  7. [7]

    Learning without forgetting,

    Z. Li and D. Hoiem, “Learning without forgetting,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018

    work page 2018

  8. [8]

    Overcoming catastrophic forgetting in neural networks,

    J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, et al., “Overcoming catastrophic forgetting in neural networks,” Proceedings of the National Academy of Sciences, 2017

    work page 2017

  9. [9]

    Gradient episodic memory for continual learning,

    D. Lopez-Paz and M. Ranzato, “Gradient episodic memory for continual learning,” inProc. NeurIPS, 2017

    work page 2017

  10. [10]

    Dark experience for general continual learning: A strong, simple baseline,

    P. Buzzega, M. Boschini, A. Porrello, D. Abati, and S. Calderara, “Dark experience for general continual learning: A strong, simple baseline,” in Proc. NeurIPS, 2020

    work page 2020

  11. [11]

    WhisperX: Time-accurate speech transcription of long-form audio,

    M. Bain, J. Huh, T. Han, and A. Zisserman, “WhisperX: Time-accurate speech transcription of long-form audio,” INTERSPEECH, 2023

    work page 2023

  12. [12]

    Montreal Forced Aligner: Trainable text-speech alignment using Kaldi,

    M. McAuliffe, M. Socolof, S. Mihuc, M. Wagner, and M. Sonderegger, “Montreal Forced Aligner: Trainable text-speech alignment using Kaldi,” INTERSPEECH, 2017

    work page 2017

  13. [13]

    The Kaldi speech recognition toolkit,

    D. Povey, A. Ghoshal, G. Boulianne, L. Burget, O. Glembek, N. Goel, M. Hannemann, et al., “The Kaldi speech recognition toolkit,” inProc. ASRU, 2011

    work page 2011

  14. [14]

    Tradition or Innovation: A Comparison of Modern ASR Methods for Forced Alignment

    R. Rousso, E. Cohen, J. Keshet, and E. Chodroff, “Tradition or inno- vation: A comparison of modern ASR methods for forced alignment,” arXiv preprint arXiv:2406.19363, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  15. [15]

    CTC-Segmentation of Large Corpora for German End-to-end Speech Recognition

    L. K ¨urzinger, D. Winkelbauer, L. Li, T. Watzel, and G. Rigoll, “CTC- segmentation of large corpora for German end-to-end speech recogni- tion,” arXiv preprint arXiv:2007.09127, 2020

    work page internal anchor Pith review Pith/arXiv arXiv 2007

  16. [16]

    LLM- ForcedAligner: A non-autoregressive and accurate LLM-based forced aligner for multilingual and long-form speech,

    B. Mu, X. Shi, X. Wang, H. Liu, J. Xu, and L. Xie, “LLM- ForcedAligner: A non-autoregressive and accurate LLM-based forced aligner for multilingual and long-form speech,” arXiv preprint arXiv:2601.18220, 2026

    work page arXiv 2026

  17. [17]

    CrisperWhisper: Accurate Timestamps on Verbatim Speech Transcriptions

    L. Wagner, B. Thallinger, and M. Zusag, “CrisperWhisper: Accu- rate timestamps on verbatim speech transcriptions,” arXiv preprint arXiv:2408.16589, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  18. [18]

    Whisper Has an Internal Word Aligner

    S.-L. Yeh, Y . Meng, and H. Tang, “Whisper has an internal word aligner,” arXiv preprint arXiv:2509.09987, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  19. [19]

    rVAD: An Unsupervised Segment-Based Robust Voice Activity Detection Method

    Z.-H. Tan, A. K. Sarkar, and N. Dehak, “rV AD: An unsupervised segment-based robust voice activity detection method,” arXiv preprint arXiv:1906.03588, 2019

    work page internal anchor Pith review Pith/arXiv arXiv 1906

  20. [20]

    Investigation of Whisper ASR hallucinations induced by non-speech audio,

    M. Bara ´nski, J. Jasi´nski, J. Bartolewska, S. Kacprzak, M. Witkowski, and K. Kowalczyk, “Investigation of Whisper ASR hallucinations induced by non-speech audio,” ICASSP, 2025

    work page 2025

  21. [21]

    Calm-Whisper: Reduce Whisper Hallucination On Non-Speech By Calming Crazy Heads Down

    Y . Wang, A. Alhmoud, S. Alsahly, M. Alqurishi, and M. Ravanelli, “Calm-Whisper: Reduce Whisper hallucination on non-speech by calm- ing crazy heads down,” arXiv preprint arXiv:2505.12969, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  22. [22]

    Lost in Transcription, Found in Distribution Shift: Demystifying Hallucination in Speech Foundation Models

    H. Atwany, A. Waheed, R. Singh, M. Choudhury, and B. Raj, “Lost in transcription, found in distribution shift: Demystifying hallucination in speech foundation models,” arXiv preprint arXiv:2502.12414, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  23. [23]

    AHa-Bench: Benchmarking audio hallucinations in large audio-language models,

    X. Cheng, D. Fu, C. Wen, S. Yu, Z. Wang, S. Ji, S. Arora, T. Jin, S. Watanabe, and Z. Zhao, “AHa-Bench: Benchmarking audio hallucinations in large audio-language models,” inProc. NeurIPS, 2025

    work page 2025

  24. [24]

    From Text Metrics to Model Internals: A Study of Whisper ASR Hallucination Detection

    J. Jasi ´nski, M. Bara ´nski, J. Bartolewska, M. Witkowski, and K. Kowal- czyk, “From text metrics to model internals: A study of Whisper ASR hallucination detection,” arXiv preprint arXiv:2606.23060, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  25. [25]

    Whisper Hallucination Detection and Mitigation via Hidden Representation Steering and Sparse AutoEncoders

    G. Aparin, V . Popov, T. Sadekova, and A. Yermekova, “Whisper hallu- cination detection and mitigation via hidden representation steering and sparse autoencoders,” arXiv preprint arXiv:2606.07473, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  26. [26]

    A Self-Refining Framework for Enhancing ASR Using TTS-Synthesized Data

    C.-K. Chou, C.-J. Hsu, H.-L. Chung, L.-H. Tseng, H.-C. Cheng, Y .-K. Fu, K. P. Huang, and H.-Y . Lee, “A self-refining framework for enhanc- ing ASR using TTS-synthesized data,” arXiv preprint arXiv:2506.11130, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  27. [27]

    Channel-aware domain-adaptive generative adversarial network for robust speech recognition,

    C.-C. Wang, L.-W. Chen, C.-K. Chou, H.-S. Lee, B. Chen, and H.-M. Wang, “Channel-aware domain-adaptive generative adversarial network for robust speech recognition,” ICASSP, 2025

    work page 2025

  28. [28]

    Speech-IFEval: Evaluating Instruction-Following and Quantifying Catastrophic Forgetting in Speech-Aware Language Models

    K.-H. Lu, C.-Y . Kuan, and H.-Y . Lee, “Speech-IFEval: Evaluating instruction-following and quantifying catastrophic forgetting in speech- aware language models,” arXiv preprint arXiv:2505.19037, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  29. [29]

    Scheduled sampling for sequence prediction with recurrent neural networks,

    S. Bengio, O. Vinyals, N. Jaitly, and N. Shazeer, “Scheduled sampling for sequence prediction with recurrent neural networks,” Advances in Neural Information Processing Systems, 2015

    work page 2015

  30. [30]

    Dynamic-SUPERB Phase-2: A collaboratively expanding benchmark for measuring the capabilities of spoken language models with 180 tasks,

    C.-Y . Huang, W.-C. Chen, S.-W. Yang, A. T. Liu, C.-A. Li, Y .-X. Lin, W.-C. Tseng, et al., “Dynamic-SUPERB Phase-2: A collaboratively expanding benchmark for measuring the capabilities of spoken language models with 180 tasks,” inProc. ICLR, 2025

    work page 2025

  31. [31]

    SUPERB: Speech processing universal performance benchmark,

    S.-W. Yang, P.-H. Chi, Y .-S. Chuang, C.-I. J. Lai, K. Lakhotia, Y . Y . Lin, et al., “SUPERB: Speech processing universal performance benchmark,” inProc. INTERSPEECH, 2021

    work page 2021

  32. [32]

    ML-SUPERB: Multilingual speech universal performance benchmark,

    J. Shi, D. Berrebbi, W. Chen, H.-L. Chung, E.-P. Hu, W.-P. Huang, X. Chang, et al., “ML-SUPERB: Multilingual speech universal performance benchmark,” inProc. INTERSPEECH, 2023

    work page 2023

  33. [33]

    HEAR: Holistic evaluation of audio representations,

    J. Turian, J. Shier, H. R. Khan, B. Raj, B. W. Schuller, C. J. Steinmetz, et al., “HEAR: Holistic evaluation of audio representations,” inProc. NeurIPS Competitions and Demonstrations Track, 2022

    work page 2022

  34. [34]

    AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehension

    Q. Yang, J. Xu, W. Liu, Y . Chu, Z. Jiang, X. Zhou, et al., “AIR-Bench: Benchmarking large audio-language models via generative comprehen- sion,” arXiv preprint arXiv:2402.07729, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  35. [35]

    AudioBench: A Universal Benchmark for Audio Large Language Models

    B. Wang, X. Zou, G. Lin, S. Sun, Z. Liu, W. Zhang, Z. Liu, A. Aw, and N. F. Chen, “AudioBench: A universal benchmark for audio large language models,” arXiv preprint arXiv:2406.16020, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  36. [36]

    MMAU: A massive multi-task audio understanding and reasoning benchmark,

    S. Sakshi, U. Tyagi, S. Kumar, A. Seth, R. Selvakumar, O. Nieto, R. Duraiswami, S. Ghosh, and D. Manocha, “MMAU: A massive multi-task audio understanding and reasoning benchmark,” inProc. ICLR, 2025

    work page 2025

  37. [37]

    AudioMarathon: A comprehensive benchmark for long-context audio understanding and efficiency in audio LLMs,

    P. He, Z. Wen, Y . Wang, Y . Wang, X. Liu, J. Huang, Z. Lei, et al., “AudioMarathon: A comprehensive benchmark for long-context audio understanding and efficiency in audio LLMs,” arXiv preprint arXiv:2510.07293, 2025

    work page arXiv 2025

  38. [38]

    Not in Sync: Unveiling temporal bias in audio chat models,

    J. Yao, S. Liu, Y . Wang, R. Cheng, L. Mei, B. Bi, Z. Xiong, and X. Cheng, “Not in Sync: Unveiling temporal bias in audio chat models,” arXiv preprint arXiv:2510.12185, 2025

    work page arXiv 2025

  39. [39]

    SpotSound: Enhancing Large Audio-Language Models with Fine-Grained Temporal Grounding

    L. Sun, X. Zhou, Z. Li, Y . Zhang, Y . Wang, and W. Xie, “SpotSound: Enhancing large audio-language models with fine-grained temporal grounding,” arXiv preprint arXiv:2604.13023, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  40. [40]

    ChronosAudio: A Comprehensive Long-Audio Benchmark for Evaluating Audio-Large Language Models

    K. Luo, L. Lin, Y . Zhang, M. Aloqaily, J. Tao, D. Wang, et al., “ChronosAudio: A comprehensive long-audio benchmark for evaluating audio-large language models,” arXiv preprint arXiv:2601.04876, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  41. [41]

    STAR-Bench: Probing deep spatio-temporal reasoning as audio 4D intelligence,

    Z. Liu, Z. Niu, Q. Xiao, Z. Zheng, R. Yuan, Y . Zang, Y . Cao, et al., “STAR-Bench: Probing deep spatio-temporal reasoning as audio 4D intelligence,” arXiv preprint arXiv:2510.24693, 2025

    work page arXiv 2025

  42. [42]

    Audio Set: An ontology and human-labeled dataset for audio events,

    J. F. Gemmeke, D. P. W. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, “Audio Set: An ontology and human-labeled dataset for audio events,” inProc. ICASSP, 2017

    work page 2017

  43. [43]

    AudioCaps: Generating captions for audios in the wild,

    C. D. Kim, B. Kim, H. Lee, and G. Kim, “AudioCaps: Generating captions for audios in the wild,” inProc. NAACL-HLT, 2019

    work page 2019

  44. [44]

    Clotho: An audio captioning dataset,

    K. Drossos, S. Lipping, and T. Virtanen, “Clotho: An audio captioning dataset,” inProc. ICASSP, 2020

    work page 2020

  45. [45]

    Clotho- AQA: A crowdsourced dataset for audio question answering,

    S. Lipping, P. Sudarsanam, K. Drossos, and T. Virtanen, “Clotho- AQA: A crowdsourced dataset for audio question answering,” inProc. EUSIPCO, 2022

    work page 2022

  46. [46]

    ToxicTone: A Mandarin audio dataset annotated for toxicity and toxic utterance tonality,

    Y .-X. Luo, Y .-C. Lin, M.-T. Chuang, J.-H. Chen, I.-N. Tsai, P. X. Kiew, Y .-H. Huang, C.-F. Liu, Y .-C. Chen, B.-H. Feng, W. Ren, and H.-Y . Lee, “ToxicTone: A Mandarin audio dataset annotated for toxicity and toxic utterance tonality,” inProc. INTERSPEECH, 2025

    work page 2025

  47. [47]

    How to Learn a New Language? An Efficient Solution for Self-Supervised Learning Models Unseen Languages Adaption in Low-Resource Scenario

    S.-H. Wang, Z.-C. Chen, J. Shi, M.-T. Chuang, G.-T. Lin, K.-P. Huang, D. Harwath, S.-W. Li, and H.-Y . Lee, “How to learn a new language? An efficient solution for self-supervised learning models unseen languages adaption in low-resource scenario,” arXiv preprint arXiv:2411.18217, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  48. [48]

    Parameter-efficient transfer learning for NLP,

    N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. de Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly, “Parameter-efficient transfer learning for NLP,” inProc. ICML, 2019

    work page 2019

  49. [49]

    LoRA: Low-rank adaptation of large language models,

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “LoRA: Low-rank adaptation of large language models,” inProc. ICLR, 2022

    work page 2022

  50. [50]

    Prefix-tuning: Optimizing continuous prompts for generation,

    X. L. Li and P. Liang, “Prefix-tuning: Optimizing continuous prompts for generation,” inProc. ACL, 2021

    work page 2021

  51. [51]

    The power of scale for parameter-efficient prompt tuning,

    B. Lester, R. Al-Rfou, and N. Constant, “The power of scale for parameter-efficient prompt tuning,” inProc. EMNLP, 2021

    work page 2021

  52. [52]

    BitFit: Simple parameter- efficient fine-tuning for transformer-based masked language-models,

    E. Ben-Zaken, S. Ravfogel, and Y . Goldberg, “BitFit: Simple parameter- efficient fine-tuning for transformer-based masked language-models,” in Proc. ACL, 2022

    work page 2022

  53. [53]

    Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning,

    H. Liu, D. Tam, M. Muqeeth, J. Mohta, T. Huang, M. Bansal, and C. Raffel, “Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning,” inProc. NeurIPS, 2022

    work page 2022

  54. [54]

    ASCEND: A spontaneous Chinese-English dataset for code- switching in multi-turn conversation,

    H. Lovenia, S. Cahyawijaya, G. I. Winata, P. Xu, X. Yan, Z. Liu, R. Frieske, T. Yu, W. Dai, E. J. Barezi, Q. Chen, X. Ma, B. E. Shi, and P. Fung, “ASCEND: A spontaneous Chinese-English dataset for code- switching in multi-turn conversation,” inProc. LREC, 2022

    work page 2022

  55. [55]

    Distil-Whisper: Robust Knowledge Distillation via Large-Scale Pseudo Labelling

    S. Gandhi, P. von Platen, and A. M. Rush, “Distil-Whisper: Robust knowledge distillation via large-scale pseudo labelling,” arXiv preprint arXiv:2311.00430, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  56. [56]

    Qwen2-Audio Technical Report

    Y . Chu, J. Xu, Q. Yang, H. Wei, X. Wei, Z. Guo, Y . Leng, Y . Lv, J. He, J. Lin, C. Zhou, and J. Zhou, “Qwen2-Audio technical report,” arXiv preprint arXiv:2407.10759, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  57. [57]

    Qwen2.5-Omni Technical Report

    J. Xu, Z. Guo, J. He, H. Hu, T. He, S. Bai, K. Chen, J. Wang, Y . Fan, K. Dang, B. Zhang, X. Wang, Y . Chu, and J. Lin, “Qwen2.5-Omni technical report,” arXiv preprint arXiv:2503.20215, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  58. [58]

    Qwen3-Omni Technical Report

    J. Xu, Z. Guo, H. Hu, Y . Chu, X. Wang, J. He, Y . Wang, X. Shi, T. He, X. Zhu, Y . Lv, Y . Wang, D. Guo, et al., “Qwen3-Omni technical report,” arXiv preprint arXiv:2509.17765, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  59. [59]

    MOSS-Audio Technical Report

    C. Yang, C. Yu, H. Chen, J. Zhu, J. Chen, K. Chen, W. Wang, Y . Wang, Y . Jiang, Y . Jiang, Z. Lin, Z. Chen, Z. Fei, et al., “MOSS-Audio technical report,” arXiv preprint arXiv:2606.01802, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  60. [60]

    VIBEVOICE-ASR technical report,

    Z. Peng, J. Yu, Y . Chang, Z. Wang, L. Dong, Y . Hao, Y . Tu, C. Yang, W. Wang, S. Xu, Y . Sun, H. Bao, W. Xu, et al., “VIBEVOICE-ASR technical report,” arXiv preprint arXiv:2601.18184, 2026

    work page arXiv 2026

  61. [61]

    5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignment , author=

    K.-H. Lu, Z. Chen, S.-W. Fu, C.-H. H. Yang, S.-F. Huang, C.-K. Yang, C.-E. Yu, C.-W. Chen, W.-C. Chen, C.-Y . Huang, Y .-C. Lin, Y .-X. Lin, C.-A. Fu, et al., “DeSTA2.5-Audio: Toward general-purpose large audio language model with self-generated cross-modal alignment,” arXiv preprint arXiv:2507.02768, 2025

    work page arXiv 2025