MOSS Transcribe Diarize Technical Report

Botian Jiang; Chen Yang; Hanfu Chen; Jie Zhu; Jingqi Chen; Ke Chen; Liwei Fan; MOSI.AI: Donghua Yu; Muchen Li; Qinyuan Cheng

REVIEW 4 major objections 4 minor 5 cited by

MOSS Transcribe Diarize claims that speaker-attributed, time-stamped transcription of hour-scale meetings can be done end-to-end by one multimodal language model, beating commercial pipelines in word error and attribution accuracy.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 12:45 UTC pith:ZILPTVET

load-bearing objection Strong CER/cpCER results, but the 'time-stamped' half of the SATS claim is never measured — the scoring deletes timestamps, and the paper's own future work admits it. the 4 major comments →

arxiv 2601.01554 v7 pith:ZILPTVET submitted 2026-01-04 cs.SD cs.AIeess.AS

MOSS Transcribe Diarize Technical Report

MOSI.AI: Donghua Yu , Zhengyuan Lin , Hanfu Chen , Chen Yang , Yiyang Zhang , Jingqi Chen , Ke Chen , Liwei Fan

show 16 more authors

Yi Jiang Jie Zhu Muchen Li Wenxuan Wang Yang Wang Zhe Xu Botian Jiang Yitian Gong Yuqian Zhang Wenbo Zhang Songlin Wang Zhiyu Wu Zhaoye Fei Qinyuan Cheng Shimin Li Xipeng Qiu

This is my paper

classification cs.SD cs.AIeess.AS

keywords speaker-attributed time-stamped transcriptionend-to-end speech LLMlong-context audio modelingspeaker diarizationtimestamp predictionmeeting transcriptionsynthetic multi-speaker datacpCER evaluation

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that a single multimodal language model can do what today usually requires stitching together separate speech-recognition and speaker-diarization systems: transcribing who said what and when, for recordings up to 90 minutes, in one forward pass. It introduces MOSS Transcribe Diarize, trained on real and simulated multi-speaker audio, with a 128k-token context that helps the model keep speaker identities and topic references coherent across long meetings. On three benchmarks—long meeting recordings, multi-guest podcasts, and overlap-rich movie clips—the model reports lower word-error, speaker-attributed error, and the gap between the two than leading commercial systems. A sympathetic reader would care because it points to a simpler, less brittle architecture for meeting transcription and conversational analytics.

Core claim

The central claim is that Speaker-Attributed, Time-Stamped Transcription (SATS) can be formulated end-to-end, and that this formulation fixes the failure modes of cascaded systems. MOSS Transcribe Diarize takes audio, encodes it with a speech encoder, projects the embeddings into a pretrained text LLM, and autoregressively outputs words interleaved with speaker labels and formatted timestamps. Timestamps are represented as ordinary text tokens inserted between audio chunks rather than as positional indices, which the paper argues makes hour-scale timestamp generation feasible. Trained on in-the-wild audio plus simulated overlapping conversations, the model claims the best cpCER and Δcp score

What carries the argument

The load-bearing mechanism is the joint autoregressive decoder: a speech encoder feeds a pretrained text LLM through a learned projection layer, and temporal information is encoded as explicit formatted timestamp strings placed between audio encoder chunks. This lets one model assign words to speakers and times without a separate alignment stage. The 128k-token context window, covering roughly 90 minutes of audio, is what preserves global speaker memory; a controllable synthetic mixture simulator supplies the dense overlap and turn-taking patterns needed for attribution learning when real multi-speaker data is scarce.

Load-bearing premise

The chief load-bearing premise is that timestamps represented as ordinary text tokens between audio chunks stay accurate over hour-scale recordings; the evaluation never measures timestamp precision, so an error there would undercut the 'time-stamped' half of the claim even if words and speaker labels are right.

What would settle it

Take a 60-minute multi-speaker meeting with manually labeled speaker turn boundaries. Run MOSS Transcribe Diarize and compare predicted turn-start/end times to the reference boundaries with a 0.5-second tolerance; if alignment accuracy falls well below the CER/cpCER gains, the timestamp component is not established.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

Meeting assistants can get speaker-attributed, timestamped transcripts in one pass, avoiding cascading errors from ASR-to-diarization hand-offs.
Audio up to 90 minutes can be processed without chunking, preserving speaker identity and coreference over dozens of turns and reducing identity drift.
The model natively emits segment-level timestamps, so transcript retrieval and skimmability do not require a separate forced-alignment step.
The consistently small Δcp across benchmarks suggests speaker attribution errors add little extra degradation beyond transcription errors, even with dense overlaps.
Simulated conversation data can be used to supply speaker-attribution supervision when real multi-speaker recordings are limited.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Editorial inference: The paper's timestamp claim is untested—no timestamp or boundary error metric appears in the evaluation, so the precision of the text-token timestamps should be validated before relying on exact timecodes in downstream tools.
Editorial inference: The timestamp-as-text representation could be made more reliable by adding an explicit time-regression objective or by evaluating with a boundary-tolerance metric, which would directly test the load-bearing assumption.
Editorial inference: Because the evaluation covers Mandarin meetings, English podcasts, and mixed-language overlaps, the hardest untested regime is very long audio with many simultaneous speakers and severe channel noise; the simulator's 80% overlap cap may not cover those extremes.
Editorial inference: If the single-pass long-context approach holds, a natural next step is streaming inference with compact speaker memory, since the current design trades latency for the benefit of full context.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Strong CER/cpCER results, but the 'time-stamped' half of the SATS claim is never measured — the scoring deletes timestamps, and the paper's own future work admits it.

read the letter

You asked what I think of this one. The punchline: MOSS shows credible ASR + speaker attribution numbers, and it may be a serious meeting-transcription product. But the advertised time-stamped capability is untested. The paper defines only CER, cpCER and Δcp; Appendix A.2's normalization strips square-bracket content that isn't speaker IDs, which removes the timestamp tokens. So even a perfect CER/cpCER score is consistent with arbitrarily bad timestamps. Section 5 lists 'finer-grained timestamp evaluation' as future work. That's a self-concession on the central claim.

What's genuinely good: the system is a clean recombination of known pieces — a pretrained audio encoder, projection, a text LLM, and textual timestamp tokens. The 128k context for up to 90 minutes is a useful engineering target. The training data description, especially the property-aware simulator with overlap caps, cross-fades, and SNR ranges, is concrete and well motivated. And the head-to-head with commercial APIs is informative; if the numbers hold, it's a strong product.

The soft spots are proportional. First, the timestamp gap is load-bearing, not cosmetic: the title says 'Transcribe Diarize,' but the abstract and introduction repeatedly stress time-stamped transcription as the key differentiator. No boundary error, no timestamp MAE, no segment-level alignment metric anywhere. Second, the AISHELL-4 sentence in §3.1 — 'use the averaged channel of the far-field signals for both training and evaluation' — could be read as training on the eval set. I suspect they mean they use that channel for both purposes across the standard split, but the wording needs fixing. Third, no error bars, ablations, or training details. The two in-house benchmarks aren't released yet, so two of three evaluations are currently unverifiable.

None of that proves the model is bad. The CER/cpCER margins are plausible for a well-trained industrial system. But as a scientific report, it argues for a capability it never measures. That's the core problem.

Who should read it: practitioners sizing up commercial options, and people interested in how far MLLMs can go on long audio. For researchers, it's a cautionary example of evaluation not matching the claim.

My recommendation: send it to peer review — the system is important enough, and referees should demand timestamp metrics, the AISHELL-4 split clarification, and an ablation of the long-context component. Without those, it shouldn't be accepted.

Referee Report

4 major / 4 minor

Summary. This paper presents MOSS Transcribe Diarize, a unified multimodal large language model for Speaker-Attributed, Time-Stamped Transcription (SATS). The architecture couples a speech encoder with a projection module into a pretrained text LLM and uses a 128k-token context window claimed to support up to 90-minute audio in a single pass. Timestamps are represented as formatted text tokens inserted between audio encoder chunks. Training uses real multilingual conversational audio and simulated mixtures produced by a property-aware simulator. The evaluation compares the model against Doubao, ElevenLabs, GPT-4o, Gemini 2.5 Pro, and Gemini 3 Pro on AISHELL-4 and two in-house benchmarks (Podcast and Movies), reporting CER, cpCER, and Δcp. The authors report consistent improvements in cpCER and Δcp and conclude that end-to-end long-context modeling outperforms commercial systems.

Significance. If the reported results are reproducible, the paper would demonstrate a meaningful step toward end-to-end SATS at meeting scale, going beyond modular ASR+diarization pipelines and addressing long-range speaker consistency with a 128k context. The decision to open-source the two in-house evaluation sets is a positive step for comparability. However, the current manuscript does not provide evidence for the timestamp component of the SATS claim and leaves several evaluation-protocol questions unresolved, so the significance is conditional on additional verification.

major comments (4)

[§2, §4.1.2, §A.2, §5] The title and abstract advertise 'Time-Stamped Transcription,' and §2 states that inserting formatted timestamp text between audio encoder chunks 'enables accurate timestamp generation over hour-scale audio.' No evaluation supports this. §4.1.2 defines only CER, cpCER, and Δcp; none measures timestamp or boundary precision. Worse, the normalization in §A.2 deletes non-speaker square-bracket annotations, which removes timestamp tokens such as [0.11] from hypotheses before scoring. Thus even a perfect CER/cpCER result is compatible with arbitrarily inaccurate timestamps. The conclusion in §5 lists 'finer-grained timestamp evaluation' as future work, confirming the absence. This is a load-bearing gap: the paper's central claim includes timestamping, so the evidence is incomplete.
[§3.1] The sentence 'We use the averaged channel of the far-field signals for both training and evaluation' is ambiguous. If AISHELL-4 was used as both a training and an evaluation set, the AISHELL-4 row of Table 2 is invalid due to train/evaluation overlap. If the intended meaning is that the same channel-averaging procedure was applied in both training and evaluation data, this should be stated unambiguously and the evaluation split clarified. Either way, the current wording is a potential contamination red flag for the paper's flagship long-form benchmark.
[§4.2, Table 2] All numbers in Table 2 are point estimates with no error bars, no number of evaluation runs, and no significance tests. Several differences are small (e.g., Podcast Δcp 2.50 for MOSS vs 2.61 for Doubao; Movies cpCER 13.36 vs 14.73), so without variance or repeated trials the claimed advantage cannot be distinguished from noise. The paper should report at least confidence intervals, per-sample distributions, or a paired test.
[§1, §4.2] The first contribution asserts that long-context modeling 'improves SATS metrics,' and §4.2 attributes the results to long-context, end-to-end modeling. No ablation is provided that isolates the 128k context from other design choices (e.g., a chunked baseline with the same model or a shorter context window), nor is there an ablation of the simulated-data augmentation. Without these controls, the causal attribution to long-context modeling is unsupported.

minor comments (4)

[§A.1] The evaluation prompt text for Gemini and MOSS appears garbled/mojibake in the current PDF. This must be fixed so readers can see exactly what instructions were used, especially for the commercial baselines whose output format constraints led to exclusions in Table 2.
[§1, §2, §3.2] There are several typographical artifacts: 'iautomatic' in §1, stray '⚶' symbols in §2 and §3.2, and a missing space in 'it' on page 5 ('it' should likely be 'it is' or similar). These should be cleaned up.
[§1] The claim of being the 'first unified multimodal model' for joint word recognition, speaker attribution, and timestamp prediction is hard to verify and seems premature. Consider softening to 'to our knowledge' with a more complete related-work comparison of prior timestamp-outputting models.
[Table 2] The table note explains GPT-4o and Gemini 3 Pro exclusions, but the reader cannot tell whether the remaining baselines were given identical prompts, output formatting constraints, or post-processing. A sentence on how baselines were adapted to the required speaker-attributed format would improve fairness assessment.

Circularity Check

1 steps flagged

Timestamp component is not measured: normalization deletes timestamp tokens before scoring.

specific steps

other [Section 4.1.2 Metrics and Appendix A.2 Output Normalization for Evaluation]
"Remove non-speaker square-bracket annotations. We delete any square-bracketed spans [...] that are not speaker IDs, using the regex \[(?!S\d+\]).*?\]. This keeps only speaker tags of the form [S1], [S01], etc., and removes other bracketed markers such as events (e.g., [event]). After normalization, each hypothesis/reference contains only speaker identifiers and plain transcript text for scoring."

The paper's central claim is that MOSS jointly performs transcription, speaker attribution, and timestamp prediction, yet the only reported metrics are CER, cpCER, and Δcp, computed after deleting all non-speaker square-bracketed spans. Timestamp tokens such as [0.11] are exactly such spans, so the scoring procedure is invariant to timestamp accuracy by construction. A model with arbitrarily wrong timestamps could achieve the reported cpCER/Δcp numbers. Section 5 confirms the gap by listing 'finer-grained timestamp evaluation' as future work. Thus the 'time-stamped' part of the SATS claim is not derived from any measured evidence in the paper.

full rationale

This is an empirical system report, not a mathematical derivation, so the classic circular-fit and self-reference patterns are largely inapplicable. The comparison to commercial systems on AISHELL-4 (a public benchmark) provides independent evidence for the CER/cpCER/Δcp claims, and the in-house Podcast/Movies sets, while not yet released, are at least described as external benchmarks. No load-bearing self-citation chain appears: the only author-overlapping reference (SpeechGPT, Ref. [22]) is used as a generic architectural prior, not as a uniqueness or validity argument. The main issue is the definitional/measurement gap: the task is defined as speaker-attributed, time-stamped transcription, but the evaluation discards timestamp tokens before scoring. This makes the reported numbers unable to substantiate the timestamp component of the headline claim. That is a partial circularity/validity gap rather than a fitted-input circularity, so a score of 3 is appropriate rather than 6 or higher.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

The report depends on several hand-set simulation constants and on untested assumptions about timestamp encoding and synthetic-data transfer. Since no training or inference details are given, the ledger captures the assumptions visible in the text.

free parameters (5)

Overlap cap in simulator = 80% of shorter segment
Hand-set simulation parameter in §3.2; controls overlap difficulty.
SNR range for noise/reverb augmentation = 0–15 dB
Chosen in §3.2; affects robustness but no sensitivity analysis.
Cross-fade duration = 50 ms
Used in §3.2 for perceptual continuity; hand-picked.
Segment count and gap distribution = Not reported (log-normal weights, Gaussian gaps)
Simulator distributions in §3.2; exact parameters omitted.
Context window length = 128k tokens
Design choice claimed to cover 90 minutes; no scaling analysis is provided.

axioms (5)

ad hoc to paper Textual timestamp tokens between audio encoder chunks preserve accurate temporal localization over long audio
Stated in §2; no timestamp evaluation is provided anywhere in the paper.
domain assumption Property-aware simulated mixtures transfer to real multi-speaker conversations
Training relies heavily on simulated data in §3.2; no ablation separates synthetic vs. real contributions.
domain assumption cpCER with optimal label permutation is a sufficient proxy for diarization quality
Used as the main speaker-attribution metric in §4.1.2; ignores timestamp/boundary accuracy.
domain assumption Averaged far-field AISHELL-4 channel is an appropriate input for both training and evaluation
§3.1 states this without clarifying train/test split separation, raising leakage risk.
domain assumption Commercial baselines were given fair prompts and normalization
Appendix A.1/A.2 shows custom prompts for Gemini that render as garbled text, and normalization strips annotations that may be part of baseline outputs.

pith-pipeline@v1.3.0-alltime-deepseek · 6995 in / 13451 out tokens · 131843 ms · 2026-08-03T12:45:52.466600+00:00 · methodology

0 comments

read the original abstract

Speaker-Attributed, Time-Stamped Transcription (SATS) aims to transcribe what is said and to precisely determine the timing of each speaker, which is particularly valuable for meeting transcription. Existing SATS systems rarely adopt an end-to-end formulation and are further constrained by limited context windows, weak long-range speaker memory, and the inability to output timestamps. To address these limitations, we present MOSS Transcribe Diarize, a unified multimodal large language model that jointly performs Speaker-Attributed, Time-Stamped Transcription in an end-to-end paradigm. Trained on extensive real wild data and equipped with a 128k context window for up to 90-minute inputs, MOSS Transcribe Diarize scales well and generalizes robustly. Across comprehensive evaluations, it outperforms state-of-the-art commercial systems on multiple public and in-house benchmarks.

discussion (0)

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

GigaChat Audio: Time-aware Large Audio Language Model
eess.AS 2026-07 conditional novelty 6.0

Interleaving periodic time markers with continuous audio tokens, plus duration-mixture synthetic training, yields stable temporal grounding for an audio LLM on inputs up to 120 minutes.
Grounding Spoken LLMs in Multi-Speaker Audio via Diarization Conditioning
eess.AS 2026-06 unverdicted novelty 6.0

Dixtral uses diarization conditioning on a Whisper-based encoder within Voxtral to outperform baselines on multi-speaker transcription and match or exceed on QA tasks.
DM-ASR: Diarization-aware Multi-speaker ASR with Large Language Models
eess.AS 2026-04 unverdicted novelty 6.0

DM-ASR reformulates multi-speaker ASR as multi-turn dialogue generation conditioned on diarization results, achieving competitive benchmark performance with relatively small models and limited data.
Balancing ASR and diarization in end-to-end LLMs for multi-talker speech recognition
eess.AS 2026-06 unverdicted novelty 4.0

LLM-based multi-talker ASR with dual-encoder, feature interleaving, length-aware speaker loss, and adaptive ASR threshold achieves 18% and 24% relative gains over baselines on AliMeeting and Aishell4.
MOSS-Audio Technical Report
cs.SD 2026-06 unverdicted novelty 4.0

MOSS-Audio is an audio-language model using a 12.5 Hz encoder, DeepStack cross-layer injection, time markers, and an event-preserving annotation pipeline for unified audio understanding.

Reference graph

Works this paper leans on

22 extracted references · 4 linked inside Pith · cited by 5 Pith papers

[1]

Whisperx: Time-accurate speech transcription of long-form audio

Max Bain, Jaesung Huh, Tengda Han, and Andrew Zisserman. Whisperx: Time-accurate speech transcription of long-form audio. arXiv preprint arXiv:2303.00747, 2023

Pith/arXiv arXiv 2023
[2]

Pyannote.audio: neural building blocks for speaker diarization

Hervé Bredin, Ruiqing Yin, Juan Manuel Coria, Gregory Gelly , Pavel Korshunov , Marvin Lavechin, Diego Fustes, Hadrien Titeux, Wassim Bouaziz, and Marie-Philippe Gill. Pyannote.audio: neural building blocks for speaker diarization. In ICASSP, pages 7124–7128, 2020

2020
[3]

The ami meeting corpus: A pre-announcement

Jean Carletta, Simone Ashby , Sophie Bourban, Mike Flynn, Maël Guillemot, Thomas Hain, Jaroslav Kadlec, Vasilis Karaiskos, Wessel Kraaĳ, Melissa Kronenthal, et al. The ami meeting corpus: A pre-announcement. In Machine Learning for Multimodal Interaction, 2005

2005
[4]

J. Chen, Y . Wang, S. Watanabe, J. Le Roux, and J. R. Hershey. Continuous speech separation: Dataset and analysis. In Proceedings of ICASSP, 2020

2020
[5]

Timemarker: A versatile video-llm for long and short video understanding with superior temporal localization ability

Shimin Chen, Xiaohan Lan, Yitian Yuan, Zequn Jie, and Lin Ma. Timemarker: A versatile video-llm for long and short video understanding with superior temporal localization ability. arXiv preprint arXiv:2411.18211, 2024

Pith/arXiv arXiv 2024
[6]

Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models

Yunfei Chu, Zhifang Jin, Weihan Xu, Yuchen Wei, et al. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models. arXiv preprint arXiv:2311.07919, 2023

Pith/arXiv arXiv 2023
[7]

AISHELL-4: An open source dataset for speech enhancement, separation, recognition and speaker diarization in conference scenario

Ying Fu, Li Cheng, Shuo Lv , Yuting Jv , Yu Kong, Yan Hu, Lei Xie, Chenglin Zhu, Jian Wu, Hao Bu, Xiong Xu, Jun Du, and Jian Chen. AISHELL-4: An open source dataset for speech enhancement, separation, recognition and speaker diarization in conference scenario. In Proceedings of Interspeech, pages 3660–3664, 2021

2021
[8]

End-to-end neural speaker diarization with self-attention

Yusuke Fujita, Takuya Yoshioka, Xuankai Chang, and Shinji Watanabe. End-to-end neural speaker diarization with self-attention. In Proceedings of ASRU, 2019

2019
[9]

The icsi meeting corpus

Adam Janin, Don Baron, Jane Edwards, Dan Ellis, David Gelbart, Nelson Morgan, Barbara Peskin, Thilo Pfau, Eliz- abeth Shriberg, and Andreas Stolcke. The icsi meeting corpus. In Proceedings of ICASSP, 2003

2003
[10]

Serialized output training for end-to-end overlapping speech recognition

Naoyuki Kanda, Yashesh Gaur, Xiaofei Wang, Zhong Meng, and Takuya Yoshioka. Serialized output training for end-to-end overlapping speech recognition. In Interspeech, pages 2797–2801, 2020

2020
[11]

From simulated mixtures to simulated conversations as training data for neural diarization

Federico Landini, Jan Profant, Mireia Díez, and Lukáš Burget. From simulated mixtures to simulated conversations as training data for neural diarization. In Proceedings of Interspeech, pages 143–147, 2022

2022
[12]

Levenshtein

Vladimir I. Levenshtein. Binary codes capable of correcting deletions, insertions and reversals. Soviet Physics Doklady, 1966

1966
[13]

Montreal forced aligner: A trainable text-speech alignment system

Michael McAuliffe, Michaela Socolof, Sarah Mihuc, Michael Wagner, and Morgan Sonderegger. Montreal forced aligner: A trainable text-speech alignment system. In Proceedings of Interspeech, 2017

2017
[14]

T. Park, I. Medennikov , K. Dhawan, W. Wang, H. Huang, N. R. Koluguri, K. C. Puvvada, J. Balam, and B. Ginsburg. Sortformer: Seamless integration of speaker diarization and asr by bridging timestamps and tokens. arXiv preprint, 2024

2024
[15]

T. J. Park, H. Huang, C. Hooper, N. Koluguri, K. Dhawan, I. Medennikov , A. Jukic, J. Balam, and B. Ginsburg. Property-aware multi-speaker data simulation: A probabilistic modelling technique for synthetic data generation. In Proceedings of the CHiME-2023 Workshop, 2023

2023
[16]

Robust speech recognition via large-scale weak supervision

Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey , and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. ICML, 2023

2023
[17]

Train short, infer long: Speech-llm enables zero- shot streamable joint asr and diarization on long audio

Mohan Shi, Xiong Xiao, Ruchao Fan, Shaoshi Ling, and Jinyu Li. Train short, infer long: Speech-llm enables zero- shot streamable joint asr and diarization on long audio. arXiv preprint arXiv:2511.16046, 2025

arXiv 2025
[18]

X-vectors: Robust dnn embeddings for speaker recognition

David Snyder, Daniel Garcia-Romero, Gregory Sell, Daniel Povey , and Sanjeev Khudanpur. X-vectors: Robust dnn embeddings for speaker recognition. In Proceedings of ICASSP, 2018

2018
[19]

Salmonn: T owards generic hearing abilities for large language models

Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhe Chen, et al. Salmonn: T owards generic hearing abilities for large language models. arXiv preprint arXiv:2310.13289, 2023. 8

Pith/arXiv arXiv 2023
[20]

Wang et al

Q. Wang et al. Diarizationlm: Speaker diarization post-processing with large language models. In Proc. Interspeech, 2024

2024
[21]

H. Yin, Y . Chen, C. Deng, L. Cheng, H. Wang, C.-H. Tan, Q. Chen, W. Wang, and X. Li. Speakerlm: End-to-end versatile speaker diarization and recognition with multimodal large language models. arXiv preprint, 2025

2025
[22]

Speechgpt: Empow- ering large language models with intrinsic cross-modal conversational abilities

Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yunfei Zhou, and Xipeng Qiu. Speechgpt: Empow- ering large language models with intrinsic cross-modal conversational abilities. In EMNLP, 2023. 9 Appendix Appendix Contents A Additional Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 A.1 Evalu...

2023

[1] [1]

Whisperx: Time-accurate speech transcription of long-form audio

Max Bain, Jaesung Huh, Tengda Han, and Andrew Zisserman. Whisperx: Time-accurate speech transcription of long-form audio. arXiv preprint arXiv:2303.00747, 2023

Pith/arXiv arXiv 2023

[2] [2]

Pyannote.audio: neural building blocks for speaker diarization

Hervé Bredin, Ruiqing Yin, Juan Manuel Coria, Gregory Gelly , Pavel Korshunov , Marvin Lavechin, Diego Fustes, Hadrien Titeux, Wassim Bouaziz, and Marie-Philippe Gill. Pyannote.audio: neural building blocks for speaker diarization. In ICASSP, pages 7124–7128, 2020

2020

[3] [3]

The ami meeting corpus: A pre-announcement

Jean Carletta, Simone Ashby , Sophie Bourban, Mike Flynn, Maël Guillemot, Thomas Hain, Jaroslav Kadlec, Vasilis Karaiskos, Wessel Kraaĳ, Melissa Kronenthal, et al. The ami meeting corpus: A pre-announcement. In Machine Learning for Multimodal Interaction, 2005

2005

[4] [4]

J. Chen, Y . Wang, S. Watanabe, J. Le Roux, and J. R. Hershey. Continuous speech separation: Dataset and analysis. In Proceedings of ICASSP, 2020

2020

[5] [5]

Timemarker: A versatile video-llm for long and short video understanding with superior temporal localization ability

Shimin Chen, Xiaohan Lan, Yitian Yuan, Zequn Jie, and Lin Ma. Timemarker: A versatile video-llm for long and short video understanding with superior temporal localization ability. arXiv preprint arXiv:2411.18211, 2024

Pith/arXiv arXiv 2024

[6] [6]

Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models

Yunfei Chu, Zhifang Jin, Weihan Xu, Yuchen Wei, et al. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models. arXiv preprint arXiv:2311.07919, 2023

Pith/arXiv arXiv 2023

[7] [7]

AISHELL-4: An open source dataset for speech enhancement, separation, recognition and speaker diarization in conference scenario

Ying Fu, Li Cheng, Shuo Lv , Yuting Jv , Yu Kong, Yan Hu, Lei Xie, Chenglin Zhu, Jian Wu, Hao Bu, Xiong Xu, Jun Du, and Jian Chen. AISHELL-4: An open source dataset for speech enhancement, separation, recognition and speaker diarization in conference scenario. In Proceedings of Interspeech, pages 3660–3664, 2021

2021

[8] [8]

End-to-end neural speaker diarization with self-attention

Yusuke Fujita, Takuya Yoshioka, Xuankai Chang, and Shinji Watanabe. End-to-end neural speaker diarization with self-attention. In Proceedings of ASRU, 2019

2019

[9] [9]

The icsi meeting corpus

Adam Janin, Don Baron, Jane Edwards, Dan Ellis, David Gelbart, Nelson Morgan, Barbara Peskin, Thilo Pfau, Eliz- abeth Shriberg, and Andreas Stolcke. The icsi meeting corpus. In Proceedings of ICASSP, 2003

2003

[10] [10]

Serialized output training for end-to-end overlapping speech recognition

Naoyuki Kanda, Yashesh Gaur, Xiaofei Wang, Zhong Meng, and Takuya Yoshioka. Serialized output training for end-to-end overlapping speech recognition. In Interspeech, pages 2797–2801, 2020

2020

[11] [11]

From simulated mixtures to simulated conversations as training data for neural diarization

Federico Landini, Jan Profant, Mireia Díez, and Lukáš Burget. From simulated mixtures to simulated conversations as training data for neural diarization. In Proceedings of Interspeech, pages 143–147, 2022

2022

[12] [12]

Levenshtein

Vladimir I. Levenshtein. Binary codes capable of correcting deletions, insertions and reversals. Soviet Physics Doklady, 1966

1966

[13] [13]

Montreal forced aligner: A trainable text-speech alignment system

Michael McAuliffe, Michaela Socolof, Sarah Mihuc, Michael Wagner, and Morgan Sonderegger. Montreal forced aligner: A trainable text-speech alignment system. In Proceedings of Interspeech, 2017

2017

[14] [14]

T. Park, I. Medennikov , K. Dhawan, W. Wang, H. Huang, N. R. Koluguri, K. C. Puvvada, J. Balam, and B. Ginsburg. Sortformer: Seamless integration of speaker diarization and asr by bridging timestamps and tokens. arXiv preprint, 2024

2024

[15] [15]

T. J. Park, H. Huang, C. Hooper, N. Koluguri, K. Dhawan, I. Medennikov , A. Jukic, J. Balam, and B. Ginsburg. Property-aware multi-speaker data simulation: A probabilistic modelling technique for synthetic data generation. In Proceedings of the CHiME-2023 Workshop, 2023

2023

[16] [16]

Robust speech recognition via large-scale weak supervision

Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey , and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. ICML, 2023

2023

[17] [17]

Train short, infer long: Speech-llm enables zero- shot streamable joint asr and diarization on long audio

Mohan Shi, Xiong Xiao, Ruchao Fan, Shaoshi Ling, and Jinyu Li. Train short, infer long: Speech-llm enables zero- shot streamable joint asr and diarization on long audio. arXiv preprint arXiv:2511.16046, 2025

arXiv 2025

[18] [18]

X-vectors: Robust dnn embeddings for speaker recognition

David Snyder, Daniel Garcia-Romero, Gregory Sell, Daniel Povey , and Sanjeev Khudanpur. X-vectors: Robust dnn embeddings for speaker recognition. In Proceedings of ICASSP, 2018

2018

[19] [19]

Salmonn: T owards generic hearing abilities for large language models

Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhe Chen, et al. Salmonn: T owards generic hearing abilities for large language models. arXiv preprint arXiv:2310.13289, 2023. 8

Pith/arXiv arXiv 2023

[20] [20]

Wang et al

Q. Wang et al. Diarizationlm: Speaker diarization post-processing with large language models. In Proc. Interspeech, 2024

2024

[21] [21]

H. Yin, Y . Chen, C. Deng, L. Cheng, H. Wang, C.-H. Tan, Q. Chen, W. Wang, and X. Li. Speakerlm: End-to-end versatile speaker diarization and recognition with multimodal large language models. arXiv preprint, 2025

2025

[22] [22]

Speechgpt: Empow- ering large language models with intrinsic cross-modal conversational abilities

Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yunfei Zhou, and Xipeng Qiu. Speechgpt: Empow- ering large language models with intrinsic cross-modal conversational abilities. In EMNLP, 2023. 9 Appendix Appendix Contents A Additional Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 A.1 Evalu...

2023