Pith. sign in

REVIEW 4 major objections 3 minor 36 references

MOVER: Combining Multiple Meeting Recognition Systems

T0 review · 4 major / 3 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MOVER is the first approach to combine meeting recognition outputs that differ in both diarization and ASR, improving tcpWER by 9.55% and 8.51% over state-of-the-art single systems on two benchmarks.

desk verdict The idea is the right kind of idea, but the submission is unreadable: mojibake full text and a hep-ex header, so the claimed gains are unverifiable. read the letter →

arxiv 2508.05055 v1 pith:FQFRVXD5 submitted 2025-08-07 eess.AS

Naoyuki Kamo , Tsubasa Ochiai , Marc Delcroix , Tomohiro Nakatani This is my paper · ORCID
classification eess.AS
keywords MOVERmeetingrecognitionsystemcombinationdiarizationautomaticspeechtcpWERCHiME-8DASRNOTSOFAR-1
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MOVER (Meeting recognizer Output Voting Error Reduction) is a five-stage method for combining complete meeting recognition hypotheses — diarization plus transcripts — rather than combining only speaker labels or only words. Earlier combination tools DOVER and ROVER each handle one side of the problem; MOVER is positioned as the first to handle both at once, so the systems being combined can disagree on who spoke, when, and what was said. Tested on the CHiME-8 DASR task and the multi-channel track of NOTSOFAR-1, MOVER combines multiple diverse systems and improves the official tcpWER metric by 9.55% and 8.51% relative to the state-of-the-art single systems. If the result is correct, it gives meeting recognition a drop-in way to turn a collection of independent systems into a stronger one without retraining any of them.

What carries the argument

The carrying mechanism is MOVER's five-stage combination pipeline: speaker alignment maps each system's speaker labels onto a common set of speakers; segment grouping finds which time intervals from different systems correspond to the same speech; word and timing combination merges the hypotheses into one word sequence with one timing. Earlier methods DOVER and ROVER combine only diarization outputs or only ASR outputs respectively; MOVER's pipeline is what lets both be combined at once, so errors that differ across systems can cancel by voting.

What would settle it

Run MOVER on a development set in which one system's speaker labels are deliberately permuted on every recording; if the aligned output does not beat the better single system's tcpWER, the combination claim fails.

Watch

Extended reading notes

Core claim

The central claim is that hypotheses from meeting recognition systems that differ in both diarization and ASR can be combined into one hypothesis that beats every individual system on tcpWER. The method aligns speakers across outputs, groups the corresponding segments, and combines words and timings by voting, producing a single consensus transcript with one set of speaker labels and time boundaries. On the two evaluated tasks, the combined hypothesis achieves relative tcpWER improvements of 9.55% (CHiME-8 DASR) and 8.51% (NOTSOFAR-1 multi-channel) over the best individual systems.

Load-bearing premise

MOVER's gains depend on its speaker-alignment and segment-grouping stages reliably matching hypotheses that disagree in both time boundaries and speaker labels, so that the combination removes more errors than the alignment process introduces.

Editorial extensions

If this is right

  • Combining several meeting recognizers with MOVER yields a single output that scores better on tcpWER than any of the individual systems.
  • The combination works without retraining or modifying the component systems, so it can be applied to off-the-shelf recognizers.
  • The method accepts systems with different speaker labelings and segment boundaries, removing a prior restriction in system-combination work.
  • The gains replicate across two distinct tasks, showing the approach is not tuned to a single benchmark.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the gain mechanism is error diversity, MOVER should improve more as the combined systems become more different; combining near-identical systems should add little or nothing. This can be tested directly by controlling the overlap between system outputs.
  • The same alignment-plus-voting structure could be lifted to other output formats that carry speaker labels and time intervals, such as multimodal diarization that uses video or motion cues.
  • A finer analysis separating alignment errors from word errors would show where the remaining tcpWER comes from and whether confidence-weighted voting pushes the gains further.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. The paper proposes MOVER (Meeting recognizer Output Voting Error Reduction), a five-stage system combination method for meeting recognition that fuses hypotheses that differ in both time intervals and speaker labels, combining diarization and ASR outputs jointly. The abstract reports relative tcpWER improvements of 9.55% and 8.51% over state-of-the-art systems on the CHiME-8 DASR task and the multi-channel track of the NOTSOFAR-1 task, respectively. However, the submitted full text is unreadable mojibake with a running header that identifies a different arXiv paper (hep-ex 2508.05058), so the five-stage algorithm, equations, experimental configurations, baselines, and per-system results cannot be inspected. The central empirical claim is therefore supported only by the abstract's self-reported numbers.

Significance. If the reported gains are genuine, MOVER would be a meaningful contribution, as it is claimed to be the first combination method that accommodates systems differing simultaneously in both diarization and ASR hypotheses. The choice of tcpWER on two official public benchmarks is appropriate and provides a standard, comparable metric. At the same time, the significance cannot be evaluated from the submitted materials: no method description, no experimental details, no code, and no supplementary reproducibility artifacts are visible. The abstract-level claims are plausible but entirely unverifiable in the current form.

major comments (4)
  1. [Full text (entire submission)] The supplied full text is not readable: it consists of mojibake, and the running header reads 'arXiv:2508.05058v1 [hep-ex] 7 Aug 2025,' which is not the identifier of this paper. As a result, the five-stage algorithm, all equations, the experimental setup, the baseline descriptions, the per-system results, and the tables cannot be inspected. The central empirical claim therefore rests entirely on the abstract. This is a verifiability failure that blocks any method-level assessment; it must be fixed before the paper can be evaluated.
  2. [Abstract] The headline numbers (relative tcpWER improvements of 9.55% and 8.51%) are reported without statistical support: no error bars, confidence intervals, significance tests, or per-system breakdowns. For a combination method, one needs to see the tcpWER of each individual system and the fused output to confirm that the combination beats every component. Please provide a full results table with individual system scores and some measure of variance (e.g., bootstrap confidence intervals).
  3. [Abstract] The component systems and the 'state-of-the-art' baseline are not identified. If one of the combined systems is also the SOTA baseline, then the relative improvement may be partly circular because the baseline contributes to the fused output. Please disclose the identities of all combined systems and whether the baseline is among them, and discuss the degree of output diversity across systems.
  4. [Method (unreadable)] The five-stage pipeline involves speaker alignment, segment grouping, and word/timing fusion, which inevitably contain thresholds and voting weights. No sensitivity analysis or ablation of the five stages is visible. The abstract gives no indication that the method is robust to parameter choices. Please provide an ablation study (e.g., leave-one-stage-out) and a sensitivity analysis of the key thresholds.
minor comments (3)
  1. [Abstract] The acronym tcpWER is used without definition at first use; please expand (e.g., 'total concatenated-permutation Word Error Rate') for readers outside the meeting-recognition subfield.
  2. [Introduction (wherever visible)] The claim of being the 'first approach' is strong and should be carefully positioned against prior system-combination work (e.g., ROVER, DOVER, DOVER-Lap, and other recent fusion methods).
  3. [Full text] The arXiv identifier in the running header does not match this submission. Please ensure the correct PDF is uploaded; the current material appears to be a corrupted or mislabeled file.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity identified; the central claim is an empirical benchmark result against external official metrics, and the available text provides no evidence that any prediction reduces to a fitted input or self-citation.

full rationale

The only readable portion of the manuscript is the abstract, which claims that MOVER combines meeting recognition hypotheses differing in diarization and ASR, and reports relative tcpWER improvements of 9.55% and 8.51% over state-of-the-art systems on the CHiME-8 DASR task and the NOTSOFAR-1 multi-channel track. These are external, officially scored benchmarks, not quantities derived from MOVER's own parameters by construction. The abtract does not state that a parameter is fitted to the evaluation set, nor does it invoke a self-citation as the load-bearing justification for the result. The garbled full text prevents any detailed equation-level check, but under the instruction not to manufacture circularity from unreadable material, there is no quotable step in which an output equals an input by definition or in which a fitted value is renamed as a prediction. The concerns raised in the reader's take about alignment failure modes or system overlap with baselines are correctness/verifiability risks, not evidence of circularity; they do not show that the claimed improvement is forced by the paper's own construction. Therefore the appropriate finding is no significant circularity, score 0.

Assumptions & free parameters 1 free parameters · 2 assumptions · 0 invented entities

The central claim rests on two domain assumptions (error complementarity of the combined systems, and official evaluation protocol) plus undisclosed pipeline hyperparameters. No new physical or formal entities are introduced; MOVER is an algorithm name, not a postulated entity. The paper contributes no free-floating constants beyond the hidden combination thresholds.

free parameters (1)
  • MOVER stage thresholds and voting weights (speaker alignment, segment grouping, word and timing fusion)
    Not disclosed in the abstract. Such combination pipelines typically tune alignment tolerances and voting weights on development data, so the reported gains carry hidden degrees of freedom that cannot be audited from the abstract.
assumptions (2)
  • domain assumption The component systems' outputs are sufficiently diverse and complementary that voting and alignment reduce rather than accumulate errors.
    Underlies any ensemble system-combination method; the abstract reports no per-system diversity analysis or correlation of errors.
  • domain assumption The tcpWER values are computed on official evaluation sets using the official CHiME-8 DASR and NOTSOFAR-1 scoring protocol.
    The headline gains are only meaningful if the evaluation protocol and scoring tools are the official ones; not verifiable from the abstract.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MOVER: Combining Multiple Meeting Recognition Systems." pith.science (2026). https://pith.science/paper/FQFRVXD5

@misc{pith2026250805055,
  author       = {Pith},
  title        = {Pith review of: MOVER: Combining Multiple Meeting Recognition Systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FQFRVXD5}},
  note         = {Machine review of arXiv:2508.05055}
}
read the original abstract

In this paper, we propose Meeting recognizer Output Voting Error Reduction (MOVER), a novel system combination method for meeting recognition tasks. Although there are methods to combine the output of diarization (e.g., DOVER) or automatic speech recognition (ASR) systems (e.g., ROVER), MOVER is the first approach that can combine the outputs of meeting recognition systems that differ in terms of both diarization and ASR. MOVER combines hypotheses with different time intervals and speaker labels through a five-stage process that includes speaker alignment, segment grouping, word and timing combination, etc. Experimental results on the CHiME-8 DASR task and the multi-channel track of the NOTSOFAR-1 task demonstrate that MOVER can successfully combine multiple meeting recognition systems with diverse diarization and recognition outputs, achieving relative tcpWER improvements of 9.55 % and 8.51 % over the state-of-the-art systems for both tasks.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

36 extracted references · 35 canonical work pages

  1. [1]

    Prabhavalkar, T

    R. Prabhavalkar, T. Hori, T. N. Sainath, R. Schl \"u ter, and S. Watanabe, ``End-to-end speech recognition: A survey,'' IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023

    work page 2023

  2. [2]

    Radford, J

    A. Radford, J. W. Kim, T. Xu, G. Brockman, C. Mcleavey, and I. Sutskever, ``Robust speech recognition via large-scale weak supervision,'' in Proc. ICML, vol. 202, 2023, pp. 28\,492--28\,518

    work page 2023

  3. [3]

    Haeb-Umbach, J

    R. Haeb-Umbach, J. Heymann, L. Drude, S. Watanabe, M. Delcroix, and T. Nakatani, ``Far-field automatic speech recognition,'' Proc. IEEE, vol. 109, no. 2, pp. 124--148, 2020

    work page 2020

  4. [4]

    Kraaij, T

    W. Kraaij, T. Hain, M. Lincoln, and W. Post, ``The AMI meeting corpus,'' in Proc.ternational Conference on Methods and Techniques in Behavioral Research, 2005, pp. 1--4

    work page 2005

  5. [5]

    Cornell, M

    S. Cornell, M. S. Wiesner, S. Watanabe, D. Raj, X. Chang, P. Garcia et al., ``The CHiME-7 DASR challenge: Distant meeting transcription with multiple devices in diverse scenarios,'' in Proc. International Workshop on Speech Processing in Everyday Environments (CHiME 2023), 2023, pp. 1--6

    work page 2023

  6. [6]

    Cornell, T

    S. Cornell, T. J. Park, H. Huang, C. Boeddeker, X. Chang, M. Maciejewski et al., ``The CHiME-8 DASR challenge for generalizable and array agnostic distant automatic speech recognition and diarization,'' in Proc. International Workshop on Speech Processing in Everyday Environments (CHiME 2024), 2024, pp. 1--6

    work page 2024

  7. [7]

    Vinnikov, A

    A. Vinnikov, A. Ivry, A. Hurvitz, I. Abramovski, S. Koubi, I. Gurvich et al., `` NOTSOFAR-1 challenge: New datasets, baseline, and tasks for distant meeting transcription,'' in Proc. Interspeech, 2024, pp. 5003--5007

    work page 2024

  8. [8]

    Y. Fu, L. Cheng, S. Lv, Y. Jv, Y. Kong, Z. Chen, Y. Hu, L. Xie, J. Wu, H. Bu, X. Xu, J. Du, and J. Chen, ``Aishell-4: An open source dataset for speech enhancement, separation, recognition and speaker diarization in conference scenario,'' in Interspeech 2021, 2021, pp. 3665--3669

    work page 2021

Show all 36 references
  1. [9]

    F. Yu, S. Zhang, Y. Fu, L. Xie, S. Zheng, Z. Du et al., `` M2MeT : The ICASSP 2022 multi-channel multi-party meeting transcription challenge,'' in Proc. ICASSP, 2022, pp. 6167--6171

    2022

  2. [10]

    F. Yu, S. Zhang, P. Guo, Y. Fu, Z. Du, S. Zheng et al., ``Summary on the ICASSP 2022 multi-channel multi-party meeting transcription grand challenge,'' in Proc. ICASSP, 2022, pp. 9156--9160

    2022

  3. [11]

    Liang, M

    Y. Liang, M. Shi, F. Yu, Y. Li, S. Zhang, Z. Du et al., ``The second multi-channel multi-party meeting transcription challenge ( M2MeT 2.0): A benchmark for speaker-attributed ASR ,'' in Proc. ASRU, 2023, pp. 1--8

    2023

  4. [12]

    S. Niu, R. Wang, J. Du, G. Yang, Y. Tu, S. Wu et al., ``The USTC-NERCSLIP systems for the CHiME-8 NOTSOFAR-1 challenge,'' in Proc. International Workshop on Speech Processing in Everyday Environments (CHiME 2024), 2024, pp. 31--36

    2024

  5. [13]

    Mitrofanov, T

    A. Mitrofanov, T. Prisyach, T. Timofeeva, S. Novoselov, M. Korenevsky, Y. Khokhlov et al., `` STCON system for the CHiME-8 challenge,'' in Proc. International Workshop on Speech Processing in Everyday Environments (CHiME 2024), 2024, pp. 13--17

    2024

  6. [14]

    N. Kamo, N. Tawara, A. Ando, T. Kano, H. Sato, R. Ikeshita et al., `` NTT Multi-Speaker ASR System for the DASR Task of CHiME-8 Challenge ,'' in Proc. International Workshop on Speech Processing in Everyday Environments (CHiME 2024), 2024, pp. 69--74

    2024

  7. [15]

    Huang, Y

    K. Huang, Y. Li, Z. Wang, H. Wang, W. Rao, Z. Sun et al., ``The NPU-TEA system for the CHiME-8 NOTSOFAR-1 challenge,'' in Proc. International Workshop on Speech Processing in Everyday Environments (CHiME 2024), 2024, pp. 45--48

    2024

  8. [16]

    C. B. Boeddeker, T. Cord-Landwehr, T. von Neumann, and R. Haeb-Umbach, ``Multi-stage diarization refinement for the CHiME-7 DASR scenario,'' in Proc. International Workshop on Speech Processing in Everyday Environments (CHiME 2023), 2023, pp. 51--56

    2023

  9. [17]

    Karafiat, K

    M. Karafiat, K. Veselý, I. Szoke, L. Mosner, K. Benes, M. Witkowski et al., `` BUT CHiME-7 system description,'' in Proc. International Workshop on Speech Processing in Everyday Environments (CHiME 2023), 2023, pp. 67--72

    2023

  10. [18]

    K. Deng, X. Zheng, and P. Woodland, ``The university of Cambridge system for the CHiME-7 DASR task,'' in Proc. International Workshop on Speech Processing in Everyday Environments (CHiME 2023), 2023, pp. 73--76

    2023

  11. [19]

    D. Raj, P. Denisov, Z. Chen, H. Erdogan, Z. Huang, M. He et al., ``Integration of speech separation, diarization, and recognition for multi-speaker meetings: System description, comparison, and analysis,'' in Proc. SLT, 2021, pp. 897--904

    2021

  12. [20]

    D. Raj, L. P. Garcia-Perera, Z. Huang, S. Watanabe, D. Povey, A. Stolcke et al., `` DOVER-Lap : A method for combining overlap-aware diarization outputs,'' Proc. SLT, 2021

    2021

  13. [21]

    J. G. Fiscus, ``A post-processing system to yield reduced word error rates: Recognizer Output Voting Error Reduction ( ROVER ) ,'' in Proc. ASRU, 1997, pp. 347--354

    1997

  14. [22]

    H. Xu, D. Povey, L. Mangu, and J. Zhu, ``Minimum Bayes risk decoding and system combination based on a recursion for edit distance,'' Computer Speech & Language, vol. 25, no. 4, pp. 802--828, 2011

    2011

  15. [23]

    Roark, M

    B. Roark, M. Saraclar, and M. Collins, ``Discriminative n-gram language modeling,'' Computer Speech & Language, vol. 21, no. 2, pp. 373--392, 2007

    2007

  16. [24]

    Yoshioka, I

    T. Yoshioka, I. Abramovski, C. Aksoylar, Z. Chen, M. David, D. Dimitriadis et al., ``Advances in online audio-visual meeting transcription,'' in Proc. ASRU, 2019, pp. 276--283

    2019

  17. [25]

    Von Neumann, C

    T. Von Neumann, C. Boeddeker, T. Cord-Landwehr, M. Delcroix, and R. Haeb-Umbach, ``Meeting recognition with continuous speech separation and transcription-supported diarization,'' in Proc. ICASSPW, 2024, pp. 775--779

    2024

  18. [26]

    Zhou, Ensemble Methods: Foundations and Algorithms

    Z.-H. Zhou, Ensemble Methods: Foundations and Algorithms. 1em plus 0.5em minus 0.4em Chapman and Hall/CRC, 2012

    2012

  19. [27]

    J. S. Garofolo, J. G. Fiscus, and C. D. Laprun, The rich transcription 2004 spring meeting recognition evaluation. 1em plus 0.5em minus 0.4em US Department of Commerce, National Institute of Standards and Technology, 2004

    2004

  20. [28]

    Sankoff and J

    D. Sankoff and J. B. Kruskal, Time Warps, String Edits, and Macromolecules: The Theory and Practice of Sequence Comparison. 1em plus 0.5em minus 0.4em Addison-Wesley, 1983

    1983

  21. [29]

    Stolcke and T

    A. Stolcke and T. Yoshioka, `` DOVER : A method for combining diarization outputs,'' in Proc. ASRU, 2019, pp. 757--763

    2019

  22. [30]

    McAuliffe, M

    M. McAuliffe, M. Socolof, S. Mihuc, M. Wagner, and M. Sonderegger, ``Montreal forced aligner: Trainable text-speech alignment using kaldi,'' in Proc. Interspeech, 2017, pp. 498--502

    2017

  23. [31]

    von Neumann, C

    T. von Neumann, C. Boeddeker, M. Delcroix, and R. Haeb-Umbach, `` MeetEval : A toolkit for computation of word error rates for meeting transcription systems,'' in Proc. International Workshop on Speech Processing in Everyday Environments (CHiME 2023), 2023, pp. 27--32

    2023

  24. [32]

    Watanabe, M

    S. Watanabe, M. Mandel, J. Barker, E. Vincent, A. Arora, X. Chang et al., `` CHiME-6 challenge: Tackling multispeaker speech recognition for unsegmented recordings,'' in Proc. International Workshop on Speech Processing in Everyday Environments (CHiME 2020), 2020, pp. 1--7

    2020

  25. [33]

    M. V. Segbroeck, Z. Ahmed et al., `` DiPCo---Dinner Party Corpus ,'' in Proc. Interspeech, 2020, pp. 434--436

    2020

  26. [34]

    Brandschain, D

    L. Brandschain, D. Graff, C. Cieri, K. Walker, C. Caruso, and A. Neely, ``The Mixer 6 corpus: Resources for cross-channel and text independent speaker recognition,'' in Proc. LREC, 2010, pp. 2441--2444

    2010

  27. [35]

    N. Kamo, N. Tawara, A. Ando, T. Kano, H. Sato, R. Ikeshita et al., ``Microphone array geometry independent multi-talker distant ASR : NTT system for the DASR task of the CHiME-8 challenge,'' arXiv preprint arXiv:2502.09859, 2025

    2025 arXiv

  28. [36]

    write newline

    " write newline "" initialize.prev.this.status FUNCTION begin.bib " write newline preamble empty 'skip preamble write newline if " thebibliography " longest.label * " " * write newline " [1] #1 " write newline " url@samestyle " write newline " " write newline " [2] #2 " write ...

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.