Pith. sign in

REVIEW 3 major objections 5 minor 43 references

Cocktail-Party Audio-Visual Speech Recognition

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Training on silent faces cuts cocktail-party speech errors from 119% to 39.2%.

desk verdict Useful benchmark and dataset, but the headline WER gain rests on an underspecified <unk> scoring rule that likely inflates the improvement. read the letter →

arxiv 2506.02178 v1 pith:6FH6K27I submitted 2025-06-02 cs.SD cs.CL

Thai-Binh Nguyen , Ngoc-Quan Pham , Alexander Waibel This is my paper
classification cs.SDcs.CL
keywords audio-visualspeechrecognitioncocktail-partysilent-facesegmentsdataaugmentationAV-HuBERTAVCocktailAVYTmulti-talkernoise
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that audio-visual speech recognition (AVSR) models fail in cocktail-party settings not only because of overlapping speech but because visible speakers are often silent, and standard training data never includes such silent-face segments. To fix this, the authors build AVCocktail, a single-channel English benchmark with multiple overlapping conversations around a 360-degree camera, and AVYT, a 1526-hour training set of talking-face and silent-face clips harvested from YouTube. Fine-tuning an AV-HuBERT encoder with a CTC/Attention decoder on this data plus dialog-style augmentation and added interfering speakers cuts word error rate from 119% to 39.2% in the hardest fixed-chunk evaluation, a 67% relative improvement over the strongest prior system. The central practical claim is that a model trained to associate silent faces with no output stops hallucinating text in realistic noisy video.

What carries the argument

The load-bearing mechanism is a training pipeline that concatenates shuffled clips from LRS2, VoxCeleb2, and the new AVYT sets into dialog-like samples, optionally inserting silent-face clips labeled with an unknown token and adding interfering speakers at controlled signal-to-noise ratios. Silent-face clips prevent the model from treating every visible mouth as speech, and dialog augmentation prevents it from recognizing silent segments by frame-sequence differences alone; the paper reports that full AVYT with dialog augmentation is what drops AVCocktail WER from 58.8% to 22.6%. The recognition architecture is an AV-HuBERT encoder with joint CTC/Attention decoding, fine-tuned from a self-supervised audio-visual model.

What would settle it

Take the AVYT silent-face and talking-face subsets, have humans verify the active-speaker labels and transcripts on a random sample, then retrain the AV-HuBERT model on the corrected subset and evaluate on AVCocktail fixed-chunk segmentation. If WER rises back toward the 58.8% baseline, the reported gain is partly an artifact of label noise; if it stays near 22-39%, the silent-face mechanism is real.

Watch

Extended reading notes

Core claim

The paper's central claim is that silent-face segments, not just acoustic noise, are the dominant failure mode for AVSR in cocktail-party environments, and that exposing a model to them during training transfers to real recordings. On the LRS2 benchmark with added interferers, all published baselines degrade sharply (Auto-AVSR from 1.7% to 69.6% WER with two interferers at -5 dB), while the fine-tuned AV-HuBERT model stays near 4.1% average WER. On AVCocktail, where 53.7% of fixed 10-second chunks contain a silently visible speaker, the same model reaches 39.2% WER versus 119% for the best prior audio-visual system; with ideal segmentation it reaches 18.2%. When the recognizer must decide on its own where speech starts and stops, the combination of silent-face clips and dialog augmentation is the deciding factor: including both lowers WER from 58.8% to 22.6% in the ablation.

Load-bearing premise

The result depends on the automatically generated AVYT labels being correct: talking versus silent comes from an off-the-shelf active-speaker detector plus an audio-video sync check, and transcripts come from Whisper-large without human verification, so systematic label noise could inflate the measured improvement.

Editorial extensions

If this is right

  • Models trained only on clean talking-face datasets will hallucinate when a visible speaker is silent, so silent-face data should become a standard ingredient in AVSR training.
  • The AVCocktail benchmark gives the community a single-channel, multi-party English evaluation that can expose robustness failures hidden by LRS2/LRS3.
  • The ablation shows that dialog augmentation and silent-face clips act together; adding either alone captures only part of the benefit.
  • Because the model works with fixed 10-second chunks, it does not need explicit speech-activity segmentation at inference time.
  • Published AVSR systems that look strong on clean datasets can degrade to unusable WERs under realistic cocktail-party noise, so performance on such benchmarks should be measured separately.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One consequence the authors do not develop is that the same silent-face recipe could help audio-only ASR in camera-bearing devices, since the visual channel can veto hallucinations.
  • A direct test of the mechanism would compare WER on chunks containing a silent face against chunks containing a talking face; the model should show its largest gain on the silent chunks.
  • If label noise is low, the 22.6% WER with active-speaker-detection segmentation suggests that end-to-end AVSR may replace explicit speaker diarization in meeting transcription.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper addresses cocktail-party audio-visual speech recognition by introducing AVCocktail, a human-labeled English benchmark with overlapping speakers and single-channel audio, and AVYT, a 1526-hour YouTube-derived dataset that contains both talking-face and silent-face clips. The authors fine-tune AV-HuBERT and a Conformer-based AVSR model with a data-augmentation pipeline that combines dialog-style concatenation, silent-face clips, and interferer noise. They report up to a 67% relative WER reduction over the Muavic baseline (from 119.0% to 39.2%) under fixed 10-second chunking, and their ablations attribute the gain mainly to combining silent-face clips with dialog augmentation. The datasets and models are planned for public release.

Significance. The paper makes a useful contribution: AVCocktail is a realistic evaluation resource that fills a real gap, AVYT addresses an underexplored training-data limitation in AVSR, and the ablation order in Table 3 is sensible and supports the central claim that silent-face-aware training plus dialog augmentation is the main driver of the improvement. If the evaluation protocol is clarified, the work would be a valuable demonstration that end-to-end AVSR can handle inactive-speaker segments without explicit inference-time segmentation. The planned public release of data and code is a further strength.

major comments (3)
  1. [Sec. 5, Table 2, and Sec. 3.4] The paper does not state how the <unk> silence token is treated when computing WER. Section 3.4 says silent-face clips are labeled with <unk> transcripts, and in the fixed-chunk protocol 53.7% of AVCocktail frames are silent-face segments. If the reference for these segments is empty and <unk> is stripped from hypotheses, AV1 is rewarded for abstaining on silence while baselines are charged for every hallucinated word; if <unk> is counted as a reference token, the evaluation becomes a joint speech-recognition and silence-detection task rather than transcription of speech. The authors must specify the scoring rule and, ideally, report WER both with and without <unk> in the reference and hypothesis. This is load-bearing for the headline 67% claim and also affects the numbers in Table 3.
  2. [Sec. 4 and Sec. 5, Tables 1-3] All results appear to be single training runs with no error bars, confidence intervals, or seed variation. For a paper whose main evidence is WER differences between fine-tuned models and baselines, this is a significant omission, particularly in Table 3 where the central comparison is 22.6% versus 28.5%. The large gaps make it likely that the qualitative ordering is stable, but the authors should report mean and standard deviation over at least three runs, or otherwise provide a variance estimate for the main comparisons.
  3. [Sec. 3.3] The AVYT labels are generated automatically: an off-the-shelf active speaker detector, an audio-video sync check, and Whisper-large transcription with no manual verification. The correctness of these labels is load-bearing, because the model is trained to emit <unk> on silent-face segments on the basis of this pipeline. If the ASD or sync labels are systematically wrong, or if Whisper hallucinates on silent or noisy clips, the model could be learning artifacts of the label-generation pipeline rather than genuine speech-activity awareness. Please report ASD precision/recall on AVYT, sync-filter statistics, and a sample or CER estimate of Whisper transcripts, or at least discuss this risk explicitly.
minor comments (5)
  1. [Table 1] Table 1 is very hard to parse: the Avg-5 column appears to collide with the per-SNR columns, and the Interferer rows are not clearly separated. Please reformat with explicit columns for interferer count and SNR.
  2. [Sec. 3.2] The dataset description says the pipeline extracts 'a single audio channel for each speaker'; please clarify whether this is the same shared monaural recording or separate channels, since the task definition in Section 2.1 implies a single audio stream.
  3. [Algorithm 1] Algorithm 1 does not specify n, the number of randomly added clips; the source corpus and SNR distribution of the speech noise in line 12 are also unspecified. Please provide these hyperparameters and augmentation details for reproducibility.
  4. [Sec. 3 and Sec. 3.3] Whisper-large is cited as [2] in Sections 3 and 3.3, but reference [2] is Muavic; the correct citation for Whisper-large is [28]. Also, Table 2 contains the typo '70 .8' in the AV5 row.
  5. [Sec. 3] The phrase 'retain only the English segments' in the description of Vox2 processing needs a concrete language-filtering procedure, as this could substantially affect the training data.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the headline WER gain is an externally measured benchmark result, not a derivation forced by construction.

full rationale

The paper's central claim is an empirical WER comparison on AVCocktail, a newly collected, human-transcribed benchmark. AV1 is fine-tuned on AVYT, which includes silent-face clips labeled with <unk>, and is then evaluated on the same 10-second fixed chunks as all baselines; the reported reductions are measured outcomes rather than quantities fitted to or derived from the target metric. The self-citations present ([31], [34], [35]) supply a YouTube video source and a dialog-augmentation technique, but the paper provides its own ablation study (Table 3) quantifying the contribution of dialog augmentation and silent-face data, so the central result does not reduce to an unverified self-citation. The one substantive concern is that the paper does not state how the <unk> token is treated when computing WER on fixed chunks containing 53.7% silent-face segments; if <unk> were deleted from hypotheses before scoring, part of AV1's advantage could reflect an evaluation asymmetry rather than improved recognition. However, the text does not specify such a scoring rule, so this remains an unspecified protocol detail and a correctness/interpretability risk, not a demonstrated circular step under the standards of this review.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The empirical claim rests on the new AVCocktail and AVYT artifacts plus the augmentation pipeline; no new theoretical or physical entities are introduced. The main unresolved burden is label quality in automatically generated AVYT.

free parameters (3)
  • Fixed 10-second chunk length = 10 s
    The headline 39.2% WER and the 67% relative reduction come from the fixed 10-second chunk condition in Table 2. Changing the chunk length changes how much silent-face context appears in each inference segment, and no sensitivity analysis is provided.
  • Interferer count and SNR grid = up to 2 interferers at -5, 0, 5, 10 dB
    Defines the extreme-noise condition in Table 1 and the augmentation recipe in Section 3.4. Only these hand-chosen values are tested.
  • Dialog augmentation clip count n = not reported
    Algorithm 1 samples n random additional clips for dialog and silent-face concatenation, but n is never specified, so the strongest ablation (22.6% WER) cannot be exactly reproduced.
assumptions (3)
  • domain assumption Automatic AVYT labels are reliable enough for training.
    Section 3.3 labels talking versus silent with active speaker detection and audio-video sync and transcribes with Whisper-large, but no manual verification, error analysis, or quality statistics are reported.
  • domain assumption Human transcripts of AVCocktail are correct ground truth.
    Section 3.2 says recordings were segmented and transcribed by humans, but no annotation instructions, inter-annotator agreement, or verification procedure is described.
  • domain assumption Single-channel 360-degree audio preserves enough target-speaker information for meaningful AVSR evaluation.
    Section 2.1 allows overlapping speech and noise in one audio channel. Unlike MISP, AVCocktail does not provide multi-channel or beamformed audio, so the acoustic evidence for each speaker is limited to the single recording channel.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Cocktail-Party Audio-Visual Speech Recognition." pith.science (2026). https://pith.science/paper/6FH6K27I

@misc{pith2026250602178,
  author       = {Pith},
  title        = {Pith review of: Cocktail-Party Audio-Visual Speech Recognition},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6FH6K27I}},
  note         = {Machine review of arXiv:2506.02178}
}
read the original abstract

Audio-Visual Speech Recognition (AVSR) offers a robust solution for speech recognition in challenging environments, such as cocktail-party scenarios, where relying solely on audio proves insufficient. However, current AVSR models are often optimized for idealized scenarios with consistently active speakers, overlooking the complexities of real-world settings that include both speaking and silent facial segments. This study addresses this gap by introducing a novel audio-visual cocktail-party dataset designed to benchmark current AVSR systems and highlight the limitations of prior approaches in realistic noisy conditions. Additionally, we contribute a 1526-hour AVSR dataset comprising both talking-face and silent-face segments, enabling significant performance gains in cocktail-party environments. Our approach reduces WER by 67% relative to the state-of-the-art, reducing WER from 119% to 39.2% in extreme noise, without relying on explicit segmentation cues.

Figures

Figures reproduced from arXiv: 2506.02178 by the authors.

Figure 2
Figure 2. Pipeline to generate AVYT dataset (AV5) [14] combines AV-HuBERT with Whisper-large and is trained on LRS3, Vox2, and augmented noisy data. Lastly, Whisper-large (A1), trained on 5M hours of diverse audio data, serves as a strong baseline for benchmarking audio-only perfor￾mance. 3. Data Preparation In this study, we utilize four datasets. For training, we use LRS2 (train and pretrain sets), Vox2 (train set), and AVY… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

43 extracted references · 30 canonical work pages

  1. [1]

    Introduction The visual information obtained from observing a person speak can alter the way auditory signals are perceived, a phenomenon known as the McGurk effect [1]. In cocktail-party environ- ments, even strong ASR models [2, 3] which mark a significant advance over early efforts [4] in conversational speech, still ex- perience significant performanc...

    work page 1976

  2. [2]

    Task definition Given an input sequence of audio A = {a1, a2,

    Cocktail-Party A VSR 2.1. Task definition Given an input sequence of audio A = {a1, a2, . . . , aT } and video V = {v1, v2, . . . , vT }, where T is the total number of time steps, at represents the audio feature and vt represents the visual feature at time step t. The task is to predict the tran- scription Ytarget = A VSR(A, V) = {y1, y2, . . . , yN }, w...

    work page

  3. [3]

    For training, we use LRS2 (train and pretrain sets), V ox2 (train set), and A VYT

    Data Preparation In this study, we utilize four datasets. For training, we use LRS2 (train and pretrain sets), V ox2 (train set), and A VYT. For testing, we evaluate on the LRS2 test set (including a modified version) and A VCocktail. Details of LRS2 test set, A VYT, and A VCock- tail are provided in the following subsections. About V ox2, we simple emplo...

    work page

  4. [4]

    The A V-HuBERT CTC/Attention (A V1) model uses the A V-HuBERT large [12] as the encoder, which has 24 transformer blocks, each with 16 attention heads

    Experimental setup As described in Section 2.2, we fine-tune two model ar- chitectures. The A V-HuBERT CTC/Attention (A V1) model uses the A V-HuBERT large [12] as the encoder, which has 24 transformer blocks, each with 16 attention heads. The CTC/Attention decoder is a 6-layer Transformer with the same dimensions and number of attention heads as the enco...

    work page

  5. [5]

    The WERs for models evaluated on the original LRS2 test set are shown in the column where SNR = ∞

    Results Table 1 presents the WER (%) of baseline models and our fine- tuned models on the LRS2 test set, including both the original and modified versions. The WERs for models evaluated on the original LRS2 test set are shown in the column where SNR = ∞. Overall, all models perform well on the original clean LRS2 dataset. The best-performing model in this...

    work page

  6. [6]

    We highlighted the gap between conventional datasets and real-world cocktail- party scenarios, where target speakers are not always ac- tive

    Conclusion In this study, we benchmarked SOTA A VSR models, which per- form impressively on conventional datasets like LRS2/LRS3 but struggle with cocktail-party scenarios. We highlighted the gap between conventional datasets and real-world cocktail- party scenarios, where target speakers are not always ac- tive. The presence of silent-face segments signi...

    work page

  7. [7]

    How is AI Changing Science? Research in the Era of Learning Algorithms

    Acknowledgment The authors gratefully acknowledge support from Carl Zeiss Stiftung under the project Jung bleiben mit Robotern (P2019- 01-002). This work was also partially supported by the Eu- ropean Union’s Horizon research and innovation programme (grant No. 101135798, project Meetween), the V olkswagen Foundation project “How is AI Changing Science? R...

    work page

  8. [8]

    Knowing who to listen to in speech recognition: Visually guided beamforming,

    U. Bub, M. Hunke, and A. Waibel, “Knowing who to listen to in speech recognition: Visually guided beamforming,” in1995 Inter- national Conference on Acoustics, Speech, and Signal Processing, vol. 1. IEEE, 1995, pp. 848–851

    work page 1995

Show all 43 references
  1. [9]

    Hearing lips and seeing voices,

    H. McGurk and J. MacDonald, “Hearing lips and seeing voices,” Nature, vol. 264, no. 5588, pp. 746–748, 1976

    1976

  2. [10]

    Muavic: A multilingual audio-visual corpus for robust speech recognition and robust speech-to-text translation,

    M. Anwar, B. Shi, V . Goswami, W.-N. Hsu, J. Pino, and C. Wang, “Muavic: A multilingual audio-visual corpus for robust speech recognition and robust speech-to-text translation,” in Interspeech 2023, 2023, pp. 4064–4068

    2023

  3. [11]

    Super-human per- formance in online low-latency recognition of conversational speech,

    T.-S. Nguyen, S. St ¨uker, and A. Waibel, “Super-human per- formance in online low-latency recognition of conversational speech,” arXiv preprint arXiv:2010.03449, 2020

    2010 arXiv

  4. [12]

    as the encoder and the decoder integrates a projection layer and a Transformer decoder with joint CTC/Attention training [29]. The second model is the Conformer CTC/Attention archi- tecture (A V2) proposed by [30], where the encoder consists of two Conformer blocks: one for au...

  5. [13]

    Recognition of conversational telephone speech using the janus speech engine,

    T. Zeppenfeld, M. Finke, K. Ries, M. Westphal, and A. Waibel, “Recognition of conversational telephone speech using the janus speech engine,” in IEEE International Conference on Acoustics, Speech, and Signal Processing. IEEE, 1997

    1997

  6. [14]

    See me, hear me: inte- grating automatic speech recognition and lip-reading,

    P. Duchnowski, U. Meier, and A. Waibel, “See me, hear me: inte- grating automatic speech recognition and lip-reading,” in 3rd In- ternational Conference on Spoken Language Processing (ICSLP 1994), 1994, pp. 547–550

    1994

  7. [15]

    Multimodal interfaces,

    A. Waibel, M. T. V o, P. Duchnowski, and S. Manke, “Multimodal interfaces,” Artificial Intelligence Review, vol. 10, 1996

    1996

  8. [16]

    Modeling focus of at- tention for meeting indexing,

    R. Stiefelhagen, J. Yang, and A. Waibel, “Modeling focus of at- tention for meeting indexing,” inProceedings of the seventh ACM international conference on Multimedia (Part 1), 1999, pp. 3–10

    1999

  9. [17]

    Visual track- ing for multimodal human computer interaction,

    J. Yang, R. Stiefelhagen, U. Meier, and A. Waibel, “Visual track- ing for multimodal human computer interaction,” in Proceedings of the SIGCHI conference on Human factors in computing sys- tems, 1998

    1998

  10. [18]

    Estimating focus of attention based on gaze and sound,

    R. Stiefelhagen, J. Yang, and A. Waibel, “Estimating focus of attention based on gaze and sound,” in Proceedings of the 2001 workshop on Perceptive user interfaces, 2001, pp. 1–9

    2001

  11. [19]

    Chil: Computers in the human interaction loop,

    A. Waibe11, H. Steusloff, R. Stiefelhagen et al., “Chil: Computers in the human interaction loop,” 2005

    2005

  12. [20]

    Robust self-supervised audio-visual speech recognition,

    B. Shi, W.-N. Hsu, and A. Mohamed, “Robust self-supervised audio-visual speech recognition,” in Interspeech 2022, 2022

    2022

  13. [21]

    Auto-avsr: Audio-visual speech recognition with automatic labels,

    P. Ma, A. Haliassos, A. Fernandez-Lopez, H. Chen, S. Petridis, and M. Pantic, “Auto-avsr: Audio-visual speech recognition with automatic labels,” in ICASSP, 2023, pp. 1–5

    2023

  14. [22]

    Whisper-flamingo: Integrating visual fea- tures into whisper for audio-visual speech recognition and trans- lation,

    A. Rouditchenko, Y . Gong, S. Thomas, L. Karlinsky, H. Kuehne, R. Feris, and J. Glass, “Whisper-flamingo: Integrating visual fea- tures into whisper for audio-visual speech recognition and trans- lation,” in Interspeech 2024, 2024, pp. 2420–2424

    2024

  15. [23]

    Speaker-targeted audio-visual models for speech recognition in cocktail-party environments,

    G.-L. Chao, W. Chan, and I. Lane, “Speaker-targeted audio-visual models for speech recognition in cocktail-party environments,” in Interspeech 2016, 2016, pp. 2120–2124

    2016

  16. [24]

    Audio-visual multi-talker speech recognition in a cocktail party,

    Y . Wu and et al., “Audio-visual multi-talker speech recognition in a cocktail party,” inInterspeech, 2021

    2021

  17. [25]

    Robust audio-visual asr with unified cross-modal attention,

    J. Li, C. Li, Y . Wu, and Y . Qian, “Robust audio-visual asr with unified cross-modal attention,” in ICASSP, 2023, pp. 1–5

    2023

  18. [26]

    Looking to listen at the cocktail party: a speaker-independent audio-visual model for speech separation,

    A. Ephrat, I. Mosseri, O. Lang, T. Dekel, K. Wilson, A. Hassidim, W. T. Freeman, and M. Rubinstein, “Looking to listen at the cocktail party: a speaker-independent audio-visual model for speech separation,” ACM Trans. Graph., vol. 37, no. 4, Jul. 2018. [Online]. Available: htt...

    2018

  19. [27]

    Visualvoice: Audio-visual speech sep- aration with cross-modal consistency,

    R. Gao and K. Grauman, “Visualvoice: Audio-visual speech sep- aration with cross-modal consistency,” in 2021 IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR) . IEEE, 2021, pp. 15 490–15 500

    2021

  20. [28]

    Seeing through the conversation: Audio-visual speech separation based on diffu- sion model,

    S. Lee, C. Jung, Y . Jang, J. Kim, and J. S. Chung, “Seeing through the conversation: Audio-visual speech separation based on diffu- sion model,” in ICASSP, 2024, pp. 12 632–12 636

    2024

  21. [29]

    Lip read- ing sentences in the wild,

    J. S. Chung, A. Senior, O. Vinyals, and A. Zisserman, “Lip read- ing sentences in the wild,” in2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 3444–3453

    2017

  22. [30]

    Lrs3-ted: a large-scale dataset for visual speech recognition,

    T. Afouras, J. S. Chung, and A. Zisserman, “Lrs3-ted: a large-scale dataset for visual speech recognition,” 2018. [Online]. Available: https://arxiv.org/abs/1809.00496

    2018 arXiv

  23. [31]

    V oxceleb2: Deep speaker recognition,

    J. S. Chung, A. Nagrani, and A. Zisserman, “V oxceleb2: Deep speaker recognition,” in Interspeech 2018, 2018, pp. 1086–1090

    2018

  24. [32]

    Unified cross-modal at- tention: Robust audio-visual speech recognition and beyond,

    J. Li, C. Li, Y . Wu, and Y . Qian, “Unified cross-modal at- tention: Robust audio-visual speech recognition and beyond,” IEEE/ACM Transactions on Audio, Speech, and Language Pro- cessing, vol. 32, pp. 1941–1953, 2024

    1941

  25. [33]

    The first multimodal information based speech processing (misp) challenge: Data, tasks, baselines and results,

    H. Chen and et al., “The first multimodal information based speech processing (misp) challenge: Data, tasks, baselines and results,” in ICASSP, 2022, pp. 9266–9270

    2022

  26. [34]

    Summary on the multimodal information based speech processing (misp) 2022 challenge,

    H. Chen, S. Wu, Y . Dai, Z. Wang, J. Du, C.-H. Lee, J. Chen, S. Watanabe, S. M. Siniscalchi, O. Scharenborg, D.-Y . Liu, B.-C. Yin, J. Pan, J.-Q. Gao, and C. Liu, “Summary on the multimodal information based speech processing (misp) 2022 challenge,” in ICASSP, 2023, pp. 1–2

    2022

  27. [35]

    Summary on the multimodal information- based speech processing (misp) 2023 challenge,

    H. Chen and et al., “Summary on the multimodal information- based speech processing (misp) 2023 challenge,” in ICASSPW, 2024, pp. 123–124

    2023

  28. [36]

    Robust speech recognition via large- scale weak supervision,

    A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large- scale weak supervision,” 2022. [Online]. Available: https: //arxiv.org/abs/2212.04356

    2022 arXiv

  29. [37]

    Hy- brid ctc/attention architecture for end-to-end speech recognition,

    S. Watanabe, T. Hori, S. Kim, J. R. Hershey, and T. Hayashi, “Hy- brid ctc/attention architecture for end-to-end speech recognition,” IEEE Journal of Selected Topics in Signal Processing , vol. 11, no. 8, pp. 1240–1253, 2017

    2017

  30. [38]

    End-to-end audio-visual speech recognition with conformers,

    P. Ma, S. Petridis, and M. Pantic, “End-to-end audio-visual speech recognition with conformers,” in ICASSP, 2021, pp. 7613–7617

    2021

  31. [39]

    From text segmentation to smart chaptering: A novel benchmark for structuring video transcriptions,

    F. Retkowski and A. Waibel, “From text segmentation to smart chaptering: A novel benchmark for structuring video transcriptions,” in EACL, Mar. 2024, pp. 406–419. [Online]. Available: https://aclanthology.org/2024.eacl-long.25

    2024

  32. [40]

    A light weight model for active speaker detection,

    J. Liao, H. Duan, K. Feng, W. Zhao, Y . Yang, and L. Chen, “A light weight model for active speaker detection,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2023, pp. 22 932–22 941

    2023

  33. [41]

    Out of time: Automated lip sync in the wild,

    J. S. Chung and A. Zisserman, “Out of time: Automated lip sync in the wild,” in ACCV Workshops, 2016. [Online]. Available: https://api.semanticscholar.org/CorpusID:26294509

    2016

  34. [42]

    Synthetic conversations improve multi-talker asr,

    T.-B. Nguyen and A. Waibel, “Synthetic conversations improve multi-talker asr,” inICASSP 2024 - 2024 IEEE International Con- ference on Acoustics, Speech and Signal Processing (ICASSP) , 2024, pp. 10 461–10 465

    2024

  35. [43]

    Msa-asr: Efficient multilingual speaker attribution with frozen asr models,

    T.-B. Nguyen and Waibel, “Msa-asr: Efficient multilingual speaker attribution with frozen asr models,” in ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2025, pp. 1–5

    2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.