Pith. sign in

REVIEW 5 major objections 5 minor 41 references

ViCocktail: Automated Multi-Modal Data Collection for Vietnamese Audio-Visual Speech Recognition

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read An automated pipeline builds a 269-hour Vietnamese AVSR dataset whose best model keeps 18% WER under heavy babble, versus 70% for audio-only ASR.

desk verdict First public Vietnamese AVSR dataset plus a credible transfer result; label-quality audit needed, but the noise-robustness core holds. read the letter →

arxiv 2506.04635 v1 pith:JKM2R6O7 submitted 2025-06-05 cs.CL cs.CV

Thai-Binh Nguyen , Thi Van Nguyen , Quoc Truong Do , Chi Mai Luong This is my paper
classification cs.CLcs.CV
keywords audio-visualspeechrecognitionVietnameseautomateddatacollectiondatasetpipelinelipreadingcross-lingualtransfercocktail-partynoiseAV-HuBERT
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that a fully automated pipeline can create an audio-visual speech recognition (AVSR) training set for a language with scarce resources, and that a model pretrained on English video can then be adapted to that language. It demonstrates this for Vietnamese, releasing 269 hours of automatically labeled video from YouTube talk shows and vlogs along with a 2-hour manually annotated test set. The best model, an English-initialized AV-HuBERT fine-tuned with a CTC/attention decoder, matches a strong audio-only Vietnamese ASR on clean speech (9.40% vs 7.53% WER) and stays at 18.39% WER under two added talkers at -5 dB, where the audio-only model collapses to 70.01%. If correct, this gives under-resourced languages a practical route to speech recognition that resists noise without expensive manual transcription.

What carries the argument

The load-bearing pipeline runs raw YouTube video through shot detection, face detection and tracking, an active-speaker detector, SyncNet for audio-visual synchronization, and landmark-based mouth-region extraction that crops 96x96 mouth frames synchronized with 16 kHz audio. Text comes from a Vietnamese Wav2Vec2 model that transcribes all 194,073 training utterances. On the modeling side, the key object is the AV-HuBERT large encoder, self-supervised on English video before being initialized from English (AV1) or multilingual (AV2) checkpoints and fine-tuned with a CTC/attention decoder under a weighted CTC-plus-cross-entropy loss; audio and visual features are fused before the transformer layers. An alternative Conformer encoder initialized from Auto-AVSR is included as a second architecture.

What would settle it

Sample roughly 1,000 of the 194,073 training utterances, have native Vietnamese transcribers relabel them, and compare; if the automatic labels' WER on that sample exceeds about 10%, the clean-condition comparison between the AVSR model and the teacher could be reflecting label noise rather than model quality, putting the automated-label premise in question.

Watch

Extended reading notes

Core claim

The central claim is that an automatically collected 269-hour Vietnamese dataset is sufficient to train an audio-visual model that holds up under noise when the encoder starts from an English-pretrained AV-HuBERT. On the paper's manually transcribed test set, this model (AV1) is competitive with the Vietnamese Wav2Vec2 ASR that generated the training labels under clean conditions, and it barely degrades as babble is added: in the hardest condition, two interfering speakers at -5 dB, its WER doubles to 18.39% while the audio-only Wav2Vec2 rises more than eightfold to 70.01%. The authors take this as evidence that visual speech information transfers across languages and that fully automated collection can substitute for manual labeling in bootstrapping AVSR for a new language.

Load-bearing premise

The 194,073 training transcripts are generated automatically by a Vietnamese Wav2Vec2 model with no manual correction or quality audit reported, so the entire dataset inherits whatever errors that teacher makes on casual YouTube speech.

Editorial extensions

If this is right

  • A new language can get an AVSR training set from raw video whenever a reasonably reliable ASR model exists for it, since the pipeline needs no manual transcription.
  • An English-pretrained AV-HuBERT encoder transfers to Vietnamese better than a multilingual checkpoint or training from scratch, so self-supervised audio-visual pretraining appears to carry across languages.
  • The released benchmark gives Vietnamese AVSR a public test set with nine (SNR, interferer) conditions, enabling direct comparison of future models.
  • The audio-visual model's WER roughly doubles rather than rising eightfold under the harshest babble tested, so the visual stream supplies most of the noise invariance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A likely follow-on the paper leaves implicit: any language with a decent ASR teacher could rerun this pipeline, shifting the bottleneck from manual annotation to teacher quality.
  • A testable extension the paper does not run: ablate the audio stream entirely under -5 dB babble; if WER stays near 18%, the model is essentially lip-reading and the visual stream carries the robust signal.
  • Related open question: separating data scale from initialization, for example training an English-initialized model on a much smaller Vietnamese subset, would clarify whether the 269 hours or the pretrained encoder deserves the credit.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. This paper proposes an automated pipeline for collecting audio-visual speech recognition (AVSR) data from raw YouTube videos, applies it to Vietnamese, and releases a 269-hour dataset with a manually annotated 2-hour test set. It fine-tunes AV-HuBERT, Conformer, audio-only, and visual-only models initialized from English, multilingual, or scratch checkpoints, and evaluates them under simulated cocktail-party noise. The headline result is that the audio-visual AV1 model reaches 9.40% WER with no added noise and 18.39% WER at -5 dB with two interferers, compared with 70.01% for the Wav2Vec2 ASR baseline and 57.67% for the audio-only AV-HuBERT control.

Significance. If the claims hold, the paper contributes the first open Vietnamese AVSR dataset, an open-source collection pipeline, and evidence that English AV-HuBERT initialization transfers to a low-resource language. A particular strength is the inclusion of audio-only AV-HuBERT controls (A3/A4): because they share the pre-training and fine-tuning protocol with AV1, the large gap at low SNR (18.39 vs 57.67 at -5 dB with two interferers) isolates the visual contribution much better than the off-the-shelf ASR baselines. The manual annotation of the test set also avoids circular evaluation. The main risks are the unmeasured quality of the 194,073 automatic training transcripts, the mislabeling of a naturally noisy test set as "clean," and the absence of uncertainty estimates.

major comments (5)
  1. [2.1, Table 2] The 194,073 training transcripts are generated automatically by the Vietnamese Wav2Vec2 model with no reported accuracy audit on the training distribution. The only evidence of teacher quality is the 7.53% WER of that same model (A1) on the manually annotated test set, but the test set is a separate, deliberately noisy sample. If the teacher systematically misrecognizes colloquial YouTube speech, then AV1's clean-condition WER of 9.40% is partly an imitation ceiling, so the "competitive in clean conditions" claim is confounded. The noise-robustness headline is protected by the A3/A4 audio-only controls, but the dataset-quality and clean-condition claims are not. Please report the teacher WER on a manually transcribed random sample of the training set and, preferably, an analysis of whether AV1's residual errors overlap with A1's.
  2. [3.1, Abstract, Section 4] The condition labeled "clean" is not clean. Section 3.1 states that the original test set "already includes naturally noisy samples with various noise types such as natural, music, and babble," yet the abstract and Section 4 call the (∞,0) condition "clean conditions" and "the clean set." The paper should either restrict the clean evaluation to utterances that are actually noise-free or consistently describe this condition as "no added noise." As written, the clean-condition comparison (9.40% vs 7.53%) is overstated.
  3. [4, Table 2] The headline comparison between AV1 and A1 under noise is not a controlled comparison. A1 is an off-the-shelf ASR that has not been fine-tuned on the noisy mixtures used in training, while AV1 has been fine-tuned with up to two interferers at -5 to 10 dB. The appropriate control for isolating the visual modality is A3/A4, which use the same AV-HuBERT architecture and fine-tuning regime as AV1; at -5 dB with two interferers, AV1 achieves 18.39% versus A3's 57.67%. The text should lead with this control comparison and explicitly acknowledge that the 8.3-fold versus 2-fold statement against A1 conflates modality, architecture, and training data distribution.
  4. [Table 2] All reported WERs are single runs on a 2-hour, 1,167-utterance test set, with no repeated seeds or confidence intervals. Several claims rest on moderate differences—for example, the clean-condition gap between AV1 (9.40) and A1 (7.53) is only 1.87 WER points—and could change under seed or bootstrap variability. Please provide at least two seeds for the main models or bootstrap confidence intervals on the test utterances.
  5. [4, Table 2] The claim that visual-only models are "unaffected by noise" is not supported by the table, which reports only a single WER for V1-V5 (e.g., 41.34 for V1). If the visual stream is truly invariant to the additive audio noise, that should be demonstrated by evaluating V1 under the same nine (SNR, interferer) conditions. As written, this is an assertion without the corresponding data.
minor comments (5)
  1. [Table 2] The table formatting is garbled: the header "Avg-5" is unexplained, and the Interferer "1" and "2" rows are not cleanly separated from the SNR columns. Please provide a clear table with explicit column labels and a footnote defining the "x" in the Fine tuning column.
  2. [2.1] The pipeline discards roughly half of the raw video (from 528.8 hours to 269 hours); reporting per-stage retention statistics (face detection, tracking, active speaker detection, SyncNet, lip extraction) would strengthen reproducibility.
  3. [2.1] Please report the thresholds used for SyncNet confidence and active speaker detection decisions, as the pipeline's output quality depends on these choices.
  4. [3.2] The SentencePiece subword model is trained on which text corpus? Please specify the training text and subword regularization settings beyond the vocabulary size.
  5. [References] Some references, such as [17] and [27], point to arXiv preprints or GitHub repositories without version identifiers or access dates; please add these for archival reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central AVSR claims rest on manually annotated test labels and independent audio-only controls, not on the teacher model that generated training transcripts.

full rationale

The paper's central claim—that a 269-hour automatically labeled Vietnamese AVSR dataset plus English AV-HuBERT initialization yields competitive clean performance and strong noise robustness—is not circular. Training transcripts are generated by a Vietnamese Wav2Vec2 model ([27], a prior work of the first author), but the evaluation is run on a held-out test set whose labels are manually annotated: 'To ensure an accurate evaluation of the model, all text labels in the test dataset are manually annotated.' The same Wav2Vec2 model also serves as the clean-condition audio baseline A1, so AV1's 9.40% versus A1's 7.53% WER clean comparison is mildly self-referential in that AV1 is trained to imitate A1's transcriptions; however, this is not a construction-level identity because the test targets are human labels and AV1 could in principle diverge from the teacher. The headline noise-robustness result is further insulated by the audio-only AV-HuBERT controls A3/A4, which share the same pretraining and noise augmentation as AV1 but degrade far more under interference (e.g., 57.67% versus 18.39% WER at -5 dB with two interferers), isolating the visual contribution from the label-generation process. No load-bearing step reduces an equation or fitted parameter to its own input, no uniqueness claim is imported from the authors' prior work, and no ansatz is smuggled in via self-citation. The unaudited teacher labels are a legitimate data-quality concern for clean-condition accuracy, but that is a correctness risk, not circularity.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The paper contributes a dataset and an empirical recipe, not a derivation, so there are no fitted mathematical constants. The hand-chosen evaluation protocol (SNR levels, interferer counts) is the main free choice. The remaining axioms are standard data-collection assumptions.

free parameters (2)
  • Cocktail-party SNR levels = -5, 0, 5, 10 dB
    Hand-selected to define easy-to-hard noisy conditions; the headline robustness claim is measured only at these levels.
  • Number of interferer speakers = 0, 1, 2
    Hand-selected; the strongest claim uses the two-interferer case, so the conclusion is tied to this choice.
assumptions (5)
  • domain assumption Automatic Wav2Vec2 transcriptions of YouTube speech are accurate enough to use as training labels.
    Used to label 194,073 utterances in Section 2.1; the paper reports selecting the best ASR model but no training-label verification.
  • domain assumption Mouth-region crops preserve the articulatory information needed for Vietnamese, including tone cues.
    Section 2.1 lip extraction and Section 1's claim that visual cues help distinguish tones; the visual-only WER of 41.34% is the only evidence.
  • domain assumption ASD and SyncNet filtering correctly removes off-screen or unsynchronized speakers.
    Section 2.1 relies on these two pretrained models to guarantee the face matches the audio.
  • domain assumption English and multilingual AV-HuBERT visual features transfer to Vietnamese without language-specific visual adaptation.
    Section 2.2 uses pretrained checkpoints; Section 4's finding that English init outperforms multilingual supports but does not prove transferability.
  • domain assumption Manual test annotations are accurate.
    Section 2.1 says all test labels are manually annotated, but no annotation procedure, inter-annotator agreement, or quality metric is given.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ViCocktail: Automated Multi-Modal Data Collection for Vietnamese Audio-Visual Speech Recognition." pith.science (2026). https://pith.science/paper/JKM2R6O7

@misc{pith2026250604635,
  author       = {Pith},
  title        = {Pith review of: ViCocktail: Automated Multi-Modal Data Collection for Vietnamese Audio-Visual Speech Recognition},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JKM2R6O7}},
  note         = {Machine review of arXiv:2506.04635}
}
read the original abstract

Audio-Visual Speech Recognition (AVSR) has gained significant attention recently due to its robustness against noise, which often challenges conventional speech recognition systems that rely solely on audio features. Despite this advantage, AVSR models remain limited by the scarcity of extensive datasets, especially for most languages beyond English. Automated data collection offers a promising solution. This work presents a practical approach to generate AVSR datasets from raw video, refining existing techniques for improved efficiency and accessibility. We demonstrate its broad applicability by developing a baseline AVSR model for Vietnamese. Experiments show the automatically collected dataset enables a strong baseline, achieving competitive performance with robust ASR in clean conditions and significantly outperforming them in noisy environments like cocktail parties. This efficient method provides a pathway to expand AVSR to more languages, particularly under-resourced ones.

Figures

Figures reproduced from arXiv: 2506.04635 by the authors.

Figure 1
Figure 1. Top: Original still images from videos used in the making of the dataset. Bottom: The mouth motions for ‘chao’ ` from two different speakers. The processing pipeline is summarised in [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Pipeline to generate the ViCocktail dataset. Video preparation. A CNN face detector based on the Sin￾gle Shot Scale-invariant Face Detector (S3FD) [23] is used to detect face appearances in the individual frames. Then, shot boundaries are identified using a content-based scene detection algorithm that analyzes frame differences. Within each shot, face tracking is performed using the Intersection Over Union (IOU) alg… view at source ↗
Figure 3
Figure 3. Left: Overview of the AVSR model, which consists of an encoder and a CTC/Attention decoder. Training utilizes two loss functions: CTC loss and cross-entropy loss. Middle and Right: Two variants of the encoder. Conformer-based encoder, where features are processed sepa￾rately before fusion, AV-HuBERT concatenates and fuses audio and visual features before passing them through Transformer encoder layers. The CTC/Atten… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

41 extracted references · 35 canonical work pages

  1. [1]

    ViCocktail: Automated Multi-Modal Data Collection for Vietnamese Audio-Visual Speech Recognition

    Introduction The task of audio-visual speech recognition (A VSR) is not new, with its roots tracing back to the discovery of the McGurk effect [1] in 1976, which demonstrated the strong interaction between auditory and visual speech perception. Inspired by this interplay, A VSR systems [2, 3, 4] have been developed to leverage visual cues for enhancing sp...

    work page Pith review arXiv 1976

  2. [2]

    ViCocktail dataset In this section, we describe the multi-stage pipeline for auto- matically generating a dataset for A VSR model

    Methodology 2.1. ViCocktail dataset In this section, we describe the multi-stage pipeline for auto- matically generating a dataset for A VSR model. Through this pipeline, we have collected about 269 hours of video of spoken sentences and phrases along with the corresponding facetrack from an initial dataset of about 528.8 hours of video. We use a variety ...

    work page

  3. [3]

    The first model uses a Conformer-based encoder [31] with a CTC/Attention de- coder [32]

    Both follow a sequence-to-sequence design. The first model uses a Conformer-based encoder [31] with a CTC/Attention de- coder [32]. The second model features a Transformer-based en- coder (A V-HuBERT [18]) paired with a CTC/Attention decoder. In the Conformer-based encoder (Figure 3, middle), the vi- sual extraction module uses a modified ResNet-18, where...

    work page

  4. [4]

    Results Table 2 presents the performance of different models on the ViCocktail dataset. Overall, multimodal audio-visual models (A V[1-4]) outperform both audio-only (A[1-5]) and visual-only models (V[1-5]), while audio-only models achieve better results than visual-only models. Audio-only models are highly sensitive to noise, with WER increasing signific...

    work page

  5. [5]

    natural”, “music

    Experimental setup 3.1. Data pre-processing We follow previous works [18, 31] for audio and video pre- processing. The training set consists of 269 hours of video from approximately 2475 speakers. The video is processed frame by frame, with each frame cropped to extract the mouth region of interest using a 96 × 96 bounding box. The audio is sampled at 16 ...

    work page

  6. [6]

    Lrs3-ted: a large-scale dataset for visual speech recognition,

    T. Afouras, J. S. Chung, and A. Zisserman, “Lrs3-ted: a large-scale dataset for visual speech recognition,” 2018. [Online]. Available: https://arxiv.org/abs/1809.00496

    arXiv 2018

  7. [7]

    Our data collection and preparation process is fully automated and can be extended to other languages

    Conclusions In this work, we introduced a new A VSR dataset and established a strong baseline A VSR model for the Vietnamese language. Our data collection and preparation process is fully automated and can be extended to other languages. We benchmarked vari- ous architectures and pre-trained models to determine the most effective initialization, with A V-...

    work page

  8. [8]

    Hearing lips and seeing voices,

    H. McGurk and J. MacDonald, “Hearing lips and seeing voices,” Nature, vol. 264, no. 5588, pp. 746–748, 1976

    1976

Show all 41 references
  1. [9]

    Integration of acous- tic and visual speech signals using neural networks,

    B. Yuhas, M. Goldstein, and T. Sejnowski, “Integration of acous- tic and visual speech signals using neural networks,” IEEE Com- munications Magazine, vol. 27, no. 11, pp. 65–71, 1989

    1989

  2. [10]

    See me, hear me: inte- grating automatic speech recognition and lip-reading,

    P. Duchnowski, U. Meier, and A. Waibel, “See me, hear me: inte- grating automatic speech recognition and lip-reading,” in 3rd In- ternational Conference on Spoken Language Processing (ICSLP 1994), 1994, pp. 547–550

    1994

  3. [11]

    Multimodal interfaces,

    A. Waibel, M. T. V o, P. Duchnowski, and S. Manke, “Multimodal interfaces,” Artificial Intelligence Review , vol. 10, pp. 299–319, 1996

    1996

  4. [12]

    Lip read- ing sentences in the wild,

    J. S. Chung, A. Senior, O. Vinyals, and A. Zisserman, “Lip read- ing sentences in the wild,” in2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 3444–3453

    2017

  5. [13]

    RUSA VIC corpus: Russian audio-visual speech in cars,

    D. Ivanko, A. Axyonov, D. Ryumin, A. Kashevnik, and A. Karpov, “RUSA VIC corpus: Russian audio-visual speech in cars,” in Proceedings of the Thirteenth Language Resources and Evaluation Conference. Marseille, France: European Language Resources Association, Jun. 2022, pp. 1555...

    2022

  6. [14]

    Large-scale visual speech recognition,

    B. Shillingford, Y . Assael, M. W. Hoffman, T. Paine, C. Hughes, U. Prabhu, H. Liao, H. Sak, K. Rao, L. Bennett, M. Mulville, M. Denil, B. Coppin, B. Laurie, A. Senior, and N. de Fre- itas, “Large-scale visual speech recognition,” inInterspeech 2019, 2019, pp. 4135–4139

    2019

  7. [15]

    Avas: Speech database for multimodal recognition applications,

    S. Antar, A. Sagheer, S. Aly, and M. F. Tolba, “Avas: Speech database for multimodal recognition applications,” in 13th Inter- national Conference on Hybrid Intelligent Systems (HIS 2013) , 2013, pp. 123–128

    2013

  8. [16]

    An arabic visual dataset for visual speech recognition,

    L. A. Elrefaei, T. Q. Alhassan, and S. S. Omar, “An arabic visual dataset for visual speech recognition,” Procedia Computer Sci- ence, vol. 163, pp. 400–409, 2019, 16th Learning and Technology Conference 2019Artificial Intelligence and Machine Learning: Embedding the Intellige...

    2019

  9. [17]

    Cn-cvs: A mandarin audio- visual dataset for large vocabulary continuous visual to speech synthesis,

    C. Chen, D. Wang, and T. F. Zheng, “Cn-cvs: A mandarin audio- visual dataset for large vocabulary continuous visual to speech synthesis,” in ICASSP 2023 - 2023 IEEE International Confer- ence on Acoustics, Speech and Signal Processing (ICASSP), 2023, pp. 1–5

    2023

  10. [18]

    Lipreading with densenet and resbi-lstm,

    X. Chen, J. Du, and H. Zhang, “Lipreading with densenet and resbi-lstm,” Signal, Image and Video Processing, vol. 14, pp. 981 – 989, 2020. [Online]. Available: https://api.semanticscholar.org/ CorpusID:214376123

    2020

  11. [19]

    A cascade sequence- to-sequence model for chinese mandarin lip reading,

    Y . Zhao, R. Xu, and M. Song, “A cascade sequence- to-sequence model for chinese mandarin lip reading,” in Proceedings of the 1st ACM International Conference on Multimedia in Asia , ser. MMAsia ’19. New York, NY , USA: Association for Computing Machinery, 2020. [Online]. Avai...

    2020

  12. [20]

    Auto-avsr: Audio-visual speech recognition with automatic labels,

    P. Ma, A. Haliassos, A. Fernandez-Lopez, H. Chen, S. Petridis, and M. Pantic, “Auto-avsr: Audio-visual speech recognition with automatic labels,” in ICASSP 2023 - 2023 IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023, pp. 1–5

    2023

  13. [21]

    Havrus corpus: High-speed recordings of audio-visual russian speech,

    V . Verkhodanova, A. L. Ronzhin, I. S. Kipyatkova, D. Ivanko, A. Karpov, and M. ˇZelezn´y, “Havrus corpus: High-speed recordings of audio-visual russian speech,” in International Conference on Speech and Computer, 2016. [Online]. Available: https://api.semanticscholar.org/Corp...

    2016

  14. [22]

    Towards es- timating the upper bound of visual-speech recognition: The visual lip-reading feasibility database,

    A. Fernandez-Lopez, O. Martinez, and F. M. Sukno, “Towards es- timating the upper bound of visual-speech recognition: The visual lip-reading feasibility database,” in 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017), 2017, pp. 208–215

    2017

  15. [23]

    Visual lip reading dataset in turkish,

    A. Berkol, T. T ¨umer-Sivri, N. Pervan-Akman, M. C ¸ olak, and H. Erdem, “Visual lip reading dataset in turkish,” Data, vol. 8, no. 1, 2023. [Online]. Available: https: //www.mdpi.com/2306-5729/8/1/15

    2023

  16. [24]

    The ASD model relies on both audio and visual features to deter- mine when a person in the video is actually speaking

    (ASD) and SyncNet [25] to address this problem. The ASD model relies on both audio and visual features to deter- mine when a person in the video is actually speaking. Then, the SyncNet model analyzes the delay between lip movements and the corresponding audio to check if the s...

  17. [25]

    Lip reading in the wild,

    J. S. Chung and A. Zisserman, “Lip reading in the wild,” in Com- puter Vision – ACCV 2016, S.-H. Lai, V . Lepetit, K. Nishino, and Y . Sato, Eds. Cham: Springer International Publishing, 2017, pp. 87–103

    2016

  18. [26]

    Robust self-supervised audio-visual speech recognition,

    B. Shi, W.-N. Hsu, and A. Mohamed, “Robust self-supervised audio-visual speech recognition,” in Interspeech 2022, 2022, pp. 2118–2122

    2022

  19. [27]

    Whisper-flamingo: Integrating visual fea- tures into whisper for audio-visual speech recognition and trans- lation,

    A. Rouditchenko, Y . Gong, S. Thomas, L. Karlinsky, H. Kuehne, R. Feris, and J. Glass, “Whisper-flamingo: Integrating visual fea- tures into whisper for audio-visual speech recognition and trans- lation,” in Interspeech 2024, 2024, pp. 2420–2424

    2024

  20. [28]

    Xls-r: Self-supervised cross-lingual speech representation learning at scale,

    A. Babu, C. Wang, A. Tjandra, K. Lakhotia, Q. Xu, N. Goyal, K. Singh, P. von Platen, Y . Saraf, J. Pino, A. Baevski, A. Con- neau, and M. Auli, “Xls-r: Self-supervised cross-lingual speech representation learning at scale,” in Interspeech 2022, 2022, pp. 2278–2282

    2022

  21. [29]

    Simple and effective zero-shot cross-lingual phoneme recognition,

    Q. Xu, A. Baevski, and M. Auli, “Simple and effective zero-shot cross-lingual phoneme recognition,” in Interspeech 2022, 2022, pp. 2113–2117

    2022

  22. [30]

    S3fd: Single shot scale-invariant face detector,

    S. Zhang, X. Zhu, Z. Lei, H. Shi, X. Wang, and S. Z. Li, “S3fd: Single shot scale-invariant face detector,” in Proceedings of the IEEE international conference on computer vision , 2017, pp. 192–201

    2017

  23. [31]

    A light weight model for active speaker detection,

    J. Liao, H. Duan, K. Feng, W. Zhao, Y . Yang, and L. Chen, “A light weight model for active speaker detection,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2023, pp. 22 932–22 941

    2023

  24. [32]

    Out of time: Automated lip sync in the wild,

    J. S. Chung and A. Zisserman, “Out of time: Automated lip sync in the wild,” in ACCV Workshops, 2016. [Online]. Available: https://api.semanticscholar.org/CorpusID:26294509

    2016

  25. [33]

    Dlib-ml: A machine learning toolkit,

    D. E. King, “Dlib-ml: A machine learning toolkit,” The Journal of Machine Learning Research, vol. 10, pp. 1755–1758, 2009

    2009

  26. [34]

    Vietnamese end-to-end speech recognition using wav2vec 2.0,

    T. B. Nguyen, “Vietnamese end-to-end speech recognition using wav2vec 2.0,” 09 2021. [Online]. Available: https: //github.com/vietai/ASR

    2021

  27. [35]

    Synthetic conversations improve multi-talker asr,

    T.-B. Nguyen and A. Waibel, “Synthetic conversations improve multi-talker asr,” inICASSP 2024 - 2024 IEEE International Con- ference on Acoustics, Speech and Signal Processing (ICASSP) , 2024, pp. 10 461–10 465

    2024

  28. [36]

    Msa-asr: Efficient multilingual speaker attribution with frozen asr models,

    T.-B. Nguyen and A.Waibel, “Msa-asr: Efficient multilingual speaker attribution with frozen asr models,” in ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2025, pp. 1–5

    2025

  29. [37]

    Phowhisper: Automatic speech recognition for vietnamese,

    T.-T. Le, L. T. Nguyen, and D. Q. Nguyen, “Phowhisper: Automatic speech recognition for vietnamese,” in The Second Tiny Papers Track at ICLR 2024 , 2024. [Online]. Available: https://openreview.net/forum?id=x3c3MkJfpG

    2024

  30. [38]

    End-to-end audio-visual speech recognition with conformers,

    P. Ma, S. Petridis, and M. Pantic, “End-to-end audio-visual speech recognition with conformers,” inICASSP 2021 - 2021 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICASSP), 2021, pp. 7613–7617

    2021

  31. [39]

    Hy- brid ctc/attention architecture for end-to-end speech recognition,

    S. Watanabe, T. Hori, S. Kim, J. R. Hershey, and T. Hayashi, “Hy- brid ctc/attention architecture for end-to-end speech recognition,” IEEE Journal of Selected Topics in Signal Processing , vol. 11, no. 8, pp. 1240–1253, 2017

    2017

  32. [40]

    Subword regularization: Improving neural network translation models with multiple subword candidates,

    T. Kudo, “Subword regularization: Improving neural network translation models with multiple subword candidates,” in ACL. Melbourne, Australia: Association for Computational Linguistics, Jul. 2018, pp. 66–75. [Online]. Available: https: //aclanthology.org/P18-1007/

    2018

  33. [41]

    Muavic: A multilingual audio-visual corpus for robust speech recognition and robust speech-to-text translation,

    M. Anwar, B. Shi, V . Goswami, W.-N. Hsu, J. Pino, and C. Wang, “Muavic: A multilingual audio-visual corpus for robust speech recognition and robust speech-to-text translation,” in Interspeech 2023, 2023, pp. 4064–4068

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.