Pith. sign in

REVIEW 4 major objections 5 minor 64 references

Target Speaker Lipreading by Audio-Visual Self-Distillation Pretraining and Speaker Adaptation

T0 review · 4 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A transferred self-supervised lipreading model, tuned per speaker and ensembled across lip and face views, reaches a 77.3% character error rate on the ChatCLR benchmark, below the top 2024 challenge result.

desk verdict The 77.3% benchmark claim is a non-significant gap with an unverified protocol match, but the cross-lingual data-scarcity experiment is the real contribution worth a second look. read the letter →

arxiv 2502.05758 v1 pith:JH4VDCKG submitted 2025-02-09 eess.AS

Jing-Xuan Zhang , Tingzhi Mao , Longjiang Guo , Jin Li , Lichen Zhang This is my paper · ORCID
classification eess.AS
keywords lipreadingvisualspeechrecognitionself-supervisedlearningspeakeradaptationcross-lingualtransfermodelensembleaudio-visualself-distillationChatCLR
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Lipreading usually assumes a generic speaker, but casual conversation in a home setting has large per-speaker variation, and building a system from scratch for Chinese is expensive in both data and compute. The paper argues that a pipeline of three moves fixes both problems: an audio-visual self-distillation model pretrained on English transfers to Chinese lipreading; the model is then fine-tuned per target speaker with a Kullback-Leibler divergence regularizer that limits overfitting to that speaker's small video set; and at test time the system averages predictions from lip-only and full-face models, which fail on different inputs. The reported result is a character error rate of 77.3% on the official ChatCLR evaluation set, below the top result of the 2024 Chat-scenario Chinese Lipreading Challenge. The paper's own confidence-interval analysis, however, shows that this gap is not statistically significant.

What carries the argument

The load-bearing machinery is fourfold. AV2vec pretraining is a student-teacher self-distillation: the student receives masked and modality-dropped audio/video and regresses, with a mean-squared error, to targets produced online by a teacher whose weights are the student's exponential moving average, giving a label-free visual encoder. The lipreading head is a hybrid CTC/attention encoder-decoder, trained with cross-entropy plus CTC and decoded by joint CTC/attention beam search. Speaker adaptation fine-tunes that model on the target speaker's videos with a weighted sum of the cross-entropy loss, a KLD term, and the CTC loss, where the KLD term pins the adapted model's output distribution to the speaker-independent model's, preventing overfitting on the small adaptation set. Test-time ensemble averaging over lip-ROI and full-face models exploits the observation that the two input views make different, complementary mistakes.

What would settle it

Run the authors' four-model ensemble on the official ChatCLR evaluation set under the challenge's own decoding and scoring script. If the CER comes out at or above the top leaderboard result of 77.6%, or if re-scoring under the two protocols gives non-identical numbers for the same model, the claim of beating the top result is falsified.

Watch

Extended reading notes

Core claim

The paper's central claim is that these three techniques are complementary and can be combined on top of a single pretrained visual encoder. Concretely, an audio-visual self-distillation model (AV2vec) pretrained on English LRS3, when fine-tuned on Chinese labeled video with a hybrid CTC/attention decoder, outperforms supervised-from-scratch training as target-language data shrinks. Fine-tuning that speaker-independent model on a target speaker's own videos with a KLD term ($\rho=0.1$) yields per-speaker gains for most of the twelve ChatCLR speakers. Equal-weight ensembling of lip-ROI and full-face models gives further gains, and the four-model ensemble reaches 77.3% CER on the official evaluation set. The paper is explicit that this is below the top leaderboard entry but that the difference is not significant at the 95% confidence level.

Load-bearing premise

The headline benchmark claim assumes the authors' post-competition evaluation of their final model on the official ChatCLR set followed the same protocol as the challenge leaderboard runs; the paper does not demonstrate that its decoding, text normalization, and scoring are identical.

Editorial extensions

If this is right

  • Cross-lingual transfer makes an English-pretrained audio-visual encoder usable for Chinese lipreading, with the advantage growing as the amount of target-language data decreases.
  • Per-speaker adaptation with KLD regularization improves most of the twelve target speakers' lipreading accuracy relative to the speaker-independent model.
  • Ensembling lip-ROI and full-face models reduces CER by roughly 3-4% relative to the average of the two single models, outperforming ensembling the same input type with different random seeds.
  • On the official ChatCLR evaluation set, the full four-model ensemble achieves 77.3% CER, numerically below the top challenge result, though the 95% confidence intervals overlap.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same transfer-then-adapt recipe should carry the English-pretrained encoder into other low-resource target languages, since the paper's rationale is that visemes are broadly shared across languages; a cheap test would be transferring to another language with a small lipreading corpus.
  • The equal-weight lip/face ensemble is a fixed fusion; a learned or input-adaptive fusion that suppresses the face stream when occluding hands are detected could beat the fixed average in the microphone-in-hand cases the paper shows.
  • Re-scoring under the official challenge protocol is the decisive check on the benchmark claim; until that is done, the practical takeaway is the combination's internal gains, not the leaderboard comparison.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper proposes a target-speaker lipreading system for Chinese built on the authors' AV2vec self-supervised audio-visual encoder. The contributions are: (i) cross-lingual transfer of an English-pretrained AV2vec model to Chinese lipreading, (ii) speaker-dependent adaptation of a speaker-independent model using a KLD-regularized loss, and (iii) test-time ensembling of models trained on full-face and lip-ROI inputs. Experiments on the 2024 ChatCLR Challenge task 2 report that the full ensemble achieves 77.3% CER on the official evaluation set, which is numerically lower than the top challenge result of 77.6%; the authors also note that this difference is not statistically significant at the 95% level. Additional validation-set results examine transfer under reduced target-language data, per-speaker adaptation, and ensemble combinations.

Significance. The paper addresses a practical and under-explored problem: lipreading for a target speaker in a low-resource language, using cross-lingual self-supervised pretraining and speaker adaptation. The proposal is concrete and the validation set includes useful ablations. If the benchmark claim were fully supported, the system would be a meaningful advance for Chinese visual speech recognition. The strengths include the use of a self-distillation pretraining approach that avoids multi-iteration clustering, and the explicit comparison of face vs lip-ROI inputs with a complementary ensemble. However, the headline comparison is not statistically established, and the cross-lingual transfer result is inconsistent across input types, which tempers the significance. The paper does not provide code, but the experimental protocol is mostly transparent.

major comments (4)
  1. [Abstract; Section 5.3.4, Table 5] The abstract's final sentence presents the 77.3% CER as 'lower than the top result' from the challenge, but the paper's own statistical analysis (Section 5.3.4) states that the difference from the top-performing teams is not significant at the 95% confidence level. The gap (0.3 percentage points) is well inside the reported bootstrap 95% confidence interval of ±1.5 for the 77.3% result, and Table 5 does not provide confidence intervals for the leaderboard systems. The benchmark claim therefore rests on a point estimate comparison that the authors themselves do not support. Please either replace this claim with a properly tested statement (e.g., matched-pair significance test on the same evaluation transcripts) or remove it from the abstract and conclusion.
  2. [Section 5.3.1, Figure 4] The cross-lingual transfer advantage under data scarcity is not consistent across input types. With 10 hours of Chinese data, AV2vec-tf-lip (88.9% CER) outperforms AV2vec-lip (92.9%), but AV2vec-tf-face (94.5%) is worse than AV2vec-face (88.1%). The text states that 'a similar trend emerged' for the face-based models, which is contradicted by the reported numbers. Since the first highlight claims that cross-lingual transfer enhances lipreading with limited target-language data, the claim must be qualified to the lip-ROI setting or an explanation must be given for why face-based transfer fails. This is load-bearing for the contribution on transfer learning.
  3. [Section 5.3.2, Figure 5, Table 4] The paper overstates the speaker-adaptation result. The text in Section 5.3.2 reports significant average improvements only for AV2vec-tf-lip and AV2vec-tf-face, while the non-transfer models show no significant changes; per-speaker analysis is 'mostly non-significant.' Nevertheless, Section 5.3.2 begins by stating that speaker adaptation 'effectively reduced CER,' and the highlights claim that speaker adaptation boosts specific-speaker accuracy. These statements should be tied to the model types and speakers for which the effect is significant, and the per-speaker analysis should include a summary test (e.g., sign test over 12 speakers) rather than relying on individual confidence-interval overlap.
  4. [Section 5.3.4, Table 5] The comparison to the challenge leaderboard is not demonstrated to be protocol-comparable. The final ensemble includes M4, which was trained after the challenge deadline, and the evaluation was performed post-hoc 'with the assistance of the Challenge organizers.' The paper does not specify the scoring script, text normalization, decoding configuration, or whether the same official evaluation pipeline was used for the reported 77.3% result as for the leaderboard runs. Without this information, a reader cannot exclude the possibility that differences in decoding or scoring contribute to the 0.3-point gap. Please document the post-hoc evaluation protocol and, if possible, obtain an official score from the organizers under the original protocol.
minor comments (5)
  1. [Section 5.3.4] The paragraph says 'The results are summarized in Table 6,' but the leaderboard results appear in Table 5; Table 6 is the model architecture table. Please correct the cross-reference.
  2. [Table 2] The baseline name 'REVAn' should be 'RAVEn' for consistency with the reference list and the text.
  3. [Section 5.2] The sentence 'The last 8 layers were averaged representations of the for the teacher' is missing a word; presumably 'outputs of the teacher.'
  4. [Conclusion] The phrase 'also leaded to better lipreading results' should be 'also led to better lipreading results.'
  5. [Abstract] The displayed abstract contains a typo 'di?erent speakers' (likely an OCR artifact); ensure the final version is printed correctly.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation: the new contributions are evaluated against external benchmarks, and the benchmark claim's weakness is statistical/protocol, not circularity.

full rationale

The paper's load-bearing comparisons are empirical: Table 3 and Figure 4 compare cross-lingual transfer against supervised/no-pretraining baselines on the ChatCLR validation set, Table 4 compares speaker-dependent models with and without KLD regularization, and Table 5 compares the final ensemble against the official ChatCLR leaderboard. No equation in Sections 3-4 is defined in terms of a quantity that later serves as the predicted result: the AV2vec teacher targets (Eq. 1) are generated online by EMA, the SI objective (Eq. 5) is standard CTC/attention, the SA objective (Eq. 8) is a KLD-regularized interpolation computed from the SI model's fixed predictions, and the ensemble rule (Eq. 9) averages independently trained face/lip models. The authors' prior AV2vec and face-ROI results are self-cited, but they are also re-validated here on ChatCLR and LRS3 (Tables 2 and 3), so the self-citations are not the only support for the central claims. The abstract's 'lower than the top result' assertion is statistically fragile because the paper itself reports no significant difference in Section 5.3.4, and the post-competition evaluation may not be protocol-identical to the leaderboard runs, but that is a correctness/evidence concern, not a circular derivation. Thus no self-definitional, fitted-input, or citation-imported circularity step can be exhibited, and the appropriate finding is no significant circularity.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central empirical claims rest on standard ML training and decoding losses, the reuse of the authors' AV2vec encoder, and several domain assumptions about cross-lingual viseme sharing, KLD regularization, input complementarity, and the validity of the evaluation protocol. The main hand-set hyperparameters are listed; none is fitted to the test set, but several were selected on the validation set.

free parameters (6)
  • Rho (KLD regularization weight) = 0.1
    Controls the balance between cross-entropy and KL-divergence regularization during speaker adaptation; set by hand in Section 5.2 and used in Equation (7).
  • Mu (CTC training weight) = 0.1
    Weight of the CTC loss in joint CTC/attention training (Equation 5); set to 0.1 in Section 5.2.
  • Alpha (CTC decoding weight) = 0.1
    Weight of CTC in ensemble decoding (Equation 9); set to 0.1 in Section 5.2.
  • Ensemble model weights = Equal weights (1/M for each base model)
    Equal weighting among face and lip models was chosen because it performed well in preliminary experiments (Section 4.3); any other weighting would change the 77.3% result.
  • Peak learning rates for SI and SD finetuning = 1e-3 (SI), 1e-4 (SD)
    Learning rate schedules are selected by validation set performance (Section 5.2); the SD learning rate matters for speaker adaptation results.
  • Beam size = 50
    Beam search size during decoding is set to 50 (Section 5.2) and affects all reported CER numbers.
assumptions (5)
  • domain assumption English and Chinese share a substantial portion of visemes, so a visual encoder pretrained on English transfers to Chinese lipreading.
    Motivates cross-lingual transfer in Section 1; no direct measurement of viseme overlap is provided.
  • domain assumption KL-divergence regularization between speaker-independent and speaker-dependent output distributions prevents overfitting on small target-speaker data.
    Assumed in Section 4.2 following ASR literature; the paper's own per-speaker significance results are mixed.
  • domain assumption Full face and lip-ROI inputs are complementary, so averaging their predictions improves accuracy.
    Central to the ensemble method in Section 4.3; supported by qualitative examples and aggregate CER, but not by a formal model or significance test against the best single model.
  • domain assumption The ChatCLR development set 8:2 split provides a valid held-out set for speaker adaptation.
    Used in Section 5.1; the authors acknowledge the small size (1,140 utterances) causes wide confidence intervals.
  • standard math Bootstrap confidence intervals (Bisani and Ney) are appropriate for comparing CER differences on these evaluation sets.
    The paper relies on 95% confidence intervals for significance statements in Sections 5.3.2 and 5.3.4; this is a standard method but its conclusions here are mostly non-significant.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Target Speaker Lipreading by Audio-Visual Self-Distillation Pretraining and Speaker Adaptation." pith.science (2026). https://pith.science/paper/JH4VDCKG

@misc{pith2026250205758,
  author       = {Pith},
  title        = {Pith review of: Target Speaker Lipreading by Audio-Visual Self-Distillation Pretraining and Speaker Adaptation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JH4VDCKG}},
  note         = {Machine review of arXiv:2502.05758}
}
read the original abstract

Lipreading is an important technique for facilitating human-computer interaction in noisy environments. Our previously developed self-supervised learning method, AV2vec, which leverages multimodal self-distillation, has demonstrated promising performance in speaker-independent lipreading on the English LRS3 dataset. However, AV2vec faces challenges such as high training costs and a potential scarcity of audio-visual data for lipreading in languages other than English, such as Chinese. Additionally, most studies concentrate on speakerindependent lipreading models, which struggle to account for the substantial variation in speaking styles across di?erent speakers. To address these issues, we propose a comprehensive approach. First, we investigate cross-lingual transfer learning, adapting a pre-trained AV2vec model from a source language and optimizing it for the lipreading task in a target language. Second, we enhance the accuracy of lipreading for specific target speakers through a speaker adaptation strategy, which is not extensively explored in previous research. Third, after analyzing the complementary performance of lipreading with lip region-of-interest (ROI) and face inputs, we introduce a model ensembling strategy that integrates both, signi?cantly boosting model performance. Our method achieved a character error rate (CER) of 77.3% on the evaluation set of the ChatCLR dataset, which is lower than the top result from the 2024 Chat-scenario Chinese Lipreading Challenge.

Figures

Figures reproduced from arXiv: 2502.05758 by the authors.

Figure 1
Figure 1. Illustration of our proposed method during training and testing. The dashed blue [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Illustration of AV2vec model. AV2vec contains a student and a teacher module. [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗
Figure 3
Figure 3. Illustration of the training scheme of our proposed method for the target speaker [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Lipreading results on ChatCLR validation set when varying the amount of training [PITH_FULL_IMAGE:figures/full_fig_p024_4.png]
Figure 5
Figure 5. Figure 5: Lipreading results on ChatCLR validation set before and after speaker adaptation. [PITH_FULL_IMAGE:figures/full_fig_p026_5.png]
Figure 6
Figure 6. Figure 6: Examples of full face and corresponding lip ROI video frames, along with the [PITH_FULL_IMAGE:figures/full_fig_p028_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

64 extracted references · 62 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in ":" * " " * FUNCTION f...

  2. [2]

    , author Chung, J.S

    author Afouras, T. , author Chung, J.S. , author Senior, A. , author Vinyals, O. , author Zisserman, A. , year 2022 . title Deep audio-visual speech recognition . journal IEEE Transactions on Pattern Analysis and Machine Intelligence volume 44 , pages 8717--8727

    work page 2022

  3. [3]

    , author Chung, J.S

    author Afouras, T. , author Chung, J.S. , author Zisserman, A. , year 2018 . title LRS3-TED : a large-scale dataset for visual speech recognition , in: booktitle arXiv preprint arXiv:1809.00496 , pp. pages 1--2

    arXiv 2018

  4. [4]

    , author Chung, J.S

    author Afouras, T. , author Chung, J.S. , author Zisserman, A. , year 2020 . title ASR is all you need: Cross-modal distillation for lip reading , in: booktitle IEEE International Conference on Acoustics, Speech and Signal Processing , organization IEEE . pp. pages 2143--2147

    work page 2020

  5. [5]

    , author Park, J

    author Ahn, Y.J. , author Park, J. , author Park, S. , author Choi, J. , author Kim, K.E. , year 2024 . title Sync VSR : Data-efficient visual speech recognition with end-to-end crossmodal audio token synchronization , in: booktitle Annual Conference of the International Speech Communication Association , pp. pages 867--871

    work page 2024

  6. [6]

    , author Zhou, Y

    author Baevski, A. , author Zhou, Y. , author Mohamed, A. , author Auli, M. , year 2020 . title Wav2vec 2.0 : A framework for self-supervised learning of speech representations . journal Advances in Neural Information Processing Systems volume 33 , pages 12449--12460

    work page 2020

  7. [7]

    , author Fainberg, J

    author Bell, P. , author Fainberg, J. , author Klejch, O. , author Li, J. , author Renals, S. , author Swietojanski, P. , year 2020 . title Adaptation algorithms for neural network-based speech recognition: An overview . journal IEEE Open Journal of Signal Processing volume 2 , pages 33--66

    work page 2020

  8. [8]

    , author Ney, H

    author Bisani, M. , author Ney, H. , year 2004 . title Bootstrap estimates for confidence intervals in ASR performance evaluation , in: booktitle IEEE International Conference on Acoustics, Speech and Signal Processing , organization IEEE . pp. pages 406--409

    work page 2004

Show all 64 references
  1. [9]

    , author Wang, D

    author Chen, C. , author Wang, D. , author Zheng, T.F. , year 2023 . title CN-CVS : A mandarin audio-visual dataset for large vocabulary continuous visual to speech synthesis , in: booktitle IEEE International Conference on Acoustics, Speech and Signal Processing , organizatio...

    2023

  2. [10]

    , author Zhou, H

    author Chen, H. , author Zhou, H. , author Du, J. , author Lee, C.H. , author Chen, J. , author Watanabe, S. , author Siniscalchi, S.M. , author Scharenborg, O. , author Liu, D.Y. , author Yin, B.C. , author Pan, J. , author Gao, J.Q. , author Liu, C. , year 2022 . title The f...

    2022

  3. [11]

    , author Schultz, T

    author Denby, B. , author Schultz, T. , author Honda, K. , author Hueber, T. , author Gilbert, J.M. , author Brumberg, J.S. , year 2010 . title Silent speech interfaces . journal Speech Communication volume 52 , pages 270--287

    2010

  4. [12]

    , author Luettin, J

    author Dupont, S. , author Luettin, J. , year 2000 . title Audio-visual speech modeling for continuous speech recognition . journal IEEE Transactions on Multimedia volume 2 , pages 141--151

    2000

  5. [13]

    , author Mart \' nez-Hinarejos, C.D

    author Gimeno-G \'o mez, D. , author Mart \' nez-Hinarejos, C.D. , year 2023 . title Comparing speaker adaptation methods for visual speech recognition for continuous spanish . journal Applied Sciences volume 13 , pages 6521

    2023

  6. [14]

    , author Ma, P

    author Haliassos, A. , author Ma, P. , author Mira, R. , author Petridis, S. , author Pantic, M. , year 2023 . title Jointly learning visual and auditory speech representations from raw data , in: booktitle International Conference on Learning Representations , pp. pages 1--15

    2023

  7. [15]

    , author Bolte, B

    author Hsu, W.N. , author Bolte, B. , author Tsai, Y.H.H. , author Lakhotia, K. , author Salakhutdinov, R. , author Mohamed, A. , year 2021 . title HuBERT : Self-supervised speech representation learning by masked prediction of hidden units . journal IEEE/ACM Transactions on A...

    2021

  8. [16]

    , author Shi, B

    author Hsu, W.N. , author Shi, B. , year 2022 . title u-HuBERT : Unified mixed-modal speech pretraining and zero-shot transfer to unlabeled modality . journal Advances in Neural Information Processing Systems volume 35 , pages 21157--21170

    2022

  9. [17]

    , author Li, J

    author Huang, Y. , author Li, J. , author He, L. , author Wei, W. , author Gale, W. , author Gong, Y. , year 2020 . title Rapid RNN-T adaptation using personalized speech synthesis and neural language generator. , in: booktitle Annual Conference of the International Speech Com...

    2020

  10. [18]

    , author Tran, D

    author Korbar, B. , author Tran, D. , author Torresani, L. , year 2018 . title Cooperative learning of audio and video models from self-supervised synchronization , in: booktitle Advances in Neural Information Processing Systems , pp. pages 1--12

    2018

  11. [19]

    , author Lee, E

    author Lee, C. , author Lee, E. , author Jung, S. , author Lee, S. , year 2004 . title Design and implementation of a real-time lipreading system using PCA and HMM . journal Journal of Korea Multimedia Society volume 7 , pages 1597--1609

    2004

  12. [20]

    , author Li, J

    author Li, K. , author Li, J. , author Zhao, Y. , author Kumar, K. , author Gong, Y. , year 2018 . title Speaker adaptation for end-to-end CTC models , in: booktitle 2018 IEEE Spoken Language Technology Workshop (SLT) , organization IEEE . pp. pages 542--549

    2018

  13. [21]

    , author Baevski, A

    author Lian, J. , author Baevski, A. , author Hsu, W.N. , author Auli, M. , year 2023 . title AV-data2vec: Self-supervised learning of audio-visual speech representations with contextualized target representations , in: booktitle Proceedings of the IEEE Automatic Speech Recogn...

    2023

  14. [22]

    , year 2013

    author Liao, H. , year 2013 . title Speaker adaptation of context dependent deep neural networks , in: booktitle IEEE International Conference on Acoustics, Speech and Signal Processing , organization IEEE . pp. pages 7947--7951

    2013

  15. [23]

    , author Lakomkin, E

    author Liu, X. , author Lakomkin, E. , author Vougioukas, K. , author Ma, P. , author Chen, H. , author Xie, R. , author Doulaty, M. , author Moritz, N. , author Kolar, J. , author Petridis, S. , author Pantic, M. , author Fuegen, C. , year 2023 . title SynthVSR : Scaling up v...

    2023

  16. [24]

    , author Martinez, B

    author Ma, P. , author Martinez, B. , author Petridis, S. , author Pantic, M. , year 2021 a. title Towards practical lipreading with distilled and efficient models , in: booktitle IEEE International Conference on Acoustics, Speech and Signal Processing , organization IEEE . pp...

    2021

  17. [25]

    , author Mira, R

    author Ma, P. , author Mira, R. , author Petridis, S. , author Schuller, B.W. , author Pantic, M. , year 2021 b. title LiRA : Learning visual speech representations from audio through self-supervision , in: booktitle Annual Conference of the International Speech Communication ...

    2021

  18. [26]

    , author Petridis, S

    author Ma, P. , author Petridis, S. , author Pantic, M. , year 2021 c. title End-to-end audio-visual speech recognition with C onformers , in: booktitle IEEE International Conference on Acoustics, Speech and Signal Processing , organization IEEE . pp. pages 7613--7617

    2021

  19. [27]

    , author Petridis, S

    author Ma, P. , author Petridis, S. , author Pantic, M. , year 2022 a. title Visual speech recognition for multiple languages in the wild . journal Nature Machine Intelligence volume 4 , pages 930--939

    2022

  20. [28]

    , author Wang, Y

    author Ma, P. , author Wang, Y. , author Petridis, S. , author Shen, J. , author Pantic, M. , year 2022 b. title Training strategies for improved lip-reading , in: booktitle IEEE International Conference on Acoustics, Speech and Signal Processing , organization IEEE . pp. page...

    2022

  21. [29]

    , author Zeng, Z

    author Ma, S. , author Zeng, Z. , author McDuff, D. , author Song, Y. , year 2021 d. title Contrastive learning of global and local video representations , in: editor Ranzato, M. , editor Beygelzimer, A. , editor Dauphin, Y. , editor Liang, P. , editor Vaughan, J.W. (Eds.), bo...

    2021

  22. [30]

    , author Liao, H

    author Makino, T. , author Liao, H. , author Assael, Y. , author Shillingford, B. , author Garcia, B. , author Braga, O. , author Siohan, O. , year 2019 . title Recurrent neural network transducer for audio-visual speech recognition , in: booktitle 2019 IEEE automatic speech r...

    2019

  23. [31]

    , author Ma, P

    author Martinez, B. , author Ma, P. , author Petridis, S. , author Pantic, M. , year 2020 . title Lipreading using temporal convolutional networks , in: booktitle IEEE International Conference on Acoustics, Speech and Signal Processing , organization IEEE . pp. pages 6319--6323

    2020

  24. [32]

    , author Gaur, Y

    author Meng, Z. , author Gaur, Y. , author Li, J. , author Gong, Y. , year 2019 . title Speaker adaptation for attention-based end-to-end speech recognition , in: booktitle Annual Conference of the International Speech Communication Association , pp. pages 241--245

    2019

  25. [33]

    , author Yamaguchi, Y

    author Noda, K. , author Yamaguchi, Y. , author Nakadai, K. , author Okuno, H.G. , author Ogata, T. , year 2014 . title Lipreading using convolutional neural network , in: booktitle Annual Conference of the International Speech Communication Association , pp. pages 1149--1153

    2014

  26. [34]

    , author Dalmia, S

    author Peng, Y. , author Dalmia, S. , author Lane, I. , author Watanabe, S. , year 2022 . title Branchformer: Parallel mlp-attention architectures to capture local and global context for speech recognition and understanding , in: booktitle International Conference on Machine L...

    2022

  27. [35]

    , author Stafylakis, T

    author Petridis, S. , author Stafylakis, T. , author Ma, P. , author Cai, F. , author Tzimiropoulos, G. , author Pantic, M. , year 2018 a. title End-to-end audiovisual speech recognition , in: booktitle IEEE International Conference on Acoustics, Speech and Signal Processing ,...

    2018

  28. [36]

    , author Stafylakis, T

    author Petridis, S. , author Stafylakis, T. , author Ma, P. , author Tzimiropoulos, G. , author Pantic, M. , year 2018 b. title Audio-visual speech recognition with a hybrid CTC / Attention architecture , in: booktitle IEEE Spoken Language Technology Workshop (SLT) , organizat...

    2018

  29. [37]

    , year 2012

    author Polikar, R. , year 2012 . title Ensemble learning . journal Ensemble machine learning: Methods and applications , pages 1--34

    2012

  30. [38]

    , author Afouras, T

    author Prajwal, K. , author Afouras, T. , author Zisserman, A. , year 2022 . title Sub-word level lip reading with visual attention , in: booktitle Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition , pp. pages 5162--5172

    2022

  31. [39]

    , author Axyonov, A

    author Ryumin, D. , author Axyonov, A. , author Ryumina, E. , author Ivanko, D. , author Kashevnik, A. , author Karpov, A. , year 2024 . title Audio–visual speech recognition based on regulated transformer and spatio–temporal fusion strategy for driver assistive systems . jour...

    2024

  32. [40]

    , author Ivanko, D

    author Ryumin, D. , author Ivanko, D. , author Ryumina, E. , year 2023 . title Audio-visual speech and gesture recognition by sensors of mobile devices . journal Sensors volume 23

    2023

  33. [41]

    , author Sim, K.C

    author Samarakoon, L. , author Sim, K.C. , year 2016 . title Factorized hidden layer adaptation for deep neural network based acoustic modeling . journal IEEE/ACM Transactions on Audio, Speech, and Language Processing volume 24 , pages 2241--2250

    2016

  34. [42]

    , author Moritz, N

    author Sar , L. , author Moritz, N. , author Hori, T. , author Le Roux, J. , year 2020 . title Unsupervised speaker adaptation using attention-based speaker memory for end-to-end ASR , in: booktitle IEEE International Conference on Acoustics, Speech and Signal Processing , org...

    2020

  35. [43]

    , author Braga, O

    author Serdyuk, D. , author Braga, O. , author Siohan, O. , year 2022 . title Transformer-based video front-ends for audio-visual speech recognition for single and muti-person video , in: editor Ko, H. , editor Hansen, J.H.L. (Eds.), booktitle Annual Conference of the Internat...

    2022

  36. [44]

    , author Hsu, W.N

    author Shi, B. , author Hsu, W.N. , author Lakhotia, K. , author Mohamed, A. , year 2022 . title Learning audio-visual speech representation by masked multimodal cluster prediction , in: booktitle International Conference on Learning Representations , pp. pages 1--12

    2022

  37. [45]

    , author Senior, A

    author Son Chung, J. , author Senior, A. , author Vinyals, O. , author Zisserman, A. , year 2017 . title Lip reading sentences in the wild , in: booktitle Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pp. pages 6447--6456

    2017

  38. [46]

    , author Harte, N

    author Sterpu, G. , author Harte, N. , year 2017 . title Towards lipreading sentences with active appearance models , in: booktitle International Conference on Auditory-Visual Speech Process , pp. pages 70--75

    2017

  39. [47]

    , author Saam, C

    author Sterpu, G. , author Saam, C. , author Harte, N. , year 2020 . title How to teach DNN s to pay attention to the visual modality in speech recognition . journal IEEE/ACM Transactions on Audio, Speech, and Language Processing volume 28 , pages 1052--1064

    2020

  40. [48]

    , author Li, J

    author Swietojanski, P. , author Li, J. , author Renals, S. , year 2016 . title Learning hidden unit contributions for unsupervised acoustic model adaptation . journal IEEE/ACM Transactions on Audio, Speech, and Language Processing volume 24 , pages 1450--1463

    2016

  41. [49]

    , year 2017

    author Themos S., G.T. , year 2017 . title Combining residual networks with LSTMs for lipreading , in: booktitle Annual Conference of the International Speech Communication Association , pp. pages 3652--3656

    2017

  42. [50]

    , author Shazeer, N

    author Vaswani, A. , author Shazeer, N. , author Parmar, N. , author Uszkoreit, J. , author Jones, L. , author Gomez, A.N. , author Kaiser, . , author Polosukhin, I. , year 2017 . title Attention is all you need . journal Advances in neural information processing systems volum...

    2017

  43. [51]

    , author Ye, Z

    author Wan, G. , author Ye, Z. , year 2024 . title Multi-modal knowledge transfer for target speaker lipreading with improved audio-visual pretraining and cross-lingual fine-tuning , in: booktitle IEEE International Conference on Multimedia and Expo Workshops (ICMEW) , pp. pages 1--6

    2024

  44. [52]

    , author Koutn \' k, J

    author Wand, M. , author Koutn \' k, J. , author Schmidhuber, J. , year 2016 . title Lipreading with long short-term memory , in: booktitle IEEE International Conference on Acoustics, Speech and Signal Processing , organization IEEE . pp. pages 6115--6119

    2016

  45. [53]

    , author Guo, P

    author Wang, H. , author Guo, P. , author Wan, X. , author Zhou, H. , author Xie, L. , year 2024 a. title Enhancing lip reading with multi-scale video and multi-encoder , in: booktitle 2024 IEEE International Conference on Multimedia and Expo Workshops (ICMEW) , pp. pages 1--6

    2024

  46. [54]

    , author Guo, P

    author Wang, H. , author Guo, P. , author Zhou, P. , author Xie, L. , year 2024 b. title MLCA-AVSR : Multi-layer cross attention fusion based audio-visual speech recognition , in: booktitle IEEE International Conference on Acoustics, Speech and Signal Processing , organization...

    2024

  47. [55]

    , author Hori, T

    author Watanabe, S. , author Hori, T. , author Kim, S. , author Hershey, J.R. , author Hayashi, T. , year 2017 . title Hybrid CTC /attention architecture for end-to-end speech recognition . journal IEEE Journal of Selected Topics in Signal Processing volume 11 , pages 1240--1253

    2017

  48. [56]

    , author Yang, Y

    author Xiong, R. , author Yang, Y. , author He, D. , author Zheng, K. , author Zheng, S. , author Xing, C. , author Zhang, H. , author Lan, Y. , author Wang, L. , author Liu, T. , year 2020 . title On layer normalization in the T ransformer architecture , in: booktitle Interna...

    2020

  49. [57]

    , author Lu, C

    author Xu, B. , author Lu, C. , author Guo, Y. , author Wang, J. , year 2020 . title Discriminative multi-modality speech recognition , in: booktitle Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. pages 14433--14442

    2020

  50. [58]

    , author Li, D

    author Xu, K. , author Li, D. , author Cassimatis, N. , author Wang, X. , year 2018 . title LCANet : End-to-end lipreading with cascaded attention- CTC , in: booktitle International Conference on Automatic Face and Gesture Recognition , pp. pages 548--555

    2018

  51. [59]

    , author Yao, K

    author Yu, D. , author Yao, K. , author Su, H. , author Li, G. , author Seide, F. , year 2013 . title KL -divergence regularized deep neural network adaptation for improved large vocabulary speech recognition , in: booktitle IEEE International Conference on Acoustics, Speech a...

    2013

  52. [60]

    , author Richmond, K

    author Zhang, J.X. , author Richmond, K. , author Ling, Z.H. , author Dai, L. , year 2021 . title TaLNet : Voice reconstruction from tongue and lip articulation with transfer learning from text-to-speech synthesis . journal Proceedings of the AAAI Conference on Artificial Inte...

    2021

  53. [61]

    , author Wan, G

    author Zhang, J.X. , author Wan, G. , author Ling, Z.H. , author Pan, J. , author Gao, J. , author Liu, C. , year 2023 . title Self-supervised audio-visual speech representations learning by multimodal self-distillation , in: booktitle IEEE International Conference on Acoustic...

    2023

  54. [62]

    , author Wan, G

    author Zhang, J.X. , author Wan, G. , author Pan, J. , year 2022 . title Is lip region-of-interest sufficient for lipreading? , in: booktitle Proceedings of the International Conference on Multimodal Interaction (ICMI) , p. pages 368–372

    2022

  55. [63]

    , author Yang, S

    author Zhang, Y. , author Yang, S. , author Xiao, J. , author Shan, S.S. , author Chen, X. , year 2020 . title Can we read speech beyond the lips? R ethinking ROI selection for deep visual speech recognition , in: booktitle IEEE International Conference on Automatic Face and G...

    2020

  56. [64]

    , author Zhou, L

    author Zhu, Q. , author Zhou, L. , author Zhang, Z. , author Liu, S. , author Jiao, B. , author Zhang, J. , author Dai, L. , author Jiang, D. , author Li, J. , author Wei, F. , year 2024 . title VatLM : Visual-audio-text pre-training with unified masked prediction for speech r...

    2024

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.