Pith. sign in

REVIEW 3 major objections 5 minor 53 references

Learning Speaker-Invariant Visual Features for Lipreading

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SIFLip learns speaker-invariant visual features by aligning frames to text labels and adversarially erasing speaker identity, which the paper reports cuts unseen-speaker character error rates by more than 11 points on the CMLR benchmark.

desk verdict Large, plausible gains for unseen-speaker lipreading, but the speaker-invariance claim is not directly tested and the evaluation has enough gaps that the numbers should be treated as conditional. read the letter →

arxiv 2506.07572 v1 pith:LKMMJZYR submitted 2025-06-09 cs.CV cs.CL

Yu Li , Feng Xue , Shujie Li , Jinrui Zhang , Shuang Yang , Dan Guo , Richang Hong This is my paper
classification cs.CVcs.CL
keywords lipreadingspeaker-invariantfeaturesfeaturedisentanglementcross-modallearningcontrastivegradientreversalgeneralizationsentence-level
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the usual lipreading pipeline learns speaker-specific visual attributes—lip shape, color, and texture—that create spurious correlations between vision and text, and that these correlations are a major cause of poor accuracy on speakers not seen during training. To fix this, it proposes SIFLip, a framework that strips speaker identity out of the visual features with two complementary mechanisms: an implicit module that aligns each video frame with its corresponding text label through contrastive learning, and an explicit module that trains a speaker-recognition branch and then reverses its gradient to erase speaker-specific cues. The paper reports that SIFLip lowers the character error rate on both the Chinese CMLR and English GRID datasets, with the largest gains on unseen speakers (e.g., from 43.18 to 32.16 on CMLR). If these results hold, they suggest that disentangling who is speaking from what is said is a practical route to speaker-robust lipreading without requiring audio at test time.

What carries the argument

The argument is carried by two disentanglement modules added to a standard 3D-CNN, Bi-GRU, Transformer, and seq2seq lipreading backbone. IDCFL (Implicit Disentanglement via Common Features Learning) uses the Montreal Forced Aligner to assign each video frame a text token, then computes a similarity matrix between visual and text embeddings and trains it with an InfoNCE contrastive loss $\mathcal{L}_{\mathrm{CL}}$ and a cross-entropy loss $\mathcal{L}_{\mathrm{CE}}$ against a diagonal ground-truth matrix, so that each frame's visual feature is pulled toward the embedding of its correct text token regardless of who is speaking. EDGR (Explicit Disentanglement via Gradient Reversal) pools the intermediate features, runs them through a speaker classifier with cross-entropy loss $\mathcal{L}_{\mathrm{ED}}$, and inserts a gradient reversal layer between the feature extractor and the classifier so that the backbone is updated to make the speaker classification fail. The total training loss is $\mathcal{L}_{\mathrm{total}} = \mathcal{L}_{\mathrm{PT}} + \alpha \mathcal{L}_{\mathrm{ID}} + \beta \mathcal{L}_{\mathrm{ED}}$, where $\mathcal{L}_{\mathrm{PT}}$ is the seq2seq text-prediction loss; the paper sets $\alpha=0.5$ and $\beta=2.0$ on both datasets. The text embeddings serve as the reference that is stable across speakers, and the reversed gradient is what actively erases residual speaker identity.

What would settle it

Train SIFLip on CMLR with three versions of the frame labels: the original MFA alignments, oracle boundaries produced by manual annotation, and deliberately shifted boundaries that preserve word order but move the boundaries by several frames. If the unseen-speaker CER is nearly the same in all three conditions, the frame-level alignment is not the active ingredient behind the reported generalization gains; if it changes substantially, alignment precision is load-bearing. A second check: train a speaker classifier on the visual features produced by SIFLip's encoder; if its accuracy is not near chance, the explicit disentanglement has not actually removed speaker identity.

Watch

Extended reading notes

Core claim

SIFLip's central claim is that a lipreading model can be made substantially more accurate on both seen and unseen speakers by learning visual features that are invariant to speaker identity, and that this invariance can be achieved without parallel audio at test time. The method combines two training-time losses: an implicit disentanglement loss (IDCFL) that performs frame-level cross-modal alignment between visual features and text embeddings, using text as a stable semantic anchor that is the same across speakers, and an explicit disentanglement loss (EDGR) that uses a speaker-identification sub-task whose gradient is reversed before it reaches the visual encoder, forcing the encoder to suppress speaker-specific information. On the CMLR benchmark, SIFLip reports a character error rate of 20.55 for seen speakers and 32.16 for unseen speakers, compared with 27.79 and 43.18 for the previous best method LipFormer; on GRID it reports 0.79 and 6.23 against 1.45 and 9.64. The ablation study identifies IDCFL as the larger contributor, with the full model performing best, and the paper attributes the gains to the removal of spurious correlations between lip appearance and text.

Load-bearing premise

The per-frame text labels used by the implicit disentanglement module come from the Montreal Forced Aligner, and the contrastive and cross-entropy losses treat these boundaries as ground truth; if the alignments are wrong, the model is trained to match frames to incorrect text tokens, and the paper gives no analysis of alignment accuracy.

Editorial extensions

If this is right

  • On CMLR, SIFLip reduces CER from 27.79 to 20.55 on seen speakers and from 43.18 to 32.16 on unseen speakers relative to LipFormer.
  • On GRID, SIFLip reduces CER from 1.45 to 0.79 on seen speakers and from 9.64 to 6.23 on unseen speakers relative to LipFormer.
  • Ablations show that both modules matter: removing IDCFL raises unseen-speaker CER from 32.16 to 39.02 on CMLR, while removing EDGR raises it to 33.13, and removing the gradient reversal alone raises it to 33.07.
  • The method works in both Chinese (with a cascaded pinyin-to-character decoder) and English, suggesting the disentanglement does not depend on the target language's orthography.
  • Since text supervision is only needed during training, the trained model still accepts video-only input at inference time.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension: the frame-level label alignment idea could transfer to other fine-grained video-language tasks, such as sign language recognition or cued speech, where per-frame semantic labels are available; the same InfoNCE-plus-cross-entropy alignment could strip out signer-specific appearance.
  • One implication the paper leaves implicit: the reliance on Montreal Forced Aligner means the method cannot be trained on audio-free corpora; an alignment-free variant that learns frame-to-text correspondence from a CTC or attention-based path during training would make the approach widely applicable.
  • A direct check whether the disentanglement is real: train a speaker classifier on the visual features SIFLip's encoder produces; if EDGR is working, that classifier's accuracy should be near chance, and the paper does not report such a measurement.
  • The pattern on GRID, where seen-speaker gains are small (0.31), suggests that on easy, highly constrained datasets speaker-invariance matters mainly for generalization rather than absolute accuracy; this predicts that improvement on unseen speakers should grow with the visual diversity of speakers in the training set.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes SIFLip, a lipreading framework that aims to improve generalization to unseen speakers by learning speaker-invariant visual features. Two complementary modules are introduced: an Implicit Disentanglement module (IDCFL) that performs frame-level cross-modal contrastive and cross-entropy alignment between visual features and text labels obtained from the Montreal Forced Aligner, and an Explicit Disentanglement module (EDGR) that adds a speaker-classification branch with a gradient reversal layer. The visual features are decoded by a seq2seq model. Experiments on CMLR and GRID report large improvements over prior methods on both seen and unseen speakers, with ablations showing that each module contributes to the gains.

Significance. If the speaker-invariance claim is substantiated, the work would make a useful contribution to lipreading by addressing a known generalization bottleneck. The paper has concrete strengths: it evaluates on two public datasets with seen/unseen splits, it compares against several prior methods, and it provides component-level ablations. The reported gains on unseen speakers are large and consistent. However, the central claim that the learned features are speaker-invariant is currently inferred only from downstream CER; no experiment directly measures speaker identity information in the representation, so the distinctive mechanism of the paper remains unverified.

major comments (3)
  1. [§3.3, Tables 2 and 3] The central claim that SIFLip learns speaker-invariant visual features is not directly tested. The evidence consists of downstream CER on seen/unseen speakers and ablations, but both are compatible with the auxiliary losses acting as generic regularizers: the frame-level alignment loss constrains the encoder, and the GRL task only removes information accessible to the particular speaker classifier used in Eq. (7)-(10). Please add a direct invariance diagnostic, such as training a speaker classifier on frozen features from Base, Base w/ IDCFL, Base w/ EDGR, and SIFLip and reporting its accuracy on held-out speaker identities, together with a feature-space visualization (e.g., t-SNE colored by speaker identity). Without such a measure, the title-level claim of speaker invariance is unsupported.
  2. [§3.2, Eqs. (5)-(6)] The IDCFL module treats MFA-derived frame labels as ground truth for the contrastive and cross-entropy losses, but no analysis of alignment quality is provided. MFA alignments are known to be imperfect, particularly under coarticulation or spontaneous speech, and misaligned word boundaries would assign incorrect text tokens to frames and corrupt the learning signal. The paper should report alignment accuracy or show sensitivity of the results to alignment quality, and it should discuss the requirement for audio during training, which limits applicability to datasets without audio.
  3. [§4.2, Table 1] The CMLR LipNet baseline is not LipNet as published: the text states that LipNet's CTC loss failed to converge on CMLR and that CTC was replaced with the Seq2Seq module. The CMLR entries for LipNet in Table 1 therefore correspond to a modified model, and the comparison is not against the original method as reported in the literature. Please state the modified-baseline configuration explicitly in the table or text, and clarify whether all baselines were re-run under the same data split and preprocessing. As written, the large absolute improvements over LipNet on CMLR are difficult to interpret.
minor comments (5)
  1. [§3.5] The roles of L_ID and L_ED are swapped in the optimization description: the text says that L_ED facilitates semantic alignment and implicitly disentangles speaker-specific features, while L_ID guides the branch to filter speaker-specific features and explicitly disentangle them. According to Sections 3.2 and 3.3, the reverse is true: L_ID is the implicit disentanglement loss and L_ED is the explicit speaker-classification loss. Please correct this wording.
  2. [§4.3] The sentence 'In the overall comparison experiment in Section 4.2, four datasets were used' is incorrect; the paper evaluates on two datasets, CMLR and GRID. Please fix this typo.
  3. [Table 1 and Tables 2-3] No error bars, standard deviations, or significance tests are reported for any of the results. Given that several ablation differences are small (e.g., w/o GR vs. w/o L_ED in Table 3), adding variance over multiple runs would substantially strengthen the claims.
  4. [Eq. (14)] There is a mismatched parenthesis in the definition of P(w): the expression Softmax(MLP((h_t_d)_j,(c_v_t)_j) has one more opening than closing parenthesis. Please correct the formula.
  5. [Abstract] The sentence 'Experimental results demonstrate that SIFLip significantly enhances generalization performance across multiple public datasets' appears twice in succession. Please remove the duplicate.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: SIFLip's reported CER gains are genuine held-out test predictions on CMLR and GRID against external and published baselines; author self-citations are present but not load-bearing.

full rationale

The paper's central claim is evaluated against external benchmarks, so the derivation chain is essentially self-contained. SIFLip's reported improvements (CER 20.55/32.16 on CMLR seen/unseen and 0.79/6.23 on GRID seen/unseen) are measured on held-out test speakers, and the paper states that 'during testing, only video was used,' so the IDCFL contrastive/CE losses (Eqs. 3-6) and the EDGR speaker-classification loss with gradient reversal (Eqs. 7-10) are training-time auxiliary objectives that cannot force the test-time CER by construction. The per-frame text labels come from an external force aligner (MFA) applied to ground-truth text; the model is not predicting quantities it was fitted on. Hyperparameters alpha and beta are tuned on CMLR, which is conventional practice, not a tautology. The authors do cite their own prior work ([24], [43], [46] share co-authors with the present paper), but these citations are used as related work or as a baseline to be compared against, not as a load-bearing premise; no uniqueness theorem is invoked, and no fitted parameter is renamed as a prediction. One genuine presentation defect exists: Section 3.5 swaps the descriptions of L_ID and L_ED relative to their definitions in Eqs. (6) and (8), calling the cross-modal alignment loss 'explicit' and the speaker-classification loss 'implicit'. This internal inconsistency could confuse readers about which module does what, but it does not alter the equations or the held-out evaluation and is not a circular reduction. The skeptic's concern that speaker-invariance is never directly measured at the representation level is a legitimate evidence gap about the causal mechanism, but it concerns support and interpretation, not circularity: the empirical results are independently checkable and could in principle be false.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The method relies on several domain assumptions about text stability, alignment accuracy, and separability of speaker and content features. The free parameters are the loss weights alpha and beta tuned on CMLR, and the GRL scaling lambda which is not reported.

free parameters (3)
  • alpha = 0.5
    Weight for the IDCFL contrastive loss L_ID in the total loss (Eq. 16), chosen via hyperparameter search on CMLR (Section 4.5).
  • beta = 2.0
    Weight for the EDGR speaker-classification loss L_ED in the total loss (Eq. 16), chosen via hyperparameter search on CMLR (Section 4.5).
  • lambda (GRL scaling) = not reported
    Scaling factor for the reversed gradient in Eq. (10); the paper calls it a parameter but never gives its value or tuning procedure.
assumptions (4)
  • domain assumption Text embeddings are stable across speakers and can serve as a common semantic anchor.
    The IDCFL module assumes that text features are invariant to speaker identity and that aligning visual features to these text features removes speaker-specific visual attributes (Section 3.2).
  • domain assumption Frame-level labels from MFA are accurate.
    The positive pairs (X,L) require per-frame text labels from Montreal Forced Aligner; incorrect boundaries would make the contrastive loss misleading (Section 3.2, Figure 3).
  • domain assumption Speaker identity information is separable from speech content in lip features.
    The EDGR module trains a speaker classifier on visual features and uses gradient reversal to suppress that information, assuming this does not also remove speech-content features (Section 3.3).
  • standard math Standard deep learning training assumptions (loss functions, backpropagation, convergence).
    The paper uses standard cross-entropy, InfoNCE, and gradient reversal without formal guarantees.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Learning Speaker-Invariant Visual Features for Lipreading." pith.science (2026). https://pith.science/paper/LKMMJZYR

@misc{pith2026250607572,
  author       = {Pith},
  title        = {Pith review of: Learning Speaker-Invariant Visual Features for Lipreading},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LKMMJZYR}},
  note         = {Machine review of arXiv:2506.07572}
}
read the original abstract

Lipreading is a challenging cross-modal task that aims to convert visual lip movements into spoken text. Existing lipreading methods often extract visual features that include speaker-specific lip attributes (e.g., shape, color, texture), which introduce spurious correlations between vision and text. These correlations lead to suboptimal lipreading accuracy and restrict model generalization. To address this challenge, we introduce SIFLip, a speaker-invariant visual feature learning framework that disentangles speaker-specific attributes using two complementary disentanglement modules (Implicit Disentanglement and Explicit Disentanglement) to improve generalization. Specifically, since different speakers exhibit semantic consistency between lip movements and phonetic text when pronouncing the same words, our implicit disentanglement module leverages stable text embeddings as supervisory signals to learn common visual representations across speakers, implicitly decoupling speaker-specific features. Additionally, we design a speaker recognition sub-task within the main lipreading pipeline to filter speaker-specific features, then further explicitly disentangle these personalized visual features from the backbone network via gradient reversal. Experimental results demonstrate that SIFLip significantly enhances generalization performance across multiple public datasets. Experimental results demonstrate that SIFLip significantly improves generalization performance across multiple public datasets, outperforming state-of-the-art methods.

Figures

Figures reproduced from arXiv: 2506.07572 by the authors.

Figure 1
Figure 1. Different speakers exhibit significant visual vari [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of the proposed SIFLip framework. SIFLip learns speaker-invariant visual features by disentangling [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Illustration of positive and negative sample pair [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Illustration of Seq2Seq module. The Seq2Seq mod [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Heatmap of Visual and Text Feature Similarity. The [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Effects of different hyper parameter 𝛼 in Eq. (16) on CMLR dataset. (a) Seen speakers. (b) Unseen speakers. (a) Seen (b) Unseen [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Effects of different hyper parameter 𝛽 in Eq. (16) on CMLR dataset. (a) Seen speakers. (b) Unseen speakers. 4.6 Case Study To qualitatively analyze the effectiveness of the proposed model, we visualized one case from CMLR dataset, as shown in [PITH_FULL_IMAGE:figures/…
Figure 8
Figure 8. Figure 8: Visualization of prediction examples on CMLR. [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

53 extracted references · 40 canonical work pages

  1. [1]

    Ahsan Adeel, Mandar Gogate, Amir Hussain, and William M Whitmer. 2019. Lip-reading driven deep learning approach for speech enhancement. IEEE Trans- actions on Emerging Topics in Computational Intelligence 5, 3 (2019), 481–490

    work page 2019

  2. [2]

    Xi Ai and Bin Fang. 2023. Cross-Modal Language Modeling in Multi-Motion- Informed Context for Lip Reading. IEEE/ACM Transactions on Audio, Speech, and Language Processing 31 (2023), 2220–2232

    work page 2023

  3. [3]

    Alessandro Amodio, Michele Ermidoro, Davide Maggi, Simone Formentin, and Sergio Matteo Savaresi. 2018. Automatic detection of driver impairment based on pupillary light reflex. IEEE transactions on intelligent transportation systems 20, 8 (2018), 3038–3048

    work page 2018

  4. [4]

    Yannis M Assael, Brendan Shillingford, Shimon Whiteson, and Nando De Fre- itas. 2016. Lipnet: End-to-end sentence-level lipreading. arXiv preprint arXiv:1611.01599 (2016)

    arXiv 2016

  5. [5]

    Blaž Bortolato, Marija Ivanovska, Peter Rot, Janez Križaj, Philipp Terhörst, Naser Damer, Peter Peer, and Vitomir Štruc. 2020. Learning privacy-enhancing face representations through feature disentanglement. In 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020) . IEEE, 495–502

    work page 2020

  6. [6]

    Hugo Bulzomi, Marcel Schweiker, Amélie Gruel, and Jean Martinet. 2023. End- to-end neuromorphic lip-reading. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 4101–4108

    work page 2023

  7. [7]

    Hang Chen, Qing Wang, Jun Du, Gen-Shun Wan, Shi-Fu Xiong, Bao-Ci Yin, Jia Pan, and Chin-Hui Lee. 2024. Collaborative Viseme Subword and End-to-End Modeling for Word-Level Lip Reading.IEEE Transactions on Multimedia 26 (2024), 9358–9371

    work page 2024

  8. [8]

    Xuejuan Chen, Jixiang Du, and Hongbo Zhang. 2020. Lipreading with DenseNet and resBi-LSTM. Signal, Image and Video Processing 14 (2020), 981–989

    work page 2020

Show all 53 references
  1. [9]

    Yiting Cheng, Fangyun Wei, Jianmin Bao, Dong Chen, and Wenqiang Zhang

  2. [10]

    Joon Son Chung and Andrew Zisserman. 2017. Out of time: automated lip sync in the wild. In Computer Vision–ACCV 2016 Workshops: ACCV 2016 International Workshops, Taipei, Taiwan, November 20-24, 2016, Revised Selected Papers, Part II

    2017

  3. [11]

    Martin Cooke, Jon Barker, Stuart Cunningham, and Xu Shao. 2006. An audio- visual corpus for speech perception and automatic speech recognition. The Journal of the Acoustical Society of America 120, 5 (2006), 2421–2424

    2006

  4. [12]

    Wanxia Deng, Lingjun Zhao, Qing Liao, Deke Guo, Gangyao Kuang, Dewen Hu, Matti Pietikäinen, and Li Liu. 2021. Informative feature disentanglement for unsupervised domain adaptation. IEEE Transactions on Multimedia 24 (2021), 2407–2421

    2021

  5. [13]

    Han Fang, Pengfei Xiong, Luhui Xu, and Yu Chen. 2021. Clip2video: Mastering video-text retrieval via image clip. arXiv preprint arXiv:2106.11097 (2021)

    2021 arXiv

  6. [14]

    Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario March, and Victor Lempitsky. 2016. Domain-adversarial training of neural networks. Journal of machine learning research 17, 59 (2016), 1–35

    2016

  7. [15]

    Yi He, Lei Yang, Hanyi Wang, Yun Zhu, and Shilin Wang. 2024. Speaker-Adaptive Lipreading Via Spatio-Temporal Information Learning. In ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . 10411–10415

    2024

  8. [16]

    Yiyang Huang, Xuefeng Liang, and Chaowei Fang. 2021. Callip: Lipreading using contrastive and attribute learning. In Proceedings of the 29th ACM International Conference on Multimedia. 2492–2500

    2021

  9. [17]

    Minsu Kim, Hyunjun Kim, and Yong Man Ro. 2022. Speaker-Adaptive Lip Reading with User-Dependent Padding. In Computer Vision – ECCV 2022 . Springer Nature Switzerland, Cham, 576–593

    2022

  10. [18]

    Minsu Kim, Jeong Hun Yeo, Jeongsoo Choi, and Yong Man Ro. 2023. Lip reading for low-resource languages by learning and combining general speech knowledge and language-specific knowledge. In Proceedings of the IEEE/CVF International Conference on Computer Vision . 15359–15371

    2023

  11. [19]

    Yaman Kumar, Rohit Jain, Khwaja Mohd Salik, Rajiv Ratn Shah, Yifang Yin, and Roger Zimmermann. 2019. Lipper: Synthesizing thy speech using multi-view lipreading. In Proceedings of the AAAI Conference on artificial intelligence , Vol. 33. 2588–2595

    2019

  12. [20]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language Learning Speaker-Invariant Visual Features for Lipreading Conference acronym ’XX, June 03–05, 2018, Woodstock, NY models...

    2023

  13. [21]

    Zhengyang Li, Timo Lohrenz, Matthias Dunkelberg, and Tim Fingscheidt. 2023. Transformer-Based Lip-Reading with Regularized Dropout and Relaxed Attention. In 2022 IEEE Spoken Language Technology Workshop (SLT) . 723–730

    2023

  14. [22]

    Dongnan Liu, Chaoyi Zhang, Yang Song, Heng Huang, Chenyu Wang, Michael Barnett, and Weidong Cai. 2022. Decompose to adapt: Cross-domain object detection via feature disentanglement. IEEE Transactions on Multimedia 25 (2022), 1333–1344

    2022

  15. [23]

    Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. 2022. Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning. Neurocomputing 508 (2022), 293–304

    2022

  16. [24]

    Songtao Luo, Shuang Yang, Shiguang Shan, and Xilin Chen. 2023. Learning Separable Hidden Unit Contributions for Speaker-Adaptive Lip-Reading. (2023)

    2023

  17. [25]

    Pingchuan Ma, Brais Martinez, Stavros Petridis, and Maja Pantic. 2021. Towards practical lipreading with distilled and efficient models. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 7608–7612

    2021

  18. [26]

    Xinghua Ma and Shilin Wang. 2022. Chinese Mandarin Lipreading using Cas- caded Transformers with Multiple Intermediate Representations. In 2022 IEEE International Conference on Image Processing (ICIP) . 2561–2565

    2022

  19. [27]

    Brais Martinez, Pingchuan Ma, Stavros Petridis, and Maja Pantic. 2020. Lipreading using temporal convolutional networks. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 6319–6323

    2020

  20. [28]

    Ziling Miao, Hong Liu, and Bing Yang. 2020. Part-based lipreading for audio- visual speech recognition. In 2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC). IEEE, 2722–2726

    2020

  21. [29]

    Xuesong Niu, Zitong Yu, Hu Han, Xiaobai Li, Shiguang Shan, and Guoying Zhao

  22. [30]

    Stavros Petridis, Themos Stafylakis, Pingehuan Ma, Feipeng Cai, Georgios Tz- imiropoulos, and Maja Pantic. 2018. End-to-end audiovisual speech recognition. In 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 6548–6552

    2018

  23. [31]

    Javad Peymanfard, Mohammad Reza Mohammadi, Hossein Zeinali, and Nasser Mozayani. 2022. Lip reading using external viseme decoding. In2022 International Conference on Machine Vision and Image Processing (MVIP) . IEEE, 1–5

    2022

  24. [32]

    KR Prajwal, Triantafyllos Afouras, and Andrew Zisserman. 2022. Sub-word level lip reading with visual attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 5162–5172

    2022

  25. [33]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learni...

    2021

  26. [34]

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In International conference on machine learning . PMLR, 28492–28518

    2023

  27. [35]

    Sucheng Ren, Yong Du, Jianming Lv, Guoqiang Han, and Shengfeng He. 2021. Learning from the master: Distilling cross-modal advanced knowledge for lip reading. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 13325–13333

    2021

  28. [36]

    Joon Son Chung, Andrew Senior, Oriol Vinyals, and Andrew Zisserman. 2017. Lip reading sentences in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition . 6447–6456

    2017

  29. [37]

    Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. Advances in neural information processing systems 27 (2014)

    2014

  30. [38]

    Jianyi Wang, Kelvin CK Chan, and Chen Change Loy. 2023. Exploring clip for assessing the look and feel of images. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37. 2555–2563

    2023

  31. [39]

    Tao Wang, Hong Liu, Pinhao Song, Tianyu Guo, and Wei Shi. 2022. Pose-guided feature disentangling for occluded person re-identification based on transformer. In Proceedings of the AAAI conference on artificial intelligence , Vol. 36. 2540–2549

    2022

  32. [40]

    Bo Xu, Cheng Lu, Yandong Guo, and Jacob Wang. 2020. Discriminative multi- modality speech recognition. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition . 14433–14442

    2020

  33. [41]

    Jingyi Xu, Hieu Le, Mingzhen Huang, ShahRukh Athar, and Dimitris Samaras

  34. [42]

    Kai Xu, Dawei Li, Nick Cassimatis, and Xiaolong Wang. 2018. LCANet: End- to-end lipreading with cascaded attention-CTC. In 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018) . IEEE, 548–555

    2018

  35. [43]

    Feng Xue, Yu Li, Deyin Liu, Yincen Xie, Lin Wu, and Richang Hong. 2023. Lip- former: learning to lipread unseen speakers based on visual-landmark transform- ers. IEEE Transactions on Circuits and Systems for Video Technology 33, 9 (2023), 4507–4517

    2023

  36. [44]

    Feng Xue, Tian Yang, Kang Liu, Zikun Hong, Mingwei Cao, Dan Guo, and Richang Hong. 2023. Lcsnet: End-to-end lipreading with channel-aware feature selection. ACM Transactions on Multimedia Computing, Communications and Applications 19, 1s (2023), 1–21

    2023

  37. [45]

    Xiaobing Zhang, Haigang Gong, Xili Dai, Fan Yang, Nianbo Liu, and Ming Liu

  38. [46]

    Yuanhang Zhang, Shuang Yang, Shiguang Shan, and Xilin Chen. 2024. ES3: Evolving Self-Supervised Learning of Robust Audio-Visual Speech Representa- tions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 27069–27079

    2024

  39. [47]

    Ya Zhao, Rui Xu, and Mingli Song. 2019. A cascade sequence-to-sequence model for chinese mandarin lip reading. In Proceedings of the ACM Multimedia Asia . 1–6

    2019

  40. [48]

    Y. Zhao, R. Xu, X. Wang, P. Hou, H. Tang, and M. Song. 2019. Hearing Lips: Improving Lip Reading by Distilling Speech Recognizers

    2019

  41. [49]

    Benjia Zhou, Zhigang Chen, Albert Clapés, Jun Wan, Yanyan Liang, Sergio Escalera, Zhen Lei, and Du Zhang. 2023. Gloss-free sign language translation: Improving from visual-language pretraining. In Proceedings of the IEEE/CVF International Conference on Computer Vision . 20871–20881

    2023

  42. [2019]

    Proceedings of the AAAI Conference on Artificial Intelligence 33, 01 (Jul

    Understanding Pictograph with Facial Features: End-to-End Sentence- Level Lip Reading of Chinese. Proceedings of the AAAI Conference on Artificial Intelligence 33, 01 (Jul. 2019), 9211–9218. doi:10.1609/aaai.v33i01.33019211

    2019 doi

  43. [2020]

    In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16

    Video-based remote physiological measurement via cross-verified feature disentangling. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16 . Springer, 295–310

    2020

  44. [2021]

    In Proceedings of the IEEE/CVF international conference on computer vision

    Variational feature disentangling for fine-grained few-shot classification. In Proceedings of the IEEE/CVF international conference on computer vision . 8812– 8821

  45. [2023]

    In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Cico: Domain-aware sign language retrieval via cross-lingual contrastive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 19016–19026

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.