Pith. sign in

REVIEW 2 major objections 5 minor 35 references

Optimal Transport-based Semantic Alignment for LLM-based Audio-Visual Speech Recognition

T0 review · 2 major / 5 minor · reviewed 2026-07-13 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Optimal transport can align audio and lip features to an LLM’s linguistic space before fusion, improving audio-visual speech recognition in clean and noisy conditions.

desk verdict Solid engineering paper: OT soft-alignment of Whisper/AV-HuBERT features to LLaMA token space gives consistent, modest WER gains on LRS3-TED; the training-only text reference is a real but not fatal soft spot. read the letter →

arxiv 2607.09001 v1 pith:EFJDPJGW submitted 2026-07-10 cs.SD eess.AS

Xugang Lu , Peng Shen , Yu Tsao , Hisashi Kawai This is my paper
classification cs.SDeess.AS
keywords audio-visualASRoptimaltransportsemanticalignmentLLM-AVSRfeaturefusioncontrastivelearningLRS3-TEDmultimodalspeechrecognition
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

LLM-based audio-visual speech recognition usually fuses pretrained audio and visual encoders as soft prompts without first closing the gap between those features and the language model’s token space. This paper claims that gap is a real limit on fusion quality, and that optimal transport can fix it. Acoustic and visual frame sequences are treated as distributions and matched to the target text’s embeddings inside the LLM; the resulting soft coupling matrices then supervise contrastive learning so both modalities become more linguistically coherent and consistent with each other. After that alignment step, the fused features condition the LLM more effectively for transcription. On the LRS3-TED benchmark the method reports lower word error rates than strong baselines across clean speech and several noise levels, which matters for anyone who needs speech systems that keep working when the microphone is compromised but the face is still visible.

What carries the argument

Entropy-regularized optimal transport couplings (with optional virtual buckets for non-informative frames) between modality features and linguistic embeddings; those couplings serve as soft pseudo-labels that supervise a contrastive alignment loss before channel-wise fusion into the LLM.

What would settle it

Train identical systems with and without the OT alignment loss (or replace OT couplings with random or purely temporal soft labels) on the same LRS3-TED split and check whether the reported WER gains at -5 dB, 0 dB, 5 dB, and clean disappear; if they do, the claimed mechanism is not carrying the result.

Watch

Extended reading notes

Core claim

The paper establishes that estimating optimal-transport couplings between Whisper acoustic features, AV-HuBERT visual features, and the LLM’s own token embeddings, then using those couplings as soft pseudo-labels for contrastive alignment before multimodal fusion, produces more effective soft prompts for LLM-based audio-visual speech recognition and yields state-of-the-art word error rates on LRS3-TED under clean and noisy test conditions.

Load-bearing premise

The method assumes that alignments computed against target text embeddings available only during training still produce audio-visual features that stay useful at inference, when no text is present.

Editorial extensions

If this is right

  • Multimodal soft prompts for LLM-AVSR should be linguistically anchored before fusion rather than only projected and concatenated.
  • OT-derived soft correspondences can replace hard temporal pairing as supervision for cross-modal contrastive learning in AVSR.
  • Virtual buckets (or unbalanced mass) can absorb silence, noise, and non-linguistic frames so they do not force false audio-text or visual-text matches.
  • Channel-wise concatenation after such alignment remains a strong, simple fusion choice under noisy training on LRS3-TED.
  • Gains should appear most clearly at low SNRs where complementary visual information must be integrated cleanly with degraded audio.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If training-time text is only a regularizer, cheaper alignment targets (phoneme embeddings, frozen text encoder, or self-distilled LLM hidden states) might recover most of the gain without full OT at every step.
  • The same OT-to-LLM-token recipe could transfer to other prompt-conditioned multimodal ASR or captioning stacks that currently fuse heterogeneous encoders by projection alone.
  • Systematic ablations that measure post-training audio-text and visual-text coupling quality at inference would separate true semantic anchoring from generic contrastive regularization.
  • Hyper-parameter interactions among OT regularization, temperature, and alignment weight suggest the method may need adaptive or learned OT schedules for stable scaling.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes an OT-based semantic alignment module for LLM-AVSR. Whisper acoustic and AV-HuBERT visual features are treated as empirical measures and coupled, via entropy-regularized OT with optional virtual buckets, to LLaMA token embeddings of the target transcript; the resulting soft couplings supervise a contrastive alignment loss that is added to the autoregressive ASR objective. After alignment, features are fused (channel concatenation preferred) and projected as soft prompts to a LoRA-tuned LLaMA3.2-3B decoder. On LRS3-TED the method reports consistent WER gains over the authors’ no-alignment baseline and over published LLM-AVSR systems under clean and noisy SNRs, with best reported numbers 7.16 / 2.34 / 1.11 / 0.73 at −5 / 0 / 5 dB / clean.

Significance. If the gains are genuinely attributable to transferable linguistic anchoring rather than generic auxiliary regularization, the work supplies a clean, modular way to reduce the audio–visual–text discrepancy that is left unaddressed by most current LLM-AVSR pipelines. The approach re-uses frozen pretrained encoders, requires no architectural change to the LLM, and is evaluated under a realistic multi-SNR protocol on the standard LRS3-TED benchmark. The empirical package (fusion ablations, virtual-bucket and α sweeps, with/without alignment) is more thorough than many concurrent LLM-AVSR papers and the absolute numbers are competitive with recent SOTA. The contribution is therefore of practical interest to the AVSR and multimodal LLM communities, provided the mechanistic claim can be better substantiated.

major comments (2)
  1. The central mechanistic claim—that OT alignment to the LLM linguistic space produces features that remain beneficial at inference—is only weakly supported. Eqs. (2)–(3) and (10)–(11) and Fig. 1 compute couplings exclusively against target-text embeddings zt that exist only in training; at inference only audio and video are present (§II-F). The paper never measures whether Proj_a / Proj_v (or fused h_av) stay closer to the linguistic manifold once the text reference is removed, nor does it compare against a non-OT contrastive or reconstruction regularizer of similar capacity. Without such evidence the SOTA attribution in Table V (and the abstract) may overstate the role of OT semantic alignment versus ordinary auxiliary supervision. A simple diagnostic (e.g., average cosine distance of fused features to held-out token embeddings, or an ablation that replaces OT soft labels by uniform / ra
  2. Table V and the surrounding text present three hyper-parameter “settings” that interact strongly (α, λ, s̃, τ). The best numbers are selected after the fact and the no-align baseline (8.34 / 2.82 / 1.29 / 0.89) already uses the same noisy-training and C_Concat recipe. Given the modest absolute deltas (often <0.5 WER) and the authors’ own admission that the hyper-parameter landscape is nontrivial (§IV), the SOTA claim needs either (i) a fixed hyper-parameter protocol chosen on validation before test evaluation or (ii) error bars / multiple seeds so that the ranking versus MMS-LLaMA is statistically credible.
minor comments (5)
  1. Eq. (7) writes the Gibbs kernel as exp(-S_ij/λ) while S is defined as cosine similarity (Eq. 4). Standard Sinkhorn OT uses a cost matrix C = -S (or 1-S); the sign convention should be stated explicitly to avoid implementation ambiguity.
  2. Tables I–II show that C_Concat is preferred mainly under noisy training; a short discussion of why channel concatenation outperforms Cross_att / Q-former in the low-SNR regime would help readers.
  3. Notation oscillates between za / Za, La = Lv = L, and Lq; a single consistent symbol table would improve readability.
  4. The virtual-bucket construction (Eq. 9, Fig. 2) is only lightly ablated (Table III). A sentence relating it to unbalanced OT would clarify the connection claimed in the text.
  5. Minor typos: “A VSR” / “A V-HuBERT” spacing, “Interspeech” capitalization, and the arXiv identifier formatting in the header.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: OT couplings are a standard training-time auxiliary loss; claimed gains are held-out WER, not a quantity forced by the fit.

full rationale

This is an empirical LLM-AVSR methods paper, not a first-principles derivation. The load-bearing claim is improved word error rate on LRS3-TED (clean and noisy SNRs) when an OT alignment module is added before fusion (Table V vs. the authors’ own no-align baseline and external systems such as MMS-LLaMA). OT couplings Q^{a→t}, Q^{v→t} are computed from current projected features and target-token embeddings available only in training (Eqs. 2–3, 10–11), then used as soft pseudo-labels for a contrastive alignment loss added to the usual AR ASR objective (Eq. 14). That is ordinary supervised auxiliary regularization: the evaluation metric (WER on held-out transcriptions, inference without target text) is not the same quantity as the training soft labels, so the reported gains do not reduce by construction to the OT inputs. Self-citations [18], [19] are prior related OT-for-ASR work by overlapping authors and appear only as background; they are not invoked as uniqueness theorems or as the sole justification of the method. Virtual buckets and Sinkhorn are taken from external literature. No fitted parameter is renamed as a prediction, no ansatz is smuggled in as a forced result, and no step equates the claimed SOTA WER to the training coupling by identity. Mechanism concerns (whether training-only text anchors transfer at inference) are correctness/transfer questions, not circularity.

Assumptions & free parameters 5 free parameters · 5 assumptions · 2 invented entities

The central claim is an empirical systems result. It rests on standard OT math, frozen pretrained encoders, LoRA fine-tuning of LLaMA, and several hand-chosen scalars (α, λ, τ, virtual-bucket margin) that control whether alignment helps. No new physical entities; the main modeling inventions are the training-time linguistic reference measure and the virtual-bucket filter for non-informative frames.

free parameters (5)
  • alignment loss weight α = 0.1 / 0.3 (reported settings)
    Scales L_align vs L_AR in Eq. (14); swept in Table IV (0.05–0.9); performance depends on the chosen band ~0.1–0.5.
  • OT entropy regularization λ = 0.05–0.2 (settings 1–3)
    Controls softness of Sinkhorn couplings in Eqs. (2)–(3); varied across settings 0.05–0.2 with interacting effects.
  • contrastive temperature τ
    Softmax temperature in Eq. (11); listed among interacting hyper-parameters selected empirically.
  • virtual-bucket similarity margin s̃ = 0.5
    Threshold that routes non-informative frames into unmatched mass (Eq. 9); fixed at 0.5 in reported settings.
  • LoRA rank and scale, LR schedule = rank=16, scale=32, lr=1e-4
    Fine-tuning knobs (rank 16, scale 32, Adam 1e-4, 5k warmup / 30k steps) inherited from prior LLM-AVSR setups and not re-justified.
assumptions (5)
  • domain assumption Entropy-regularized OT with cosine-similarity cost yields meaningful soft correspondences between continuous speech/visual frames and discrete LLM token embeddings.
    Invoked throughout §II.B–D; standard OT math is used, but the semantic validity of this cost for AVSR is assumed, not proved.
  • domain assumption Frozen Whisper and AV-HuBERT encoders already contain features that can be linearly projected into a space where linguistic alignment is useful.
    §II.A freezes both encoders; only projections, fusion, LoRA, and alignment loss are trained.
  • ad hoc to paper Training-time access to target token embeddings is a valid teacher signal for inference-time audio-visual-only decoding.
    Core of the Align module in Fig. 1 and Eqs. (1)–(3); target path is training-only.
  • domain assumption Channel-wise concatenation after alignment is an adequate fusion operator for conditioning the LLM.
    Chosen empirically after Tables I–II; treated as the baseline fusion thereafter.
  • standard math Sinkhorn iterations converge to a usable coupling for the sequence lengths used (L ≈ L_t after projection).
    Standard entropy-regularized OT theory (Cuturi/Peyré–Cuturi) assumed in §II.B.
invented entities (2)
  • Virtual buckets for non-informative AV frames in OT matching
    purpose: Absorb mass from noise/silence/outlier frames so they are not forced to align to linguistic tokens (Eq. 9, Fig. 2).
    Adapted from prior matching literature the paper cites, but introduced here as part of the AVSR alignment pipeline; only weak ablation support (Table III).
  • OT coupling matrices as soft pseudo-labels for A↔text and V↔text contrastive losses
    purpose: Supervise extraction of linguistically coherent multimodal features before fusion (Eqs. 10–11).
    The specific use of OT Q as soft labels into the LLM token space is the paper’s methodological core; evidence is only the end-task WER ablations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Optimal Transport-based Semantic Alignment for LLM-based Audio-Visual Speech Recognition." pith.science (2026). https://pith.science/paper/EFJDPJGW

@misc{pith2026260709001,
  author       = {Pith},
  title        = {Pith review of: Optimal Transport-based Semantic Alignment for LLM-based Audio-Visual Speech Recognition},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EFJDPJGW}},
  note         = {Machine review of arXiv:2607.09001}
}
read the original abstract

Large language model (LLM)-based audio-visual speech recognition (LLM-AVSR) has recently demonstrated strong robustness in adverse acoustic environments by leveraging complementary audio and visual information. Existing approaches typically employ independently pretrained acoustic and visual encoders, whose outputs are projected and fused as soft prompts to condition an LLM for speech recognition. However, most methods perform multimodal fusion without explicitly addressing the representational discrepancy between audio, visual and text modalities, potentially limiting the effectiveness of cross-modal integration. In this paper, we propose an optimal transport (OT)-based semantic alignment framework for LLM-AVSR. The proposed method explicitly bridges the modality gap by aligning the acoustic and visual representations with reference to the linguistic embedding space of the LLM before multimodal fusion. Specifically, OT is used to estimate probabilistic coupling matrices that characterize structured correspondences between modality-specific features and linguistic embeddings. The resulting OT couplings are further utilized as soft pseudo-labels to supervise contrastive learning, encouraging the extraction of semantically coherent and cross-modal consistent audio-visual representations. By anchoring multimodal features to the linguistic space of the LLM, the proposed framework facilitates more effective multimodal fusion and decoding. We implement the proposed framework using a Whisper-based acoustic encoder, an AV-HuBERT-based visual encoder, and a LLaMA3.2-3B decoder. Experiments conducted on the LRS3-TED benchmark demonstrate consistent improvements over strong baselines and achieve state-of-the-art performance under both clean and noisy evaluation conditions across a wide range of signal-to-noise ratios (SNRs).

Figures

Figures reproduced from arXiv: 2607.09001 by the authors.

Figure 1
Figure 1. The proposed LLM-based audio-visual ASR (LLM-AVSR) model framework: (a) Model architecture, (b) OT-based alignment module, (c) Modules [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Virtual buckets for aligning non-informative features. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

35 extracted references · 3 linked inside Pith

  1. [1]

    MIR-GAN: Refining Frame- Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition,

    Y . Hu, C. Chen, R. Li, H. Zou, E. Chng, “MIR-GAN: Refining Frame- Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition,” ACL, pp. 1610–11625, 2023

    2023

  2. [2]

    Learning Video Temporal Dynamics With Cross-Modal Attention For Robust Audio-Visual Speech Recognition,

    S. Kim, K. Jang, S. Bae, H. Kim, S. Yun, “Learning Video Temporal Dynamics With Cross-Modal Attention For Robust Audio-Visual Speech Recognition,” IEEE-SLT, pp. 447–454, 2024

    2024

  3. [3]

    End-to-end audio-visual speech recognition with conformers,

    P. Ma, S. Petridis, and M. Pantic, “End-to-end audio-visual speech recognition with conformers,” IEEE-ICASSP, pp. 7613–7617, 2021

    2021

  4. [4]

    Auto-avsr: Audio-visual speech recognition with automatic labels,

    P. Ma, A. Haliassos, A. Fernandez-Lopez, H. Chen, S. Petridis, and M. Pantic, “Auto-avsr: Audio-visual speech recognition with automatic labels,” IEEE-ICASSP, pp. 1–5, 2023

    2023

  5. [5]

    Large language models are strong audiovisual speech recognition learners,

    U. Cappellazzo, M. Kim, H. Chen, P. Ma, S. Petridis, D. Falavigna, A. Brutti, and M. Pantic, “Large language models are strong audiovisual speech recognition learners,” IEEE-ICASSP, pp. 1-–5, 2025

    2025

  6. [6]

    Where Visual Speech Meets Language: VSP-LLM Framework for Efficient and Context-Aware Visual Speech Processing,

    J. Yeo, S. Han, M. Kim, Y . Ro, “Where Visual Speech Meets Language: VSP-LLM Framework for Efficient and Context-Aware Visual Speech Processing,” EMNLP (Findings), pp. 11391–11406, 2024

    2024

  7. [7]

    Hy- brid ctc/attention architecture for end-to-end speech recognition,

    S. Watanabe, T. Hori, S. Kim, J. R. Hershey, and T. Hayashi, “Hy- brid ctc/attention architecture for end-to-end speech recognition,” IEEE Journal of Selected Topics in Signal Processing, vol. 11, no. 8, pp. 1240–1253, 2017

    2017

  8. [8]

    Whisper-Flamingo: Integrating Visual Features into Whisper for Audio-Visual Speech Recognition and Translation,

    A. Rouditchenko, Y . Gong, S. Thomas, L. Karlinsky, H. Kuehne, R. Feris, J. Glass, “Whisper-Flamingo: Integrating Visual Features into Whisper for Audio-Visual Speech Recognition and Translation,” Inter- speech, pp. 2420–2424, 2024

    2024

Show all 35 references
  1. [9]

    MMS-LLaMA: Efficient LLM-based Audio-Visual Speech Recognition with Minimal Multimodal Speech Tokens,

    J. Yeo, H. Rha, S. Park, Y . Ro, “MMS-LLaMA: Efficient LLM-based Audio-Visual Speech Recognition with Minimal Multimodal Speech Tokens,” ACL, pp. 20724–20735, 2025

    2025

  2. [10]

    Omni- avsr: Towards unified multimodal speech recognition with large language models,

    U. Cappellazzo, X. Liu, P. Ma, S. Petridis, and M. Pantic, “Omni- avsr: Towards unified multimodal speech recognition with large language models,” arXiv preprint arXiv:2511.07253, 2025

    2025

  3. [11]

    V ALLR: Visual ASR Language Model for Lip Reading,

    M. Thomas, E. Fish, R. Bowden, “V ALLR: Visual ASR Language Model for Lip Reading,” arXiv preprint arXiv:2503.21408, 2025

    2025

  4. [12]

    Zero-A VSR: Zero-Shot Audio-Visual Speech Recognition with LLMs by Learning Language- Agnostic Speech Representations,

    J. Yeo, M. Kim, C. Kim, S. Petridis, Y . Ro, “Zero-A VSR: Zero-Shot Audio-Visual Speech Recognition with LLMs by Learning Language- Agnostic Speech Representations,” arXiv preprint arXiv:2503.06273, 2025

    2025 arXiv

  5. [13]

    Uncovering the Visual Contribution in Audio-Visual Speech Recognition,

    Z. Lin, N. Harte, “Uncovering the Visual Contribution in Audio-Visual Speech Recognition,” IEEE-ICASSP, pp. 1–5, 2025

    2025

  6. [14]

    Align before Fuse: Vision and Language Representation Learning with Mo- mentum Distillation,

    J. Li, R. R. Selvaraju, A. Gotmare, S. Joty, C. Xiong, S. Hoi, “Align before Fuse: Vision and Language Representation Learning with Mo- mentum Distillation,” NeurIPS, pp. 9694–9705, 2021

    2021

  7. [15]

    Optimal Transport for Domain Adaptation,

    N. Courty, R. Flamary, D. Tuia and A. Rakotomamonjy, “Optimal Transport for Domain Adaptation,” In IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 39, no. 9, pp. 1853–1865, 2014

    2014

  8. [16]

    From Word Embeddings To Document Distances,

    M. Kusner, Y . Sun, N. Kolkin, K. Weinberge, “From Word Embeddings To Document Distances,” in Proceedings of the 32nd International Conference on Machine Learning (ICML), vol. 37, pp. 957-965, 2015

    2015

  9. [17]

    Word rotator’s distance,

    S. Yokoi, R. Takahashi, R. Akama, J. Suzuki, and K. Inui, “Word rotator’s distance,” In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 2944–2960, 2020

    2020

  10. [18]

    Cross-modal Alignment with Optimal Transport for CTC-based ASR,

    X. Lu, S. Shen, Y . Tsao, H. Kawai, “Cross-modal Alignment with Optimal Transport for CTC-based ASR,” IEEE-ASRU, 2023

    2023

  11. [19]

    Hierarchical Cross-Modality Knowledge Transfer with Sinkhorn Attention for CTC-based ASR,

    X. Lu, S. Shen, Y . Tsao, H. Kawai, “Hierarchical Cross-Modality Knowledge Transfer with Sinkhorn Attention for CTC-based ASR,” IEEE-ICASSP, 2024

    2024

  12. [20]

    LA VCap: LLM-based Audio-Visual Captioning using Optimal Transport,

    K. Rho, H. Lee, V . Iverson, J. Chung, “LA VCap: LLM-based Audio-Visual Captioning using Optimal Transport,” arXiv preprint arXiv:2501.09291, 2025

    2025 arXiv

  13. [21]

    Computational Optimal Transport: With Applications to Data Science,

    G. Peyr ´e and M. Cuturi, “Computational Optimal Transport: With Applications to Data Science,” Foundations and Trends® in Machine Learning, vol. 11 (5-6), pp 355-607, 2019

    2019

  14. [22]

    Robust Speech Recognition via Large-Scale Weak Supervision,

    A. Radford, J. Kim, T. Xu, G. Brockman, C. McLeavey, I. Sutskever, “Robust Speech Recognition via Large-Scale Weak Supervision,” ICML, pp. 28492–28518, 2023

    2023

  15. [23]

    wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations,

    A. Baevski, Y . Zhou, A. Mohamed, M. Auli, “wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations,” NeurIPS, 2020

    2020

  16. [24]

    HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units,

    W. Hsu, B. Bolte, Y . Tsai, K. Lakhotia, R. Salakhutdinov, A. Mohamed, “HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units,” IEEE ACM Trans. Audio Speech Lang. Process, vol. 29, pp. 3451–3460, 2021

    2021

  17. [25]

    Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction,

    B. Shi, W. Hsu, K. Lakhotia, A. Mohamed, “Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction,” ICLR, 2022

    2022

  18. [26]

    https://huggingface.co/meta-llama/Llama-3.2-3B

  19. [27]

    https://huggingface.co/collections/Qwen/qwen3-vl

  20. [28]

    LoRA: Low-Rank Adaptation of Large Language Models,

    E. Hu, Y . Shen, P. Wallis, Z. Zhu, Y . Li, S. Wang, L. Wang, W. Chen, “LoRA: Low-Rank Adaptation of Large Language Models,” ICLR, 2022

    2022

  21. [29]

    Sinkhorn Distances: Lightspeed Computation of Optimal Transport,

    Marco Cuturi, “Sinkhorn Distances: Lightspeed Computation of Optimal Transport,” In Advances in Neural Information Processing Systems, pp. 2292-2300, 2013

    2013

  22. [30]

    Learning to Align Sequential Actions in the Wild,

    W. Liu, B. Tekin, H. Coskun, V . Vineet, P. Fua, M. Pollefeys, “Learning to Align Sequential Actions in the Wild,” CVPR, pp. 2171–2181, 2022

    2022

  23. [31]

    SuperGlue: Learning Feature Matching With Graph Neural Networks,

    P. Sarlin, D. DeTone, T. Malisiewicz, A. Rabinovich, “SuperGlue: Learning Feature Matching With Graph Neural Networks,” CVPR, pp. 4937–4946, 2020

    2020

  24. [32]

    Unsupervised Learning of Visual Features by Contrasting Cluster Assignments,

    M. Caron, I. Misra, J. Mairal, P. Goyal, P. Bojanowski, A. Joulin, “Unsupervised Learning of Visual Features by Contrasting Cluster Assignments,” NeurIPS, 2020

    2020

  25. [33]

    Cross-Modal Global Interaction and Local Alignment for Audio-Visual Speech Recognition,

    Y . Hu, R. Li, C. Chen, H. Zou, Q. Zhu, E. Chng, “Cross-Modal Global Interaction and Local Alignment for Audio-Visual Speech Recognition,” IJCAI, pp. 5076–5084, 2023

    2023

  26. [34]

    SALMONN: Towards Generic Hearing Abilities for Large Language Models,

    C. Tang, W. Yu, G. Sun, X. Chen, T. Tan, W. Li, L. Lu, Z. Ma, C. Zhang, “SALMONN: Towards Generic Hearing Abilities for Large Language Models,” ICLR, 2024

    2024

  27. [35]

    Lrs3-ted: a large-scale dataset for visual speech recognition,

    T. Afouras, J. S. Chung, and A. Zisserman, “Lrs3-ted: a large-scale dataset for visual speech recognition,” arXiv preprint arXiv:1809.00496, 2018

    2018 arXiv

Pith tools

Reviewed July 13, 2026 · model on record in the stance chip above.