Pith. sign in

REVIEW 3 major objections 5 minor 38 references

UBG-Net: An Uncertainty-aware Bayesian Gating Network for Robust Audio-Visual Speech Recognition

T0 review · 3 major / 5 minor · reviewed 2026-07-09 · glm-5.2

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Noise-Aware Bayesian Gating Lifts Audio-Visual Speech Recognition in Cocktail-Party Noise

desk verdict Novel uncertainty coupling for AVSR, but gains are small and missing variance estimates under stochastic inference read the letter →

arxiv 2607.06892 v1 pith:MUVMCG6L submitted 2026-07-08 eess.AS

Jinjie Fu , Hang Chen , Wu Guo , Zhijun Zhang , Kuiliang Li , Peng Gao This is my paper · ORCID
classification eess.AS
keywords bayesianubg-netuncertainty-awarefusionnetworkaudio-visualdistributionduhv
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Audio-visual speech recognition systems struggle when audio is corrupted by overlapping speakers or background noise, because standard fusion methods cannot tell clean signal from garbage. This paper proposes UBG-Net, which couples two kinds of uncertainty: aleatoric uncertainty (measuring inherent data noise per modality) and epistemic uncertainty (measuring model confidence). The key idea is that the learned variance of each modality's features is fed as a context vector into a Bayesian gating network, so the gate knows how noisy each stream is and can downweight corrupted features accordingly. At inference time, the Bayesian network produces multiple candidate transcripts via Monte Carlo sampling; a hierarchical voting scheme picks the most frequent candidate and breaks ties using beam-search confidence scores. On the AVCocktail and noise-augmented LRS2 benchmarks, the method reduces word error rates relative to the baseline, with the largest gains in heavily corrupted conditions.

What carries the argument

The central mechanism is the Modality Uncertainty-aware Bayesian Fusion (MUBF): an MLP predicts mean and variance for each modality's features (capturing aleatoric uncertainty), the variance is concatenated with the mean into a context vector, and this context drives a Bayesian linear gating network whose weights carry distributions (capturing epistemic uncertainty). The gate output modulates the fused multimodal features before they reach the decoder. At inference, Monte Carlo sampling of the Bayesian weights yields multiple transcripts, and DUHV applies majority voting with confidence-score tie-breaking to select the final output.

What would settle it

If the learned per-modality variance does not correlate with actual signal-to-noise quality under corruption, or if decoupling aleatoric and epistemic modeling yields no measurable difference from independent modeling, the premise that data noise guides model confidence collapses. The ablation table partially addresses this, showing isolated aleatoric or epistemic modeling underperforms their joint injection.

Watch

Extended reading notes

Core claim

The paper claims that explicitly injecting signal-level aleatoric uncertainty as a contextual cue into a Bayesian gating network to model epistemic uncertainty (the MUBF mechanism), combined with a distribution uncertainty-aware hierarchical voting strategy (DUHV), achieves superior robustness for audio-visual speech recognition compared to SOTA baselines on the AVCocktail and LRS2 datasets.

Load-bearing premise

The central assumption is that the variance predicted by a simple MLP genuinely reflects actual signal corruption quality, so that feeding it into the Bayesian gating layers meaningfully couples data noise with model confidence. If the network learns a variance that does not track real noise, the gating mechanism modulates features on a spurious signal.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The manuscript proposes UBG-Net, an uncertainty-aware Bayesian gating framework for robust audio-visual speech recognition (AVSR). The core methodological contribution is the Modality Uncertainty-aware Bayesian Fusion (MUBF) mechanism, which models aleatoric uncertainty (data noise) via modality-dependent MLP encoders and injects the resulting distribution parameters as a context vector into a Bayesian gating network that models epistemic uncertainty (model parameter uncertainty). A secondary contribution is the Distribution Uncertainty-aware Hierarchical Voting (DUHV) strategy, which selects the final transcript from Monte Carlo (MC) samples by majority voting, using inference scores as a tie-breaker. Experiments are conducted on the AVCocktail and noise-augmented LRS2 datasets, with ablation studies isolating the contributions of the aleatoric and epistemic components and the voting strategy.

Significance. The paper addresses a practically important problem: robust AVSR under signal corruption and distribution shifts. The conceptual framing—using aleatoric uncertainty as a contextual cue for epistemic uncertainty rather than treating them independently—is reasonable and not standard in the AVSR literature. The method is built on established tools (variational inference, reparameterization, AV-HuBERT backbone), and the ablation design (Table 2) is structured to isolate individual components. The inclusion of Whisper and Qwen3-Omni as reference points contextualizes task difficulty. The work is a reasonable engineering contribution to robust multimodal speech recognition.

major comments (3)
  1. Section 4.1 and Tables 1–2: The manuscript states in footnote 3 that 'all reported results are averaged over 10 independent runs to ensure statistical reliability,' yet no standard deviations, confidence intervals, or significance tests are reported anywhere. The improvements over the baseline are small in absolute terms: on LRS2 (Table 1), the average WER improves from 4.1% to 3.9% (0.2% absolute), and several individual conditions show no improvement or slight regression (e.g., 2 interfering speakers at 5 dB: 3.2→3.2; clean: 2.1→2.2). On AVCocktail (Table 2), Gold segmentation improves by 0.7% absolute (22.6→21.9). The ablation differences are even smaller: removing epistemic uncertainty changes Gold WER by 0.4% (21.9→22.3), removing aleatoric by 0.2% (21.9→22.1). Given that the inference procedure is explicitly stochastic (MC sampling with K=5), the absence of variance estimates makes
  2. Section 2.1.1, Eq. (1): The central claim that aleatoric uncertainty serves as a 'signal-quality indicator' is not empirically validated. The learned variance (σ_t^m)^2 is produced by a simple MLP encoder and constrained by a KL divergence toward a standard Gaussian prior (Eq. 7, L_KL-Feat). There is no analysis showing that the predicted variance correlates with actual signal corruption (e.g., SNR, noise level, or silent face segments). Without such evidence, it remains unclear whether the gating network in Eq. (4) is modulating features based on genuine signal quality or on a spurious learned signal. A scatter plot or correlation analysis between predicted variance and known noise levels would substantially strengthen the load-bearing claim that aleatoric uncertainty guides epistemic modeling.
  3. Section 2.1.2, Eq. (5): The fusion formula H_t^final = H_t^deep + g_t ⊙ Concat(h_t^v, h_t^a) adds the gated multimodal features to the deep representation H_t^deep from the pre-trained backbone. However, the dimensionality and origin of H_t^deep relative to the concatenated features h_t^v and h_t^a are not clearly specified. If H_t^deep is the output of the AV-HuBERT encoder (which already fuses audio and visual streams), the relationship between this already-fused representation and the separately computed h_t^v, h_t^a needs clarification. This affects the interpretability of the gating mechanism: is the BGN re-weighting features that the backbone has already fused?
minor comments (5)
  1. Table 1: The 'Avg' column header is rendered as 'A vg' with a space. Also, the SNR column header '∞' (clean condition) is placed under the SNR grouping but the column appears to be the last data column; alignment is slightly confusing.
  2. Section 3.2: The loss coefficients β_1 and β_2 are warmed up to target values of 1e-7 and 1e-4, respectively. The value of β_1 = 1e-7 is extremely small; it would help to briefly justify why such a small weight is appropriate for the KL term on the Bayesian weights.
  3. Figure 1: The figure is dense and the text labels are small. The distinction between the training path and inference path could be made clearer, perhaps by using color coding or separating the two paths into sub-figures.
  4. Section 4.2, Figure 2: The x-axis labels '0', '1', '≥2' are described in the caption but the figure itself would benefit from clearer tick labels (e.g., 'Det', 'K=1', 'K=2', etc.) for readability.
  5. Reference [8] (Nguyen et al., Interspeech 2025) and [30] (Qwen3-Omni, arXiv 2025) are cited; the Qwen3-Omni reference lists the arXiv ID as 2509.17765, which appears to be a future-dated preprint. Please verify the citation accuracy.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found; derivation is self-contained standard Bayesian deep learning evaluated against external benchmarks.

full rationale

The paper's derivation chain is self-contained and does not reduce to its inputs by construction. The MUBF mechanism (Eqs. 1-7) follows standard variational inference: aleatoric uncertainty is modeled via MLP encoders with reparameterization (Eqs. 1-2), the context vector concatenates distribution parameters (Eq. 3), Bayesian linear layers produce a gating vector (Eq. 4-5), and the loss combines standard CTC/attention losses with KL divergence terms (Eq. 7). No equation is defined in terms of the quantity it claims to predict. The DUHV strategy (Eqs. 8-10) is a straightforward voting procedure with no circular dependency. The paper cites external work (Blundell et al. [17] for Bayesian NNs, Kingma et al. [20,23] for reparameterization, Kendall & Gal [10] for uncertainty taxonomy) — all standard, independently verifiable results, not self-citations. The baseline [8] (Nguyen et al.) appears to be by different authors. Results are evaluated against external benchmarks (AVCocktail, LRS2) with independent baselines (Whisper, Qwen3-Omni). The skeptic's concern about missing variance estimates is a correctness/empirical rigor issue, not circularity. No step in the derivation chain is self-definitional, no prediction is a renamed fit, and no load-bearing claim rests on a self-citation chain.

Assumptions & free parameters 5 free parameters · 3 assumptions · 2 invented entities

The ledger captures the key hyperparameters tuned for the model, the core domain assumptions about uncertainty dependency, and the newly introduced architectural components with their supporting evidence.

free parameters (5)
  • beta_1 = 1e-7
    Loss coefficient for KL-Weight, linearly warmed up to target value.
  • beta_2 = 1e-4
    Loss coefficient for KL-Feat, linearly warmed up to target value.
  • K = 5
    Number of Monte Carlo samples during inference, determined empirically as optimal.
  • DoRA r, alpha = r=8, alpha=16
    Hyperparameters for the Weight-Decomposed Low-Rank Adaptation modules.
  • Bias init = -5.0
    Initialization for Bayesian linear biases to encourage sparsity.
assumptions (3)
  • domain assumption Aleatoric and epistemic uncertainties are not independent and have an intrinsic dependency that can be modeled.
    The paper's core contribution rests on the premise that data noise should explicitly inform model confidence estimation, contradicting prior work that treats them in isolation.
  • domain assumption Standard Gaussian prior is appropriate for Bayesian network weights.
    Used in Eq. 6 for KL divergence; standard practice but an assumption nonetheless.
  • domain assumption AV-HuBERT pre-trained representations are robust enough to serve as the deep feature backbone.
    The method builds upon and freezes these representations, assuming their quality is sufficient for the fusion module to enhance.
invented entities (2)
  • Modality Uncertainty-aware Bayesian Fusion (MUBF) independent evidence
    purpose: Mechanism to inject aleatoric uncertainty into Bayesian gating for epistemic uncertainty modeling.
    Ablation study (Table 2) shows performance improvement when using MUBF compared to isolated uncertainty modeling.
  • Distribution Uncertainty-aware Hierarchical Voting (DUHV) independent evidence
    purpose: Strategy to select optimal transcripts from Monte Carlo samples by majority voting with confidence tie-breaking.
    Ablation study (Table 2) validates its contribution over standard majority voting or best confidence selection.

how reviews work

0 comments
Cite this review

Pith. "Pith review of UBG-Net: An Uncertainty-aware Bayesian Gating Network for Robust Audio-Visual Speech Recognition." pith.science (2026). https://pith.science/paper/MUVMCG6L

@misc{pith2026260706892,
  author       = {Pith},
  title        = {Pith review of: UBG-Net: An Uncertainty-aware Bayesian Gating Network for Robust Audio-Visual Speech Recognition},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MUVMCG6L}},
  note         = {Machine review of arXiv:2607.06892}
}
read the original abstract

Audio-Visual speech recognition systems often degrade in real-world scenarios due to signal corruption and distribution shifts. To address this, we propose a unified uncertainty-modeling framework, namely the uncertainty-aware Bayesian gating network (UBG-Net). UBG-Net features a Modality Uncertainty-aware Bayesian Fusion (MUBF) mechanism that injects signal-level aleatoric uncertainty into a Bayesian network to model epistemic uncertainty, thereby ensuring robust fusion of pre-trained backbone features. For inference, we introduce Distribution Uncertainty-aware Hierarchical Voting (DUHV) to select transcripts from Monte Carlo samples, prioritizing frequency and using inference scores in case of a tie. Experiments on the AVCocktail and LRS2 datasets demonstrate the overall superiority of UBG-Net compared to SOTA baselines. Ablation studies confirm that MUBF and DUHV effectively filter noise, enhancing fusion and decoding robustness.

Figures

Figures reproduced from arXiv: 2607.06892 by the authors.

Figure 1
Figure 1. Framework Overview. (1) AV-HuBERT Encoder: A pre-trained backbone employed for deep multimodal representation extraction. (2) Modality Uncertainty-aware Bayesian Fusion: A Bayesian gating network that modulates multimodal streams using injected aleatoric uncertainty parameters. (3) AV-HuBERT Decoder: The decoding module, where the top and bottom parts denote the training and inference stages, respectively. tailed de… view at source ↗
Figure 2
Figure 2. Effect of MC sample size on WER across ASD, Fix, and Gold sets. 0 denotes deterministic decoding, 1 represents a single stochastic sample, and ≥ 2 indicates voting results. (frequency-based). As shown in [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

38 extracted references · 38 canonical work pages

  1. [1]

    This mechanism is particularly crucial in noisy environments where acoustic signals are corrupted, yet visual information remains reliable

    Introduction Audio-Visual Speech Recognition (A VSR) leverages visual cues, such as lip movements, to compensate for audio degradation[1, 2]. This mechanism is particularly crucial in noisy environments where acoustic signals are corrupted, yet visual information remains reliable. However, most existing A VSR models rely on fixed audio–visual fusion parad...

    work page

  2. [2]

    UBG-Net: An Uncertainty-aware Bayesian Gating Network for Robust Audio-Visual Speech Recognition

    Proposed UBG-Net The proposed UBG-Net for A VSR, as depicted in Fig. 1, in- cludes three parts: the A V-HuBERT encoder for multimodal deep representation extracting, the modality uncertainty-aware Bayesian fusion block and the A V-HuBERT decoder. Our in- novations focus on the latter two parts, and we will provide de- 1We did not include LRS3 in our evalu...

    work page Pith review arXiv 2026

  3. [3]

    Datasets and metrics For training, we use a combination of datasets including LRS2, V ox2, and A VYT [8]

    Experimental Setup 3.1. Datasets and metrics For training, we use a combination of datasets including LRS2, V ox2, and A VYT [8]. For evaluation, we use two benchmarks: •Simulated LRS2 [19]:Following the protocol of Nguyen et al. [8], we introduce up to two background interfering speakers and apply different signal-to-noise ratios (SNRs) of {−5,0,5,10}dB....

    work page

  4. [4]

    Comparison with baseline We use algorithms in paper [8] as the baseline, which can achieve SOTA performance on A VCocktail and LRS23

    Experimental Results 4.1. Comparison with baseline We use algorithms in paper [8] as the baseline, which can achieve SOTA performance on A VCocktail and LRS23. Addi- tionally, we include Whisper (large-v3) [29] and Qwen3-Omni (30B-A3B-Instruct) [30] as reference points to contextualize the task difficulty. As shown in the Table 1 and Table 2, these genera...

    work page

  5. [5]

    This indicates thatK= 5is the optimal operating point for balancing recognition accuracy and inference latency

    work page

  6. [6]

    Conclusion In this work, we proposed a novel uncertainty-aware A VSR framework featuring a Bayesian fusion mechanism that is modality-uncertainty-aware. By explicitly injecting aleatoric uncertainty as a perceptual context to the Bayesian gating net- work, our method effectively decouples optimization objec- tives, enabling the epistemic uncertainty model...

    work page

  7. [7]

    No AI tool was used to gen- erate the core scientific contributions, and all authors take full responsibility for the manuscript’s content

    Generative AI Use Disclosure Generative AI tools were used in a limited manner strictly for language polishing and formatting. No AI tool was used to gen- erate the core scientific contributions, and all authors take full responsibility for the manuscript’s content

    work page

  8. [8]

    An overview of deep-learning-based audio-visual speech enhancement and separation,

    D. Michelsanti, Z.-H. Tan, S.-X. Zhang, Y . Xu, M. Yu, D. Yu, and J. Jensen, “An overview of deep-learning-based audio-visual speech enhancement and separation,”IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 1368–1396, 2021

    work page 2021

Show all 38 references
  1. [9]

    Hearing lips and seeing voices,

    H. McGurk and J. MacDonald, “Hearing lips and seeing voices,” Nature, vol. 264, no. 5588, pp. 746–748, 1976

    1976

  2. [10]

    A review of recent ad- vances on deep learning methods for audio-visual speech recog- nition,

    D. Ivanko, D. Ryumin, and A. Karpov, “A review of recent ad- vances on deep learning methods for audio-visual speech recog- nition,”Mathematics, vol. 11, no. 12, p. 2665, 2023

    2023

  3. [11]

    Auto-A VSR: Audio-visual speech recognition with automatic labels,

    P. Ma, A. Haliassos, A. Fernandez-Lopez, H. Chen, S. Petridis, and M. Pantic, “Auto-A VSR: Audio-visual speech recognition with automatic labels,” inIEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5

    2023

  4. [12]

    Whisper-Flamingo: Integrating Visual Features into Whisper for Audio-Visual Speech Recognition and Translation,

    A. Rouditchenko, Y . Gong, S. Thomas, L. Karlinsky, H. Kuehne, R. Feris, and J. Glass, “Whisper-Flamingo: Integrating Visual Features into Whisper for Audio-Visual Speech Recognition and Translation,” inInterspeech 2024, 2024, pp. 2420–2424

    2024

  5. [13]

    Robust audio-visual ASR with unified cross-modal attention,

    J. Li, C. Li, Y . Wu, and Y . Qian, “Robust audio-visual ASR with unified cross-modal attention,” inIEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5

    2023

  6. [14]

    AMG-A VSR: Adap- tive modality guidance for audio-visual speech recognition via progressive feature enhancement,

    Z. Zhao, D. Guo, W. Ou, H. Liu, and Y . Lin, “AMG-A VSR: Adap- tive modality guidance for audio-visual speech recognition via progressive feature enhancement,” inThe 16th Asian Conference on Machine Learning (Conference Track), 2024

    2024

  7. [15]

    Cocktail-Party Audio-Visual Speech Recognition,

    T.-B. Nguyen, N.-Q. Pham, and A. Waibel, “Cocktail-Party Audio-Visual Speech Recognition,” inInterspeech 2025, 2025, pp. 1828–1832

    2025

  8. [16]

    Cross-modal global interaction and local alignment for audio-visual speech recognition,

    Y . Hu, R. Li, C. Chen, H. Zou, Q. Zhu, and E. S. Chng, “Cross-modal global interaction and local alignment for audio-visual speech recognition,” inProceedings of the 32nd International Joint Conference on Artificial Intelligence (IJCAI),

  9. [17]

    Available: https://www.ijcai.org/proceedings/ 2023/0564.pdf

    [Online]. Available: https://www.ijcai.org/proceedings/ 2023/0564.pdf

    2023

  10. [18]

    What uncertainties do we need in Bayesian deep learning for computer vision?

    A. Kendall and Y . Gal, “What uncertainties do we need in Bayesian deep learning for computer vision?” inAdvances in Neural Information Processing Systems, vol. 30, 2017

    2017

  11. [19]

    A survey of un- certainty in deep neural networks,

    J. Gawlikowski, C. R. N. Tassi, M. Ali, J. Lee, M. Humt, J. Feng, A. Kruspe, R. Triebel, P. Jung, R. Roscheret al., “A survey of un- certainty in deep neural networks,”Artificial Intelligence Review, vol. 56, no. Suppl 1, pp. 1513–1589, 2023

    2023

  12. [20]

    Proxy- driven robust multimodal sentiment analysis with incomplete data,

    A. Zhu, M. Hu, X. Wang, J. Yang, Y . Tang, and N. An, “Proxy- driven robust multimodal sentiment analysis with incomplete data,” inProceedings of the 63rd Annual Meeting of the Asso- ciation for Computational Linguistics (Volume 1: Long Papers), 2025, pp. 22 123–22 138

    2025

  13. [21]

    Embrac- ing unimodal aleatoric uncertainty for robust multimodal fusion,

    Z. Gao, X. Jiang, X. Xu, F. Shen, Y . Li, and H. T. Shen, “Embrac- ing unimodal aleatoric uncertainty for robust multimodal fusion,” inIEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR), 2024, pp. 26 876–26 885

    2024

  14. [22]

    Uncertainty-aware audiovisual activity recognition using deep bayesian variational inference,

    M. Subedar, R. Krishnan, P. L. Meyer, O. Tickoo, and J. Huang, “Uncertainty-aware audiovisual activity recognition using deep bayesian variational inference,” inProceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 6301– 6310

    2019

  15. [23]

    Latent distribution decouple for uncertain-aware mul- timodal multi-label emotion recognition,

    J. Huang, J. Zhong, Q. Lei, J. Gao, Y . Yang, S. Wang, P. Li, and K. Wei, “Latent distribution decouple for uncertain-aware mul- timodal multi-label emotion recognition,” inFindings of the As- sociation for Computational Linguistics: ACL 2025, 2025, pp. 24 123–24 138

    2025

  16. [24]

    COLD fusion: Calibrated and or- dinal latent distribution fusion for uncertainty-aware multimodal emotion recognition,

    M. K. Tellamekala, S. Amiriparian, B. W. Schuller, E. Andr ´e, T. Giesbrecht, and M. Valstar, “COLD fusion: Calibrated and or- dinal latent distribution fusion for uncertainty-aware multimodal emotion recognition,”IEEE Transactions on Pattern Analysis and Machine Intelligence,...

    2023

  17. [25]

    Weight uncertainty in neural network,

    C. Blundell, J. Cornebise, K. Kavukcuoglu, and D. Wierstra, “Weight uncertainty in neural network,” inInternational confer- ence on machine learning. PMLR, 2015, pp. 1613–1622

    2015

  18. [26]

    Uncertainty estimation in autore- gressive structured prediction,

    A. Malinin and M. Gales, “Uncertainty estimation in autore- gressive structured prediction,” inInternational Conference on Learning Representations (ICLR), 2021. [Online]. Available: https://openreview.net/forum?id=jN5y-zb5Q7m

    2021

  19. [27]

    Lip read- ing sentences in the wild,

    J. Son Chung, A. Senior, O. Vinyals, and A. Zisserman, “Lip read- ing sentences in the wild,” inIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 6447–6456

    2017

  20. [28]

    Auto-Encoding Variational Bayes,

    D. P. Kingma and M. Welling, “Auto-Encoding Variational Bayes,” inInternational Conference on Learning Representations (ICLR), 2014

    2014

  21. [29]

    Deep variational information bottleneck,

    A. A. Alemi, I. Fischer, J. V . Dillon, and K. Murphy, “Deep variational information bottleneck,” inInternational Conference on Learning Representations (ICLR), 2017. [Online]. Available: https://openreview.net/forum?id=HyxQzBceg

    2017

  22. [30]

    Practical variational inference for neural networks,

    A. Graves, “Practical variational inference for neural networks,” inAdvances in Neural Information Processing Systems, vol. 24, 2011

    2011

  23. [31]

    Variational dropout and the local reparameterization trick,

    D. P. Kingma, T. Salimans, and M. Welling, “Variational dropout and the local reparameterization trick,” inAdvances in Neural In- formation Processing Systems, vol. 28, 2015

    2015

  24. [32]

    Learning audio-visual speech representation by masked multimodal cluster prediction,

    B. Shi, W.-N. Hsu, K. Lakhotia, and A. Mohamed, “Learning audio-visual speech representation by masked multimodal cluster prediction,” inInternational Conference on Learning Representations (ICLR), 2022. [Online]. Available: https: //openreview.net/forum?id=Z1Qlm11uOM

    2022

  25. [33]

    DoRA: Weight-decomposed low-rank adaptation,

    S.-Y . Liu, C.-Y . Wang, H. Yin, P. Molchanov, Y .-C. F. Wang, K.-T. Cheng, and M.-H. Chen, “DoRA: Weight-decomposed low-rank adaptation,” inForty-first International Conference on Machine Learning, 2024

    2024

  26. [34]

    LoRA: Low-rank adaptation of large language models,

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “LoRA: Low-rank adaptation of large language models,” inInternational Conference on Learning Representations (ICLR), 2022. [Online]. Available: https://openreview.net/forum?id=nZeVKeeFYf9

    2022

  27. [35]

    Understanding the difficulty of train- ing deep feedforward neural networks,

    X. Glorot and Y . Bengio, “Understanding the difficulty of train- ing deep feedforward neural networks,” inProceedings of the thirteenth international conference on artificial intelligence and statistics. JMLR Workshop and Conference Proceedings, 2010, pp. 249–256

    2010

  28. [36]

    Decoupled weight de- cay regularization,

    I. Loshchilov and F. Hutter, “Decoupled weight de- cay regularization,” inInternational Conference on Learn- ing Representations (ICLR), 2019. [Online]. Available: https://openreview.net/forum?id=Bkg6RiCqY7

    2019

  29. [37]

    Robust speech recognition via large-scale weak supervision,

    A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” inInternational conference on machine learning. PMLR, 2023, pp. 28 492–28 518

    2023

  30. [38]

    Qwen3-omni technical report,

    J. Xu, Z. Guo, H. Hu, Y . Chu, X. Wang, J. He, Y . Wang, X. Shi, T. He, X. Zhu, Y . Lv, Y . Wang, D. Guo, H. Wang, L. Ma, P. Zhang, X. Zhang, H. Hao, Z. Guo, B. Yang, B. Zhang, Z. Ma, X. Wei, S. Bai, K. Chen, X. Liu, P. Wang, M. Yang, D. Liu, X. Ren, B. Zheng, R. Men, F. Zhou,...

    2025 arXiv

Pith tools

Reviewed July 9, 2026 · model on record in the stance chip above.